Pith. sign in

REVIEW 19 cited by

DistillSpec: Improving Speculative Decoding via Knowledge Distillation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.08461 v2 pith:UVGNT6JV submitted 2023-10-12 cs.CL cs.AIcs.LG

Yongchao Zhou , Kaifeng Lyu , Ankit Singh Rawat , Aditya Krishna Menon , Afshin Rostamizadeh , Sanjiv Kumar , Jean-François Kagy , Rishabh Agarwal This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords modeldistillspecdrafttargetdecodingdistillationperformancestandard
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Speculative decoding (SD) accelerates large language model inference by employing a faster draft model for generating multiple tokens, which are then verified in parallel by the larger target model, resulting in the text generated according to the target model distribution. However, identifying a compact draft model that is well-aligned with the target model is challenging. To tackle this issue, we propose DistillSpec that uses knowledge distillation to better align the draft model with the target model, before applying SD. DistillSpec makes two key design choices, which we demonstrate via systematic study to be crucial to improving the draft and target alignment: utilizing on-policy data generation from the draft model, and tailoring the divergence function to the task and decoding strategy. Notably, DistillSpec yields impressive 10 - 45% speedups over standard SD on a range of standard benchmarks, using both greedy and non-greedy sampling. Furthermore, we combine DistillSpec with lossy SD to achieve fine-grained control over the latency vs. task performance trade-off. Finally, in practical scenarios with models of varying sizes, first using distillation to boost the performance of the target model and then applying DistillSpec to train a well-aligned draft model can reduce decoding latency by 6-10x with minimal performance drop, compared to standard decoding without distillation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

    cs.CL 2026-08 conditional novelty 7.0 of 10

    AdaMTP uses entropy-based segmentation to adaptively mask multi-token prediction losses, improving quality and speed over fixed-horizon multi-token prediction.

  2. Weak-to-Strong Generalization via Direct On-Policy Distillation

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Transferring the log-ratio of a small model's pre-RL and post-RL checkpoints provides a dense implicit reward that improves stronger student models at a fraction of the cost of direct RL.

  3. Adaptive Accompaniment with ReaLchords

    cs.SD 2025-06 conditional novelty 6.0 of 10

    An online melody-to-chord accompaniment model, fine-tuned with reinforcement learning and distillation from a future-seeing teacher, recovers from cold starts and mid-song perturbations better than MLE baselines.

  4. Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism

    cs.DC 2025-05 conditional novelty 6.0 of 10

    On unified-memory edge hardware, Ghidorah partitions Medusa-style speculative decoding across CPU and GPU with all-column weight splits, sparse ARM kernels, and profile-based tuning, reporting up to 7.6x decode speedup.

  5. Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission

    cs.DC 2025-05 conditional novelty 6.0 of 10

    An uncertainty-aware hybrid language model skips and compresses uplink token transmissions, achieving up to 206 times higher token throughput with 97.4% accuracy in simulation.

  6. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AdaServe customizes speculative decoding per request to satisfy heterogeneous per-token latency SLOs, claiming up to 4.3x fewer violations and 1.9x higher goodput than baselines.

  7. Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A CPU/GPU heterogeneous speculative decoding system that reports 1.79x to 10.1x speedups for 7B-13B LLMs on consumer-grade hardware.

  8. Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A small on-device language model can skip sending most tokens to a remote large language model when its temperature-perturbation uncertainty is low, cutting uplink load by 45.93% and speeding token throughput 2.54x.

  9. Continuous Speculative Decoding for Autoregressive Image Generation

    cs.CV 2024-11 reject novelty 6.0 of 10

    Continuous speculative decoding accelerates continuous autoregressive image generation by over 2x while approximately maintaining output quality.

  10. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

    cs.CL 2026-07 conditional novelty 5.0 of 10

    AngelSpec + DFly pair a chat MTP drafter with a code/math block-diffusion drafter and load-aware verification pruning, reaching up to 2.4x AR throughput on Hy3-A21B.

  11. Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A gated LoRA and a small sampler let an autoregressive LLM draft multiple future tokens per step, and self-speculative verification converts those drafts into up to roughly 5x fewer generation steps.

  12. Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs

    cs.CL 2025-06 reject novelty 5.0 of 10

    ASTS extends locally typical sampling with semantic scoring and dynamic thresholds, reporting improved perplexity, MAUVE, and diversity on story and summarization tasks.

  13. Scaling Laws for Speculative Decoding

    cs.CL 2025-05 reject novelty 5.0 of 10

    Speculative decoding acceptance rate and throughput are described by empirical log-linear scaling laws in pretraining tokens, draft depth, and batch size, yielding the Scylla recipe for faster LLM inference.

  14. Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding

    cs.CL 2025-02 conditional novelty 5.0 of 10

    CoSD fuses two LLMs at decoding time: it keeps the draft model's tokens unless a confidence rule or a decision tree over token probabilities says the assistant model's token is better.

  15. Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration

    cs.LG 2024-11 conditional novelty 5.0 of 10

    A CTC-based draft model with sequence-level training and a CTC transform for candidate cleaning improves accepted-token counts and inference speedup in speculative decoding of LLMs.

  16. FastDraft: How to Train Your Draft

    cs.CL 2024-11 conditional novelty 5.0 of 10

    FastDraft trains small vocabulary-compatible draft models from scratch in under 24 hours, yielding up to 2x wall-clock and 3x memory-bound speedups for speculative decoding.

  17. Energy-Efficient Wireless LLM Inference via Uncertainty and Importance-Aware Speculative Decoding

    cs.LG 2025-08 reject novelty 4.0 of 10

    This paper proposes filtering cloud-verification requests by combining token-level uncertainty with attention-based importance, claiming energy savings up to 40.7% in wireless hybrid LLM inference.

  18. Automatic Task Detection and Heterogeneous LLM Speculative Decoding

    cs.CL 2025-05 conditional novelty 4.0 of 10

    TaskSpec clusters user queries into tasks, fine-tunes a separate small draft model for each task, and routes prompts to the right draft model to raise token acceptance in speculative decoding.

  19. Deploying Foundation Model Powered Agent Services: A Survey

    cs.DC 2024-12 accept novelty 4.0 of 10

    This survey proposes a layered framework (execution, resource, model, agent, application) for deploying foundation-model-powered agent services across edge-cloud environments, and reviews optimization techniques at ea...

Pith tools