Pith. sign in

REVIEW 3 cited by

Accelerating Production LLMs with Combined Token/Embedding Speculators

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.19124 v2 pith:L7LZX44F submitted 2024-04-29 cs.CL

Accelerating Production LLMs with Combined Token/Embedding Speculators

Davis Wertheimer , Joshua Rosenkranz , Thomas Parnell , Sahil Suneja , Pavithra Ranganathan , Raghu Ganti , Mudhakar Srivatsa This is my paper
classification cs.CL
keywords acceleratinginferencebasedraftmodelmodelspredictproduction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

This technical report describes the design and training of novel speculative decoding draft models, for accelerating the inference speeds of large language models in a production environment. By conditioning draft predictions on both context vectors and sampled tokens, we can train our speculators to efficiently predict high-quality n-grams, which the base model then accepts or rejects. This allows us to effectively predict multiple tokens per inference forward pass, accelerating wall-clock inference speeds of highly optimized base model implementations by a factor of 2-3x. We explore these initial results and describe next steps for further improvements.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

    cs.LG 2026-06 unverdicted novelty 7.0

    Develops theory for acceptance in speculative decoding under greedy/relaxed/tree criteria, with exact KL certificates and margin bounds, evaluated on Qwen3 models.

  2. An Empirical Study of Speculative Decoding on Software Engineering Tasks

    cs.SE 2026-04 unverdicted novelty 7.0

    Speculative decoding accelerates LLM inference on SE tasks without accuracy loss, with model-based methods suiting code generation and model-free methods suiting repository-level repair and editing.

  3. Speculative Decoding with a Speculative Vocabulary

    cs.CL 2026-02 conditional novelty 6.0

    SpecVocab dynamically selects a per-step candidate vocabulary for the speculative-decoding draft model, beating static 32K-token vocabularies on acceptance length and throughput (up to +8.1% vs EAGLE-3).