REVIEW 4 cited by
Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
We present a novel inference scheme, self-speculative decoding, for accelerating Large Language Models (LLMs) without the need for an auxiliary model. This approach is characterized by a two-stage process: drafting and verification. The drafting stage generates draft tokens at a slightly lower quality but more quickly, which is achieved by selectively skipping certain intermediate layers during drafting. Subsequently, the verification stage employs the original LLM to validate those draft output tokens in one forward pass. This process ensures the final output remains identical to that produced by the unaltered LLM. Moreover, the proposed method requires no additional neural network training and no extra memory footprint, making it a plug-and-play and cost-effective solution for inference acceleration. Benchmarks with LLaMA-2 and its variants demonstrated a speedup up to 1.99$\times$.
Forward citations
Cited by 4 Pith papers
-
Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
A post-conversion training objective that reconstructs the original attention function of MLA-converted draft models improves speculative-decoding acceptance in most of 64 matched task cells.
-
Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
On an Apple-silicon laptop, speculative decoding gave a 1.61x speedup for one 0.5B-to-3B pairing, while three of five pairings were slower because of framework overhead and serial Metal verification.
-
Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphs
Spiffy speeds up diffusion LLM inference up to about 3x (and up to 7.9x with parallel decoding) by verifying multiple candidate unmasked states in one batched model call, while preserving greedy output.
-
RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding
RAD selects redundant attention layers via self-speculative decoding throughput, replaces them with SSM blocks, and distills them into hybrid models that beat the original on GSM8K and CRUX.
Discussion (0). Continue with ORCID to comment.