{"paper":{"title":"Accelerating Large Language Model Decoding with Speculative Sampling","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Speculative sampling generates multiple tokens per transformer call by drafting sequences from a smaller model and verifying them with rejection sampling that matches the target distribution exactly.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Charlie Chen, Geoffrey Irving, Jean-Baptiste Lespiau, John Jumper, Laurent Sifre, Sebastian Borgeaud","submitted_at":"2023-02-02T18:44:11Z","abstract_excerpt":"We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model. This is combined with a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics. We benchmark speculative sampling with Chinchilla, a 70 billion p"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model, achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Speculative sampling accelerates LLM decoding 2-2.5x by letting a draft model propose short sequences that the target model scores in parallel, then applies modified rejection sampling to keep the exact target distribution.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Speculative sampling generates multiple tokens per transformer call by drafting sequences from a smaller model and verifying them with rejection sampling that matches the target distribution exactly.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"410fdf7b93229e0572ea1e04357819a74fd374865f4023418fa69d87ee9353ee"},"source":{"id":"2302.01318","kind":"arxiv","version":1},"verdict":{"id":"6be6da2d-8aed-435c-940e-c30badc3424b","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-11T07:23:19.309503Z","strongest_claim":"We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model, achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself.","one_line_summary":"Speculative sampling accelerates LLM decoding 2-2.5x by letting a draft model propose short sequences that the target model scores in parallel, then applies modified rejection sampling to keep the exact target distribution.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model.","pith_extraction_headline":"Speculative sampling generates multiple tokens per transformer call by drafting sequences from a smaller model and verifying them with rejection sampling that matches the target distribution exactly."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2302.01318/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":18,"sample":[{"doi":"","year":2021,"title":"Vivit: Avideovisiontransformer","work_id":"31194d14-e3e7-41f9-995f-fc580e37c0b0","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1901,"title":"T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners.Advances in neural information processi","work_id":"8e180b3a-a506-460e-b38b-5d27a2fb6ab2","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Evaluating Large Language Models Trained on Code","work_id":"042493e9-b26f-4b4e-bbde-382072ca9b08","ref_index":4,"cited_arxiv_id":"2107.03374","is_internal_anchor":true},{"doi":"","year":null,"title":"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale","work_id":"98201f98-f4e5-4d1c-9ed7-b795e3c8f76c","ref_index":5,"cited_arxiv_id":"2208.07339","is_internal_anchor":true},{"doi":"","year":2010,"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","work_id":"e96730e3-129b-4db6-b981-15ab7932e297","ref_index":6,"cited_arxiv_id":"2010.11929","is_internal_anchor":true}],"resolved_work":18,"snapshot_sha256":"5a32f7e842cb7f417ad0524bbac599ad5e13e5dcab2ba23d72943d8c368ed890","internal_anchors":7},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}