Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SPEED-Bench is a benchmark for speculative decoding that claims speedups depend heavily on the prompt distribution and serving regime, and that standardized diverse workloads are needed to measure them fairly.

desk verdict A genuinely useful benchmark for speculative decoding, with a diversity-driven split that is broader but not proven representative — worth reviewing. read the letter →

arxiv 2604.09557 v2 pith:EEWGKQH2 submitted 2026-02-10 cs.DC cs.AI

Talor Abramovich , Maor Ashkenazi , Izzy Putterman , Benjamin Chislett , Tiyasa Mitra , Bita Darvish Rouhani , Ran Zilberstein , Yonatan Geifman This is my paper · ORCID
classification cs.DCcs.AI
keywords speculativedecodingLLMinferencebenchmarksemanticdiversitythroughputevaluationacceptanceratedraftlengthdataselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SPEED-Bench is a benchmark for speculative decoding (SD), a technique that uses a small draft model to guess several tokens ahead and then verifies them in one pass, speeding up LLM inference. The paper argues that SD speedups depend heavily on what text the model is given and on the serving conditions, so existing evaluations—often small, repetitive, and run one request at a time—mislead. To fix that, SPEED-Bench provides a qualitative split whose prompts are chosen to maximize semantic diversity, and a throughput split with fixed input-length buckets from 1k to 32k tokens to support batch-size scaling. A unified measurement framework runs the same tokenized sequences in production-grade serving engines, and the paper uses it to show that synthetic token inputs overestimate throughput, that optimal draft length shifts with batch size, and that vocabulary pruning degrades some domains more than others. A sympathetic reader would take away that SD comparisons need standardized, diverse, and realistic workloads to be trustworthy.

What carries the argument

The load-bearing object is the greedy subset-selection algorithm with local swap refinement: it minimizes the sum of pairwise cosine similarities between row-normalized prompt embeddings, so the chosen prompts span the semantic space rather than clustering. For the throughput regime, the key identity is Speedup = (t_ar * AL) / t_sd, which separates domain-dependent acceptance length (AL) from system-dependent per-step latencies; the Throughput Split provides realistic workloads for measuring those latencies, allowing domain-specific speedups to be estimated without exhaustive testing.

What would settle it

Sample, say, 1,000 prompts uniformly from logged production traffic for a coding assistant and a chat service, run a fixed drafter-target pair through the same measurement framework, and compare category-level acceptance lengths and the ranking across domains with SPEED-Bench's results; if the rankings diverge materially, the diversity-selection premise fails. Similarly, if random-token prompts on a mixture-of-experts model activate experts as evenly as real prompts do, the claimed synthetic-input distortion collapses.

Watch

Extended reading notes

Core claim

The paper argues that speculative decoding speed cannot be judged on narrow or synthetic workloads; evaluation needs semantically diverse, realistic data and production-grade measurement. Its Qualitative Split selects prompts by minimizing pairwise cosine similarity among embeddings via greedy selection with local swap refinement, cutting average semantic similarity by about 40% versus an earlier benchmark. A Throughput Split adds real prompts in fixed input-length buckets (1k–32k tokens) across three entropy tiers, enabling throughput-latency Pareto curves. A measurement framework normalizes tokenization and chat templates so all engines process identical sequences. Empirically, random-toke

Load-bearing premise

The whole benchmark rests on the assumption that maximizing semantic diversity in embedding space (cosine similarity) produces a prompt set whose diversity matches the real-world data distribution that matters for speculative decoding; nothing in the paper independently verifies that connection.

Editorial extensions

If this is right

  • If the benchmark is adopted, published SD speedups will become comparable across methods, because all measurements run on the same diverse prompts and the same underlying engine implementations.
  • Synthetic token inputs should be retired for SD benchmarking: the paper measures a 23% average overestimate in throughput, and shows random inputs also distort expert routing in mixture-of-experts models.
  • Optimal draft length is not a fixed property of a drafter; practitioners need to re-tune it for their batch size, since shorter drafts win in compute-bound high-batch regimes.
  • Vocabulary-pruned drafters should be expected to lose accuracy on multilingual, retrieval-augmented, and summarization workloads even when math and coding look fine.
  • Drafter training context length matters: models trained on short sequences degrade sharply beyond their training input length, and long-context scaling like YaRN recovers much of the accuracy.
  • The authors identify a boundary on their own framework: above about 256 concurrent requests, the Python client-side loop can become a bottleneck, so the highest-concurrency regime is not yet fully covered.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the selection algorithm optimizes embedding-space cosine distance, but embedding diversity is only a proxy for the task-level properties that determine acceptance rates; a validation against randomly sampled production traffic would strengthen the claim of representativeness.
  • Editorial inference: the speedup decomposition suggests a calibration protocol: measuring per-step latencies once per hardware/engine/batch configuration and reusing them with domain-specific ALs could let the community extrapolate speedups to new domains cheaply.
  • Editorial inference: the paper's finding that low-diversity benchmarks hide drafter differences implies that previously published comparisons on small, repetitive datasets may need re-checking, especially for multilingual and long-context claims.
  • Boundary noted by the authors: their measurement client can become a bottleneck above roughly 256 concurrent requests, so the very-high-concurrency regime is not fully covered.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SPEED-Bench proposes a benchmark suite for speculative decoding (SD) with two data splits and a measurement framework. The Qualitative Split selects 880 prompts from 18 public sources by greedily minimizing pairwise cosine similarity among prompt embeddings (Eq. 2, Algorithm 1), yielding 11 categories with 80 samples each. The Throughput Split provides fixed-ISL buckets (1k–32k) across three entropy classes for throughput-latency evaluation at batch sizes up to 512. The framework integrates with vLLM, TensorRT-LLM, SGLang, and SpecBench, and the paper reports experiments on Llama 3.3 70B, GPT-OSS 120B, DeepSeek R1, Qwen3 235B, and Qwen3-Next, covering acceptance rates, speedups, vocabulary pruning, synthetic-input artifacts, draft-length selection, and training-ISL effects. The paper claims SPEED-Bench will establish a unified evaluation standard for practical SD comparisons.

Significance. If the dataset and measurement pipeline are sound, SPEED-Bench is a useful step toward standardized SD evaluation. Its strengths include: the use of production-grade engines rather than high-level stubs; the Throughput Split, which addresses a real gap in batch-size/ISL coverage; the validation of a proxy formula for domain-specific speedups (Appendix H); and the demonstrations that synthetic random inputs distort both SD and baseline MoE throughput (Appendix G) and that vocabulary pruning degrades multilingual and RAG performance. The paper is also transparent about its conflict of interest and about acknowledged limitations (e.g., the GIL at BS>256, the unconfirmed causes of long-context EAGLE3 degradation). The main risk is whether the Qualitative Split's diversity-maximized subset is representative enough of real workloads to support the paper's strong claim of a 'unified evaluation standard.'

major comments (3)
  1. [§5, Eq. (2), Algorithm 1; Appendix C] The Qualitative Split is constructed by minimizing pairwise cosine similarity in an embedding space. This is a coverage/diversity objective, not a representativeness objective: it selects mutually dissimilar prompts that tend to lie in low-density regions and down-weights common, clustered prompt types. Appendix C shows that the optimized subset is stable across random seeds, but stability under resampling is not the same as unbiasedness relative to the source distribution or to real deployment workloads. Since the aggregate AL/speedup numbers in Table 1 and the comparisons with SpecBench in §8.3 rest entirely on this subset, the paper should either (a) validate that aggregate conclusions (draft-method rankings, category orderings) agree with those obtained from a uniform/stratified sample of the same source pools, or (b) compare the selected prompts against a held-out realistic workload
  2. [Table 1; §8.3 and Figure 5] The central quantitative results are reported without confidence intervals, standard errors, or repeated-run variation. With 80 samples per category and, apparently, a single measurement per cell, small differences between methods (e.g., the mean AL values of 2.44 for Vanilla and EAGLE3 on Llama 3.3 70B at Temperature=0; the mean AL differences across domains) cannot be assessed statistically. The claims that SPEED-Bench 'corrects' the SpecBench impression, or that one drafter outperforms another, need error bars or significance tests. This is especially important because the stated purpose of the benchmark is to enable reliable cross-method comparisons.
  3. [§5, 'Selection Algorithm'] The entire Qualitative Split depends on a single pre-trained embedder, OpenAI's text-embedding-3-large, used to compute cosine similarities. There is no evidence that the selected subset is stable across embedding models or that this embedding geometry is aligned with the task properties that actually determine SD acceptance rates and acceptance lengths. I request an ablation: recompute the selection with at least one other embedder (or an ensemble) and report subset overlap and the resulting differences in per-category AL/speedups. Without such evidence, the benchmark's diversity claims and the subsequent empirical conclusions are tied to a single, proprietary embedding choice.
minor comments (5)
  1. [Abstract and Conclusion] The abstract says 'unified evaluation standard' and the conclusion says 'establishes a unified evaluation ecosystem.' This is stronger than what the evidence currently supports, given the representativeness concern above; consider using 'a step toward' or 'a benchmark for.'
  2. [§5 vs Appendix A] The text says the Qualitative Split aggregates '18 publicly available datasets,' while Table 2 reports '24 distinct data sources' for SPEED-Bench. Please clarify whether this counts only the Qualitative Split or the union of both splits.
  3. [Table 2] Minor typo: 'Lanuages' should be 'Languages.' Also, the table lists 'Programming Lanuages' and 'Lanuages in Multilingual' with inconsistent capitalization.
  4. [References] The reference list contains duplicate entries for See et al. 2017 (2017a and 2017b with the same title and pages). Please deduplicate and use a single citation.
  5. [§7] The note that the asyncio/GIL introduces overhead at BS>256 is honest, but it would help to state explicitly which experimental points in Figures 6 and 7 are within the validated regime.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is constructed by explicit, non-fitted selection rules and its claims are supported by direct measurements against external baselines.

full rationale

SPEED-Bench is a benchmark and measurement paper, not a derivation. The Qualitative Split is deliberately constructed to maximize embedding-space diversity (Eq. 2, Algorithm 1); this is a transparent design choice, not a fitted parameter later relabeled as a prediction. Acceptance lengths, speedups, and throughputs are measured directly from engine outputs, not derived from the selection objective. The comparisons with SpecBench in §8.3 are independent empirical measurements on differently sourced data, and the finding that low-diversity benchmarks hide differences is a consequence of the construction but is presented as a demonstration, not as a deduced law. The proxy Equation 3 in Appendix H is an algebraic identity, but it is used explicitly and checked against directly measured end-to-end speedups (Table 10), so it is not a hidden redefinition. The paper contains self-referential elements—NVIDIA employment, TensorRT-LLM, NVIDIA draft checkpoints—but these are disclosed conflicts of interest and are not used as evidence for the central claims. The stated limitations (GIL overhead at BS>256, unconfirmed causes of long-context degradation) are candid and do not hide circularity. The representativeness question raised about the diversity-maximizing selection is a validity concern, not a circularity concern; it does not reduce any claimed result to its own inputs. Therefore no circular step meets the evidentiary bar.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim of benchmark representativeness rests on design choices (embedding-based diversity, entropy labels, and the measurement framework) that are assumptions rather than measured facts. The paper provides some validation but does not independently establish these assumptions.

assumptions (3)
  • domain assumption Maximizing semantic diversity in the embedding space (OpenAI text-embedding-3-large) yields a dataset representative of real-world SD workloads.
    The selection algorithm in Section 5 minimizes pairwise cosine similarity, but there is no independent evidence that this diversity metric aligns with the dimensions that affect speculative decoding accuracy.
  • domain assumption The entropy categorization (Low/Mixed/High) correctly reflects the difficulty of speculative decoding across domains.
    The Throughput Split groups data by domain entropy based on a taxonomy from Li et al., but AL measurements only partially validate this ordering (Appendix J shows GPT-OSS EAGLE3 deviates from expected ordering).
  • domain assumption The measurement framework accurately captures streaming response timings without engine-specific overheads.
    Section 7 notes a GIL limitation at BS>256, so timing accuracy is limited at very high concurrency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding." pith.science (2026). https://pith.science/paper/EEWGKQH2

@misc{pith2026260409557,
  author       = {Pith},
  title        = {Pith review of: SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EEWGKQH2}},
  note         = {Machine review of arXiv:2604.09557}
}
read the original abstract

Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optimizations, SD performance is inherently data-dependent, meaning that diverse and representative workloads are essential for accurately measuring its effectiveness. Existing benchmarks suffer from limited task diversity, inadequate support for throughput-oriented evaluation, and a reliance on high-level implementations that fail to reflect production environments. To address this, we introduce SPEED-Bench, a comprehensive suite designed to standardize SD evaluation across diverse semantic domains and realistic serving regimes. SPEED-Bench offers a carefully curated Qualitative data split, selected by prioritizing semantic diversity across the data samples. Additionally, it includes a Throughput data split, allowing speedup evaluation across a range of concurrencies, from latency-sensitive low-batch settings to throughput-oriented high-load scenarios. By integrating with production engines like vLLM and TensorRT-LLM, SPEED-Bench allows practitioners to analyze system behaviors often masked by other benchmarks. We highlight this by quantifying how synthetic inputs overestimate real-world throughput, identifying batch-size dependent optimal draft lengths and biases in low-diversity data, and analyzing the caveats of vocabulary pruning in state-of-the-art drafters. We release SPEED-Bench to establish a unified evaluation standard for practical comparisons of SD algorithms.

Figures

Figures reproduced from arXiv: 2604.09557 by the authors.

Figure 1
Figure 1. Overview of the SPEED-Bench ecosystem. (Left) Curation of the Qualitative split, utilizing a custom selection algorithm on prompt embeddings to maximize semantic diversity across categories. (Middle) Construction of the Throughput Split, where data is aggregated and processed into fixed Input Sequence Length (ISL) buckets (1k-32k) across three domain difficulties, supporting large batch sizes (up to 512 per ISL and … view at source ↗
Figure 2
Figure 2. Comparison of average semantic similarity between samples (lower is better). SPEED-Bench achieves lower similarity than both random selection and SpecBench across all categories. with Local Swap Refinement (see Algorithm 1). We initialize S with a random index and iteratively append i ∗ = argmini /∈S P j∈S x ⊤ i xj . To escape local minima, we then iteratively swap iout ∈ S with iin ∈/ S if the swap strictly decreas… view at source ↗
Figure 3
Figure 3. Average AL on the Qualitative Split. External drafting scales better across DLs. the SpecBench framework excels at evaluating methods us￾ing native PyTorch/HuggingFace, SPEED-Bench focuses on the viability of these methods in deployment. To support a holistic pipeline, we demonstrate how SpecBench models can be evaluated within our framework. The supplementary material includes an example for SpecBench’s Medusa, and… view at source ↗
Figures from the paper (21 more)
Figure 5
Figure 5. Figure 5: Average AL across selected categories in SpecBench vs SPEED-Bench. Target model is Llama 3.3 70B. DL = 7. Full results are in Appendix K. narios. Unlike methods that focus on latency at BS = 1, SPEED-Bench enables the construction of throughput￾latency Pareto curves, p…
Figure 4
Figure 4. Figure 4: Average AL across selected categories using GPT-OSS 120B and EAGLE3 drafters (full vs. pruned vocabulary), DL = 3. GLE3 applies vocabulary pruning (usually to 32k tokens), mitigating the computational bottlenecks of the final projec￾tion layer. While effective for stan…
Figure 6
Figure 6. Figure 6: Throughput as a function of user TPS, comparing random input tokens to the Throughput Split (8k). Target is GPT-OSS 120B with EAGLE3 drafter, measured on TensorRT-LLM. DL = 3. Points represent BS from 1 to 128. 0 100 200 300 400 User TPS 0 5000 10000 15000 Output TPS p…
Figure 7
Figure 7. Figure 7: Throughput as a function of user TPS, comparing DL = 1, 3 on the Throughput Split (2k). Target is GPT-OSS 120B with EAGLE3, measured on vLLM. Points represent BS from 2 to 512. in Appendix F: random inputs fail to trigger realistic expert routing in the MoE target mode…
Figure 9
Figure 9. Figure 9: Pairwise similarity matrices for the ’Translation/Multilingual’ category. SpecBench (left) shows dense blocks of high similarity, indicating redundant data. SPEED-Bench (right) shows a dispersed, low-similarity distribution, demonstrating better semantic diversity [PI…
Figure 8
Figure 8. Figure 8: Average AL as a function of ISLs, comparing training ISLs. Target is GPT-OSS 120B with EAGLE3 drafters, measured on vLLM. Dotted lines denote YaRN scaling. Legend labels (1k, 2k, 4k) indicate the maximum ISL used during training. DL = 3. Throughput Split focused on lar…
Figure 10
Figure 10. Figure 10: display the pairwise cosine similarity matrices for two categories: Translation/Multilingual and Math, respectively. In these heatmaps, darker green values indicate high semantic similarity (redundancy), while lighter yellow values indicate low similarity (diversity).…
Figure 9
Figure 9. Figure 9: Pairwise similarity matrices for the ’Translation/Multilingual’ category. SpecBench (left) shows dense blocks of high similarity, indicating redundant data. SPEED-Bench (right) shows a dispersed, low-similarity distribution, demonstrating better semantic diversity [PI…
Figure 11
Figure 11. Figure 11: illustrates the activation frequency of the top-k experts for a middle layer (Layer 17) in GPT-OSS 120B during the prefill of 8k ISL inputs at a batch size of 32. While SPEED-Bench inputs result in a relatively uniform activation profile, random tokens lead to signifi…
Figure 10
Figure 10. Figure 10: display the pairwise cosine similarity matrices for two categories: Translation/Multilingual and Math, respectively. In these heatmaps, darker green values indicate high semantic similarity (redundancy), while lighter yellow values indicate low similarity (diversity).…
Figure 12
Figure 12. Figure 12: tracks the total number of unique experts activated across layers of the model. Notably, processing random tokens fails to activate 20-30% of available experts in certain layers. This lack of coverage is interesting given the high volume of tokens (32 × 8000), confirm…
Figure 11
Figure 11. Figure 11: illustrates the activation frequency of the top-k experts for a middle layer (Layer 17) in GPT-OSS 120B during the prefill of 8k ISL inputs at a batch size of 32. While SPEED-Bench inputs result in a relatively uniform activation profile, random tokens lead to signifi…
Figure 13
Figure 13. Figure 13: presents the average AL as a function of ISL for three setups. For Vanilla SD (Llama 3.3 70B) and Native MTP (Qwen3-Next), we observe the expected behavior: Low Entropy prompts (e.g., coding, sorting) yield the highest ALs. High Entropy prompts (e.g., creative writing…
Figure 12
Figure 12. Figure 12: tracks the total number of unique experts activated across layers of the model. Notably, processing random tokens fails to activate 20-30% of available experts in certain layers. This lack of coverage is interesting given the high volume of tokens (32 × 8000), confirm…
Figure 14
Figure 14. Figure 14: Average AL across all categories in SpecBench vs. SPEED-Bench. Target model is Llama 3.3 70B. DL=7, BS=32. L. Inference Engine Comparison In Section 8.4, we briefly discussed the performance differences between inference backends. Here we provide the full comparison b…
Figure 13
Figure 13. Figure 13: presents the average AL as a function of ISL for three setups. For Vanilla SD (Llama 3.3 70B) and Native MTP (Qwen3-Next), we observe the expected behavior: Low Entropy prompts (e.g., coding, sorting) yield the highest ALs. High Entropy prompts (e.g., creative writing…
Figure 15
Figure 15. Figure 15: compares the throughput of TensorRT-LLM and vLLM. Both frameworks are orchestrated in Python, which can introduce host synchronization overhead and kernel launch latency compared to C++ implementations. To mitigate this, both engines leverage CUDA Graphs to capture an…
Figure 14
Figure 14. Figure 14: Average AL across all categories in SpecBench vs. SPEED-Bench. Target model is Llama 3.3 70B. DL=7, BS=32. M. Inference Engine Comparison In Section 8.4, we briefly discussed the performance differences between inference backends. Here we provide the full comparison b…
Figure 16
Figure 16. Figure 16: AL Stability across various models. Average AL measured on the Throughput Split buckets (1k–32k). Target is GPT-OSS 120B, with three EAGLE3 drafters. Carefully configured RoPE scaling can ensure stability over all context lengths. 24 [PITH_FULL_IMAGE:figures/full_fig…
Figure 15
Figure 15. Figure 15: compares the throughput of TensorRT-LLM and vLLM. Both frameworks are orchestrated in Python, which can introduce host synchronization overhead and kernel launch latency compared to C++ implementations. To mitigate this, both engines leverage CUDA Graphs to capture an…
Figure 16
Figure 16. Figure 16: AL Stability across various models. Average AL measured on the Throughput Split buckets (1k–32k). Target is GPT-OSS 120B, with three EAGLE3 drafters. Carefully configured RoPE scaling can ensure stability over all context lengths. 28 [PITH_FULL_IMAGE:figures/full_fig…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    PSD is a training-free framework that jointly optimizes spatial unmasking and temporal speculative decoding in diffusion LLMs to reach up to 5.5x tokens per forward pass while preserving accuracy comparable to greedy ...

  2. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0 of 10

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  3. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

    cs.AI 2026-07 accept novelty 6.0 of 10

    DSpark's semi-autoregressive drafter plus load-aware confidence scheduling raises accepted draft length and shifts the production serving Pareto frontier by 60-85% higher per-user speed at matched throughput versus MTP-1.

  4. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs

    cs.AI 2026-07 conditional novelty 5.5 of 10

    Iterative Puzzle plus KD, RL, quantization, and MTP compresses Nemotron-3-Super to 75B total / 9B active parameters with ~2× interactive throughput and 8× 1M-context concurrency while retaining most parent accuracy.

  5. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0 of 10

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

Reference graph

Works this paper leans on

57 extracted references · 23 linked inside Pith · cited by 5 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Pard: Accelerating llm inference with low-cost parallel draft model adaptation

    An, Z., Bai, H., Liu, Z., Li, D., and Barsoum, E. Pard: Accelerating llm inference with low-cost parallel draft model adaptation. arXiv preprint arXiv:2504.18583, 2025

    arXiv 2025

  4. [4]

    Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues

    Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., et al. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762, 2024

    arXiv 2024

  5. [5]

    Nvidia nemotron 3: Efficient and open intelligence

    Blakeman, A., Grattafiori, A., Basant, A., Gupta, A., Khattar, A., Renduchintala, A., Vavre, A., Shukla, A., Bercovich, A., Ficek, A., et al. Nvidia nemotron 3: Efficient and open intelligence. arXiv preprint arXiv:2512.20856, 2025

    arXiv 2025

  6. [6]

    Long code arena: a set of benchmarks for long-context code models

    Bogomolov, E., Eliseeva, A., Galimzyanov, T., Glukhov, E., Shapkin, A., Tigina, M., Golubev, Y., Kovrigin, A., van Deursen, A., Izadi, M., and Bryksin, T. Long code arena: a set of benchmarks for long-context code models. arXiv preprint arXiv:2406.11612, 2024

    arXiv 2024

  7. [7]

    Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., Soricut, R., Specia, L., and Tamchyna, A. s. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp.\ 12--58, Baltimore, Maryland, USA, June 2014. A...

    2014

  8. [8]

    D., Chen, D., and Dao, T

    Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning, pp.\ 5209--5235. PMLR, 2024

    2024

Show all 57 references
  1. [9]

    Accelerating large language model decoding with speculative sampling

    Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    2023 arXiv

  2. [10]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., B...

    2021

  3. [11]

    Sequoia: Scalable, robust, and hardware-aware speculative decoding

    Chen, Z., May, A., Svirschevski, R., Huang, Y., Ryabinin, M., Jia, Z., and Chen, B. Sequoia: Scalable, robust, and hardware-aware speculative decoding. CoRR, 2024

    2024

  4. [12]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  5. [13]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 3029--3051, 2023

    2023

  6. [14]

    X., and Wen, J.-R

    Dong, Z., Tang, T., Li, J., Zhao, W. X., and Wen, J.-R. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. arXiv preprint arXiv:2309.13345, 2023

    2023 arXiv

  7. [15]

    Break the sequential dependency of llm inference using lookahead decoding

    Fu, Y., Bailis, P., Stoica, I., and Zhang, H. Break the sequential dependency of llm inference using lookahead decoding. In Forty-first International Conference on Machine Learning

  8. [16]

    The llama 3 herd of models, 2024

    Grattafiori, A., Dubey, A., Jauhri, A., and et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  10. [18]

    REST : Retrieval-based speculative decoding

    He, Z., Zhong, Z., Cai, T., Lee, J., and He, D. REST : Retrieval-based speculative decoding. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol...

    2024 doi

  11. [19]

    Moesd: Unveil speculative decoding's potential for accelerating sparse moe

    Huang, Z., Zhu, L., Zhan, Z., Hu, T., Mao, W., Yu, X., Liu, Y., and Zhang, T. Moesd: Unveil speculative decoding's potential for accelerating sparse moe. arXiv preprint arXiv:2505.19645, 2025

    2025

  12. [20]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  13. [21]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  14. [22]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  15. [23]

    M., Ghaddar, A., Sun, Q., Ma, L., Luo, Y., Li, D., Coates, M., Hao, J., and Zhang, Y

    Li, D., Zhou, J., Brunswic, L. M., Ghaddar, A., Sun, Q., Ma, L., Luo, Y., Li, D., Coates, M., Hao, J., and Zhang, Y. Omni-thinker: Scaling multi-task rl in llms with hybrid reward and task scheduling, 2025 a . URL https://arxiv.org/abs/2507.14783

    2025

  16. [24]

    Competition-level code generation with alphacode

    Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d'Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D., S...

    2022 arXiv

  17. [25]

    EAGLE : Speculative sampling requires rethinking feature uncertainty

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE : Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024 a

    2024

  18. [26]

    EAGLE-2 : Faster inference of language models with dynamic draft trees

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE-2 : Faster inference of language models with dynamic draft trees. In Empirical Methods in Natural Language Processing, 2024 b

    2024

  19. [27]

    EAGLE-3 : Scaling up inference acceleration of large language models via training-time test

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE-3 : Scaling up inference acceleration of large language models via training-time test. In Annual Conference on Neural Information Processing Systems, 2025 b

    2025

  20. [28]

    Repobench: Benchmarking repository-level code auto-completion systems, 2024 a

    Liu, T., Xu, C., and McAuley, J. Repobench: Benchmarking repository-level code auto-completion systems, 2024 a . URL https://arxiv.org/abs/2306.03091

    2024 arXiv

  21. [29]

    Chatqa: Surpassing gpt-4 on conversational qa and rag

    Liu, Z., Ping, W., Roy, R., Xu, P., Lee, C., Shoeybi, M., and Catanzaro, B. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint arXiv:2401.10225, 2024 b

    2024 arXiv

  22. [30]

    X., Sha, J., Wang, S., and Wen, J.-R

    Luo, W., Zhao, W. X., Sha, J., Wang, S., and Wen, J.-R. Mmath: A multilingual benchmark for mathematical reasoning. arXiv preprint arXiv:2505.19126, 2025

    2025 arXiv

  23. [31]

    Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Confere...

    2024

  24. [32]

    Y., Singh, S., Tang, X., von Werra, L., and Longpre, S

    Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., and Longpre, S. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124, 2023

    2023 arXiv

  25. [33]

    Tensorrt‑llm: High‑performance inference for large language models

    NVIDIA . Tensorrt‑llm: High‑performance inference for large language models. https://github.com/NVIDIA/TensorRT-LLM, 2023. Accessed: 2026‑01‑06

    2023

  26. [34]

    gpt-oss-120b & gpt-oss-20b model card, 2025

    OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925

    2025 arXiv

  27. [35]

    Paech, S. J. Eq-bench creative writing benchmark v3. https://github.com/EQ-bench/creative-writing-bench, 2025

    2025

  28. [36]

    Mcif: Multimodal crosslingual instruction-following benchmark from scientific talks, 2025

    Papi, S., Züfle, M., Gaido, M., Savoldi, B., Liu, D., Douros, I., Bentivogli, L., and Niehues, J. Mcif: Multimodal crosslingual instruction-following benchmark from scientific talks, 2025. URL https://arxiv.org/abs/2507.19634

    2025

  29. [37]

    Ya RN : Efficient context window extension of large language models

    Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Ya RN : Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=wHBfxhZu1u

    2024

  30. [38]

    Humanity's last exam, 2025

    Phan, L., Gatti, A., Han, Z., and et al. Humanity's last exam, 2025. URL https://arxiv.org/abs/2501.14249

    2025 arXiv

  31. [39]

    Project gutenberg

    Project Gutenberg . Project gutenberg. https://www.gutenberg.org

  32. [40]

    E.-H., May, A., Chen, T., and Chen, B

    Sadhukhan, R., Chen, J., Chen, Z., Tiwari, V., Lai, R., Shi, J., Yen, I. E.-H., May, A., Chen, T., and Chen, B. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In The Thirteenth International Conference on Learning Repr...

  33. [42]

    J., and Manning, C

    See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1073--1083, Vancouver, Canada, July 2017 b . Associ...

    2017 doi

  34. [43]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  35. [44]

    Ada-leval: Evaluating long-context llms with length-adaptable benchmarks

    Wang, C., Duan, H., Zhang, S., Lin, D., and Chen, K. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol...

    2024

  36. [45]

    Coser: Coordinating llm-based persona simulation of established roles, 2025

    Wang, X., Wang, H., Zhang, Y., Yuan, X., Xu, R., tse Huang, J., Yuan, S., Guo, H., Chen, J., Wang, W., Xiao, Y., and Zhou, S. Coser: Coordinating llm-based persona simulation of established roles, 2025. URL https://arxiv.org/abs/2502.09082

    2025

  37. [46]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 0 95266--95290, 2024 b

    2024

  38. [47]

    M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Zhang, M., Zhang, Z., Ouyang, W., Xu, K., Chen, W., Fu, J., and Peng, J

    Wang, Z. M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Zhang, M., Zhang, Z., Ouyang, W., Xu, K., Chen, W., Fu, J., and Peng, J. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv: ...

    2023 arXiv

  39. [48]

    L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: St...

    2020

  40. [49]

    Writingbench: A comprehensive benchmark for generative writing, 2025

    Wu, Y., Mei, J., Yan, M., Li, C., Lai, S., Ren, Y., Wang, Z., Zhang, J., Wu, M., Jin, Q., and Huang, F. Writingbench: A comprehensive benchmark for generative writing, 2025. URL https://arxiv.org/abs/2503.05244

    2025

  41. [50]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Compu...

    2024 doi

  42. [51]

    Parallelspec: Parallel drafter for efficient speculative decoding

    Xiao, Z., Zhang, H., Ge, T., Ouyang, S., Ordonez, V., and Yu, D. Parallelspec: Parallel drafter for efficient speculative decoding. arXiv preprint arXiv:2410.05589, 2024

    2024 arXiv

  43. [52]

    Mimo-v2-flash technical report, 2026

    Xiaomi, L.-C. Mimo-v2-flash technical report, 2026. URL https://arxiv.org/abs/2601.02780

    2026 arXiv

  44. [53]

    Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024

    2024 arXiv

  45. [54]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

    2025 arXiv

  46. [55]

    Longspec: Long-context lossless speculative decoding with efficient drafting and verification

    Yang, P., Du, C., Zhang, F., Wang, H., Pang, T., Du, C., and An, B. Longspec: Long-context lossless speculative decoding with efficient drafting and verification. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, 2025 b

    2025

  47. [56]

    Improving massively multilingual neural machine translation and zero-shot translation

    Zhang, B., Williams, P., Titov, I., and Sennrich, R. Improving massively multilingual neural machine translation and zero-shot translation. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computat...

    2020 doi

  48. [57]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36: 0 46595--46623, 2023

    2023

  49. [58]

    L., Huang, J., Yu, C

    Zheng, L., Yin, L., Xie, Z., Sun, C. L., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. Sglang: Efficient execution of structured language model programs. In Conference on Neural Information Processing Systems (NeurIPS), 2...

    2024 doi arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.