Pith. sign in

REVIEW 5 major objections 5 minor 38 references

Scaling Laws for Speculative Decoding

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims speculative decoding efficiency follows three log-linear scaling laws—over pretraining tokens, draft decoder count, and batch size—and that applying them yields up to 2x throughput.

desk verdict Useful qualitative recipe buried under quantitative scaling laws that the paper's own figures don't support. read the letter →

arxiv 2505.07858 v1 pith:ZLIMPOPV submitted 2025-05-08 cs.CL cs.AI

Siyuan Yan , Mo Zhu , Guo-qing Jiang , Jianfei Wang , Jiaxing Chen , Wentai Zhang , Xiang Liao , Xiao Cui
show 3 more authors
Chen Zhang Zhuoran Song Ran Zhu
This is my paper · ORCID
classification cs.CLcs.AI
keywords speculativedecodingscalinglawsacceptanceratedraftmodelrooflineTopKtreebatchsizethroughput
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that speculative decoding—generating draft tokens with a small model and verifying them in parallel with the large model—is governed by simple log-linear scaling laws, just as LLM pretraining loss is. Specifically, the draft's acceptance rate grows logarithmically with pretraining tokens (α≈0.08, β≈5.05) and with draft decoder count (α≈0.74, β≈4.61), while throughput grows logarithmically with batch size only when the number of parallel TopK verification paths is chosen optimally. The paper derives an optimal-TopK formula topk(b)=27904√(1+0.034/b)−27897 from a roofline model that balances compute and memory traffic. If these laws are right, draft-model training budgets, decoder depth, and batch-adaptive tree sizing can all be set by formula instead of by search. The authors combine the three laws into a system called Scylla and report acceptance rates 1.5–2.2 points higher than a strong baseline and roughly 2x decoding throughput in deployment.

What carries the argument

The mechanism that carries the argument is a pair of empirical laws plus a roofline model. The pretrain and capacity laws are log-linear regressions of measured acceptance rate against log10 of pretraining tokens and decoder count, fitted over seven token budgets (1B–100B) and decoder depths (1, 2, 5, 10). The batch-size law comes from a roofline analysis: the paper hand-counts FLOPs and memory reads/writes for every component of a Scylla forward pass (draft FC layer, QKV projections, self-attention, MLP, LM head), forms the arithmetic intensity I(b,topk)=W_comp/W_mem, and asserts that maximum throughput occurs exactly where I equals the hardware critical intensity I_crit=P_peak/B_mem. Solving that equality for Qwen2.5-72B with 10K prefill yields the optimal-TopK formula. The roofline model is what turns the batch-size law from empirical curve-fitting into a closed-form, hardware-aware prediction.

What would settle it

Run Qwen2.5-72B on the same H800 setup at batch sizes 1, 4, 16, 32, and 64 and measure throughput for each TopK value from 10 to 80; if the argmax TopK deviates systematically from 27904√(1+0.034/b)−27897, for instance if the optimal width stays flat or falls with batch size, Theorem 1.3 is falsified. Independently, re-run the pretrain-token sweep on a single fixed corpus; an acceptance-rate slope far from 0.08 per decade, or a downturn past 50B tokens, would falsify Theorem 1.1.

Watch

Extended reading notes

Core claim

The central claim is that acceptance rate and throughput in speculative decoding are predictable, log-linear functions of three controllable resources. Theorems 1.1 and 1.2 state that acceptance rate equals α·log10(T_pretrain)+β with (α,β)≈(0.08,5.05), and α·log10(D)+β with (α,β)≈(0.74,4.61), where T_pretrain is pretraining tokens and D is draft decoder count. Theorem 1.3 states that throughput equals α·log2(b)+β with (α,β)≈(286.79,7.54) along the optimal TopK path topk(b)=27904√(1+0.034/b)−27897. The paper argues that pretraining data, decoder depth, and batch size can be scaled independently, that pretrain-token scaling beats SFT-data scaling at about half the training iterations, and that the combined system Scylla achieves a 1.5–2.2-point acceptance-rate gain at temperature T=0 and up to 2× throughput versus a fixed-policy baseline, with throughput gains concentrated at large batch sizes where a fixed tree width collapses.

Load-bearing premise

The entire batch-size law rests on the assumption that hand-counted arithmetic intensity correctly predicts real performance and that the best tree width is exactly where it equals the hardware's critical compute-to-memory ratio; the resulting constants were never verified against measured optimal tree widths on actual hardware.

Editorial extensions

If this is right

  • Pretraining a draft model on roughly 10–50B tokens is a cheap, inference-free way to raise acceptance rate, with gains continuing though saturating past 50B tokens.
  • Each additional log-unit of draft decoder depth buys a predictable acceptance-rate increase, so draft capacity can be budgeted against added inference cost.
  • Batch-adaptive TopK selection prevents the throughput collapse that fixed tree widths cause at large batches, keeping arithmetic intensity near the critical point.
  • At batch 64, a system following the laws keeps a 1.21x speedup over no speculative decoding while a fixed-policy baseline falls below it.
  • Pretrain scaling plus the same SFT data reaches a given acceptance rate with 4.5x training iterations versus 8x for SFT-only scaling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the log-linear form is general, the same pretrain-token and decoder-count curves could be used to predict when a draft model is good enough before running expensive verification experiments.
  • The functional form topk(b)∝√(1+C/b) likely transfers across dense models and GPUs, but the constant C should be recalibrated per model and accelerator; treating C=0.034 as universal would be an overreach the paper does not make.
  • The laws were measured on Transformer-style autoregressive draft models; testing them on Medusa-style heads or early-exit drafts would reveal whether they are laws of speculative decoding or of this architecture family.
  • Since the paper leaves RLHF out, a natural extension is to prepend a preference-optimization stage to the draft model; if the pretrain+SFT curves hold, RLHF should shift the intercept β upward without changing the log slope.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes three log-linear scaling laws for speculative decoding: acceptance rate as a function of pretraining-token volume (Theorem 1.1), acceptance rate as a function of draft-model decoder count (Theorem 1.2), and decoding throughput/optimal top-k path as functions of batch size (Theorem 1.3). On the basis of these laws, the authors build Scylla, a draft-model training and inference configuration that scales pretraining data, decoder depth, and batch-adaptive top-k selection. The paper reports that Scylla achieves higher acceptance rates than EAGLE2/EAGLE3 on several benchmarks and about 2x throughput over EAGLE2 on an industrial inference engine.

Significance. If the quantitative scaling laws were established, the paper would provide actionable guidance for draft-model pretraining budgets, decoder-capacity choices, and batch-dependent tree sizes in speculative decoding, and the Scylla system would be a useful demonstration. The empirical study covers multiple backbones and benchmarks, and the industrial throughput comparison is a useful data point. However, the central quantitative claims are not recoverable from the paper's own figures and tables: the stated constants in Theorems 1.1 and 1.2 do not match the displayed data points, and Theorem 1.3's throughput equation contradicts the measured throughput in Table 5. The validation of the optimal-top-k formula is circular because it is derived from a roofline model and then 'validated' against that same model's output. Since the scaling laws are the load-bearing contribution, these problems substantially weaken the paper's main claim.

major comments (5)
  1. [§3.1, Theorem 1.1, Figure 1 (left)] The stated constants α≈0.08 and β≈5.05 are not supported by the paper's own data. Using the seven points visible in Figure 1 (left), which are also given in §4.1 (x = 1, 2, 5, 10, 20, 50, 100 B; y = 5.13, 5.15, 5.29, 5.33, 5.30, 5.43, 5.40), an ordinary least-squares fit yields a slope of roughly 0.15 and an intercept of roughly 5.00, not (0.08, 5.05). The discrepancy is not a unit artifact: rescaling the x-axis by a constant changes only the intercept, while the slope is off by nearly a factor of two. The paper provides no fitting procedure, raw data, or repeated-runs summary that would recover the stated coefficients, so Theorem 1.1 as written is not established by the displayed evidence.
  2. [§3.1, Theorem 1.2, Figure 1 (middle)] The four points in Figure 1 (middle) (D = 1, 2, 5, 10; y = 5.03, 5.24, 5.65, 5.83) fit a log-linear line with slope approximately 0.83 and intercept approximately 5.02, whereas Theorem 1.2 states α≈0.74 and β≈4.61. With only four points and no error bars, an R² of 0.99 is not strong evidence of a precise law. If these are the data behind the theorem, the stated constants are not recoverable; if other data were used, that data and the fitting procedure must be reported.
  3. [§3.2, §4.3, Theorem 1.3, Table 5] Equation (3) predicts roughly 7.5 tokens/s at b = 1 and 1728 tokens/s at b = 64, while Table 5 reports Scylla + Opt.topk at 250 and 2150 tokens/s, respectively. The equation therefore cannot be a fit to the measured system throughput; it must be a fit to the theoretical roofline curves of Figure 5. The paper should state this distinction explicitly and provide a direct measured-versus-predicted comparison. As written, the claim that Theorem 1.3 is 'validated' by the regression is misleading, because the displayed throughput values in Table 5 do not match the theorem.
  4. [§3.2, Eq. (7), Figure 1 (right)] The optimal-top-k formula in Eq. (4) is derived by imposing I(b, topk,opt) = I_crit, and the claimed validation of Eq. (4) is performed by fitting the same roofline model's theoretical curves (Figure 1 right and Figure 5). This is circular: the model is used to generate the data that then confirms the model. There is no independent hardware measurement of the optimal top-k value for Qwen2.5-72B across batch sizes. Without such a measurement, Theorem 1.3's top-k law is not empirically validated.
  5. [§3, Theorems 1.1–1.3] The three statements are called 'Theorems' but are empirical curve fits with no derivation, no standard errors, and no model-selection analysis. The pretraining effect itself is small: the acceptance rate changes from 5.13% to 5.43% over two orders of magnitude of tokens, and the paper reports no repeated-run variability or error bars. Since the design decisions behind Scylla (10B pretraining, decoder scaling, batch-adaptive top-k) are justified by these quantitative laws, the lack of a reproducible fitting protocol is a load-bearing gap.
minor comments (5)
  1. [Abstract and §4.4] '1.5-2.2 higher acceptance rate' is ambiguous: the reported numbers are 1.47–2.22 percentage-point improvements over EAGLE2, not a 1.5–2.2x multiplicative increase. Please state which quantity is meant.
  2. [Figure 1] The right panel uses a linear batch-size axis while the caption says 'log scale'; clarify the axis transformation. Also specify whether the points in each panel are single-benchmark values, means over benchmarks, or different from the values quoted in the text.
  3. [Appendix E, Eq. (8)] Equation (8) introduces an extra parameter k described only as 'ranging from 0.9 to 1.2'. It should be stated how k is determined, how many configurations are used, and whether the fitted relationship is identified rather than an arbitrary parametric form.
  4. [§4.4, Table 5] The text says observed throughput discrepancies arise from communication overhead and kernel utilization but still uses Table 5 to support the scaling law. Provide a side-by-side comparison of theoretical and measured throughput for the same batch sizes so the reader can judge the size of the discrepancy.
  5. [§6 and 'Code will be released later'] The quantitative claims depend entirely on fits, but no raw data, fitting scripts, or code are provided. The statement in §6 that the scaling laws are 'verified only in Transformer-based Scylla' should also be reflected in the abstract and conclusion, where the laws are presented as general.

Circularity Check

3 steps flagged · score 6.0 of 10

Theorems 1.1 and 1.2 are in-sample regressions presented as validation; Theorem 1.3's optimal-topk formula is defined by Eq. 7 and then 'validated' against the same roofline curves.

  1. fitted input called prediction [Theorem 1.1/1.2 and Section 4.1-4.2, Figure 1 Left/Middle, Figures 6-7]
    "Acceptance rate =α· log10(Tpretrain) +β; α∼ 0.08, β ∼ 5.05 (1) ... These results validate that scaling pretraining data enhances performance but with diminishing returns at larger scales. The observed trend follows a log-linear scaling law (Theorem 1.1) with strong statistical validity (R2 = 0.89), indicating a strong logarithmic correlation between acceptance rate and pretraining tokens."

    The constants α and β are least-squares fits to the very points plotted in Figure 1 Left (and similarly in Figure 1 Middle for Theorem 1.2). The text then cites the R² of that same fit as 'validation' of the law. There is no held-out prediction, no separate measurement, and no independent test: evaluating the formula at the fitted token budgets merely inverts the regression that produced α and β. The same structure is used for Theorem 1.2, where the decoder-count points in Figure 1 Middle are fitted and then reported as confirming the log-linear law.

  2. self definitional [Section 3.2, Eq. (7), Theorem 1.3, and Section 4.3, Figure 1 Right]
    "It is demonstrated that maximum throughput consistently occurs when the arithmetic intensity approaches Icrit. Optimal topk Configuration. To maintain optimal throughput during batch size scaling, we derive the optimal topk criterion: I(b, topk,opt) := Icrit (7) ... By solving Equation 7 under Qwen2.5-72B architecture with a 10K prefill input, we formalize empirical scaling laws in Theorem 1.3. The right panel of Figure 1 validates the inverse-batch square root scaling law between batch size (b) and TopK-path (topk) fitted in Equation 4."

    Equation 7 defines the optimal topk as the solution of I(b, topk) = Icrit within the paper's own roofline model. Figure 5, from which the 'Optimal Top-k Path' curve is taken, is explicitly labeled 'Theoretical throughput of Qwen2.5-72B (single-GPU)', and its maxima are computed from the same analytical FLOP/memory formulas. Therefore Equation 4 is a fitted re-expression of that defining equation, and the claimed 'validation' in Figure 1 Right is a regression on the same theoretical curve, not an independent measurement of optimal topk on hardware.

1 more flagged steps
  1. fitted input called prediction [Section 4.3, Eq. (3) and following sentence]
    "Furthermore, we establish a log-linear scaling law (Equation 3) for maximum throughput as a function of batch size. The regression validates Theorem 1.3’s efficacy in maximizing throughput during batch size scaling while avoiding throughput degradation."

    The throughput law α ≈ 286.79, β ≈ 7.54 is obtained by regression on the theoretical maximum-throughput curves shown in Figure 5 and Figure 1 Right, which are generated by the same roofline model used to define I(b, topk). Calling the regression a 'validation' is in-sample: it only checks that the fitted line reproduces the model output from which it was estimated. The real-system throughputs in Table 5 are presented separately and are attributed to communication and kernel effects rather than used to verify the stated α and β constants.

full rationale

The paper's genuinely independent content is the external system comparison: acceptance rate versus EAGLE/EAGLE2/EAGLE3 in Figure 2 and Table 4, and throughput versus EAGLE2 in Table 5. These empirical benchmarks do not reduce to the fitted scaling-law formulas, so the paper is not wholly circular. However, the three stated scaling laws are not independently predicted. Theorem 1.1 and Theorem 1.2 are least-squares fits to the same data points that the text cites as confirmatory evidence; reporting R² of the fit does not validate the law out of sample. Theorem 1.3's optimal-topk formula is defined by Equation 7 as the point where arithmetic intensity equals Icrit in the paper's roofline model, and the 'validation' of that formula regresses Equation 4 onto the same theoretical curves. The throughput law in Equation 3 is likewise a regression on those theoretical maxima. Thus the quantitative scaling claims are partly descriptive fits or restatements of their defining equations, while the system-level Scylla results retain independent empirical content. This yields a partial circularity score of 6.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claims rest on fitted coefficients for the three scaling laws, plus a roofline model whose hardware fidelity is assumed. No new physical entities are introduced. The most load-bearing assumptions are the log-linear functional forms and the I = I_crit optimality condition.

free parameters (8)
  • alpha_pretrain = 0.08
    Slope of log-linear fit of acceptance rate versus log10 pretraining tokens (Theorem 1.1), fitted to 7 points in Figure 1 Left.
  • beta_pretrain = 5.05
    Intercept of the Theorem 1.1 fit on pretraining tokens.
  • alpha_capacity = 0.74
    Slope of log-linear fit of acceptance rate versus log10 decoder count (Theorem 1.2), fitted to points at decoder count 1, 2, 5, and 10.
  • beta_capacity = 4.61
    Intercept of the Theorem 1.2 fit on decoder count.
  • alpha_batch_throughput = 286.79
    Slope of log-linear fit of maximum throughput versus log2 batch size (Theorem 1.3), fitted to theoretical roofline curves.
  • beta_batch_throughput = 7.54
    Intercept of the Theorem 1.3 throughput fit.
  • topk_scale_constants = 27904, 27897, 0.034
    Constants in optimal topk(b) = 27904*sqrt(1 + 0.034/b) - 27897, solved from the roofline model for Qwen2.5-72B with 10K prefill; no generalization evidence.
  • acceptance_topk_k = 0.9 to 1.2
    Scaling factor in the empirical tacc(topk) formula (Eq. 8, Appendix E), fitted to experimental configurations.
assumptions (5)
  • standard math Speculative decoding token acceptance probability is min(1, P(t)/P_hat(t)), and rejected tokens are sampled from the residual distribution (Section 2).
    Standard result from speculative decoding literature, used as the basis for defining acceptance rate.
  • domain assumption The FLOPs and memory-access formulas in Tables 3, 7, and 8 accurately model the real kernels on NVIDIA H800 hardware (Section 3.2, Appendix C).
    The roofline model and Theorem 1.3 depend on these hand-derived formulas; no calibration against measured kernel efficiency is reported.
  • ad hoc to paper The empirical log-linear form (acceptance rate linear in log tokens or log decoders) is the correct functional form for scaling (Theorems 1.1 and 1.2).
    The scaling laws are not derived; the log-linear form is assumed and then fitted to data.
  • ad hoc to paper The optimal TopK-path is exactly the point where arithmetic intensity equals the critical intensity I_crit (Eq. 7).
    Used to derive Eq. 4; not proven from first principles and not directly measured on real hardware.
  • domain assumption Single-epoch pretraining and two-epoch SFT on ShareGPT reveal scaling behavior that transfers across model families (Section 4.1).
    Scaling laws are fit on Vicuna and Llama2 models and assumed to hold for Qwen2.5, Llama3, and 70B-class models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Laws for Speculative Decoding." pith.science (2026). https://pith.science/paper/ZLIMPOPV

@misc{pith2026250507858,
  author       = {Pith},
  title        = {Pith review of: Scaling Laws for Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZLIMPOPV}},
  note         = {Machine review of arXiv:2505.07858}
}
read the original abstract

The escalating demand for efficient decoding in large language models (LLMs) is particularly critical for reasoning-intensive architectures like OpenAI-o3 and DeepSeek-R1, which depend on extended chain-of-thought reasoning. This study investigates speculative decoding techniques through dense LLM architectures to establish foundational insights for accelerating reasoning tasks. While speculative decoding methods leveraging parallel draft-verification cycles have emerged as promising acceleration techniques, the scaling laws governing decoding efficiency remain under-explored compared to conventional backbone LLMs developed through Pretraining->SFT->RLHF training paradigms. In this work, we discover Log-linear Scaling Laws (Theorem 1.1, 1.2 and 1.3) governing draft model acceptance rate (or decoding speed) across three dimensions: pretraining token volume, draft model capacity, and decoding batch size. Building on these laws, we achieve Scylla, which coordinates multi-dimensional scaling for popular LLMs (Llama2/3, Qwen2.5). Empirical validation shows Scylla achieves 1.5-2.2 higher acceptance rate than EAGLE2 and 0.3 higher than EAGLE3 at temperature T = 0, with peak performance gains on summarization and QA tasks (Figure 2). Industrial inference engine deployments demonstrate 2X decoding throughput improvements over EAGLE2 (Table 5), validating the transformative potential of systematic scaling for efficient LLM inference. Code will be released later.

Figures

Figures reproduced from arXiv: 2505.07858 by the authors.

Figure 1
Figure 1. (Left): Scaling law of acceptance rate on MT [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Acceptance rate comparison between Scylla and previous SOTA methods across several [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Speculative decoding architecture used for scaling laws. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: (a) Architecture comparison between draft model of EAGLE and Scylla. We enhance [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: It is demonstrated that maximum throughput consistently occurs when the arithmetic [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Scaling up pretraining tokens. 4.1 Scaling Up Pretraining Data In this section, we examine how scaling up pretraining tokens impacts acceptance rate [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Scaling up draft model capacity. 4.4 Scylla Built on Scaling Laws Building on established scaling laws, we achieve Scylla through scaling pretraining with 10B tokens, draft model capacity and decoding batch size, which demonstrates superior performance across all evalu…
Figure 8
Figure 8. Figure 8: Throughput variations across different topk values while accounting for acceptance rate fluctuations on batch sizes of 16, 32 and 64. Using this formula, we systematically evaluate throughput variations across different topk values while accounting for acceptance rate …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 8 canonical work pages

  1. [1]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    arXiv 2001

  2. [2]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    arXiv 2022

  3. [3]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  4. [4]

    Medusa: Simple LLM inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024

    arXiv 2024

  5. [5]

    EAGLE: Speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024

    arXiv 2024

  6. [6]

    EAGLE-2: Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858, 2024

    arXiv 2024

  7. [7]

    EAGLE-3: Scaling up inference acceleration of large language models via training-time test

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840, 2025

    arXiv 2025

  8. [8]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

Show all 38 references
  1. [9]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  2. [10]

    Blockwise parallel decoding for deep autoregressive models

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018

    2018

  3. [11]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  4. [12]

    Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.See https://vicuna. lmsys. org (accessed 14 April 2023)...

    2023

  5. [13]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  6. [14]

    The Llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 10

    2024 arXiv

  7. [15]

    Judging LLM-as-a-judge with MT-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  8. [16]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  9. [17]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  10. [18]

    Alpaca: A strong, replicable instruction- following model

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/a...

    2023

  11. [19]

    Abstractive text sum- marization using sequence-to-sequence RNNs and beyond

    Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text sum- marization using sequence-to-sequence RNNs and beyond. arXiv preprint arXiv:1602.06023, 2016

    2016 arXiv

  12. [20]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computa...

    2019

  13. [21]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    2023 arXiv

  14. [22]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  15. [23]

    Online speculative decoding

    Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. arXiv preprint arXiv:2310.07177, 2023

    2023 arXiv

  16. [24]

    Lookahead: An inference acceleration framework for large language model with lossless generation accuracy

    Yao Zhao, Zhitian Xie, Chen Liang, Chenyi Zhuang, and Jinjie Gu. Lookahead: An inference acceleration framework for large language model with lossless generation accuracy. In Proceed- ings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6344–6355, 2024

    2024

  17. [25]

    Ouroboros: Speculative decoding with large model enhanced drafting

    Weilin Zhao, Yuxiang Huang, Xu Han, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. Ouroboros: Speculative decoding with large model enhanced drafting. arXiv e-prints, pages arXiv–2402, 2024

    2024

  18. [26]

    Break the sequential dependency of LLM inference using lookahead decoding

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding. arXiv preprint arXiv:2402.02057, 2024

    2024 arXiv

  19. [27]

    DistillSpec: Improving speculative decoding via knowledge distillation

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Ros- tamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. DistillSpec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08461, 2023

    arXiv

  20. [28]

    CLLMs: Consistency large language models

    Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. CLLMs: Consistency large language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  21. [29]

    Sequoia: Scalable, robust, and hardware-aware speculative decoding

    Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint arXiv:2402.12374, 2024

    2024 arXiv

  22. [30]

    SSSD: Simply-scalable speculative decoding

    Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Lorenz K Müller, and Lukas Cavigelli. SSSD: Simply-scalable speculative decoding. arXiv preprint arXiv:2411.05894, 2024

    2024 arXiv

  23. [31]

    Better & faster large language models via multi-token prediction

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737, 2024

    2024 arXiv

  24. [32]

    Clover: Regressive lightweight speculative decoding with sequential knowledge

    Bin Xiao, Chunan Shi, Xiaonan Nie, Fan Yang, Xiangwei Deng, Lei Su, Weipeng Chen, and Bin Cui. Clover: Regressive lightweight speculative decoding with sequential knowledge. arXiv preprint arXiv:2405.00263, 2024. 11

    2024 arXiv

  25. [33]

    Clover-2: Accurate inference for regressive lightweight speculative decoding

    Bin Xiao, Lujun Gui, Lei Su, and Weipeng Chen. Clover-2: Accurate inference for regressive lightweight speculative decoding. arXiv preprint arXiv:2408.00264, 2024

    2024 arXiv

  26. [34]

    Learning harmonized represen- tations for speculative sampling

    Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized represen- tations for speculative sampling. arXiv preprint arXiv:2408.15766, 2024

    2024 arXiv

  27. [35]

    Sinkhorn distance minimization for knowledge distillation

    Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, and Houqiang Li. Sinkhorn distance minimization for knowledge distillation. In LREC-COLING, pages 14846–14858, 2024

    2024

  28. [36]

    Multi-level optimal transport for universal cross-tokenizer knowledge distillation on language models

    Xiao Cui, Mo Zhu, Yulei Qin, Liang Xie, Wengang Zhou, and Houqiang Li. Multi-level optimal transport for universal cross-tokenizer knowledge distillation on language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23724–23732, 2025

    2025

  29. [37]

    Sinkd: Sinkhorn distance minimization for knowledge distillation

    Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, and Houqiang Li. Sinkd: Sinkhorn distance minimization for knowledge distillation. TNNLS, 2024

    2024

  30. [38]

    Kangaroo: Lossless self-speculative decoding for accelerating llms via double early exiting

    Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Duyu Tang, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding for accelerating llms via double early exiting. Advances in Neural Information Processing Systems, 37:11946–11965, 2024. 12 A Pretraining Data ...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.