Pith. sign in

REVIEW 3 major objections 5 minor 53 references

Utility-Driven Speculative Decoding for Mixture-of-Experts

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For mixture-of-experts LLMs, speculative decoding causes up to 1.5x slowdowns because draft tokens activate extra experts, and the paper's utility-driven Cascade caps the loss at 5%.

desk verdict Real problem, sensible controller, and believable numbers, but the 'utility predicts speedup' validation is circular and the temporal-locality assumption is under-tested; still deserves peer review. read the letter →

arxiv 2506.20675 v1 pith:PFEIO5AT submitted 2025-06-17 cs.DC cs.AIcs.LG

Anish Saxena , Po-An Tsai , Hritvik Taneja , Aamer Jaleel , Moinuddin Qureshi This is my paper
classification cs.DCcs.AIcs.LG
keywords speculativedecodingMixture-of-ExpertsMoEservingmemorybandwidthadaptivespeculationlengthutilitydatamovementLLMinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding, in which a small drafter proposes $K$ tokens and the main model verifies them together, helps dense LLMs because the verification step loads the same weights regardless of token count. The paper argues that this premise breaks for mixture-of-experts (MoE) models: each draft token routes to its own subset of experts, so $K$ draft tokens collectively activate more weights and can raise verification time by 2-3x, causing slowdowns of up to 1.5x even when token throughput improves. The paper defines speculation utility as the ratio of the gain in effective token rate (the average number of tokens emitted per iteration) to the verification-time overhead, and proves that the time per output token under speculation equals the no-speculation time divided by that utility. Cascade uses this ratio to decide when to speculate, to test candidate $K$ values over short windows, and to hill-climb toward the utility-maximizing $K$. On five MoE models and seven workloads, Cascade caps worst-case slowdown at about 5% and improves throughput over fixed-$K$ speculation by 7-14%.

What carries the argument

The central mechanism is the speculation utility, the ratio of the benefit of speculation (the effective token rate, ETR, the average number of tokens emitted per iteration) to its cost (the ratio of a speculative iteration's time to a no-speculation iteration's time): $U = ETR_{spec} / (t_{iter,spec}/t_{iter,base})$. The paper proves that the time per output token under speculation is $t_{base}/U$, so the $K$ that maximizes measured utility is also the $K$ that minimizes latency. To turn this into a controller, Cascade runs short trial phases (about four iterations each, at most four trials), computes utility at the end of each trial, and commits to the best $K$ for a set phase of about sixteen iterations; it exits testing early when utility is consistently falling or has converged, sets $K=0$ when utility is below one, and doubles the set phase whenever speculation has been switched off, which cuts the worst-case testing overhead from 14% to 5%. The design relies on the empirical observation that utility, though variable, has temporal locality over short intervals, so recent trials predict the near future.

What would settle it

Run Cascade on a workload where utility changes faster than the control cycle, for instance a request stream that alternates between high-affinity and low-affinity token phases every few tokens, and compare the $K$ chosen at the end of the four-iteration trial with the $K$ that actually maximizes utility over the next sixteen iterations; if the two disagree in a substantial fraction of windows, the temporal-locality assumption that the design rests on is violated.

Watch

Extended reading notes

Core claim

The paper's central claim is that speculative decoding is not practical for MoEs under the usual always-on policy because the verification of $K$ draft tokens activates a larger set of experts, increasing data movement and verification latency by 2-3x and sometimes producing slowdowns up to 1.5x. The paper further claims that this cost-benefit balance varies with task, model, request, and even iteration, so no static $K$ can serve real workloads. Its positive claim is that the right control signal is the speculation utility, $U_{spec} = ETR_{spec} / (t_{iter,spec}/t_{iter,base})$, the ratio of the effective token rate (tokens emitted per iteration) to the verification-time overhead, and that maximizing $U$ is equivalent to minimizing the time per output token because $t_{spec} = t_{base}/U_{spec}$. Cascade operationalizes the ratio with a test-and-set policy that briefly tries several $K$ values, chooses the one with highest measured utility, disables speculation ($K=0$) when utility is below one, and exponentially backs off testing when speculation is consistently harmful. Evaluated across five MoE models, seven tasks, and both n-gram and a learned drafter, Cascade limits the worst-case slowdown to about 5% (compared with up to 54% for static $K$) and delivers 7-14% average throughput gains over static-$K$ schemes.

Load-bearing premise

The load-bearing premise is that the utility measured over a short trial window of about four iterations stays representative for the following set phase of about sixteen iterations, so the periodic test-and-set loop is predicting the near future rather than acting on stale information.

Editorial extensions

If this is right

  • A serving system should treat $K=0$ as an explicit state, because even the most conservative one-token speculation can slow some MoE tasks by 25%, and a policy that cannot turn speculation off is unsafe.
  • Static per-task profiles are inadequate for mixed workloads, since the same model can need different $K$ values for code, math, and extraction; Cascade's per-interval adaptation matches or beats the best static $K$ on every evaluated task.
  • Because utility is computed only from measured throughput and iteration times, the same controller applies to model-free drafters, learned drafters, and MoE variants with shared or quantized experts, without retraining or offline profiling.
  • The proved identity $t_{spec} = t_{base}/U$ gives the online search a direct target: choosing the $K$ with the highest measured utility is not a proxy for latency but the definition of minimizing it.
  • Cascade's worst-case slowdown of 5% means speculation can be left on by default and managed dynamically, instead of being disabled globally or profiled per task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A similar test-and-set loop could tune other serving knobs, such as batch width, expert caching policy, or quantization level, whenever the knob has a measurable benefit-to-cost ratio with short-term locality.
  • The control cycle is fixed at about twenty iterations, so its real-time duration scales with model and hardware speed; on slower devices the same policy would react more slowly in wall-clock terms, suggesting the cycle length should be tied to measured iteration time rather than a fixed iteration count.
  • Average utility can hide rare dips, and the paper reports transient losses up to 33% during test phases; a tail-latency service-level objective would require tracking a low quantile of utility rather than the mean.
  • Because verification cost depends on expert affinity, routing draft tokens toward already-loaded or shared experts is a complementary lever that would raise the optimal $K$; predicting that cost from router statistics, rather than measuring it after the fact, is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies speculative decoding for Mixture-of-Experts (MoE) LLMs and argues that unlike in dense models, speculation is often harmful because draft tokens activate additional experts, increasing memory traffic and verification time by up to 3x. To address this, the authors propose Cascade, a runtime framework that maintains a utility metric equal to the ratio of effective-token-rate (ETR) gains to iteration-time overhead, periodically tests several speculation lengths K over short trial phases, and then fixes the best K for a longer set phase, disabling speculation when utility falls below one. Cascade is implemented in vLLM and evaluated on five MoEs with n-gram and EAGLE drafters across seven workloads. The main claimed results are a worst-case slowdown of only 5% (versus up to 1.5x slowdown with static K) and a 7-14% throughput improvement over static-K schemes.

Significance. If the main claims hold, the paper addresses a real and timely problem: speculative decoding is not automatically useful for MoEs, and prior dynamic-K methods assume near-zero overspeculation penalties. The end-to-end experiments span five different MoE architectures and fourteen model-task combinations, and the authors are transparent about hyperparameter sensitivity. The finding that speculation can cause 1.5x slowdowns in MoEs while helping dense models is a useful negative result. However, the core validation of the utility metric is circular, and the worst-case slowdown claim is contradicted by data reported in the paper itself, so the significance is currently lower than the abstract suggests.

major comments (3)
  1. [§4, Definition 4.1 and Theorem 4.2, Eq. (1)-(4), Fig. 8] Theorem 4.2 is an identity, not a predictive result: utility is defined as ETR_spec divided by the normalized iteration time, and TPOT_spec = t_iter_spec / ETR_spec, so t_base / t_spec = U_spec algebraically. Consequently, the R^2 = 99.4% in Fig. 8 only confirms that the authors computed the two quantities from the same measurements; it does not empirically validate utility as a predictor of speculation performance. This overclaim is load-bearing because the paper's design rationale, 'utility robustly predicts speculation performance,' rests on this validation. The authors should either remove the prediction claim and present utility as a direct algebraic reformulation of normalized TPOT, or provide an out-of-sample validation.
  2. [§2.7, §5.3, §5.5, §7.5] The entire 5% slowdown bound depends on an unvalidated locality assumption: a 4-iteration trial of each K (t=4, M=4, T=16) is assumed to forecast the utility-maximizing K for the following S=16 set phase, and adaptive back-off extends the set phase to 32, 64, ... iterations after K=0. The only evidence is windowed utility plots over 16-iteration windows (Figs. 6 and 7) and a coarse sensitivity sweep (Section 7.5). The paper never measures prediction error at the actual decision horizon (20 iterations, and longer with back-off), so it does not establish how often stale trials cause Cascade to keep speculation on during a harmful phase or miss a beneficial phase. I would like to see a quantitative prediction-error analysis, e.g., measuring the agreement between the best-K chosen by a trial and the best-K measured over the subsequent set phase, and reporting accuracy as a function of the back-off length.
  3. [§7.1 and abstract/intro claims] The paper repeatedly claims that Cascade 'limits worst-case slowdown to just 5%,' but Section 7.1 states that among sampled requests, 'the maximum loss was 33% and occurred only in 3 instances.' A 33% per-request degradation is a worst-case slowdown of 33%, not 5%. This is not a cosmetic wording issue: the abstract and the conclusion present the 5% number as the paper's central guarantee. The authors should either redefine what the 5% bound applies to (e.g., average over a task), report the true worst case, or explain why the 33% instances are excluded from the claimed bound.
minor comments (5)
  1. [§3, Table 1] The column header 'P' is never defined; I assume it denotes the total parameter count, but it should be stated explicitly.
  2. [§1 and §7] The phrase 'All-3GM' in Fig. 13 appears to be an abbreviation for 'All-3', and it is not defined in the text. Please define all workload abbreviations consistently.
  3. [§7] The sentence 'Cascade improves average performance across all MoEs except DeepSeek, which sees a marginal 1% gain' is ambiguous about whether the comparison is against the best static-K baseline or against the no-speculation baseline; please clarify.
  4. [§4, Fig. 8] Because the R^2 value is an algebraic identity, the caption should not describe utility as a 'predictor'; it should instead say that utility is a rescaling of normalized TPOT.
  5. [§5.3] The paragraph describing the first trial and K_start says the manager 'chooses dynamically by scanning recent history,' but it does not specify how far back the history extends or what happens when no non-zero K has been tried yet. Please make the initialization policy precise.

Circularity Check

1 steps flagged · score 6.0 of 10

Utility validation is definitional: U = t_base/t_spec by construction, so Fig. 8's R^2=99.4% is an identity check, not an empirical prediction; Cascade's end-to-end comparison to static-K is independent.

  1. self definitional [Section 4, Definition 4.1, Theorem 4.2, and Fig. 8]
    "Definition 4.1 (Speculation Utility). The ratio of the benefit of speculation (ETR improvement) to its cost (speculation overheads). ... Theorem 4.2. If ... t_spec = t_base/U_spec. ... Fig. 8: Speedup as a function of measured utility for 5 MoEs ... Utility reliably predicts performance (R2 = 99.4%)."

    In the proof, benefit = ETR_spec and cost = t_iter,spec/t_iter,base, while TPOT = t_iter/ETR (Eq. 1). Substituting gives U_spec = ETR_spec / (t_iter,spec/t_iter,base) = (t_iter,base)/(t_iter,spec/ETR_spec) = t_base/t_spec. So 'speedup equals utility' is an algebraic identity from the definitions; the scatter plot and R^2=99.4% merely confirm that the measured quantities satisfy the arithmetic. The statement 'maximizing utility directly minimizes TPOT' is therefore true by construction, not an empirical finding. This makes the utility-validation contribution circular, though the Cascade system comparison against static-K baselines is not.

full rationale

The only load-bearing reduction to its own inputs is the utility metric itself. By Definition 4.1 and Eqs. (1)-(4), speculation utility is exactly the ratio of baseline TPOT to speculative TPOT, so Fig. 8's high R^2 is an in-sample identity rather than evidence that utility predicts performance. The paper's central engineering claims (slowdown bounded near 5%, 7-14% throughput gains over static K, additive benefits of dynamic disabling, adaptive back-off, and hill-climbing) are evaluated empirically against static-K baselines and do not reduce to the definition. The temporal-locality assumption behind test-and-set is a predictive heuristic that is only qualitatively supported by windowed plots; this is an unvalidated assumption and a correctness risk, but not a circularity. No load-bearing self-citation or imported uniqueness theorem appears; references to prior speculative-decoding work are external and not used to justify Cascade's design. Score 6 reflects the partial circularity of the utility-as-predictor claim while recognizing the independent end-to-end evaluation.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claims rest on five premises: the memory-bandwidth-bound decode model, the definitional utility identity, the local stationarity of ETR and cost, the baseline measurement protocol, and the assumption that non-expert overheads stay small. The first and third are empirical; the second is a tautology. No data or code is shipped to independently verify the first and third across other GPU generations or schedulers.

free parameters (7)
  • trial phase duration t = 4
    Length of each trial run for a K value during the test phase. Chosen by hand; sensitivity evaluated in Section 7.5.
  • set phase duration S = 16
    Length of the production phase after selecting K. Chosen by hand; sensitivity evaluated in Section 7.5.
  • maximum test phase length T (number of trials M) = 16 (4 trials of t=4)
    Bounding exploration cost; hand-chosen to keep the test phase short relative to the set phase.
  • early-exit convergence threshold = 10%
    Stops hill-climbing when utility across successive K values differs by less than 10%; hand-chosen in Section 5.6.
  • baseline refresh interval = every 100 iterations
    No-speculation baseline iteration time is refreshed infrequently to compute the cost ratio; hand-chosen in Section 5.3.
  • initial baseline measurement length = 4 iterations
    First few decode iterations run without speculation to set the baseline time; hand-chosen in Section 5.3.
  • maximum speculation length K_max = 3
    Evaluation cap; stated as yielding at-best marginal benefits beyond this, but the exact value is hand-chosen.
assumptions (6)
  • domain assumption Decode iteration time is memory-bandwidth-bound and proportional to the volume of active expert weights fetched.
    Invoked in Sections 2.1-2.4 to derive the 2-3x verification overhead; if the GPU were compute-bound or expert weights were cached, the cost model would fail.
  • standard math The utility ratio U = ETR_spec / (t_iter,spec / t_iter,base) determines TPOT by t_spec = t_base / U.
    Theorem 4.2 derives this algebraically from the definitions in Definition 4.1; it is true by construction, not an empirical law.
  • domain assumption ETR and verification cost exhibit temporal locality over short iteration intervals, making a 4-iteration trial predictive of the next 16-iteration set phase.
    This stationarity premise is the basis of the test-and-set policy in Section 5.3 and is supported only by windowed plots (Figs. 6-7), not by a formal prediction-horizon test.
  • domain assumption The no-speculation baseline iteration time measured in the first few decode iterations and refreshed every 100 iterations remains representative during speculation.
    Utility's cost ratio depends on this baseline; if the baseline drifts, utility is biased (Section 5.3).
  • domain assumption Drafting, rejection sampling, and CPU telemetry overheads remain small relative to expert data movement.
    The paper's overhead breakdown (Section 2.3) measures this on one GPU; the framework assumes it holds across platforms.
  • domain assumption The router's expert selection for the K draft tokens is roughly independent, so the bucket-and-balls analysis in Section 2.4 estimates unique expert activations.
    Used to explain the 2-3x data-movement increase; the paper notes affinity weakens the effect, so the model is approximate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Utility-Driven Speculative Decoding for Mixture-of-Experts." pith.science (2026). https://pith.science/paper/PFEIO5AT

@misc{pith2026250620675,
  author       = {Pith},
  title        = {Pith review of: Utility-Driven Speculative Decoding for Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFEIO5AT}},
  note         = {Machine review of arXiv:2506.20675}
}
read the original abstract

GPU memory bandwidth is the main bottleneck for low-latency Large Language Model (LLM) inference. Speculative decoding leverages idle GPU compute by using a lightweight drafter to propose K tokens, which the LLM verifies in parallel, boosting token throughput. In conventional dense LLMs, all model weights are fetched each iteration, so speculation adds no latency overhead. Emerging Mixture of Experts (MoE) models activate only a subset of weights per token, greatly reducing data movement. However, we show that speculation is ineffective for MoEs: draft tokens collectively activate more weights, increasing data movement and verification time by 2-3x. When token throughput gains fail to offset this overhead, speculation causes slowdowns up to 1.5x, making it infeasible. Even when useful, the optimal K varies by task, model, and even between requests and iterations. Thus, despite widespread use in dense LLMs, speculation remains impractical in leading MoEs. We present Cascade, a utility-driven framework that selectively enables speculation to avoid slowdowns and dynamically tunes K to accelerate MoE serving. Cascade uses a lightweight metric, speculation utility, the ratio of token gains to verification cost, which shows iteration-level locality, enabling periodic decisions via short test and longer set phases. For each request, Cascade disables speculation if utility drops below one during testing, and when utility exceeds one, tests multiple K-values to choose the utility-maximizing K for the set phase. We implement Cascade in vLLM and evaluate it on five popular MoEs with workloads spanning code, math, extraction, and mixed tasks. Cascade limits slowdown to 5% (vs. 1.5x) and improves throughput by 7-14% over static K, making speculative decoding practical for MoEs.

Figures

Figures reproduced from arXiv: 2506.20675 by the authors.

Figure 1
Figure 1. (a) Speculation in dense LLMs incurs no added memory traffic (b) Speculation in MoEs increases data-movement during [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Speculation improves TPOT without increasing [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 2
Figure 2. Unlike dense models, MoEs activate only a subset [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Top: TPOT and ETR speedup with n-gram speculation on dense LLAMA-3 (left, green) and Mixtral MoE (right, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: TPOT improvement across popular MoE models for different same-request and mixed-request tasks at varying [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Iteration-level variation of ETR and speculation cost [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Utility variation for different requests, averaged for window of 16 iterations, for selected model, task, and speculation [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Speedup as a function of measured utility for 5 [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 10
Figure 10. Figure 10: Test-and-Set Policy: Periodically, the speculation [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 9
Figure 9. Figure 9: Overview of utility-driven speculation. The specu [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 12
Figure 12. Figure 12: Hill climbing search: Leverage the directionality [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 11
Figure 11. Figure 11: Adaptive Back-Off: If utility at 𝐾 = 1 is persistently below 1, test less frequently to minimize cost. To remain practical, Cascade must impose minimal overhead even when speculation consistently fails to provide benefits. To achieve this, we implement an adaptive bac…
Figure 13
Figure 13. Figure 13: TPOT improvement of Cascade and static-K schemes on 5 MoE and 7 tasks with n-gram speculation. Cascade [PITH_FULL_IMAGE:figures/full_fig_p009_13.png]
Figure 14
Figure 14. Figure 14: System architecture of speculation in vLLM. We [PITH_FULL_IMAGE:figures/full_fig_p009_14.png]
Figure 17
Figure 17. Figure 17: The performance of Cascade with EAGLE specula [PITH_FULL_IMAGE:figures/full_fig_p010_17.png]
Figure 16
Figure 16. Figure 16: Utility variation for math+code+extraction mixed [PITH_FULL_IMAGE:figures/full_fig_p010_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 22 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  3. [3]

    Sean Betts. 2023. Peering inside GPT-4: Understanding its mixture of experts (MOE) architecture. https://medium.com/@seanbetts/peering-inside-gpt-4- understanding-its-mixture-of-experts-moe-architecture-2a42eb8bdcb3

    work page 2023

  4. [4]

    Oscar Brown, Zhengjie Wang, Andrea Do, Nikhil Mathew, and Cheng Yu. 2024. Dynamic Depth Decoding: Faster Speculative Decoding for LLMs. arXiv preprint arXiv:2409.00142

    arXiv 2024

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-shot Learners. Advances in neural information processing systems 33 (2020), 1877–1901

    work page 2020

  6. [6]

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv preprint arXiv:2401.10774

    arXiv 2024

  7. [7]

    Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E Gonzalez, Matei Zaharia, and Ion Stoica. 2025. MoE-Lightning: High- Throughput MoE Inference on Memory-constrained GPUs. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 . 715–730

    work page 2025

  8. [8]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Lau- rent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318

    arXiv 2023

Show all 53 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  2. [10]

    Yunfei Cheng, Aonan Zhang, Xuanyu Zhang, Chong Wang, and Yi Wang. 2024. Recurrent Drafter for Fast Speculative Decoding in Large Language Models. arXiv preprint arXiv:2403.09919

    2024 arXiv

  3. [11]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  4. [12]

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. 2024. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. arXiv preprint arXiv:2401.06066

    2024 arXiv

  5. [13]

    Databricks. 2024. Introducing DBRX: A New State-of-the-Art Open LLM. https: //www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

    2024

  6. [14]

    Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai Li, and Yiran Chen. 2024. SiDA-MoE: Sparsity-Inspired Data- Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models. Pro- ceedings of Machine Learning and Systems 6 ...

    2024

  7. [15]

    Hugging Face. 2025. Large Language Model Text Generation Inference. https: //github.com/huggingface/text-generation-inference

    2025

  8. [16]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39. http://jmlr.org/papers/v23/21- 0998.html

    2022

  9. [17]

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2023. Breaking the Sequential Dependency of LLM Inference Using Lookahead Decoding. https://lmsys.org/ blog/2023-11-21-lookahead-decoding/

    2023

  10. [18]

    Google. 2024. Our next-generation model: Gemini 1.5. https: //blog.google/technology/ai/google-gemini-next-generation-model-february- 2024/#sundar-note

    2024

  11. [19]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, et al . 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  12. [20]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  13. [21]

    Beichen Huang, Yueming Yuan, Zelei Shao, and Minjia Zhang. 2025. MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators. arXiv preprint arXiv:2504.02658

    2025 arXiv

  14. [22]

    Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. 2023. Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference. arXiv preprint arXiv:2303.06182

    2023 arXiv

  15. [23]

    Kaiyu Huang, Hao Wu, Zhubo Shi, Han Zou, Minchen Yu, and Qingjiang Shi

  16. [24]

    Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. 2024. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference. arXiv:2308.12066 [cs.LG]

    2024 arXiv

  17. [25]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  18. [26]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  19. [27]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202). PMLR, 19274–19286

    2023

  20. [28]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. arXiv preprint arXiv:2406.16858

    2024 arXiv

  21. [29]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv preprint arXiv:2401.15077

    2024 arXiv

  22. [30]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv preprint arXiv:2503.01840

    2025 arXiv

  23. [31]

    Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang, Qinghan Chen, Shuhuai Lin, April Yang, Zhihao Zhang, Zhuoming Chen, et al . 2025. AdaServe: SLO-Customized LLM Serving with Fine-Grained Speculative Decod- ing. arXiv preprint arXiv:2501.12162

    2025 arXiv

  24. [32]

    Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, and Winston Hu. 2024. PEARL: Parallel Speculative Decoding with Adaptive Draft Length. arXiv preprint arXiv:2408.11850

    2024 arXiv

  25. [33]

    Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2024. Optimizing Speculative Decoding for Serving Large Language Models Using Goodput. arXiv preprint arXiv:2406.14066

    2024 arXiv

  26. [34]

    Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz. 2024. Dynamic speculation lookahead accelerates speculative decoding of large language models. arXiv preprint arXiv:2405.04304

    2024 arXiv

  27. [35]

    Meta. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    2025

  28. [36]

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al

  29. [37]

    Nvidia. 2025. TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference. https://github.com/NVIDIA/TensorRT-LLM

    2025

  30. [38]

    Apoorv Saxena. 2023. Prompt Lookup Decoding. https://github.com/ apoorvumang/prompt-lookup-decoding/

    2023

  31. [39]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538

    2017 arXiv

  32. [40]

    Andrii Skliar, Ties van Rozendaal, Romain Lepert, Todor Boinovski, Mart van Baalen, Markus Nagel, Paul Whatmough, and Babak Ehteshami Bejnordi. 2024. Mixture of cache-conditional experts for efficient mobile device inference. arXiv preprint arXiv:2412.00099

    2024 arXiv

  33. [41]

    Benjamin Spector and Chris Re. 2023. Accelerating LLM Inference with Staged Speculative Decoding. arXiv preprint arXiv:2308.04623

    2023 arXiv

  34. [42]

    Snowflake Staff. 2024. Snowflake Arctic - LLM for enterprise AI. https://www.snowflake.com/blog/arctic-open-efficient-foundation-language- models-snowflake/

    2024

  35. [43]

    NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prang...

    2022 arXiv

  36. [44]

    Qwen Team. 2024. QWEN1.5-Moe: Matching 7B model performance with 1/3 activated parameters. https://qwenlm.github.io/blog/qwen-moe/

    2024

  37. [45]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all 12 Utility-Driven Speculative Decoding for Mixture-of-Experts you need. Advances in neural information processing systems

    2017

  38. [46]

    Zhuofan Wen, Shangtong Gui, and Yang Feng. 2024. Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration. Advances in Neural Information Processing Systems 37 (2024), 92082–92100

    2024

  39. [47]

    Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. 2024. MoE-Pruner: Pruning Mixture-of-Experts Large Lan- guage Model using the Hints from Its Router. arXiv preprint arXiv:2410.12013

    2024 arXiv

  40. [48]

    Ziyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Rui Wang, and Zhaopeng Tu. 2024. Draft Model Knows When to Stop: A Self-Verification Length Policy for Speculative Decoding. arXiv preprint arXiv:2411.18462

    2024 arXiv

  41. [49]

    Weilin Zhao, Yuxiang Huang, Xu Han, Wang Xu, Chaojun Xiao, Xinrong Zhang, Yewei Fang, Kaihuo Zhang, Zhiyuan Liu, and Maosong Sun. 2024. Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding. arXiv preprint arXiv:2402.13720

    2024 arXiv

  42. [50]

    Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low- bit Quantization for Efficient and Accurate LLM Serving. Proceedings of Machine Learning and Systems 6 (2024), 196–209

    2024

  43. [51]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judg- ing LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623. 13

    2023

  44. [2024]

    arXiv preprint arXiv:2409.02060

    OLMoE: Open Mixture-of-Experts Language Models. arXiv preprint arXiv:2409.02060

    arXiv

  45. [2025]

    arXiv preprint arXiv:2503.05096

    SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding. arXiv preprint arXiv:2503.05096

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.