Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SpecRouter casts LLM inference as adaptive routing: at each step it selects a chain of draft and verifier models to minimize predicted latency per token, while keeping output identical to the target model.

desk verdict Plausible adaptive multi-level speculative decoding design with no real evaluation; the quality guarantee is underspecified and the experiments are a placeholder. read the letter →

arxiv 2505.07680 v1 pith:UP3BPFM5 submitted 2025-05-12 cs.LG cs.DC

Hang Wu , Jianian Zhu , Yinghui Li , Haojie Wang , Biao Hou , Jidong Zhai This is my paper · ORCID
classification cs.LGcs.DC
keywords speculativedecodingadaptiveroutingmodelchainschedulingmulti-levelverificationKVcacherollbacklatencyoptimizationLLMservingacceptanceprobability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SpecRouter claims that the static draft-target pairing used in speculative decoding leaves most of the available speedup on the table. The paper reframes inference as an adaptive routing problem: at each generation step, a scheduler chooses a chain of models (one drafter and possibly several intermediate verifiers, ending at the designated target model) that minimizes predicted latency per accepted token, and the chosen chain can change as the workload and system load shift. If correct, this makes speculative decoding self-tuning across heterogeneous model pools, removing the manual search for good draft-target pairs and keeping output identical to the target model under deterministic sampling. The paper reports speedups over the autoregressive baseline that stay above the compared two-level and vanilla three-level schemes at every batch size tested. A sympathetic reader would take the central message to be that adaptivity, not a bigger fixed chain, is what unlocks consistent acceleration.

What carries the argument

The machinery is the dynamic chain scheduler and its simplified cost model, Eq. 7: it ranks candidate chains by predicted effective time per target token, using smoothed per-model latencies $T_i$ and acceptance probabilities $\alpha_{ij} \approx f(\mathrm{SimScore}(M_i,M_j))$, where $\mathrm{SimScore}$ is $1 - \mathbb{E}[D_{\mathrm{TV}}]$ between the two models' output distributions. The acceptance values decide how far a draft is expected to propagate; the cost model decides whether a longer chain with fast early models pays off. The supporting mechanism is the StateManager's logical validity mask, which marks rolled-back KV cache entries invalid instead of physically moving data, then truncates only the common minimum rollback length across the batch. Together these components let the system adapt the inference path online without sacrificing consistency.

What would settle it

Record, for every candidate chain on the four datasets, the realized acceptance rate per adjacent model pair and the measured effective time per accepted token. If, over repeated runs, the chain with the lowest predicted effective time is not the chain with the lowest measured effective time, or if the $\alpha$ estimates deviate systematically from the realized acceptance rates, the adaptive-selection mechanism is not the source of the claimed speedup.

Watch

Extended reading notes

Core claim

The central discovery is a cost-aware model-chain scheduler that converts speculative decoding from a fixed two-stage pipeline into a multi-hop route chosen on the fly. For each candidate chain ending at the target model, the scheduler estimates acceptance probabilities between adjacent models from the smoothed total variation distance of their output distributions, plugs measured per-model token times and those acceptance estimates into a per-cycle cost model, and selects the chain with the lowest predicted effective time per target token. The same step is then executed with staged verification, so intermediate models can reject bad drafts before the target model pays the verification cost, and a logical validity mask makes KV-cache rollback cheap and correct across heterogeneous models. On the preliminary evaluation, Third-level SpecRouter reaches speedups such as 1.91x at batch size 16, versus 1.31x for two-level and 1.20x for vanilla three-level speculative decoding, and beats both baselines at all reported batch sizes.

Load-bearing premise

The load-bearing premise is that the effective acceptance probability between two models can be estimated from their smoothed average distribution-distance score through an unspecified mapping, and that the simplified cost model correctly ranks which chain will be fastest; if either piece fails, the selected chain is suboptimal and the reported speedups evaporate.

Editorial extensions

If this is right

  • Serving systems with a pool of heterogeneous models no longer need offline search over draft-verifier pairs; the scheduler performs that search online.
  • Intermediate verifiers can reject low-quality drafts before the target model runs, shifting part of the verification cost from the largest model to smaller ones.
  • Logical KV-cache rollback with a validity mask keeps multi-level speculation consistent under continuous batching, making the approach usable in production serving.
  • The reported speedups hold across batch sizes from 1 to 64, whereas the static baselines degrade at some batch sizes.
  • Under greedy decoding, generated sequences are identical to target-model-only decoding, so acceleration does not change output quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Outside the paper's own evidence: the cleanest way to validate the adaptive claim is to log realized acceptance rates and compare them against the $\alpha$ estimates from the similarity mapping; the paper does not report that comparison.
  • A direct extension the design implies: adding a new model to the pool should require no retuning, because the scheduler should adopt it only when its profiled speed and similarity make it competitive.
  • The routing formulation also suggests that the same cost model could rank candidate token trees rather than single chains, making the scheduler applicable to tree-based drafting without a separate mechanism.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. SpecRouter proposes an adaptive multi-level speculative decoding system that dynamically selects a chain of draft and verifier models to reduce inference latency while preserving the target model's output quality. The system uses a ModelChainScheduler that estimates inter-model acceptance probabilities from smoothed Total Variation Distance (Eqs. 5-6) and predicts effective latency per token (Eq. 7) to choose an optimal chain. A multi-level verification framework with intermediate verifiers is described, along with a StateManager that uses a logical validity mask for KV-cache rollbacks. The evaluation in Section 6 reports speedups in Table 2 for batch sizes 1-64 relative to an autoregressive baseline.

Significance. If the central guarantees held, the adaptive chain scheduling idea could be a useful contribution to LLM serving, and the logical validity mask is a sensible mechanism for heterogeneous KV-cache rollback. However, the paper does not establish these claims: the multi-level verification protocol as described changes the output distribution, and the experiments are explicitly preliminary and lack the detail needed to support the speedup claims. The core correctness issue would need to be resolved before the contribution can be assessed.

major comments (4)
  1. [§4.3 and §3.1] The multi-level verification protocol is inconsistent with the quality-preservation guarantee. Under the described protocol, an intermediate model can reject a drafted token before the target model verifies it (the VerifyProcessor repeats for each level and stops at the first rejection). In greedy decoding, a token is correct only if it equals the target's argmax; an intermediate model with different logits can reject a token the target would have accepted. Because the target never sees that token, the final output diverges from the Target-Model-Only baseline, contradicting the 'rigorously preserving output quality' claim in Section 3.1 and the Output Quality metric in Section 5. No lossless rejection rule is specified; the only way to know the target's verdict is to run the target.
  2. [§4.2, Eq. (7), Algorithm 1] The chain-selection model is unvalidated. The mapping f from SimScore to acceptance probability is left unspecified ('e.g., calibrated sigmoid') with no calibration procedure, and Eq. (7) is a simplified cost model whose accuracy is never tested. Because the scheduler's optimal-chain decision depends entirely on these estimates, an uncalibrated f or an inaccurate cost model would select suboptimal chains and invalidate the claimed speedups. No sensitivity analysis or ablation against ground-truth acceptance rates is provided.
  3. [§2.2, Eq. (2)] The acceptance probability is stated as α ≈ E[Σ min(p,q)] = 1 - E[DTV]. This identity is exact for the speculative-sampling acceptance rule, but the paper assumes deterministic/greedy decoding for output equivalence (Section 5). Under greedy decoding, the acceptance probability is q(argmax p), not 1 - DTV, so the scheduler's α estimates are not grounded in the actual decoding rule. This further undermines the reliability of the chain selection, independent of the calibration of f.
  4. [§6, Table 2] The experimental evidence is insufficient. Table 2 reports speedups across batch sizes but omits error bars, number of runs, dataset-specific results (despite four datasets listed in Section 5), and configuration details for the baselines. The baselines 'Second-level SD' and 'Third-level SD' in Table 2 do not correspond to the baselines defined in Section 5 (SSD-Smallest, SSD-Tuned). The abstract itself calls the experiments 'preliminary,' and Section 6.1 says 'we immediately support comprehensive experiments' without providing them. The claim of consistent superiority across batch sizes is not substantiated.
minor comments (6)
  1. [§5] The baseline definitions (SSD-Smallest, SSD-Tuned) do not appear in Table 2; the table's column labels 'Second-level SD' and 'Third-level SD' are undefined, making it impossible to interpret the comparison.
  2. [§5] The text mentions a target model 'e.g., Llama-2-70b' but the experiments use Llama-2-7b-chat-hf as the target; clarify whether the 70B model was ever evaluated.
  3. [§2.2] The citation '[28?]' contains a typographical artifact; the reference is incomplete.
  4. [Figure 2 and Eq. (7)] Figure 2 uses 'T_cascade' while Eq. (7) uses 'T_eff' for the same quantity; unify the notation.
  5. [§6.1] The sentence 'we immediately support comprehensive experiments' is unclear and appears to be a placeholder; please replace it with an actual description of the experiments performed or omit it.
  6. [§5] The paper does not provide the code or a link to the implementation, despite claiming a 16K-line implementation; consider releasing it for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the core chain-selection model and reported speedups are not derived from fitted parameters or self-citations.

full rationale

SpecRouter's central derivation chain is self-contained. The scheduler's effective-time model (Eq. 7) is a cost model assembled from profiled per-token latencies T_i and estimated acceptance probabilities alpha_ij, where alpha_ij is derived from a smoothed Total Variation Distance via an explicitly stated but uncalibrated mapping f (Section 4.2). This is a modeling assumption with unvalidated accuracy, but it is not circular: the alpha estimates do not come from the speedup numbers in Table 2, and the reported speedups are presented as measured experimental outcomes, not as outputs of the cost model. The paper adapts the standard speculative-decoding formula (Eq. 2) from prior work, which is independent support under its stated assumptions, and the self-citation [55] is used only as a general KV-cache reference, not as a load-bearing justification for chain selection or for the claimed speedups. The internal tension between intermediate-model rejection and the promised target-identical output is a correctness concern, not a circular-derivation concern, and no fitted parameter is renamed as a prediction. The most notable limitations are the uncalibrated mapping f, the lack of validation of Eq. 7 against measured acceptance rates, and the 'preliminary experiments' caveat in Sections 5 and 6; these affect confidence in the results but do not make the derivation circular.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central design depends on a small number of hand-chosen hyperparameters (W, f, EMA constants), the standard SD theory from prior work, and an unvalidated cost model. The only invented entity is the logical validity mask, which has no independent empirical or formal support in this paper.

free parameters (3)
  • Speculative draft window size W = 4 (Figure 2 example; global default)
    The number of tokens drafted per step is set as a global configuration; the paper does not report how W was chosen in experiments.
  • Mapping function f from SimScore to acceptance probability = not specified (e.g., calibrated sigmoid)
    Section 4.2 states alpha_ij is estimated as f(SimScore) but neither the functional form nor calibration procedure is given; the scheduler's predictions depend on it.
  • EMA smoothing constants alpha_time and alpha_sim = not specified
    Section 4.2 defines exponentially weighted moving averages for timings and similarities but does not report the smoothing constants used.
assumptions (3)
  • standard math Speculative decoding acceptance theory: acceptance probability alpha approximately equals 1 minus DTV(p, q), and expected accepted tokens follow the geometric series (Eqs. 2 to 4).
    Quoted from prior work (Leviathan et al.) and used as the basis for the scheduler's predictions.
  • ad hoc to paper The simplified chain efficiency model (Eq. 7) adequately approximates real latency per token.
    The paper states a simplified model can be expressed and gives no derivation or validation; all chain selection depends on this cost model.
  • domain assumption All models in the pool share the same tokenizer as the target model.
    Section 5 restricts experiments to Llama-family models because of consistent tokenizer availability; the entire multi-level draft and verify scheme assumes token-id compatibility.
invented entities (1)
  • Logical validity mask (cache_mask)
    purpose: Decouples logical validity of KV-cache entries from physical storage to support low-overhead rollbacks in multi-level speculation.
    Introduced in Section 4.4; correctness is argued conceptually (mask ignored by attention) but there is no standalone evaluation or formal proof, and its overhead is not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models." pith.science (2026). https://pith.science/paper/UP3BPFM5

@misc{pith2026250507680,
  author       = {Pith},
  title        = {Pith review of: SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UP3BPFM5}},
  note         = {Machine review of arXiv:2505.07680}
}
read the original abstract

Large Language Models (LLMs) present a critical trade-off between inference quality and computational cost: larger models offer superior capabilities but incur significant latency, while smaller models are faster but less powerful. Existing serving strategies often employ fixed model scales or static two-stage speculative decoding, failing to dynamically adapt to the varying complexities of user requests or fluctuations in system performance. This paper introduces \systemname{}, a novel framework that reimagines LLM inference as an adaptive routing problem solved through multi-level speculative decoding. \systemname{} dynamically constructs and optimizes inference "paths" (chains of models) based on real-time feedback, addressing the limitations of static approaches. Our contributions are threefold: (1) An \textbf{adaptive model chain scheduling} mechanism that leverages performance profiling (execution times) and predictive similarity metrics (derived from token distribution divergence) to continuously select the optimal sequence of draft and verifier models, minimizing predicted latency per generated token. (2) A \textbf{multi-level collaborative verification} framework where intermediate models within the selected chain can validate speculative tokens, reducing the verification burden on the final, most powerful target model. (3) A \textbf{synchronized state management} system providing efficient, consistent KV cache handling across heterogeneous models in the chain, including precise, low-overhead rollbacks tailored for asynchronous batch processing inherent in multi-level speculation. Preliminary experiments demonstrate the validity of our method.

Figures

Figures reproduced from arXiv: 2505.07680 by the authors.

Figure 1
Figure 1. , comprising several key components. The central control plane is the ChainRouter; it receives generation requests, orchestrates the multi-level speculative decoding workflow (prefill, draft, verify, rollback), interacts with the ModelChainScheduler to get optimal model chains, dispatches tasks to the Executor, and manages the overall generation state and termination logic. Resource management is handled by the Mode… view at source ↗
Figure 2
Figure 2. Dynamic model chain selection example. Based on profiled times (𝑇𝑖) and acceptance probabili￾ties (𝛼𝑖 , derived from SimScore), the scheduler predicts the effective time per target token (𝑇eff) for candidate chains (using logic based on Eq. 7, conceptualized in Alg. 1’s prediction function) and selects the chain with the minimum predicted time. 4.3 The Processors and Execution Stages The Executor routes requests to … view at source ↗
Figure 3
Figure 3. Illustration of the logical cache_mask decou￾pling validity from physical KV cache storage. Invalid items (Mask=0) are ignored during attention, enabling correct handling of asynchronous progress after roll￾backs without immediate data movement. (1) Logical Rollback: The RollbackProcessor imme￾diately updates cache_mask for sequence 𝑏, setting the last 𝑟𝑏 entries to 0. The attention mechanism in the next step uses t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpecMemo: Speculative Decoding is in Your Pocket

    cs.LG 2025-05 conditional novelty 5.0 of 10

    SpecMemo prunes speculative-decoding token trees and pre-allocates memory to run multi-turn LLM chatbots on GPUs with as little as 8GB, retaining about 96% of throughput while cutting buffer memory by 65%.

Reference graph

Works this paper leans on

55 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    Scaling laws for generative mixed-modal language models

    Aghajanyan, A., Yu, L., Conneau, A., Hsu, W.-N., Hambardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning (2023), PMLR, pp. 265– 279

    work page 2023

  2. [2]

    S., Ramjee, R., and Tumanov, A

    Agrawal, A., Kedia, N., Mohan, J., Panwar, A., Kwatra, N., Gula- vani, B. S., Ramjee, R., and Tumanov, A. Vidur: A large-scale simu- lation framework for llm inference. Proceedings of Machine Learning and Systems 6 (2024), 351–366

    work page 2024

  3. [3]

    In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp

    Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gula- vani, B., Tumanov, A., and Ramjee, R.Taming{Throughput-Latency} tradeoff in{LLM} inference with{Sarathi-Serve}. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 117–134

    work page 2024

  4. [4]

    D., Chen, D., and Dao, T

    Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv: 2401.10774 (2024)

    arXiv 2024

  5. [5]

    On the entropy of language models in getting semantic from tokens

    Chen, H., Yang, X., Jiaying, Z., and Wang, W. On the entropy of language models in getting semantic from tokens

    work page

  6. [6]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  7. [7]

    Advances in Neural Information Processing Systems 37 (2024), 86226–86242

    Chen, Z., Y ang, X., Lin, J., Sun, C., Chang, K., and Huang, J.Cascade speculative drafting for even faster llm inference. Advances in Neural Information Processing Systems 37 (2024), 86226–86242

    work page 2024

  8. [8]

    Nvidia a100 gpu: Performance & innovation for gpu computing

    Choqette, J., and Gandhi, W. Nvidia a100 gpu: Performance & innovation for gpu computing. In 2020 IEEE Hot Chips 32 Symposium (HCS) (2020), IEEE Computer Society, pp. 1–43

    work page 2020

Show all 55 references
  1. [9]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24 , 240 (2023), 1–113

    2023

  2. [10]

    Unified scaling laws for routed language models

    Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al. Unified scaling laws for routed language models. In International conference on machine learning (2022), PMLR, pp. 4057–4086

    2022

  3. [11]

    Train- ing verifiers to solve math word problems, 2021

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Train- ing verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168 9 (2021)

    2021 arXiv

  4. [12]

    J., Gonzalez, J

    Crankshaw, D., W ang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., and Stoica, I. Clipper: A{Low-Latency} online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17) (2017), pp. 613–627

    2017

  5. [13]

    Increasing transformer token length with a maximum entropy principle method

    Cukier, R. Increasing transformer token length with a maximum entropy principle method. arXiv preprint arXiv:2408.10277 (2024)

    2024 arXiv

  6. [14]

    Graphrouter: A graph-based router for llm selections

    Feng, T., Shen, Y., and You, J. Graphrouter: A graph-based router for llm selections. In The Thirteenth International Conference on Learning Representations (2024)

    2024

  7. [15]

    {ServerlessLLM}:{Low-Latency} serverless inference for large language models

    Fu, Y., Xue, L., Huang, Y., Brabete, A.-O., Ustiugov, D., Patel, Y., and Mai, L. {ServerlessLLM}:{Low-Latency} serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 135–153

    2024

  8. [16]

    The llama 3 herd of models

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al- Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  9. [17]

    In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) (2024), pp

    Hu, Q., Ye, Z., Wang, Z., Wang, G., Zhang, M., Chen, Q., Sun, P., Lin, D., W ang, X., Luo, Y., et al.Characterization of large language model development in the datacenter. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) (2024), pp. 709–729

    2024

  10. [18]

    Specserve: Efficient and slo-aware large language model serving with adaptive speculative decoding

    Huang, K., Wu, H., Shi, Z., Zou, H., Yu, M., and Shi, Q. Specserve: Efficient and slo-aware large language model serving with adaptive speculative decoding. arXiv preprint arXiv:2503.05096 (2025)

    2025

  11. [19]

    Hunter, J. S. The exponentially weighted moving average. Journal of quality technology 18, 4 (1986), 203–210

    1986

  12. [20]

    Jain, S., and Verma, R. K. A taxonomy and survey on grid-based rout- ing protocols designed for wireless sensor networks. ACM Computing Surveys 56, 8 (2024), 1–41

    2024

  13. [21]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.Scaling laws for neural language models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  14. [22]

    H., Gonza- lez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonza- lez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (2023)....

    2023

  15. [23]

    Lebovitz, L., Cavigelli, L., Magno, M., and Muller, L. K. Efficient inference with model cascades. Transactions on Machine Learning Research (2023)

    2023

  16. [24]

    In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp

    Lee, W., Lee, J., Seo, J., and Sim, J.{InfiniGen}: Efficient generative inference of large language models with dynamic{KV} cache man- agement. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 155–172

    2024

  17. [25]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning (2023), PMLR, pp. 19274–19286

    2023

  18. [26]

    Static batching of irregular workloads on gpus: Framework and application to efficient moe model inference, 2025

    Li, Y., Li, Y., Zhang, J., Chen, B., Chen, X., Duan, L., Jin, Y., Li, Z., Liu, X., W ang, H., W ang, W., W ang, Y., Y ang, J., Zhang, P., Zheng, L., and Yu, W. Static batching of irregular workloads on gpus: Framework and application to efficient moe model inference, 2025

    2025

  19. [27]

    In International Conference on Machine Learning (2024)

    Li, Y., Wei, F., Zhang, C., and Zhang, H.EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning (2024)

    2024

  20. [28]

    EAGLE-3: Scaling up in- ference acceleration of large language models via training-time test, 2025

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE-3: Scaling up in- ference acceleration of large language models via training-time test, 2025

    2025

  21. [29]

    Adaserve: Slo-customized llm serving with fine-grained speculative decoding

    Li, Z., Chen, Z., Delacourt, R., Oliaro, G., W ang, Z., Chen, Q., Lin, S., Yang, A., Zhang, Z., Chen, Z., et al. Adaserve: Slo-customized llm serving with fine-grained speculative decoding. arXiv preprint arXiv:2501.12162 (2025)

    2025 arXiv

  22. [30]

    Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache.arXiv preprint arXiv:2401.02669 (2024)

    Lin, B., Zhang, C., Peng, T., Zhao, H., Xiao, W., Sun, M., Liu, A., Zhang, Z., Li, L., Qiu, X., et al. Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache.arXiv preprint arXiv:2401.02669 (2024)

    2024 arXiv

  23. [31]

    Parrot: Efficient serving of{LLM-based} applications with semantic variable

    Lin, C., Han, Z., Zhang, C., Y ang, Y., Y ang, F., Chen, C., and Qiu, L. Parrot: Efficient serving of{LLM-based} applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 929–945

    2024

  24. [32]

    Deepseek-v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  25. [33]

    In The Thirteenth International Conference on Learning Representations (2025)

    Liu, T., Li, Y., Lv, Q., Liu, K., Zhu, J., Hu, W., and Sun, X.Pearl: Parallel speculative decoding with adaptive draft length. In The Thirteenth International Conference on Learning Representations (2025)

    2025

  26. [34]

    A., Xie, G., Zhan, J., Zhang, H., Hjálmt `ysson, G., and Greenberg, A

    Maltz, D. A., Xie, G., Zhan, J., Zhang, H., Hjálmt `ysson, G., and Greenberg, A. Routing design in operational networks: A look from the inside. ACM SIGCOMM Computer Communication Review 34 , 4 (2004), 27–40

    2004

  27. [35]

    Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Y ang, L., Shi, X., et al.Specinfer: Accelerat- ing large language model serving with tree-based speculative infer- ence and verification. In Proceedings of the 29th ACM International Con...

    2024

  28. [36]

    E., Kadous, M

    Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., and Stoica, I. Routellm: Learning to route llms from preference data. In The Thirteenth International Conference on Learning Representations (2024)

    2024

  29. [37]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 (2019)

    2019 arXiv

  30. [38]

    Mooncake: Trading more storage for less computation—a {KVCache-centric} architecture for serving{LLM} chatbot

    Qin, R., Li, Z., He, W., Cui, J., Ren, F., Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: Trading more storage for less computation—a {KVCache-centric} architecture for serving{LLM} chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25) (2025), pp. 155–170

    2025

  31. [39]

    S., Jha, S., Kalbarczyk, Z

    Qiu, H., Banerjee, S. S., Jha, S., Kalbarczyk, Z. T., and Iyer, R. K. {FIRM}: An intelligent fine-grained resource management framework for{SLO-Oriented} microservices. In 14th USENIX symposium on operating systems design and implementation (OSDI 20) (2020), pp. 805– 825

    2020

  32. [40]

    W., Tay, Y., Ruder, S., Zhou, D., et al

    Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., et al. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057 (2022)

    2022 arXiv

  33. [41]

    In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles(2024), pp

    Song, Y., Mi, Z., Xie, H., and Chen, H.Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles(2024), pp. 590– 606

    2024

  34. [42]

    Llumnix: Dynamic scheduling for large language model serving

    Sun, B., Huang, Z., Zhao, H., Xiao, W., Zhang, X., Li, Y., and Lin, W. Llumnix: Dynamic scheduling for large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implemen- tation (OSDI 24) (2024), pp. 173–191

    2024

  35. [43]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  36. [44]

    N., Kaiser, Ł., and Polosukhin, I

    V aswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Ad- vances in neural information processing systems 30 (2017)

    2017

  37. [45]

    In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations (2020), pp

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: syst...

    2020

  38. [46]

    Loongserve: Efficiently serving long-context large language models with elastic se- quence parallelism

    Wu, B., Liu, S., Zhong, Y., Sun, P., Liu, X., and Jin, X. Loongserve: Efficiently serving long-context large language models with elastic se- quence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (2024), pp. 640–654

    2024

  39. [47]

    Advances in Neural Information Processing Systems 37 (2024), 128082–128117

    Yin, M., Chen, M., Huang, K., and W ang, M.A theoretical perspective for speculative decoding algorithm. Advances in Neural Information Processing Systems 37 (2024), 128082–128117

    2024

  40. [48]

    S., Kim, G.-W., Kim, S., and Chun, B.-G

    Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) (2022), pp. 521–538

    2022

  41. [49]

    Fast and live model auto scaling with o(1) host caching, 2024

    Zhang, D., W ang, H., Liu, Y., Wei, X., Shan, Y., Chen, R., and Chen, H. Fast and live model auto scaling with o(1) host caching, 2024

    2024

  42. [50]

    Leveraging uncertainty estimation for efficient llm routing

    Zhang, T., Mehradfar, A., Dimitriadis, D., and Avestimehr, S. Leveraging uncertainty estimation for efficient llm routing. arXiv preprint arXiv:2502.11021 (2025)

    2025 arXiv

  43. [51]

    Judging llm-as-a-judge with mt- bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt- bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

    2023

  44. [52]

    L., Huang, J., Yu, C

    Zheng, L., Yin, L., Xie, Z., Sun, C. L., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems 37 (2024), 62557–62583

    2024

  45. [53]

    {DistServe}: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. {DistServe}: Disaggregating prefill and decoding for goodput- optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) (2024), pp. 193–210

    2024

  46. [54]

    A survey on efficient inference for large language models

    Zhou, Z., Ning, X., Hong, K., Fu, T., Xu, J., Li, S., Lou, Y., Wang, L., Yuan, Z., Li, X., et al. A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294 (2024)

    2024 arXiv

  47. [55]

    arXiv preprint arXiv:2503.08461 (2025)

    Zhu, J., Wu, H., W ang, H., Li, Y., Hou, B., Li, R., and Zhai, J.Fastcache: Optimizing multimodal llm serving through lightweight kv-cache compression framework. arXiv preprint arXiv:2503.08461 (2025)

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.