Pith. sign in

REVIEW 5 major objections 5 minor 34 references

Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Scheduling speculative-decoding requests with both predicted output length and token acceptance rate, rather than output length alone, reduces average inference latency by roughly 39%.

desk verdict A plausible scheduling idea for speculative decoding, but the 39% claim is built on thin empirical support and a stability trigger that is not specified enough to check. read the letter →

arxiv 2505.17074 v1 pith:76V55KLH submitted 2025-05-20 cs.CL cs.AIcs.LG

Ruixiao Li , Fahao Chen , Peng Li This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords speculativedecodingLLMinferencerequestschedulingtokenacceptanceratesemi-clairvoyantleastattainedserviceshortestjobfirstlatencyminimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding speeds up LLM inference by having a small model draft several candidate tokens that the large model verifies in parallel, so a request's execution time depends on how many drafted tokens are accepted, not just on how many tokens it outputs. The paper's claim is that schedulers which estimate execution time purely from predicted output length are systematically wrong for such workloads, and that a scheduler can do much better by tracking each request's token acceptance rate as it decodes. It proposes LAPS-SD, a two-phase scheduler: while acceptance rates are still unstable, it preempts requests across multiple priority queues (LAS-style); once a request's acceptance rate settles, it estimates the remaining execution time with a formula combining predicted length and acceptance rate, and switches to shortest-job-first without preemption. Experiments on three datasets place the average latency reduction at about 39% versus length-prediction-based SJF and LAS. A sympathetic reader would care because it is a concrete, parameter-light way to bring classic scheduling ideas to LLM serving systems that already use speculative decoding.

What carries the argument

The load-bearing mechanism is the two-state, multi-queue scheduling algorithm LAPS-SD with its execution-time estimator. Requests begin as 'non-perceptible' in the highest-priority queue; after each speculative round the accumulated service determines which of $K$ exponentially sized priority queues they sit in, allowing preemption of young requests at low switching cost because few KV pairs have been generated. When the acceptance rate has been stable for $\gamma$ consecutive rounds (difference below a threshold $\delta$), the request becomes 'perceptible,' and its remaining time is estimated by $$\tilde{T}_i = \frac{n L_i}{n A_i + 1} T_{\text{SSM}} + \frac{L_i}{n A_i + 1} T_{\text{LLM}},$$ where $n$ is the number of speculative tokens per round and $A_i$ the stabilized average acceptance rate. Perceptible requests are scheduled SJF within their queue and are never preempted, eliminating the switching overhead that grows with KV-cache size. The key property the algorithm exploits is the empirical stabilization of acceptance rates over decoding.

What would settle it

Run LAPS-SD on a workload constructed so that token acceptance rates oscillate throughout the entire decoding process, or let requests complete before the $\gamma$-round stability window elapses; if the average latency is no better than pure LAS or pure LP-SJF, the stabilization premise fails. In particular, compare the estimated execution time of Eq. (6) against measured times after stabilization on a large sample of requests; an average error well above the reported 6.84% would indicate the stability detector is not working.

Watch

Extended reading notes

Core claim

The central discovery is that speculative-decoding requests become 'perceptible' during execution: after an initial volatile phase, their token acceptance rate stabilizes, so the total execution time can be estimated accurately enough to run shortest-job-first scheduling. The paper shows that execution time is shaped by both output length and acceptance rate—more precisely, the number of LLM verification passes scales as $L_i/(nA_i+1)$ per request, where $L_i$ is output length, $A_i$ is average acceptance rate, and $n$ is the speculative window. Existing length-prediction-only schedulers (LP-SJF) ignore the acceptance-rate factor and therefore misorder requests; preemption-only schedulers (LAS) avoid misordering but pay increasing KV-cache switching costs. LAPS-SD runs a LAS-like preemptive multi-queue during the volatile phase, then commits each request to a non-preemptive SJF order once stability is detected, and the authors report a 39% average latency reduction over these baselines.

Load-bearing premise

The method assumes that a request's token acceptance rate becomes stable and predictable early enough in decoding that the fixed average acceptance rate used in the execution-time estimate is accurate; if acceptance rates keep fluctuating, the shortest-job-first ordering is based on wrong job sizes.

Editorial extensions

If this is right

  • LLM serving systems that support speculative decoding can use acceptance-rate monitoring, not just predicted output length, to estimate request execution times.
  • Schedulers can safely combine preemption and SJF: preempt only while estimates are unreliable, then commit to non-preemptive order, avoiding both head-of-line blocking and excessive KV-cache switching costs.
  • The optimal number of priority queues is not fixed: it decreases as request count grows and as switching costs rise, so deployments should tune $K$ per workload.
  • Execution-time estimation error is around 6.84% on average across the tested datasets, and the paper attributes the residual error to acceptance-rate prediction accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same semi-clairvoyant principle—preempt while estimates are unreliable, then commit to SJF—should transfer to any workload with an early volatile phase followed by stabilization, such as auto-regressive decoding with adaptive sampling temperatures or multi-step reasoning traces.
  • The stability thresholds $\gamma$ and $\delta$ are left unspecified in the paper; a practical deployment would need to set them per model family, and an adaptive scheme that learns these thresholds online could further close the gap to the optimal schedule.
  • The current experimental setup fixes batch size to 1; a natural extension is to group perceptible requests with similar $\tilde{T}_i$ into batches, trading some latency for throughput while keeping the SJF ordering.
  • The 39% figure compares against two baselines; a stronger test would be against a clairvoyant scheduler that knows each request's true execution time from the start, which would isolate the cost of the volatile phase.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper proposes LAPS-SD, a scheduling algorithm for speculative-decoding LLM requests. The method assumes that a request's token acceptance rate is volatile in early decoding and stabilizes later; while a request is non-perceptible it is managed with LAS-style preemption across K priority queues, and once its acceptance rate is judged stable, the algorithm predicts output length (via an existing method) and acceptance rate (from recent history), estimates the remaining execution time using Eq. (6), and schedules perceptible requests by the SJF principle. The paper formulates a non-preemptive average-latency minimization problem in Section 3, describes the algorithm in Section 4, and evaluates it in Section 5 on three datasets against LP-SJF and LAS, reporting an average latency reduction of about 39% and an average execution-time estimation error of 6.84%.

Significance. The paper addresses a real and under-studied problem: scheduling speculative-decoding requests whose execution time depends on both output length and token acceptance rate. The two-phase design—preemption while acceptance rates are volatile, then SJF after stabilization—is a sensible idea that is motivated by the example curves in Figure 3. The paper also makes an honest effort to evaluate the execution-time estimator, and the 39% figure is an empirical measurement rather than a parameter fitted to the target result. If the latency reduction is reproducible under fully specified hyperparameters and with run-to-run variance reported, this would be a useful contribution to LLM serving systems. However, the current evidence under-supports the headline quantitative claim and the optimality wording in the abstract.

major comments (5)
  1. [Section 4.3, Eq. (6)] The central premise—that acceptance rates stabilize early enough and remain stable for the rest of decoding—is not quantitatively established. The stability condition uses parameters gamma and delta that are never specified, and Figure 3 shows only three example curves. This matters because the perceptible phase uses the fixed acceptance rate A_i in Eq. (6) as a predictor for the remaining execution, so without evidence on how early stabilization occurs, the semi-clairvoyant stage may start too late to affect latency or may start while A_i is still volatile.
  2. [Section 5.2, Figure 7] The reported aggregate estimation error (6.84% overall; 7.63%, 11.21%, and 8.51% per dataset) is not the right metric for validating SJF ordering. The scatter plots show individual real/estimated time ratios spanning roughly 0.6 to 1.6, i.e., per-request errors of about 40–60%. Since SJF is sensitive to pairwise order rather than mean error, these errors can invert scheduling decisions even when the mean error is small; the paper should report order-preservation statistics or error quantiles by output length.
  3. [Section 5.2, Figure 6] The number of priority queues K is a free parameter, and the optimal K varies with dataset and request count (e.g., K=6 for 10 Chatbot requests vs. K=4 for 10 MiniThinky requests). The main comparison in Figure 5 does not state which K was used for LAPS-SD, nor whether K was tuned per dataset. Without this information the 39% claim is not reproducible, and the reported advantage could be partly due to per-dataset tuning of K.
  4. [Abstract, Section 3] The statement that LAPS-SD "minimizes" average inference latency is not supported by any optimality proof or lower-bound argument. The formulation in Eqs. (2)–(5) is a non-preemptive problem statement, and Algorithm 1 is a heuristic. The text should either provide a formal guarantee for the idealized model or soften the claim to say that LAPS-SD reduces latency in the evaluated settings.
  5. [Section 4.3, Eq. (6)] The notation in Eq. (6) is internally inconsistent with the definitions in the surrounding text. The text defines T_LLM as the speculation time per token and T_SSM as the verification cost per round, but Eq. (6) places T_SSM in the "Speculation Time" term and T_LLM in the "Verification Time" term. Since Eq. (6) is the basis of the execution-time estimate and the claimed 6.84% accuracy, the paper must clarify which quantity is actually used in the implementation and align the text and equation.
minor comments (5)
  1. [Section 4.2] The threshold relation Sup_j = M^(j-1) * Sup_1 introduces a parameter M that is never defined or reported; please define M and explain how it was set in the experiments.
  2. [Section 5] Figures 5–7 show no error bars, confidence intervals, or repeated-run information, so it is unclear whether the reported differences between LAPS-SD and the baselines are stable across random seeds or workload samplings.
  3. [Section 2.3 and References] The statement that LAS-based scheduling is "adopted by [Leviathan et al., 2023]" is questionable: that reference is a speculative-decoding paper, not a scheduling paper, and the LAS scheduling policy is credited to [Rai et al., 2003].
  4. [References] The reference "[Z et al., 2024]" is incomplete (missing full author list, title, and venue) and appears to duplicate the later [Zheng et al., 2024] entry; please merge or correct.
  5. [Section 4.3] Minor wording: "prioritize scheduling perception requests" should read "prioritize scheduling perceptible requests," and the pseudocode in Algorithm 1 omits the details of how the stability check and queue migration are actually implemented.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: LAPS-SD's execution-time estimates are empirical and its latency reduction is measured, not fitted.

full rationale

The paper's central claim (39% average latency reduction) is an experimental outcome reported in Section 5.2 (Figure 5), not a quantity derived from the method's definitions. The execution-time estimator in Eq. (6) combines an externally predicted output length L_i (from [Z et al., 2024], a prior method by other authors) with a per-request acceptance rate A_i measured from that request's own decoding history. This is an adaptive forecasting rule, not an identity: A_i is the average of observed acceptance rates over gamma rounds after a stability test (Section 4.3), and the estimated time is used only as a scheduling key for SJF ordering. Nothing in Eq. (6) or Algorithm 1 sets the average inference latency equal to the estimated time, and no fitted parameter is renamed as the 39% result. The stability threshold parameters gamma and delta are left unspecified, which harms reproducibility and could affect whether the SJF phase orders requests correctly, but that is a correctness/robustness limitation rather than circular reasoning. The only self-citation ([Chen et al., 2025], by two of the present authors) appears in an introductory survey sentence about existing speculative-decoding serving systems and is not load-bearing for the scheduling algorithm or the empirical result. The tuning of K in Figure 6 is a design choice that affects performance, but the reported latency is measured end-to-end against fixed baselines, so the claimed reduction is not forced by construction. The paper is therefore self-contained with respect to its empirical claims, and no circular step can be exhibited.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on several hyperparameters and the empirical stabilization property. The free parameters are not fixed in the text, and the stabilizing-acceptance-rate assumption is the main load-bearing premise.

free parameters (5)
  • Number of priority queues K = Not specified for main experiments; optimal values range from 4 to 6 in Figure 6
    K controls the tradeoff between SJF approximation and preemption overhead. The paper shows the optimal K depends on dataset and request count, but does not state which K is used for the headline 39% result, suggesting possible selection on the test data.
  • Stability window gamma = Not specified
    The number of consecutive speculative decoding rounds used to compare acceptance rates in the stability test (Section 4.3). Its value affects when requests become perceptible, and no sensitivity analysis is given.
  • Stability threshold delta = Not specified
    The maximum allowed difference in acceptance rate over gamma rounds for declaring a request stable. Not specified, and no analysis of how it impacts latency.
  • Queue threshold base M = Not specified
    The exponential factor for queue size ranges (Sup_j = M^(j-1) * Sup_1). M is introduced in Section 4.2 but never assigned a value.
  • Speculative length n = Not specified
    The number of tokens generated by the SSM per round in Eq. (6). This is a system parameter that directly enters the execution-time estimate.
assumptions (3)
  • domain assumption The token acceptance rate of a request becomes stable and predictable after an initial unstable phase (Figure 3, Section 4.1).
    The entire transition from LAS-style scheduling to SJF-style scheduling depends on this stabilization property. It is demonstrated on three example requests but not established generally or theoretically.
  • domain assumption The batch size is set to 1, so serving is a single-server queue and preemption cost is the KV-cache switching overhead (Section 3).
    The problem formulation and experiments assume batch size 1. The paper claims extension to larger batches is easy but provides no experiment or argument.
  • domain assumption The execution time of a request is well approximated by Eq. (6), which assumes a constant acceptance rate A_i per round and averages over the number of accepted tokens.
    This per-round expectation ignores the variance in accepted tokens and the fact that acceptance rate changes over time; the paper does not validate this approximation against a queuing model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency." pith.science (2026). https://pith.science/paper/76V55KLH

@misc{pith2026250517074,
  author       = {Pith},
  title        = {Pith review of: Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/76V55KLH}},
  note         = {Machine review of arXiv:2505.17074}
}
read the original abstract

Speculative decoding accelerates Large Language Model (LLM) inference by employing a small speculative model (SSM) to generate multiple candidate tokens and verify them using the LLM in parallel. This technique has been widely integrated into LLM inference serving systems. However, inference requests typically exhibit uncertain execution time, which poses a significant challenge of efficiently scheduling requests in these systems. Existing work estimates execution time based solely on predicted output length, which could be inaccurate because execution time depends on both output length and token acceptance rate of verification by the LLM. In this paper, we propose a semi-clairvoyant request scheduling algorithm called Least-Attained/Perceived-Service for Speculative Decoding (LAPS-SD). Given a number of inference requests, LAPS-SD can effectively minimize average inference latency by adaptively scheduling requests according to their features during decoding. When the token acceptance rate is dynamic and execution time is difficult to estimate, LAPS-SD maintains multiple priority queues and allows request execution preemption across different queues. Once the token acceptance rate becomes stable, LAPS-SD can accurately estimate the execution time and schedule requests accordingly. Extensive experiments show that LAPS-SD reduces inference latency by approximately 39\% compared to state-of-the-art scheduling methods.

Figures

Figures reproduced from arXiv: 2505.17074 by the authors.

Figure 1
Figure 1. The illustration depicts different scheduling algorithms for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The ratio of switching costs to the inference time of re [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The average acceptance rate of three example requests [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The queue structure in the proposed scheduling algorithm. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The average inference latency with different scheduling algorithms. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The impact of the number of priority queue. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The estimation accuracy of the execution time. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 24 canonical work pages

  1. [1]

    Tam- ing throughput-latency tradeoff in llm inference with sarathi-serve

    [Agrawal et al., 2024] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gula- vani, Alexey Tumanov, and Ramachandran Ramjee. Tam- ing throughput-latency tradeoff in llm inference with sarathi-serve. In 18th USENIX Symposium on Operat- ing Systems Design and Implementation (OSDI 24), pages 117–134,

    work page 2024

  2. [5]

    Language models are few-shot learners

    [Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877–1901,

    2020

  3. [7]

    Accelerating large language model de- coding with speculative sampling

    [Chen et al., 2023] Charlie Chen, Sebastian Borgeaud, Ge- offrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model de- coding with speculative sampling. arXiv preprint arXiv:2302.01318,

    arXiv 2023

  4. [8]

    Luan, Zhou Su, and Jing Deng

    [Chen et al., 2025] Fahao Chen, Peng Li, Tom H. Luan, Zhou Su, and Jing Deng. Spin: Accelerating large lan- guage model inference with heterogeneous speculative models. In Proceedings of IEEE International Confer- ence on Computer Communications (INFOCOM) , Lon- don, United Kingdom, May

    work page 2025

  5. [10]

    Specdec++: Boosting speculative de- coding via adaptive candidate lengths

    [Huang et al., 2024] Kaixuan Huang, Xudong Guo, and Mengdi Wang. Specdec++: Boosting speculative de- coding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715,

    arXiv 2024

  6. [11]

    The effect of schedul- ing and preemption on the efficiency of llm inference serv- ing

    [Kim et al., 2024] Kyoungmin Kim, Kijae Hong, Caglar Gulcehre, and Anastasia Ailamaki. The effect of schedul- ing and preemption on the efficiency of llm inference serv- ing. arXiv preprint arXiv:2411.07447,

    arXiv 2024

  7. [12]

    Efficient memory management for large language model serving with pagedattention

    [Kwon et al., 2023] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sympo- sium on Operating Systems Principles , pages 611–626,

    work page 2023

  8. [13]

    Incorporating spec- ulative execution into scheduling of control-flow-intensive designs

    [Lakshminarayana et al., 2000] Ganesh Lakshminarayana, Anand Raghunathan, and Niraj K Jha. Incorporating spec- ulative execution into scheduling of control-flow-intensive designs. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 19(3):308–324,

    work page 2000

Show all 34 references
  1. [15]

    Al- paserve: Statistical multiplexing with model parallelism for deep learning serving

    [Li et al., 2023] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. Al- paserve: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Op...

    2023

  2. [16]

    Specpim: Accel- erating speculative inference on pim-enabled system via architecture-dataflow co-exploration

    [Li et al., 2024] Cong Li, Zhe Zhou, Size Zheng, Jiaxi Zhang, Yun Liang, and Guangyu Sun. Specpim: Accel- erating speculative inference on pim-enabled system via architecture-dataflow co-exploration. In Proceedings of the 29th ACM International Conference on Architectural Supp...

    2024

  3. [17]

    [Liu et al., 2024b] X. Liu, C. Daniel, L. Hu, W. Kwon, Z. Li, X. Mo, A. Cheung, Z. Deng, I. Stoica, and H. Zhang. Op- timizing speculative decoding for serving large language models using goodput. arXiv preprint arXiv:2406.14066,

    arXiv

  4. [18]

    Specinfer: Accelerating large language model serv- ing with tree-based speculative inference and verification

    [Miao et al., 2024] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serv- ing with tree-based speculative inference and verification...

    2024

  5. [19]

    Exegpt: Constraint-aware resource scheduling for llm inference

    [Oh et al., 2024] Hyungjun Oh, Kihong Kim, Jaemin Kim, Sungkyun Kim, Junyeol Lee, Du-seong Chang, and Jiwon Seo. Exegpt: Constraint-aware resource scheduling for llm inference. In Proceedings of the 29th ACM Interna- tional Conference on Architectural Support for Program- ming...

    2024

  6. [20]

    Splitwise: Efficient generative llm infer- ence using phase splitting

    [Patel et al., 2024] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, ´I˜nigo Goiri, Saeed Maleki, and Ri- cardo Bianchini. Splitwise: Efficient generative llm infer- ence using phase splitting. In 2024 ACM/IEEE 51st An- nual International Symposium on Computer Archi...

    2024

  7. [21]

    Efficient interactive llm serving with proxy model-based sequence length prediction

    [Qiu et al., 2024] Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T Kalbarczyk, Tamer Basar, and Rav- ishankar K Iyer. Efficient interactive llm serving with proxy model-based sequence length prediction. InInterna- tional...

    2024

  8. [22]

    Analysis of las scheduling for job size distributions with high variance

    [Rai et al., 2003] Idris A Rai, Guillaume Urvoy-Keller, and Ernst W Biersack. Analysis of las scheduling for job size distributions with high variance. In Proceedings of the 2003 ACM SIGMETRICS international conference on Measurement and modeling of computer systems , pages 218–228,

    2003

  9. [24]

    Specexec: Massively parallel speculative de- coding for interactive LLM inference on consumer de- vices

    [Svirschevski et al., 2024] Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative de- coding for interactive LLM inference on consumer de- vices. In The Thirty-eighth Annual Conference on Neural Inform...

    2024

  10. [25]

    Llama: Open and efficient founda- tion language models

    [Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- oth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:23...

    2023 arXiv

  11. [26]

    Minions: Accelerating large language model inference with adap- tive and collective speculative decoding

    [Wang et al., 2024] Siqi Wang, Hailong Yang, Xuezhu Wang, Tongxuan Liu, Pengbo Wang, Xuning Liang, Kejie Ma, Tianyu Feng, Xin You, Yongjun Bao, et al. Minions: Accelerating large language model inference with adap- tive and collective speculative decoding. arXiv preprint arXiv...

    2024 arXiv

  12. [27]

    Fast distributed infer- ence serving for large language models

    [Wu et al., 2023] Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed infer- ence serving for large language models. arXiv preprint arXiv:2305.05920,

    2023 arXiv

  13. [28]

    Mini- thinky dataset

    [Xuan Son NGUYEN, 2024] Xuan Son NGUYEN. Mini- thinky dataset. https://huggingface.co/datasets/ngxson/ MiniThinky-dataset,

    2024

  14. [29]

    Perllm: Person- alized inference scheduling with edge-cloud collaboration for diverse llm services

    [Yang et al., 2024] Zheming Yang, Yuanhao Yang, Chang Zhao, Qi Guo, Wenkai He, and Wen Ji. Perllm: Person- alized inference scheduling with edge-cloud collaboration for diverse llm services. arXiv preprint arXiv:2405.14636,

    2024 arXiv

  15. [30]

    Tree of thoughts: Deliberate problem solving with large language models

    [Yao et al., 2024] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Informa- tion Processing Systems, 36,

    2024

  16. [31]

    Adap- tive batch budget for llm inference

    [Yes ¸ilet al., 2024] C ¸ a˘grı Yes ¸il, Berhan Turku Ay, Funda Ay Ak, ¨Oyk¨u Berfin Mercan, and O˘guzhan Nefeso˘glu. Adap- tive batch budget for llm inference. In 2024 9th Interna- tional Conference on Computer Science and Engineering (UBMK), pages 219–223. IEEE,

    2024

  17. [32]

    Re- sponse length perception and sequence scheduling: An llm-empowered llm inference pipeline

    [Z et al., 2024] Zheng Z, Ren X, and et al Xue F. Re- sponse length perception and sequence scheduling: An llm-empowered llm inference pipeline. Advances in Neu- ral Information Processing Systems, 36,

    2024

  18. [33]

    Response length perception and sequence scheduling: An llm-empowered llm inference pipeline

    [Zheng et al., 2024] Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. Response length perception and sequence scheduling: An llm-empowered llm inference pipeline. Advances in Neural Information Processing Systems, 36,

    2024

  19. [34]

    Toolqa: A dataset for llm question answering with external tools

    [Zhuang et al., 2024] Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. Advances in Neu- ral Information Processing Systems, 36, 2024

    2024

  20. [2000]

    Fast inference from transformers via speculative decoding

    [Leviathan et al., 2023] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Ma- chine Learning, pages 19274–19286. PMLR,

    2023

  21. [2003]

    Accelerating LLM inference with staged speculative decoding

    [Spector and Re, 2023] Benjamin Frederick Spector and Christopher Re. Accelerating LLM inference with staged speculative decoding. In Workshop on Efficient Systems for Foundation Models of ICML 2023,

    2023

  22. [2020]

    Lee, Deming Chen, and Tri Dao

    [Cai et al., 2024] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning,

    2024

  23. [2021]

    Speculative streaming: Fast llm infer- ence without auxiliary models

    [Bhendawade et al., 2024] Nikhil Bhendawade, Irina Be- lousova, Qichen Fu, Henry Mason, Mohammad Rastegari, and Mahyar Najibi. Speculative streaming: Fast llm infer- ence without auxiliary models

    2024

  24. [2023]

    Program synthesis with large language models

    [Austin et al., 2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

    2021 arXiv

  25. [2024]

    Chatbot instruc- tion prompts

    [Alessandro Palla, 2023] Alessandro Palla. Chatbot instruc- tion prompts. https://huggingface.co/datasets/alespalla/ chatbot instruction prompts,

    2023

  26. [2025]

    Glide with a cape: A low-hassle method to accelerate speculative decoding

    [Du et al., 2024] Cunxiao Du, Jing Jiang, Xu Yuanchen, Ji- awei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. In Forty-first International Conference on Machine Learning,

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.