Pith. sign in

REVIEW 4 major objections 6 minor 59 references

This paper claims that LLM query routing can jointly optimize accuracy, cost, and time-to-first-token by simulating a serving framework's token batches, delivering up to 40% higher accuracy–cost utility at the same latencies as standard loa

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:02 UTC pith:FG3H36WX

load-bearing objection Worth a serious review if the authors add a held-out split and a head-to-head with [27]; the SFS idea is solid and the paper is honestly written. the 4 major comments →

arxiv 2607.18253 v1 pith:FG3H36WX submitted 2026-05-13 cs.AI

Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

Shivam Patel , Akaash R. Parthasarathy , Ankur Mallick , Gauri Joshi This is my paper
classification cs.AI
keywords LLM query routingtime-to-first-tokenlatency-aware routingaccuracy-cost tradeoffcontinuous batchingserving framework simulationqueueing analysisdynamic workloads
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing LLM query routers—which choose a model per query to balance response quality and price—fail because they ignore the latency a query will actually experience, which depends on the current prefill and decode workload at each model instance. It proposes to close that gap with a lightweight estimator, the Serving Framework Simulation (SFS), that simulates continuous batching exactly as the serving framework would process token batches, predicting time-to-first-token (TTFT) for an incoming query at each instance. The router then maximizes accuracy–cost utility among instances predicted to meet the query's TTFT target. Experiments on three transformer models and four task types report that this joint optimization yields up to 40% higher accuracy–cost utility than load-balancing baselines at matched latencies, with a sub-millisecond estimation overhead. If correct, latency-aware routing would let service operators meet per-query response-time targets without a separate load-balancing layer erasing the quality–cost gains.

Core claim

The paper claims that time-to-first-token (TTFT) in an LLM serving pool is predictable enough to route on, and that routing on it changes the accuracy–cost outcome. SFS snapshots each instance's resident requests, remaining prefill tokens, and estimated decode lengths, then deterministically simulates continuous-batching token batches until the new query emits its first decode token. Batch times come from a per-instance calibrated linear model of batch composition (token counts, context-scaled decode attention, quadratic prefill attention). Summing these batch times gives the TTFT estimate used in routing. The paper reports under 5% mean absolute percentage error on TTFT at ~0.14 ms overhead

What carries the argument

The Serving Framework Simulation (SFS) estimator, together with the token-batch processing-time model it relies on. SFS simulates the serving framework's batching and scheduling policy on a workload snapshot—remaining prefill tokens and estimated decode lengths of resident requests plus the new query—and sums predicted processing times of consecutive token batches up to the new query's first decode token (TTFT). Batch times come from a per-instance calibrated linear regression whose terms capture dense token computation, decode-side attention and KV-cache reads (linear in context length), and prefill-side attention (linear in prefill chunk times context, plus quadratic within the chunk). For

Load-bearing premise

The load-bearing premise is that the time to process a token batch on a GPU is well approximated by a per-instance calibrated linear function of token counts, context lengths, and prefill/decode composition (Eq. 9); if memory-bandwidth contention, tensor-parallel communication, scheduler nondeterminism, or preemption materially break that linear model, SFS's TTFT estimates—and therefore the set of instances deemed to meet each query's latency target—will be wrong.

What would settle it

Run the SFS router on a held-out workload that mixes many long-context decode sequences with concurrent prefills on a tensor-parallel multi-GPU instance, and compare predicted versus measured TTFT; if the mean absolute percentage error departs substantially from the reported ~5%, or if a router with oracle (measured) TTFT outperforms SFS by a large margin, the claim that lightweight workload simulation drives the 40% utility gain would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If TTFT can be estimated accurately at routing time, per-query latency SLOs no longer need to be delegated to a separate load-balancing layer; the same routing decision can carry quality, cost, and latency commitments.
  • Routers can exploit the heterogeneous accuracy–cost structure of a model pool under load, sending latency-sensitive or long-prompt queries to faster instances while still using larger models when their quality gain justifies the cost.
  • The Lagrangian variant of the objective gives operators a single dial δ to trade average latency against accuracy–cost utility, so latency targets and quality/cost can be tuned without re-training the router.
  • The average-case LPS estimator provides a planning tool: from arrival rate, service capacity, and parallelism cap, operators can predict queueing delay before the router has live workload snapshots.
  • Because the simulation only needs to continue to the first decode token, predictions are robust to moderate errors in decode-length estimates, keeping the estimator light enough for the routing critical path.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same SFS simulation could be extended to end-to-end response latency (not just TTFT) by continuing the simulation to sequence completion, making the router relevant for batch and long-generation workloads where TTFT is not the binding SLO.
  • The accuracy of routing hinges on the decode-length estimator as much as on the batch-time model: if resident decode-length estimates are systematically biased, the prefill/decode interference picture degrades; a testable extension is to feed the router ground-truth future decode lengths and measure how much of the 40% gain is attributable to the simulator versus the predictor.
  • The LPS formula suggests a capacity-planning heuristic: for a given arrival burstiness, the parallelism cap k (controlled by serving configuration) sets how quickly waiting time grows with load; operators could size k using the router's own calibrated metrics rather than throughput averages.
  • The paper assumes a single router; if multiple independent routers send queries to shared instances, each router's snapshot-based simulation would need to account for other routers' decisions—an interaction the paper lists as future work but which the SFS estimator's snapshot mechanism could partially address.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SFS, a serving-framework simulation based time-to-first-token (TTFT) estimator for LLM query routing. SFS uses per-instance workload snapshots, predicted decode lengths, and a calibrated token-batch processing-time model (Eq. 9) to simulate token batches until the arriving query's first decode token, producing a TTFT estimate. This estimate is used in a routing objective (Eq. 3) that maximizes accuracy-cost utility subject to per-query TTFT constraints, with a Lagrangian variant (Eq. 5) for exploring the utility-latency tradeoff. Experiments on Qwen3-0.6B/8B/32B with four task types compare SFS-based routing against Round Robin, Shortest Queue, and Latency-Agnostic baselines, reporting up to 40% improvement in OnTimeUtility at matched latencies, 33% higher AUC, and sub-millisecond estimator overhead.

Significance. If validated, this is a practical contribution to LLM serving: it connects accuracy-cost routing with system-level latency management, and the simulator-based estimator is lightweight and interpretable. The paper includes a public code repository and measures estimator overhead, which supports reproducibility. However, the empirical claims currently rest on an experimental protocol that does not clearly separate calibration from evaluation, omits comparisons with the closest prior latency-aware router and with the paper's own simpler throughput estimator, and reports no error bars. These gaps are load-bearing for the central claim that SFS-based joint optimization improves routing utility.

major comments (4)
  1. [§5 / §4.1] The experimental protocol uses the same 2.5K-query-per-task sample to calibrate the β coefficients in Eq. (9), train the LightGBM accuracy and output-length predictors, and estimate prefill throughput (§5). No held-out split is described, so the reported TTFT MAPE (Figs. 2–3) and the OnTimeUtility gains in Figs. 5–7 may be in-sample. This matters because the feasible set J(i) in Eq. (3) is built from SFS TTFT estimates; optimistic estimates can inflate the router's advantage. Please specify and use a clear train/test split, report held-out MAPE for Eq. (9), and ideally evaluate routing on queries not used in any calibration or training step.
  2. [§5 Baselines] The paper identifies [27] as the closest prior latency-aware router and introduces a simpler throughput-based estimator in Eq. (6), but neither is included in the routing comparisons of Figs. 5–7. Without [27], the claim of improved latency-aware routing over prior work is not directly supported; without Eq. (6) in the routing loop, the experiments do not show that the simulation complexity of SFS is necessary for the observed gains. Please add these two comparisons, or justify their omission explicitly.
  3. [§5, Figs. 5–7] The central quantitative claims—33% AUC gain, 46% improvement at 5 qps, and 40% higher utility than Shortest Queue—are based on curves generated from stochastic arrival processes, but no error bars, confidence intervals, or number of seeds are reported. Because OnTimeUtility in Eq. (4) averages over random arrivals, routing decisions, and realized latencies, the observed differences might not be statistically significant. Please report means with standard deviations or confidence intervals over multiple independent runs and, where possible, significance tests.
  4. [§4.1, Eq. (9)] The token-batch processing-time model assumes an additive linear form and omits effects such as memory-bandwidth contention, tensor-parallel communication, scheduler nondeterminism, and preemption, although Section F states that preemption behavior is simulated. Validation is only shown on the calibration data (Fig. 3) with approximately 4% MAPE and no held-out split. Since systematic bias in TTFT estimates—not just noise—can change the feasible set J(i) and hence routing decisions, please validate Eq. (9) on held-out batch compositions and under the routing workloads used in Figs. 5–7.
minor comments (6)
  1. [Abstract] There is a typo: “intoa latency-aware router” should be “into a latency-aware router.”
  2. [Fig. 2] The left and right panels would be easier to read if axes were labeled consistently and the units (ms) appeared on both panels; the caption mentions outliers but does not describe how many points they represent.
  3. [Fig. 6] The caption text contains garbled characters and is not typeset correctly; it should read “Utilities plotted against TTFT latencies by varying δ in Eq. (5).”
  4. [§4.2 / Appendix E] The LPS average-case estimator in Eq. (10) is derived under Poisson arrivals and memoryless service, but Figure 10 shows highly variable output lengths. The appendix acknowledges this limitation, but the main text should state more prominently that Eq. (10) is only an approximation for heavy-tailed workloads and should be validated in routing experiments.
  5. [Fig. 8] The overhead estimate reports a mean of 0.139 ms but no distribution or percentile information; a box plot or tail statistic would help support the “sub-millisecond” claim.
  6. [§5] The paper says “2.5K queries from each task” are used for calibration/training, but it is unclear whether the routing simulation consumes these same queries as arriving requests. Please state explicitly how the arrival workload relates to the calibration/training set.

Circularity Check

0 steps flagged

No construction-level circularity: SFS is an empirically calibrated simulator; reported gains are not forced by the equations. Minor self-citations and an in-sample calibration caveat do not rise to circularity.

full rationale

The derivation chain is self-contained. SFS TTFT (Eq. 8) is a sum of predicted token-batch processing times (Eq. 9), with coefficients calibrated from observed batch times; this is empirical parameter fitting, not a definitional reduction in which a fitted parameter is renamed as a prediction. The routing objective (Eq. 3) and OnTimeUtility (Eq. 4) use realized TTFT and realized utility, so the reported 33-46% improvements over load-balancing baselines are not equal to any fitted quantity by construction. The average-case LPS waiting time (Eq. 10) is derived in Appendix E from an M/M/1-style argument, not imported from a self-citation. Self-citations ([4], [37], [38]) appear only in related-work summaries and do not carry load-bearing premises. The main caveat is that the same 2.5K-query set is used to calibrate β, train accuracy/output-length predictors, and run the simulated arrivals, with no held-out split described; this is a potential overfitting/validation weakness, not circularity, because no equation reduces to its own input.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The central routing claim rests on fitted per-instance estimators (batch-time coefficients, throughput, decode time, accuracy/output-length predictors), standard queueing assumptions for the average-case formula, and the domain assumption that token-batch composition faithfully determines GPU processing time. No new physical entities are introduced.

free parameters (6)
  • β_{0..3,j} token-batch time coefficients = not reported
    Eq. (9); calibrated per model instance from observed token-batch processing times; core of SFS TTFT prediction.
  • θ^pre_j prefill throughput = not reported
    Used in throughput estimator Eq. (6) and average-case Eq. (11); estimated from calibration queries.
  • Tbar^dec_j average decode batch time = not reported
    Used in Eq. (6) and Eq. (11); estimated per instance.
  • LightGBM accuracy predictor = not reported
    Section B.4; trained on LLM-as-a-judge scores; supplies accuracy estimates for utility Eq. (2).
  • LightGBM output-length predictor = not reported
    Section B.4; trained on completion-token counts; supplies predicted decode lengths used for cost and decode workload.
  • LPS parameters μ_j, k, α_j = not reported
    Appendix E; service capacity from peak throughput, parallelism k from observed average, arrival rate from workload; used only in the average-case estimator.
axioms (6)
  • standard math PASTA and M/M/1 stationarity
    Used in Appendix E to derive E[W_{i,j}] = (α/μ)^k / (μ - α); assumes Poisson arrivals see time averages.
  • domain assumption Memoryless service requirement
    Appendix E assumption 2: per-query service requirement is exponential/geometric in time average; known to be an approximation for LLM generation lengths.
  • ad hoc to paper Token-batch processing time model (Eq. 9) is sufficient
    The paper assumes batch processing time is a linear function of token counts, context lengths, and prefill/decode composition with per-instance fitted coefficients, ignoring kernel-level memory bandwidth and communication effects.
  • domain assumption Deterministic simulation of batching policy
    Section 4.1: SFS replays vLLM-style continuous batching from workload snapshots; scheduler nondeterminism, preemption, and memory allocation are assumed captured.
  • domain assumption LLM-as-a-judge scores are a valid quality measure
    Section B.4/D: accuracy estimator trained on Gemini judge scores; utility uses these scores as ground truth.
  • domain assumption Synthetic TTFT targets reflect realistic SLOs
    Section B.2: tau_i generated from seeded uniform plus linear function of prompt length.

pith-pipeline@v1.3.0-alltime-deepseek · 20612 in / 12410 out tokens · 108060 ms · 2026-08-02T14:02:20.692149+00:00 · methodology

0 comments
read the original abstract

Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost. However, current query routers are largely latency-agnostic and do not consider the generation latency experienced by queries at model instances. In practice, latency is often controlled by load-balancing policies such as round-robin or join-the-shortest-queue, which do not account for model accuracy or inference cost. Incorporating query latency into routing is challenging as it depends not only on the query's prompt length, but also on the current prefill and decode workload at the model instance and the scheduling and batching policy of the serving framework. We design a lightweight latency estimator that simulates autoregressive token batch processing in the serving framework and estimates the time-to-first-token (TTFT) of queries. We incorporate this latency estimator into a latency-aware router that jointly optimizes latency, accuracy, and cost when assigning queries to model instances. Our experimental results indicate that this joint optimization yields up to 40% improvement in accuracy--cost utility while maintaining the same latencies as standard load-balancing approaches.

Figures

Figures reproduced from arXiv: 2607.18253 by Akaash R. Parthasarathy, Ankur Mallick, Gauri Joshi, Shivam Patel.

Figure 1
Figure 1. Figure 1: TTFT (time-to-first-token) for autoregressive generation. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: TTFT estimation using the prefill-throughput estimator (left) and the proposed Serving Framework Simulation (SFS) estima￾tor (right) for Qwen3-0.6B (hosted on one H100 GPU). The prefill throughput estimator does not consider decode interference, while SFS simulates autoregressive generation to calculate latencies. where the estimated waiting time Wci,j (t) is the sum of the remaining prefill tokens of resi… view at source ↗
Figure 3
Figure 3. Figure 3: Token-batch processing time estimator. The estimator in eq. (9) closely matches observed batch times, with ≈ 4% mean absolute percentage er￾ror across token-batch compositions (for Qwen3-0.6B hosted on one H100 GPU). The intercept term (β0,j ) captures fixed per-batch overheads. The first workload term (with coeff. β1,j ) captures token-wise dense-layer computation, which scales linearly with the number of… view at source ↗
Figure 4
Figure 4. Figure 4: Observed TTFT for sequence generation on Qwen3-0.6B (on single H100 GPU) compared with theo￾retical TTFT under the Limited Processor Sharing (LPS) model where k queries can be served in parallel. We estimate the service capacity µj (in queries per second) of model instance j from its peak generation output throughput. Consider the average arrival rate αj queries per second, with ρj = αj/µj < 1. Under Poiss… view at source ↗
Figure 5
Figure 5. Figure 5: OnTimeUtility across varying arrival rate (in qps). Latency-aware routing (SFS) balances utility and latency to maintain high OnTimeUtility, demon￾strating 33% higher AUC. Overview of results. We examine latency-aware routing from two complementary perspectives. First, we vary the offered query load under the latency-constrained routing objective in eq. (3). Second, we vary the latency penalty parameter δ … view at source ↗
Figure 6
Figure 6. Figure 6: Utilities plotted against TTFT latencies by varying δ in eq. (5). Latency-aware routing provides a controllable utility–latency tradeoff. Varying latency emphasis. To analyze the trade-off be￾tween utility and latency, we vary δ values in eq. (5) and obtain average utility and latency values for queries (following the Poisson arrival process with rate = 8 qps) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: OnTimeUtility for routers under query arrivals from a Markov-modulated Poisson process with high-to-low arrival-rate ratio r= 3, 6. Our proposed approach improves OnTimeUtility across query arrival processes. 0.00 0.05 0.10 0.15 0.20 0.25 Simulation latency (ms) 0 500 1000 1500 2000 Number of Queries Mean = 0.139 ms [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Query token length distributions across the tasks in our evaluations. Alpaca [46], GovReport-Summarization [21], HotpotQA [49], and WritingPrompts [13] tasks present a diverse range of input lengths (∼ 10 to 104 tokens), simulating real workloads and a comprehensive evaluation of our routing scheme and latency estimator. 0 200 400 600 qwen3-0.6b alpaca mean=199 0 200 400 600 govreport-summ. mean=1051 0 50 … view at source ↗
Figure 10
Figure 10. Figure 10: Response token lengths for examined Qwen3-0.6B/8B/32B language models [42] across tasks. Response lengths vary based on the nature of the task (creative generation, summarization, etc.), covering a broad range of ∼ 102 to 103 tokens. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Limited Processor Sharing Scheme [52]. A new job directly begins serving by the processor if fewer than k jobs were already present, otherwise queued (in FCFS manner) until the number of preceding jobs at the server drops below k. This appendix derives the queueing-delay estimator used in Sec￾tion 4.2. We use a limited processor sharing (LPS) abstraction from queueing theory [52], in which at most k jobs … view at source ↗
Figure 12
Figure 12. Figure 12: OnTimeUtility with varying accuracy–cost balance (λ) for our approach and latency agnostic routing. Consistent improvement across varying accuracy–cost bal￾ances indicates the generalizability of our approach. 3 4 5 6 7 8 9 Queries Per Second 50 60 70 80 90 100 TTFT SLO Attainment (%) SFS (Ours) Shortest Queue Latency-Agnostic Round Robin [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: Fraction of queries routed to different model instances across approaches. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 6 linked inside Pith

  1. [1]

    Gulavani, Ramachan- dran Ramjee, and Alexey Tumanov

    Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S. Gulavani, Ramachan- dran Ramjee, and Alexey Tumanov. Vidur: A large-scale simulation framework for llm inference. InMLSys, May 2024

    2024

  2. [2]

    Gulavani, Alexey Tumanov, and Ramachandran Ramjee

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in llm inference with sarathi-serve. InProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, OSDI’24, 9 USA, 2024. USENIX Association. ISBN 978-1-939133-40-3

    2024

  3. [3]

    Alibaba cloud model studio: Model invocation pricing, 2026

    Alibaba Cloud. Alibaba cloud model studio: Model invocation pricing, 2026. URLhttps://www.alibabacloud. com/help/en/model-studio/model-pricing. Alibaba Cloud Documentation Center, accessed Apr. 17, 2026

    2026

  4. [4]

    Federate the router: Learning language model routers with sparse and decentralized evaluations, 2026

    Baris Askin, Shivam Patel, Anupam Nayak, Andrea Vigano, Jiin Woo, Gauri Joshi, and Carlee Joe-Wong. Federate the router: Learning language model routers with sparse and decentralized evaluations, 2026. URL https://arxiv.org/abs/2601.22318

    arXiv 2026

  5. [5]

    Optimal scheduling algorithms for llm inference: Theory and practice.Proc

    Agrim Bari, Parikshit Hegde, and Gustavo de Veciana. Optimal scheduling algorithms for llm inference: Theory and practice.Proc. ACM Meas. Anal. Comput. Syst., 9(3), December 2025. doi: 10.1145/3771574. URL https://doi.org/10.1145/3771574

    doi:10.1145/3771574 2025

  6. [6]

    Boerner, Stephen Deems, Thomas R

    Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. Access: Advancing innovation: Nsf’s advanced cyberinfrastructure coordination ecosystem: Services & support. InPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good, PEARC ’23, page 173–176, New York, NY, USA, 2023. Association for ...

    arXiv 2023

  7. [7]

    Learning compact representations of llm abilities via item response theory, 2025

    Jianhao Chen, Chenxu Wang, Gengrui Zhang, Peng Ye, Lei Bai, Wei Hu, Yuzhong Qu, and Shuyue Hu. Learning compact representations of llm abilities via item response theory, 2025. URLhttps://arxiv.org/abs/2510. 00844

    2025

  8. [8]

    Frugalgpt: How to use large language models while reducing cost and improving performance.Transactions on Machine Learning Research, 2024

    Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance.Transactions on Machine Learning Research, 2024

    2024

  9. [9]

    Block: Balancing load in llm serving with context, knowledge and predictive scheduling, 2025

    Wei Da and Evangelia Kalyvianaki. Block: Balancing load in llm serving with context, knowledge and predictive scheduling, 2025. URLhttps://arxiv.org/abs/2508.03611

    Pith/arXiv arXiv 2025

  10. [10]

    A unified approach to routing and cascading for LLMs

    Jasper Dekoninck, Maximilian Baader, and Martin Vechev. A unified approach to routing and cascading for LLMs. InForty-second International Conference on Machine Learning, 2025

    2025

  11. [11]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Laksh- manan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  12. [12]

    Dujian Ding, Ankur Mallick, Shaokun Zhang, Chi Wang, Daniel Madrigal, Mirian Del Carmen Hipolito Garcia, Menglin Xia, Laks V. S. Lakshmanan, Qingyun Wu, and Victor Rühle. BEST-route: Adaptive LLM routing with test-time optimal compute. InForty-second International Conference on Machine Learning, 2025

    2025

  13. [13]

    Hierarchical neural story generation

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao, editors,Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1082. UR...

    doi:10.18653/v1/p18-1082 2018

  14. [14]

    The markov-modulated poisson process (mmpp) cookbook

    Wolfgang Fischer and Kathleen Meier-Hellstern. The markov-modulated poisson process (mmpp) cookbook. Performance Evaluation, 18(2):149–171, 1993. ISSN 0166-5316. doi: https://doi.org/10.1016/0166-5316(93) 90035-S. URLhttps://www.sciencedirect.com/science/article/pii/016653169390035S

    arXiv 1993

  15. [15]

    Efficient LLM scheduling by learning to rank

    Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. Efficient LLM scheduling by learning to rank. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=wlLjYl0Gi6

    2024

  16. [16]

    Gemini: A family of highly capable multimodal models, 2025

    Gemini Team. Gemini: A family of highly capable multimodal models, 2025. URLhttps://arxiv.org/abs/ 2312.11805

    Pith/arXiv arXiv 2025

  17. [17]

    Past-future scheduler for llm serving under sla guarantees

    Ruihao Gong, Shihao Bai, Siyu Wu, Yunqian Fan, Zaijun Wang, Xiuhong Li, Hailong Yang, and Xianglong Liu. Past-future scheduler for llm serving under sla guarantees. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’25, page 798–813, New York, NY, USA, 2025. As...

    arXiv 2025

  18. [18]

    Li, and Richard Socher

    Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. InInternational Conference on Learning Representations, 2018. URLhttps://openreview. net/forum?id=B1l8BtlCb. 10

    2018

  19. [19]

    Routerbench: A benchmark for multi-LLM routing system

    Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-LLM routing system. InAgentic Markets Workshop at ICML 2024, 2024. URLhttps://openreview.net/forum?id=IVXmV8Uxwh

    2024

  20. [20]

    Slo-aware scheduling for large language model inferences, 2025

    Jinqi Huang, Yi Xiong, Xuebing Yu, Wenjie Huang, Entong Li, Li Zeng, and Xin Chen. Slo-aware scheduling for large language model inferences, 2025. URLhttps://arxiv.org/abs/2504.14966

    Pith/arXiv arXiv 2025

  21. [21]

    Efficient attentions for long document summarization

    Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors,Proceedings of the 2021 Conference of the North American Chapter of th...

    2021

  22. [22]

    Predicting llm inference latency: A roofline-driven ml method

    Saki Imai, Rina Nakazawa, Marcelo Amaral, Sunyanan Choochotkaew, and Tatsuhiro Chiba. Predicting llm inference latency: A roofline-driven ml method. InML for Systems Workshop, Annual Conference on Neural Information Processing Systems, 2024

    2024

  23. [24]

    Universal model routing for efficient LLM inference

    Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Congchao Wang, Zifeng Wang, Alec Go, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. Universal model routing for efficient LLM inference. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  24. [25]

    Lightgbm: A highly efficient gradient boosting decision tree

    Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. ...

    2017

  25. [26]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY, USA, 2023. Association for Computing Machin...

    arXiv 2023

  26. [27]

    Faster, cheaper, just as good: Cost- and latency constrained routing for llms

    Javid Lakha, Minlan Yu, and Rana Shahout. Faster, cheaper, just as good: Cost- and latency constrained routing for llms. InSparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference, 2025. URLhttps://openreview.net/forum?id=pZFJLsIY2m

    2025

  27. [28]

    Deterministic non-autoregressive neural sequence modeling by iterative refinement

    Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, Brussels, Belgium, October-November 2018. As...

    doi:10.18653/v1/d18-1149 2018

  28. [29]

    Fromgenerationtojudgment: Oppor- tunities and challenges of LLM-as-a-judge

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, YuxuanJiang, CanyuChen, TianhaoWu, KaiShu, LuCheng, andHuanLiu. Fromgenerationtojudgment: Oppor- tunities and challenges of LLM-as-a-judge. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, andVioletPeng, editors,Proceedings of the 20...

    2025

  29. [30]

    Diffusion-LM improves controllable text generation

    Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022. URLhttps://openreview.net/forum?id= 3s9IrEsjLyk. 11

    2022

  30. [31]

    Throughput-optimal scheduling algorithms for llm inference and ai agents,

    Yueying Li, Jim Dai, and Tianyi Peng. Throughput-optimal scheduling algorithms for llm inference and ai agents,

  31. [32]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: //aclanthology.org/W04-1013/

    2004

  32. [33]

    Queueing, predictions, and llms: Challenges and open problems, 2025

    Michael Mitzenmacher and Rana Shahout. Queueing, predictions, and llms: Challenges and open problems, 2025. URLhttps://arxiv.org/abs/2503.07545

    Pith/arXiv arXiv 2025

  33. [34]

    Deferred prefill for throughput maximization in llm inference

    Moonmoon Mohanty, Gautham Bolar, Preetam Patil, Umamaheswari Devi, Felix George, Pratibha Moogi, and Parimal Parag. Deferred prefill for throughput maximization in llm inference. InProceedings of the 5th Workshop on Machine Learning and Systems, EuroMLSys ’25, page 100–106, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400715389. ...

    arXiv 2025

  34. [35]

    Nvidia h100 pcie gpu — product brief

    NVIDIA Corporation. Nvidia h100 pcie gpu — product brief. Technical Report PB-11133-001, NVIDIA, September 2022. URL https://www.nvidia.com/content/dam/en-zz/Solutions/gtcs22/data-center/ h100/PB-11133-001_v01.pdf

    2022

  35. [36]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  36. [37]

    Proxrouter: Proximity-weighted llm query routing for improved robustness to outliers, 2025

    Shivam Patel, Neharika Jali, Ankur Mallick, and Gauri Joshi. Proxrouter: Proximity-weighted llm query routing for improved robustness to outliers, 2025. URLhttps://arxiv.org/abs/2510.09852

    arXiv 2025

  37. [38]

    Locus: Low-dimensional model embeddings for efficient model exploration, comparison, and selection, 2026

    Shivam Patel, William Cocke, and Gauri Joshi. Locus: Low-dimensional model embeddings for efficient model exploration, comparison, and selection, 2026. URLhttps://arxiv.org/abs/2601.21082

    arXiv 2026

  38. [39]

    Queue management for slo-oriented large language model serving

    Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu, Christian Pinto, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravishankar Iyer. Queue management for slo-oriented large language model serving. In Proceedings of the 2024 ACM Symposium on Cloud Computing, SoCC ’24, page 18–35, New York, NY, USA,

    2024

  39. [40]

    Efficiently scaling transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In D. Song, M. Carbin, and T. Chen, editors,Proceedings of Machine Learning and Systems, volume 5, pages 606–624. Curan, 2023. URL https://proceedings.mlsys.org/paper_files/pa...

    2023

  40. [41]

    Kalbarczyk, Tamer Başar, and Ravishankar K

    Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T. Kalbarczyk, Tamer Başar, and Ravishankar K. Iyer. Efficient interactive llm serving with proxy model-based sequence length prediction. InThe 5th International Workshop on Cloud Intelligence / AIOps at ASPLOS 2024, volume 5, pages 1–7, San Diego, CA, U...

    2024

  41. [42]

    Qwen3 technical report, 2025

    Qwen. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  42. [43]

    Don’t stop me now: Embedding based scheduling for llms

    Rana Shahout, eran malach, Chunwei Liu, Weifan Jiang, Minlan Yu, and Michael Mitzenmacher. Don’t stop me now: Embedding based scheduling for llms. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=7JhGdZvW4T

    2025

  43. [44]

    IRT-router: Effective and interpretable multi-LLM routing via item response theory

    Wei Song, Zhenya Huang, Cheng Cheng, Weibo Gao, Bihan Xu, GuanHao Zhao, Fei Wang, and Runze Wu. IRT-router: Effective and interpretable multi-LLM routing via item response theory. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (...

  44. [45]

    Llumnix: dynamic scheduling for large language model serving

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: dynamic scheduling for large language model serving. InProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, OSDI’24, USA, 2024. USENIX Association. ISBN 978-1-939133-40-3. 12

    2024

  45. [46]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford_alpaca, 2023

    2023

  46. [47]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.761. URLhttps://aclanthology.org/2025.acl-long.761/

    doi:10.18653/v1/2025.acl-long.761 2025

  47. [48]

    Ronald W. Wolff. Poisson arrivals see time averages.Operations Research, 30(2):223–231, 1982. ISSN 0030364X, 15265463. URLhttp://www.jstor.org/stable/170165

    1982

  48. [49]

    Cohen, Ruslan Salakhutdinov, and Christo- pher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2018

    2018

  49. [50]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URLhttp...

    2017

  50. [51]

    BARTScore: Evaluating generated text as text generation

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. BARTScore: Evaluating generated text as text generation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, 2021. URLhttps://openreview.net/forum?id=5Ya8PbvpZ9

    2021

  51. [52]

    PhD thesis, Georgia Institute of Technology, August 2009

    Jiheng Zhang.Limited Processor Sharing Queues and Multi-Server Queues. PhD thesis, Georgia Institute of Technology, August 2009. URLhttps://people.orie.cornell.edu/jdai/thesis/Jiheng_Zhang_Thesis. pdf

    2009

  52. [53]

    Orca: A distributed serving system for Transformer-Based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-1. URLhttps://www.usenix.org/conference...

    2022

  53. [56]

    scores". The

    Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. EmbedLLM: Learning compact representations of large language models. InThe Thirteenth International Conference on Learning Representations, 2025. 13 A Related Works A broad range of research directions tackle efficient response generation from varying perspectives. L...

    2025

  54. [57]

    Alternatively, generation work for queries follows the memoryless distribution [34]

    Memoryless service requirement:each query has an exponential service requirement, with mean as1/µj in the time average sense. Alternatively, generation work for queries follows the memoryless distribution [34]

  55. [58]

    Limited parallelism:at most k queries can be served simultaneously, and the remaining queries wait in a FIFO queue

  56. [59]

    Saturated aggregate capacity:whenever the active decode batch is nonempty, the instance provides approxi- mately constant aggregate service rateµ, while this rate is shared across active sequences

  57. [60]

    State process and stationary distribution.Queries arrive according to a Poisson process of rateαj, so whenever a new query arrives, the state|Rj(t)| increases by one

    Stability condition:incoming rate of queries does not exceed the service capacity, i.e., the utilizationρj = αj/µj <1. State process and stationary distribution.Queries arrive according to a Poisson process of rateαj, so whenever a new query arrives, the state|Rj(t)| increases by one. For|Rj(t)| resident queries in the system, with|Aj(t)| active queries, ...

  58. [2024]

    ISBN 9798400712869

    Association for Computing Machinery. ISBN 9798400712869. doi: 10.1145/3698038.3698523. URL https://doi.org/10.1145/3698038.3698523

    arXiv

  59. [2025]

    URLhttps://arxiv.org/abs/2504.07347

    Pith/arXiv arXiv