Pith. sign in

REVIEW 2 major objections 4 minor 36 references

An online linear program with dual shadow prices routes LLM decode requests better than standard load-balancing heuristics across competing service objectives.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 22:48 UTC pith:XAUXTUJ5

load-bearing objection Solid transplant of classical bid-price control into continuous-batching LLM decode routing; large Vidur gains, but the multi-objective claim rests on unvalidated reward surrogates and pure simulation. the 2 major comments →

arxiv 2607.03948 v1 pith:XAUXTUJ5 submitted 2026-07-04 cs.AI cs.LGmath.OC

Online Linear Programming for Multi-Objective Routing in LLM Serving

Zixi Chen , Yinyu Ye , Zijie Zhou This is my paper
classification cs.AI cs.LGmath.OC
keywords online linear programmingLLM servingrequest routingbid-price controlmulti-objective SLOsKV-cache constraintscontinuous batchingshadow prices
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM serving systems must decide, as each request finishes prefill, which decode worker should take it next, subject to batch-size and KV-cache limits that bind for the whole remaining generation. Production stacks typically use simple rules such as join-the-shortest-queue or power-of-two choices; those rules ignore explicit service-level objectives and give operators little control over the latency-throughput trade-off. This paper recasts the problem as an online linear program whose objective is a weighted sum of per-request rewards that stand in for throughput, average and tail end-to-end latency, and time-to-first-token. Dual variables of that program become shadow prices on future batch slots and memory; a request is admitted only when its reward exceeds the imputed cost of the resources it will occupy. Warm-started projected gradient steps keep the prices up to date in a few milliseconds. In the Vidur simulator the resulting bid-price router substantially improves average and tail latency metrics relative to the usual heuristics while remaining competitive on throughput, and operators can move along the Pareto surface simply by changing the objective weights.

Core claim

Formulating multi-worker LLM decode routing as an online linear program with time-coupled batch and KV-cache packing constraints, then deciding admissions by comparing each request’s SLO-weighted reward against the dual shadow prices of the resources it consumes, yields large gains over standard load-balancing heuristics across multiple latency and throughput regimes, while giving operators an explicit dial for trading those objectives against one another.

What carries the argument

The bid-price control rule: at each step the dual prices of residual batch and memory capacity are updated by a short warm-started projected sub-gradient pass; a waiting request is admitted to a device only when its multi-objective reward exceeds the inner product of those prices with the resource profile it will occupy for its predicted remaining length.

Load-bearing premise

The claim rests on the idea that the hand-crafted per-request reward terms (especially the indicator rewards used for tails and the “saved-steps” rewriting of average latency) faithfully turn the desired aggregate statistical SLOs into online decisions whose greedy ranking under dual prices stays near-optimal.

What would settle it

Replace the proposed reward decomposition with a pure shortest-queue or random policy inside the same dual-price loop, or ablate the tail-indicator terms, and check whether the reported 25–45 % relative gains on average and P95/P99 end-to-end latency and TTFT disappear in the Vidur experiments under the same weight settings.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Operators can re-target the same router for different products simply by changing a handful of non-negative weights rather than rewriting heuristics.
  • Shadow prices themselves become live diagnostics that identify whether batch slots or KV-cache memory is the binding bottleneck under a given workload.
  • When the system is lightly loaded or decode lengths are short, near-zero shadow prices correctly signal that simple heuristics already suffice.
  • The same packing formulation extends, with only kernel changes, to heterogeneous GPU fleets and to mixed prefill-decode workers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same dual-price skeleton could be reused for other online packing problems inside serving stacks, such as speculative-decoding token budgets or multi-model co-location on a shared memory pool.
  • If production decode-length predictors improve, the residual gap between the online dual solution and the offline clairvoyant optimum should shrink, giving a clear metric for how much prediction accuracy is still worth buying.
  • Admission control under overload becomes a natural by-product: any request whose reward falls below its shadow-price cost can be delayed or rejected without extra machinery.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper formulates online routing of LLM decode requests to parallel workers as a multi-objective online LP under time-coupled batch-size and KV-cache constraints. It constructs per-request rewards that linearly combine throughput, QPS, average EEL/TTFT (via a non-negative “saved-steps” rewriting), and tail EEL/TTFT (via hard indicators 1{latency < t′}), then applies a bid-price policy that admits a request only when its reward exceeds the dual shadow-price cost of the resources it will occupy. Dual prices are tracked by warm-started projected mini-batch subgradient steps on an SAA dual objective so that decisions stay within millisecond budgets. The router is integrated into the Vidur simulator and shown to produce large relative gains (often 25–45 % on average/P95/P99 EEL and TTFT) over Round-Robin, LOR, Random and Power-of-2 across weight regimes, perfect/noisy decode-length prediction, stationary and rate-shift workloads, and both real and synthetic traces.

Significance. If the empirical gains survive more rigorous validation of the reward surrogates and real-system deployment, the work supplies a clean, weight-tunable alternative to the heuristic routers that dominate production LLM stacks, together with diagnostic shadow prices that identify batch versus memory bottlenecks. Strengths that raise the contribution above pure engineering are the explicit LP duality derivation, the predictable-runtime first-order dual tracker, the public Vidur integration, and the consistent multi-regime simulation results (including non-stationary arrivals). The paper therefore offers a useful bridge from classical online resource allocation to LLM serving systems.

major comments (2)
  1. [§3.1, eqs. (2)–(3)] §3.1 (reward construction, eqs. (2)–(3) and the subsequent indicator definitions): average EEL/TTFT are rewritten as non-negative “saved steps” and tails become hard indicators 1{k-tj < t′} (resp. for EEL). These are surrogates; the paper never shows that greedy ranking of the resulting rj,g,k under the dual prices of (6)–(8) approximates the true multi-objective program that uses sample quantiles and true averages. Figure 13 only demonstrates that larger ζ2 raises the empirical satisfaction rate for a fixed threshold; there is no oracle comparison that optimizes true P99, nor an ablation that replaces the indicators by other quantile surrogates. Because the 25–45 % gains in Tables 1–2 and Figures 2–3 rest on these rewards, the multi-objective claim remains incompletely supported.
  2. [§4, Tables 1–2] §4 and Appendix C: all quantitative claims are obtained inside the Vidur simulator on four A100s. While the authors correctly flag real-system integration as future work, the central assertion that the method “outperforms others based on heuristics” for LLM serving is therefore still provisional; at minimum a sensitivity study to the simulator’s constant-iteration-time assumption under PD disaggregation (or a single real-engine micro-benchmark) is needed before the magnitude of the reported improvements can be taken as established.
minor comments (4)
  1. [Abstract] Abstract closing sentence (“A big picture from our result: a science-based approach …”) is informal and should be removed or rewritten in standard scientific prose.
  2. [§4.3, Figures 2–3] Figures 2–3 and the many appendix radar plots report only normalized values; absolute latency/throughput numbers (or at least a table of raw means) would make the practical magnitude of the gains clearer.
  3. [§1] Related-work discussion of recent workload-aware and multi-LLM routers (Jain et al., Wu & Silwal, Mei et al.) is brief; a short paragraph clarifying why those methods are not used as additional baselines would strengthen the experimental design.
  4. [§3.2] Notation for residual capacities b(k) and the scarcity vector dk is introduced somewhat abruptly; a short clarifying sentence after (6) would help.

Circularity Check

0 steps flagged

No significant circularity: classical online-LP bid-price machinery applied to a new domain, with empirical claims checked against external baselines and traces.

full rationale

The paper’s derivation chain is an application of standard online linear programming and bid-price control (dual shadow prices, SAA-style dual objective, projected subgradient updates) to multi-worker LLM decode routing. The dual formulation, reduced dual, and bid-price margin are obtained from ordinary LP duality of the per-step packing LP; they are not defined in terms of the reported latency/throughput improvements. The multi-objective per-request rewards (throughput tokens, “saved-steps” rewriting of average EEL/TTFT, indicator rewards for tail thresholds) are explicit modeling surrogates chosen by the authors; they are inputs to the online LP, not quantities that the dual prices or the simulator results are forced to reproduce by construction. Dual prices are tracked online from action history the router itself generates—this is ordinary online learning, not a circular definition of the target SLOs. Empirical claims are evaluated against external heuristics (Round-Robin, LOR, Random, Power-of-2) on external request traces (LMSYS-Chat-1M and synthetic P/D workloads) inside the Vidur simulator. Self-citations to Ye’s prior online-LP work (Agrawal et al. 2014; Li et al. 2020; Li & Ye 2022) supply the methodological toolkit; under the stated rules they are independent support for the algorithm class, not a load-bearing uniqueness theorem that forbids alternatives or renames the target metrics. No equation equates a claimed improvement to a fitted constant, and no uniqueness or ansatz is smuggled in as an external fact that forces the result. Surrogate quality of the tail indicators is a validity concern, not circularity. Score 0; steps empty.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The paper rests on standard LP duality and continuous-batching mechanics already present in the literature, plus a handful of modeling choices (reward decomposition, residual-capacity forecasts, hand-chosen objective weights) that are free parameters or domain assumptions rather than new physical entities. No novel particles or forces are postulated; the only invented objects are the per-request reward terms and the SAA dual objective used for price learning.

free parameters (4)
  • objective weights (α,β,γ,σ,ζ1,ζ2)
    Hand-selected for each experimental regime to emphasize average latency, tails or throughput; different vectors produce the Pareto trade-offs shown in Figures 2–3. Central empirical claims depend on these choices.
  • tail thresholds t′1, t′2
    Used inside the indicator rewards for Tail EEL/TTFT; set by the authors (e.g., t′2=49 in Fig. 13) without an automatic calibration procedure.
  • gradient steps K, mini-batch size |Bk|, step-size ηk
    Control the accuracy/runtime of the dual update; fixed but not systematically ablated for every reported table.
  • remaining-arrival forecast bnremain(k)
    Appears in the scarcity vector dk; any misspecification directly scales all dual prices.
axioms (4)
  • standard math Standard strong duality for packing LPs and the bid-price interpretation of dual variables
    Invoked throughout §3.2 to justify comparing reward against a⊤p.
  • domain assumption Decode iteration time is essentially constant under PD disaggregation (memory-bandwidth bound)
    Stated in §2 to justify treating one decode step as one discrete period; cited profiling from Jain et al. 2024.
  • domain assumption Predicted decode length ˆoj (even with 20 % noise) is sufficiently informative for the column coefficients a(j,g,k)
    Used to build every resource column; robustness is shown empirically but never proved.
  • ad hoc to paper Aggregate tail quantiles can be driven by per-request indicator rewards 1{latency < t′}
    Introduced in §3.1 without a formal guarantee that the resulting online policy optimizes the true quantile.
invented entities (1)
  • SLO-decomposable per-request reward rj,g,k that mixes throughput, saved-steps latency, QPS and tail indicators no independent evidence
    purpose: Turns multi-objective statistical SLOs into an additive linear objective suitable for online LP
    The particular algebraic form (especially the positive “saved-steps” rewrite and the tail indicators) is constructed by the authors; no independent external validation of the surrogate is supplied.

pith-pipeline@v1.1.0-grok45 · 27654 in / 3388 out tokens · 32130 ms · 2026-07-11T22:48:14.368304+00:00 · methodology

0 comments
read the original abstract

We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints. Unlike widely used routing heuristics that are not tied to explicit service-level objectives (SLOs) and offer limited control over latency-throughput trade-offs, we introduce a multi-objective optimization framework that formulates routing as an online linear programming with interpretable decision rewards. We apply an efficient bid-price control policy based on the online linear programming that admits requests when their SLO-weighted benefit exceeds their shadow prices. To meet millisecond decision requirements, we develop a warm-started, projected first-order updates that track the evolving dual shadow prices online with predictable runtime. We integrate our router into the Vidur simulator and demonstrate substantial improvements over standard baselines across multiple SLO regimes, including end-to-end latency, time-to-first-token, throughput, and tail performance. A big picture from our result: a science-based approach outperforms others based on heuristics.

Figures

Figures reproduced from arXiv: 2607.03948 by Yinyu Ye, Zijie Zhou, Zixi Chen.

Figure 1
Figure 1. Figure 1: Illustration of Prefill/Decode (P/D) disaggregation with continuous batching. Each small box denotes a token, and col￾ors indicate distinct requests. Within each decode worker, active requests are processed token-by-token in a continuously updated batch as completed requests are replaced by newly admitted ones. of each decode device, and may release a subset of queued requests to devices. This design is su… view at source ↗
Figure 3
Figure 3. Figure 3: Real-data comparison illustrating how objective weights shift performance trade-offs. because the policy is not incentivized to protect the slowest requests. In the second setting, we emphasize tail control via (α, β, γ, σ, ζ1, ζ2) = (0, 0, 0, 0, 1, 1), which markedly improves tail end-to-end latency and tail TTFT. Robustness to non-stationary workloads. A natural con￾cern with any router that learns from … view at source ↗
Figure 2
Figure 2. Figure 2: Real-data comparison between routing policies under two representative multi-objective settings. binations are deferred to Appendix C.1. Real-data performance under representative multi￾objective settings [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Real-data comparison between routing policies under stable arrival process λ = 0.4, with a focus on average latency. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.4; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1 400 , 1 400 , 1, 1, 1  ; oˆj = oj Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized valu… view at source ↗
Figure 5
Figure 5. Figure 5: Real-data comparison between routing policies under stable arrival process λ = 0.4, with a focus on tail latency. 1. Targeted improvements: when the objective weights emphasize a particular metric family (average latency, tail latency, or throughput), our policy achieves the best performance on that targeted metric compared to all baselines. 2. Trade-offs are explicit and predictable: when we set the weigh… view at source ↗
Figure 6
Figure 6. Figure 6: Real-data comparison between routing policies under stable arrival process λ = 0.4, with a focus on throughput. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.5; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1, 1, 1, 1 400 , 1 400  ; oˆj = oj Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algo… view at source ↗
Figure 7
Figure 7. Figure 7: Real-data comparison between routing policies under overloaded arrival process λ = 0.5, with a focus on average latency. baseline policies as in the main text and report results under both perfect and noisy decode-length prediction. P/D ratio 1:4 (decode-heavy). Figures 17 and 18 report comparisons under a stable arrival rate for decode-heavy requests (P/D = 1:4), with objective weights chosen to emphasize… view at source ↗
Figure 8
Figure 8. Figure 8: Real-data comparison between routing policies under overloaded arrival process λ = 0.5, with a focus on tail latency. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.5; (α, β, γ, σ, ζ1, ζ2) = 1, 1 400 , 1 400 , 1, 1 400 , 1 400  ; oˆj = oj Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized… view at source ↗
Figure 9
Figure 9. Figure 9: Real-data comparison between routing policies under overloaded arrival process λ = 0.5, with a focus on throughput. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.4; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1, 1, 1, 1 400 , 1 400  ; oˆj ∼ Unif(0.8 · oj , 1.2 · oj ) Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0… view at source ↗
Figure 10
Figure 10. Figure 10: Real-data comparison between routing policies with noisy decode length prediction under stable arrival process λ = 0.4, with a focus on average latency. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.4; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1 400 , 1 400 , 1, 1, 1  ; oˆj ∼ Unif(0.8 · oj , 1.2 · oj ) Thr EEL TTFT QPS Tail EEL … view at source ↗
Figure 11
Figure 11. Figure 11: Real-data comparison between routing policies with noisy decode length prediction under stable arrival process λ = 0.4, with a focus on tail latency. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Real-data comparison between routing policies with noisy decode length prediction under stable arrival process λ = 0.4, with a focus on throughput. 0 1 2 3 4 5 2 94.0 94.2 94.4 94.6 94.8 95.0 95.2 95.4 Tail TTFT Quantile [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: TTFT satisfaction rate as a function of the tail weight ζ2. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.5; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1, 1, 1, 1 400 , 1 400  ; oˆj ∼ Unif(0.8 · oj , 1.2 · oj ) Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Ran… view at source ↗
Figure 14
Figure 14. Figure 14: Real-data comparison between routing policies with noisy decode length prediction under overloaded arrival process λ = 0.5, with a focus on average latency. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Real-data comparison between routing policies with noisy decode length prediction under overloaded arrival process λ = 0.5, with a focus on tail latency. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.5; (α, β, γ, σ, ζ1, ζ2) = 1, 1 400 , 1 400 , 1, 1 400 , 1 400  ; oˆj ∼ Unif(0.8 · oj , 1.2 · oj ) Thr EEL TTFT QPS Tail… view at source ↗
Figure 16
Figure 16. Figure 16: Real-data comparison between routing policies with noisy decode length prediction under overloaded arrival process λ = 0.5, with a focus on throughput. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.4; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1, 1, 1, 1 400 , 1 400  ; oˆj = oj Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 … view at source ↗
Figure 17
Figure 17. Figure 17: Synthetic data (P/D ratio 1 : 4) comparison between routing policies under stable arrival process, with a focus on average latency. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.4; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1 400 , 1 400 , 1, 1, 1  ; oˆj = oj Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0… view at source ↗
Figure 18
Figure 18. Figure 18: Synthetic data (P/D ratio 1 : 4) comparison between routing policies under stable arrival process, with a focus on tail latency. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Synthetic data (P/D ratio 1 : 4) comparison between routing policies with noisy decode length prediction under stable arrival process, with a focus on average latency. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.4; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1 400 , 1 400 , 1, 1, 1  ; oˆj ∼ Unif(0.8 · oj , 1.2 · oj ) Thr EEL TTF… view at source ↗
Figure 20
Figure 20. Figure 20: Synthetic data (P/D ratio 1 : 4) comparison between routing policies with noisy decode length prediction under stable arrival process, with a focus on tail latency. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.4; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1, 1, 1, 1 400 , 1 400  ; oˆj = oj Thr EEL TTFT QPS Tail EEL (99%) Tail TT… view at source ↗
Figure 21
Figure 21. Figure 21: Synthetic data (P/D ratio 4 : 1) comparison between routing policies under stable arrival process, with a focus on average latency. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.4; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1 400 , 1 400 , 1, 1, 1  ; oˆj = oj Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0… view at source ↗
Figure 22
Figure 22. Figure 22: Synthetic data (P/D ratio 4 : 1) comparison between routing policies under stable arrival process, with a focus on tail latency. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Synthetic data (P/D ratio 4 : 1) comparison between routing policies with noisy decode length prediction under stable arrival process, with a focus on average latency. Thr EEL TTFT QPS Tail EEL (99%) Tail TTFT (99%) 0.0 0.2 0.4 0.6 0.8 1.0 Normalized value Algorithm 1 Round Robin LOR Random (a) λ = 0.4; (α, β, γ, σ, ζ1, ζ2) = 1 135 , 1 400 , 1 400 , 1, 1, 1  ; oˆj ∼ Unif(0.8 · oj , 1.2 · oj ) Thr EEL TTF… view at source ↗
Figure 24
Figure 24. Figure 24: Synthetic data (P/D ratio 4 : 1) comparison between routing policies with noisy decode length prediction under stable arrival process, with a focus on tail latency. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_24.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 7 linked inside Pith

  1. [1]

    Langley , title =

    P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =

    2000

  2. [2]

    T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980

    1980

  3. [3]

    M. J. Kearns , title =

  4. [4]

    Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983

    1983

  5. [5]

    R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000

    2000

  6. [6]

    Suppressed for Anonymity , author=

  7. [7]

    Newell and P

    A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981

    1981

  8. [8]

    A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959

    1959

  9. [9]

    Proceedings of the 29th Symposium on Operating Systems Principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th Symposium on Operating Systems Principles , pages=

  10. [10]

    Advances in Neural Information Processing Systems , volume=

    Response length perception and sequence scheduling: An llm-empowered llm inference pipeline , author=. Advances in Neural Information Processing Systems , volume=

  11. [11]

    arXiv preprint arXiv:2404.08509 , year=

    Efficient interactive llm serving with proxy model-based sequence length prediction , author=. arXiv preprint arXiv:2404.08509 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2408.15792 , year=

    Efficient LLM Scheduling by Learning to Rank , author=. arXiv preprint arXiv:2408.15792 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2501.14743 , year=

    KVDirect: Distributed Disaggregated LLM Inference , author=. arXiv preprint arXiv:2501.14743 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2410.01035 , year=

    Don't Stop Me Now: Embedding Based Scheduling for LLMs , author=. arXiv preprint arXiv:2410.01035 , year=

    Pith/arXiv arXiv

  15. [15]

    Management science , volume=

    An analysis of bid-price controls for network revenue management , author=. Management science , volume=. 1998 , publisher=

    1998

  16. [16]

    Mathematics of Operations Research , volume=

    Bid-price controls for network revenue management: Martingale characterization of optimal bid prices , author=. Mathematics of Operations Research , volume=. 2009 , publisher=

    2009

  17. [17]

    Operations Research , volume=

    Online linear programming: Dual convergence, new algorithms, and regret bounds , author=. Operations Research , volume=. 2022 , publisher=

    2022

  18. [18]

    Proceedings of the 20th international conference on machine learning (icml-03) , pages=

    Online convex programming and generalized infinitesimal gradient ascent , author=. Proceedings of the 20th international conference on machine learning (icml-03) , pages=

  19. [19]

    arXiv preprint arXiv:2011.06327 , year=

    Near-optimal primal-dual algorithms for quantity-based network revenue management , author=. arXiv preprint arXiv:2011.06327 , year=

    Pith/arXiv arXiv 2011

  20. [20]

    Operations Research , volume=

    A dynamic near-optimal algorithm for online linear programming , author=. Operations Research , volume=. 2014 , publisher=

    2014

  21. [21]

    https://www.theverge.com/news/710867/openai-chatgpt-daily-prompts-2-billionutmsource=chatgpt.com , url=

    OpenAI says ChatGPT users send over 2.5 billion prompts every day , author=. https://www.theverge.com/news/710867/openai-chatgpt-daily-prompts-2-billionutmsource=chatgpt.com , url=

  22. [22]

    Vidur: A Large-Scale Simulation Framework For

    Agrawal, Amey and Kedia, Nitin and Mohan, Jayashree and Panwar, Ashish and Kwatra, Nipun and Gulavani, Bhargav and Ramjee, Ramachandran and Tumanov, Alexey , journal=. Vidur: A Large-Scale Simulation Framework For

  23. [23]

    Taming throughput-latency tradeoff in

    Agrawal, Amey and Kedia, Nitin and Panwar, Ashish and Mohan, Jayashree and Kwatra, Nipun and Gulavani, Bhargav S and Tumanov, Alexey and Ramjee, Ramachandran , journal=. Taming throughput-latency tradeoff in

  24. [24]

    Proceedings of Machine Learning and Systems , volume=

    Efficiently scaling transformer inference , author=. Proceedings of Machine Learning and Systems , volume=

  25. [25]

    Flexgen: High-throughput generative inference of large language models with a single

    Sheng, Ying and Zheng, Lianmin and Yuan, Binhang and Li, Zhuohan and Ryabinin, Max and Chen, Beidi and Liang, Percy and R. Flexgen: High-throughput generative inference of large language models with a single. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  26. [26]

    16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , pages=

    Orca: A distributed serving system for \ Transformer-Based \ generative models , author=. 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , pages=

  27. [27]

    arXiv preprint arXiv:2401.09670 , year=

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving , author=. arXiv preprint arXiv:2401.09670 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2509.02718 , year=

    Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving , author=. arXiv preprint arXiv:2509.02718 , year=

    arXiv

  29. [29]

    ACM SIGKDD Explorations Newsletter , volume=

    Omnirouter: Budget and performance controllable multi-llm routing , author=. ACM SIGKDD Explorations Newsletter , volume=. 2025 , publisher=

    2025

  30. [30]

    arXiv preprint arXiv:2408.13510 , year=

    Intelligent router for llm workloads: Improving performance through workload-aware scheduling , author=. arXiv preprint arXiv:2408.13510 , year=

    Pith/arXiv arXiv

  31. [31]

    1984 , publisher=

    Linear and nonlinear programming , author=. 1984 , publisher=

    1984

  32. [32]

    Operations research , volume=

    Linear programming , author=. Operations research , volume=. 2002 , publisher=

    2002

  33. [33]

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Li, Tianle and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Li, Zhuohan and Lin, Zi and Xing, Eric and others , journal=

  34. [34]

    Advances in Neural Information Processing Systems , volume=

    Simple and fast algorithm for binary integer and online linear programming , author=. Advances in Neural Information Processing Systems , volume=

  35. [35]

    2025 , month = mar, note =

    vLLM Roadmap , title =. 2025 , month = mar, note =

    2025

  36. [36]

    2025 , month = dec, note =

    2025