{"id":"cc87568d-9180-42ff-96b2-f66f84fb1e6e","arxiv_id":"2607.18253","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A serving-framework simulation that predicts time-to-first-token lets LLM routers jointly optimize accuracy, cost, and latency, improving accuracy-cost utility by up to 40% at matched latency.","lead":"This paper adds latency estimates to LLM query routing, letting a router send each query to the model that best balances accuracy, cost, and response speed. In simulated serving workloads, the approach improves accuracy-cost utility by up to 40% at the same latency as standard load-balancing policies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SFS gains may be in-sample: Eq. (9) is calibrated and evaluated without a described held-out split, and the additive batch-time model omits contention effects that affect TTFT.","rationale":"The paper's strongest claim is that joint optimization with SFS latency estimation yields up to 40% better accuracy-cost utility at matched latencies. For that to hold, the TTFT estimates must be accurate enough that the feasible set J(i) in the routing objective is not systematically biased. The weakest point in the argument is the empirical validation of the Eq. (9) token-batch time model and the downstream routing evaluation: the manuscript does not describe a held-out split between calibration/training and evaluation. The same 2.5K queries per task are said to be used for calibrating throughput, fitting beta coefficients, and training predictors, and no separate test set is mentioned. This is exactly the condition under which a low MAPE and strong OnTimeUtility results could be an artifact of in-sample fitting rather than predictive skill. The concern is not that the authors acted improperly; it is that the evidence, as presented, does not yet rule out this alternative explanation. The reader identified essentially the same weakest assumption, focusing on Eq. (9)'s linear model and the lack of cross-validation; my reading agrees. I do not see a basis to move the verdict: the paper is plausible and the experiments are directionally supportive, but the missing hold-out validation keeps the central quantitative claim conditional. The proposed temporal hold-out test would settle whether the concern actually lands: if held-out MAPE and utility gains persist, the central claim is supported; if not, the reported advantage is partly an artifact of the evaluation protocol.","tokens_in":21000,"tokens_out":4146,"duration_ms":42275,"concrete_test":"Reproduce Figure 5 and the Figure 6 tradeoff curve using a temporal hold-out: use the first 50% of each task's query stream to calibrate beta in Eq. (9) and train the accuracy/output-length predictors; evaluate SFS TTFT MAPE and OnTimeUtility on the second 50%, using the same Poisson and MMPP-2 arrival processes. Report the held-out TTFT MAPE and the SFS-vs-Shortest-Queue utility gap. If the held-out MAPE remains near the reported ~5% and the 33-46% gains persist, the concern is resolved; if MAPE degrades or the utility gap narrows materially, the calibration and evaluation are confounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the TTFT estimates used to build the feasible set J(i) in Eq. (3) are accurate under the actual serving workload. That accuracy rests on Eq. (9), which assumes token-batch processing time is an additive linear function of token counts, context lengths, and prefill/decode composition, with per-instance calibrated coefficients. The model omits memory-bandwidth contention, tensor-parallel communication, scheduler nondeterminism, and preemption, and SFS deterministically simulates vLLM from workload snapshots. The paper's validation (Figures 2-3) reports roughly 4-5% MAPE but does not state whether the evaluation queries were held out from the 2.5K-query-per-task set used to calibrate beta and train the accuracy/output-length predictors (Section 5). If calibration and evaluation share queries, the TTFT MAPE and the resulting OnTimeUtility advantages (33% AUC, 46% at 5 qps, 40% vs Shortest Queue) are upper bounds. Because routing selects instances based on estimated TTFT, systematic bias—not just noise—changes J(i) and can inflate the reported gains. The missing held-out split is therefore load-bearing, not a cosmetic omission.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SFS, a serving-framework simulation based time-to-first-token (TTFT) estimator for LLM query routing. SFS uses per-instance workload snapshots, predicted decode lengths, and a calibrated token-batch processing-time model (Eq. 9) to simulate token batches until the arriving query's first decode token, producing a TTFT estimate. This estimate is used in a routing objective (Eq. 3) that maximizes accuracy-cost utility subject to per-query TTFT constraints, with a Lagrangian variant (Eq. 5) for exploring the utility-latency tradeoff. Experiments on Qwen3-0.6B/8B/32B with four task types compare SFS-based routing against Round Robin, Shortest Queue, and Latency-Agnostic baselines, reporting up to 40% improvement in OnTimeUtility at matched latencies, 33% higher AUC, and sub-millisecond estimator overhead.","tokens_in":1563,"tokens_out":1770,"duration_ms":70375,"significance":"If validated, this is a practical contribution to LLM serving: it connects accuracy-cost routing with system-level latency management, and the simulator-based estimator is lightweight and interpretable. The paper includes a public code repository and measures estimator overhead, which supports reproducibility. However, the empirical claims currently rest on an experimental protocol that does not clearly separate calibration from evaluation, omits comparisons with the closest prior latency-aware router and with the paper's own simpler throughput estimator, and reports no error bars. These gaps are load-bearing for the central claim that SFS-based joint optimization improves routing utility.","major_comments":[{"comment":"The experimental protocol uses the same 2.5K-query-per-task sample to calibrate the β coefficients in Eq. (9), train the LightGBM accuracy and output-length predictors, and estimate prefill throughput (§5). No held-out split is described, so the reported TTFT MAPE (Figs. 2–3) and the OnTimeUtility gains in Figs. 5–7 may be in-sample. This matters because the feasible set J(i) in Eq. (3) is built from SFS TTFT estimates; optimistic estimates can inflate the router's advantage. Please specify and use a clear train/test split, report held-out MAPE for Eq. (9), and ideally evaluate routing on queries not used in any calibration or training step.","section":"§5 / §4.1"},{"comment":"The paper identifies [27] as the closest prior latency-aware router and introduces a simpler throughput-based estimator in Eq. (6), but neither is included in the routing comparisons of Figs. 5–7. Without [27], the claim of improved latency-aware routing over prior work is not directly supported; without Eq. (6) in the routing loop, the experiments do not show that the simulation complexity of SFS is necessary for the observed gains. Please add these two comparisons, or justify their omission explicitly.","section":"§5 Baselines"},{"comment":"The central quantitative claims—33% AUC gain, 46% improvement at 5 qps, and 40% higher utility than Shortest Queue—are based on curves generated from stochastic arrival processes, but no error bars, confidence intervals, or number of seeds are reported. Because OnTimeUtility in Eq. (4) averages over random arrivals, routing decisions, and realized latencies, the observed differences might not be statistically significant. Please report means with standard deviations or confidence intervals over multiple independent runs and, where possible, significance tests.","section":"§5, Figs. 5–7"},{"comment":"The token-batch processing-time model assumes an additive linear form and omits effects such as memory-bandwidth contention, tensor-parallel communication, scheduler nondeterminism, and preemption, although Section F states that preemption behavior is simulated. Validation is only shown on the calibration data (Fig. 3) with approximately 4% MAPE and no held-out split. Since systematic bias in TTFT estimates—not just noise—can change the feasible set J(i) and hence routing decisions, please validate Eq. (9) on held-out batch compositions and under the routing workloads used in Figs. 5–7.","section":"§4.1, Eq. (9)"}],"minor_comments":[{"comment":"There is a typo: “intoa latency-aware router” should be “into a latency-aware router.”","section":"Abstract"},{"comment":"The left and right panels would be easier to read if axes were labeled consistently and the units (ms) appeared on both panels; the caption mentions outliers but does not describe how many points they represent.","section":"Fig. 2"},{"comment":"The caption text contains garbled characters and is not typeset correctly; it should read “Utilities plotted against TTFT latencies by varying δ in Eq. (5).”","section":"Fig. 6"},{"comment":"The LPS average-case estimator in Eq. (10) is derived under Poisson arrivals and memoryless service, but Figure 10 shows highly variable output lengths. The appendix acknowledges this limitation, but the main text should state more prominently that Eq. (10) is only an approximation for heavy-tailed workloads and should be validated in routing experiments.","section":"§4.2 / Appendix E"},{"comment":"The overhead estimate reports a mean of 0.139 ms but no distribution or percentile information; a box plot or tail statistic would help support the “sub-millisecond” claim.","section":"Fig. 8"},{"comment":"The paper says “2.5K queries from each task” are used for calibration/training, but it is unclear whether the routing simulation consumes these same queries as arriving requests. Please state explicitly how the arrival workload relates to the calibration/training set.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile problem and the SFS estimator is a credible engineering contribution. The main barriers to acceptance are empirical rigor: the missing held-out split, missing baselines against [27] and the paper's own throughput estimator, and absent error bars. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I did not find evidence of construction-level circularity; the concerns are about validation and attribution of gains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the SFS latency estimator is a genuinely useful piece of engineering, and the routing formulation is clean, but the evaluation as written leaves two gaps that matter: no held-out split is described for the calibrated estimators, and the closest prior work [27] is never compared. I'd want those addressed before relying on the 40% utility claim.\n\nWhat's new: bridging accuracy-cost routing with workload-aware TTFT estimation via a deterministic simulation of continuous batching. The token-batch time model in Eq. (9) is simple and interpretable, and the 4% MAPE on batch times is plausible for the tested setting. The overhead figure (0.14 ms) is useful. The LPS queueing derivation in the appendix is a nice bonus for when real-time snapshots are unavailable.\n\nSoft spots: First, the held-out issue. Section 5 says 2.5K queries per task are used to calibrate throughput, token-batch coefficients, and train predictors, but nowhere does it say that the routing evaluation uses a separate set of queries. If the same queries are used, the TTFT MAPE and OnTimeUtility gains are optimistic. This is not a minor wording issue; routing decisions depend on the feasibility set J(i), so systematic bias in TTFT estimates changes the reported behavior. The authors need to state the split or redo the evaluation. Second, the absence of [27] is a real omission. [27] already does cost- and latency-constrained routing with output-length prediction; this paper's contribution is the prefill/decode workload simulation, so the comparison is necessary to show the added value. Third, no error bars or number of seeds. For a systems paper with a single GPU setup, that may be standard, but it makes the 33%/46%/40% numbers hard to assess.\n\nI don't think the additive linear model in Eq. (9) is a fatal flaw. The empirical fit suggests it captures the dominant effects for vLLM on H100, and the authors position it as a calibrated approximation. But the missing held-out split makes it hard to separate approximation error from overfitting.\n\nBottom line: this is a solid, clearly-written paper that deserves a serious referee. I'd send it to review, but with a strong request that the authors add a held-out evaluation, compare to [27], and report variance. If the held-out results hold, it's a good contribution to the LLM-serving subfield.","headline":"Worth a serious review if the authors add a held-out split and a head-to-head with [27]; the SFS idea is solid and the paper is honestly written.","tokens_in":21809,"tokens_out":2691,"would_cite":true,"duration_ms":24125,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM query routing can jointly optimize accuracy, cost, and time-to-first-token by simulating a serving framework's token batches, delivering up to 40% higher accuracy–cost utility at the same latencies as standard loa","keywords":["LLM query routing","time-to-first-token","latency-aware routing","accuracy-cost tradeoff","continuous batching","serving framework simulation","queueing analysis","dynamic workloads"],"falsifier":"Run the SFS router on a held-out workload that mixes many long-context decode sequences with concurrent prefills on a tensor-parallel multi-GPU instance, and compare predicted versus measured TTFT; if the mean absolute percentage error departs substantially from the reported ~5%, or if a router with oracle (measured) TTFT outperforms SFS by a large margin, the claim that lightweight workload simulation drives the 40% utility gain would be falsified.","tokens_in":20900,"feed_emoji":"⚡","tokens_out":7240,"duration_ms":66856,"temperature":0.7,"pith_summary":"The paper argues that existing LLM query routers—which choose a model per query to balance response quality and price—fail because they ignore the latency a query will actually experience, which depends on the current prefill and decode workload at each model instance. It proposes to close that gap with a lightweight estimator, the Serving Framework Simulation (SFS), that simulates continuous batching exactly as the serving framework would process token batches, predicting time-to-first-token (TTFT) for an incoming query at each instance. The router then maximizes accuracy–cost utility among instances predicted to meet the query's TTFT target. Experiments on three transformer models and four task types report that this joint optimization yields up to 40% higher accuracy–cost utility than load-balancing baselines at matched latencies, with a sub-millisecond estimation overhead. If correct, latency-aware routing would let service operators meet per-query response-time targets without a separate load-balancing layer erasing the quality–cost gains.","feed_headline":"Latency-aware router lifts LLM accuracy-cost utility 40%","feed_subtitle":"Time-to-first-token estimates let routers meet latency targets without sacrificing quality or cost.","key_machinery":"The Serving Framework Simulation (SFS) estimator, together with the token-batch processing-time model it relies on. SFS simulates the serving framework's batching and scheduling policy on a workload snapshot—remaining prefill tokens and estimated decode lengths of resident requests plus the new query—and sums predicted processing times of consecutive token batches up to the new query's first decode token (TTFT). Batch times come from a per-instance calibrated linear regression whose terms capture dense token computation, decode-side attention and KV-cache reads (linear in context length), and prefill-side attention (linear in prefill chunk times context, plus quadratic within the chunk). For","core_discovery":"The paper claims that time-to-first-token (TTFT) in an LLM serving pool is predictable enough to route on, and that routing on it changes the accuracy–cost outcome. SFS snapshots each instance's resident requests, remaining prefill tokens, and estimated decode lengths, then deterministically simulates continuous-batching token batches until the new query emits its first decode token. Batch times come from a per-instance calibrated linear model of batch composition (token counts, context-scaled decode attention, quadratic prefill attention). Summing these batch times gives the TTFT estimate used in routing. The paper reports under 5% mean absolute percentage error on TTFT at ~0.14 ms overhead","pith_inferences":["The same SFS simulation could be extended to end-to-end response latency (not just TTFT) by continuing the simulation to sequence completion, making the router relevant for batch and long-generation workloads where TTFT is not the binding SLO.","The accuracy of routing hinges on the decode-length estimator as much as on the batch-time model: if resident decode-length estimates are systematically biased, the prefill/decode interference picture degrades; a testable extension is to feed the router ground-truth future decode lengths and measure how much of the 40% gain is attributable to the simulator versus the predictor.","The LPS formula suggests a capacity-planning heuristic: for a given arrival burstiness, the parallelism cap k (controlled by serving configuration) sets how quickly waiting time grows with load; operators could size k using the router's own calibrated metrics rather than throughput averages.","The paper assumes a single router; if multiple independent routers send queries to shared instances, each router's snapshot-based simulation would need to account for other routers' decisions—an interaction the paper lists as future work but which the SFS estimator's snapshot mechanism could partially address."],"forward_implications":["If TTFT can be estimated accurately at routing time, per-query latency SLOs no longer need to be delegated to a separate load-balancing layer; the same routing decision can carry quality, cost, and latency commitments.","Routers can exploit the heterogeneous accuracy–cost structure of a model pool under load, sending latency-sensitive or long-prompt queries to faster instances while still using larger models when their quality gain justifies the cost.","The Lagrangian variant of the objective gives operators a single dial δ to trade average latency against accuracy–cost utility, so latency targets and quality/cost can be tuned without re-training the router.","The average-case LPS estimator provides a planning tool: from arrival rate, service capacity, and parallelism cap, operators can predict queueing delay before the router has live workload snapshots.","Because the simulation only needs to continue to the first decode token, predictions are robust to moderate errors in decode-length estimates, keeping the estimator light enough for the routing critical path."],"fun_headline_variants":["TTFT-aware router boosts utility 40%","TTFT prediction lifts LLM routing utility 40%","Router predicts time-to-first-token, gains 40% utility","Joint latency-accuracy-cost routing lifts utility 40%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the time to process a token batch on a GPU is well approximated by a per-instance calibrated linear function of token counts, context lengths, and prefill/decode composition (Eq. 9); if memory-bandwidth contention, tensor-parallel communication, scheduler nondeterminism, or preemption materially break that linear model, SFS's TTFT estimates—and therefore the set of instances deemed to meet each query's latency target—will be wrong.","fun_headline_variants_meta":{"raw":{"variants":["TTFT-aware router boosts utility 40%","TTFT prediction lifts LLM routing utility 40%","Router predicts time-to-first-token, gains 40% utility","Joint latency-accuracy-cost routing lifts utility 40%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001565,"raw_usage":{"total_tokens":6078,"prompt_tokens":723,"completion_tokens":5355,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":5297}},"tokens_in":467,"tokens_out":5355,"duration_ms":36955,"temperature":1.0,"reasoning_tokens":5297,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:02:20.692149+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the SFS router on a held-out workload that mixes many long-context decode sequences with concurrent prefills on a tensor-parallel multi-GPU instance, and compare predicted versus measured TTFT; if the mean absolute percentage error departs substantially from the reported ~5%, or if a router with oracle (measured) TTFT outperforms SFS by a large margin, the claim that lightweight workload simulation drives the 40% utility gain would be falsified.","supporting_citations":[],"review_version":1}