{"id":"803f9cd5-5862-4d7a-afe3-26be50efca68","arxiv_id":"2608.08382","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLMVisor fits a piecewise-linear formula over token counts, context length, squared tokens, and batch size to predict and attribute LLM batch latency in real time.","lead":"LLMVisor is a mathematical model that divides the time a GPU spends serving a shared batch of AI requests into a fair per-request share, fast enough to run during scheduling. The paper reports that the model is far more accurate than counting tokens alone across several large language models and GPU types.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-request attribution is never validated: an accurate batch fit does not establish the advertised additive per-request shares.","rationale":"The model is simple and the aggregate direction is plausible. The reader's concern about train/test leakage is legitimate and should be fixed, but it is an experimental-hygiene issue that can be resolved with a clean split. The attribution gap is more fundamental: no metric, baseline, or ground truth for per-request latency appears anywhere in the manuscript, so the central contribution is not falsifiable as written. An OLS fit to batch latency cannot certify a decomposition, because the loss function is invariant to many per-request reallocations. The unexplained a4 term and the unmet nonnegativity requirement make the attribution convention under-specified. I therefore join the CONDITIONAL verdict, but condition acceptance on a direct validation of per-request shares (or a clear definition of the attribution semantics and its acceptable error), not only on held-out batch prediction. The reader's weakest assumption was train/test leakage; I partially agree, since that is one necessary condition, but I do not think it is the deepest one.","tokens_in":6730,"tokens_out":6075,"duration_ms":64889,"concrete_test":"On a small validation set (e.g., 100 randomly sampled batches with B in {2,3,4}, drawn from the same workloads), measure leave-one-out marginal latency for each request by re-running the batch with that request removed on the same engine, and compare the removed request's attributed share f(i|B) with the observed batch-latency difference. Report mean absolute error and p90/p99 of the difference; if the attribution error is not comparable to the batch-level relative errors in Tables 1 and 2, the claimed per-request accuracy is not established. In addition, state the exact convention used to allocate a4 sum_i |B| and verify all shares are nonnegative on the fitted coefficients.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is per-request latency attribution, but every reported metric (R2, p90/p99 relative error) is computed on the aggregate batch-latency prediction. The per-request shares f(i|B) are produced by re-reading the same OLS coefficients from Eq. (1); they are never compared with any ground-truth per-request latency t_i, and the paper does not define how t_i would be measured inside a co-batched GPU step. This matters because an aggregate fit does not identify a unique decomposition: many different per-request allocations sum to the same batch latency, so a near-perfect batch fit can coexist with an arbitrary per-request share. The batch-level term a4 sum_i |B| = a4 B^2 in Eq. (1) is particularly telling: it is not a per-request feature, so distributing it requires an unstated convention (e.g., crediting a4 B to each request). If that convention is wrong, per-tenant accounting and SLO admission decisions inherit a systematic bias. The stated design requirement that shares be nonnegative is also not enforced by the unconstrained OLS fit. Hence the headline claim that LLMVisor decomposes batch latency into additive, per-request shares is a mathematical consequence of the linear form, not an empirically established attribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LLMVisor, a piecewise-linear latency model for LLM serving that predicts end-to-end batch latency and, by linear additivity, assigns per-request latency shares. The model uses four terms: per-request tokens, context tokens, squared tokens, and a batch-size term, with coefficients fit by OLS on short warm-up profiling runs. The authors evaluate prefill and decode latency across Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 GPUs with varying tensor parallelism, comparing against VTC and reporting near-perfect R2 and reduced p90/p99 relative errors. The paper claims microsecond-scale runtime, making the model suitable for in-scheduler use.","tokens_in":6975,"tokens_out":2912,"duration_ms":31280,"significance":"If the central claim were fully established, LLMVisor would be a useful primitive for multi-tenant LLM serving: an interpretable, additive latency model that runs at microsecond scale and can support per-tenant accounting and admission control. The roofline-guided feature choice and the decision to fit coefficients from profiling data are reasonable, and the aggregate batch-latency evaluation shows a plausible improvement over a token-count baseline. However, the paper's advertised contribution is per-request attribution, and that contribution is never directly validated. The reported metrics are all aggregate batch-level fits, and the paper does not show that the per-request shares f(i|B) correspond to any measurable per-request latency. The absence of a train/test split and the arbitrary exclusion of outliers further weaken the empirical claims. The central idea is defensible and potentially fixable, but the evidence as presented does not support the headline attribution claim.","major_comments":[{"comment":"The paper's central contribution is per-request latency attribution, but every reported metric (R2, p90/p99 relative error) is computed on the aggregate batch-latency prediction. The per-request shares f(i|B) are never compared with any ground-truth per-request latency t_i, and the vLLM internal profiler described in §3.1 measures per-step batch latency, not per-request latency inside a co-batched GPU step. An aggregate fit does not identify a unique per-request decomposition: many different allocations sum to the same batch latency, so a near-perfect batch-level fit can coexist with arbitrary or incorrect per-request shares. The paper needs to validate attribution explicitly, for example by comparing LLMVisor's shares against isolated single-request latencies or against a controlled abatement study where requests are removed one at a time.","section":"§2.1, Eq. (1)"},{"comment":"The paper fits coefficients using short warm-up profiling runs but never states that the evaluation data in Tables 1 and 2 are held out from those runs. If the evaluation batches overlap the fitting data, the near-perfect R2 and the 2.5x–4.4x error improvements are statements about in-sample fit quality, not prediction accuracy. The authors should specify the exact split between profiling and evaluation data, and report test-set metrics with confidence intervals or error bars. Without this, the claimed accuracy improvements over VTC are not established.","section":"§3.1 and Tables 1–2"},{"comment":"The batch-level term a4 * sum_i |B| equals a4 * |B|^2 and is not a per-request feature. Distributing this term across requests requires an unstated convention (for example, crediting a4 * |B| to each request), and the choice of convention directly affects per-tenant accounting and admission decisions. Moreover, the design requirement in §2 that shares be nonnegative is not enforced by the unconstrained OLS fit; some coefficients can be negative, leading to negative attributed latency for some requests. The paper should state the attribution rule for the batch term and discuss how nonnegativity is guaranteed or handled.","section":"§2.1, Eq. (1)"},{"comment":"The paper reports average improvements 'excluding the two blue-marked outliers' but Table 1 does not visibly mark any entries in blue, and no criterion is given for identifying an outlier. This makes the averaged improvement claims non-reproducible. The authors should either include all configurations in the average or provide an explicit, pre-specified outlier-removal rule.","section":"§3.2"}],"minor_comments":[{"comment":"The notation sum_i |B| is confusing: since |B| does not depend on i, the term should be written as a4 * |B|^2 or a4 * B^2, and the phrase 'captures utilization gain' should be reconciled with the quadratic dependence on batch size.","section":"§2.1"},{"comment":"The paper says the model yields 'four sets of parameters in total' but then reports results across multiple models, GPU types, and tensor-parallel sizes. It should clarify whether the coefficients are refit per configuration or shared across configurations; the latter would make the total parameter count ambiguous.","section":"§2.1"},{"comment":"The evaluation section does not report the number of profiling steps, the number of evaluation batches, or the variance of the latency measurements. Adding this information would help assess the stability of the reported R2 and percentile errors.","section":"§3.1"},{"comment":"The phrase 'up to 3.5x and 4.4x improvement' is imprecise because the improvements are average improvements computed after excluding outliers. The text should specify whether these are maximum or average improvements across configurations.","section":"§3.2–3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a workshop-level extended abstract: the core model is simple and the OLS machinery is standard, but the claimed per-request attribution is not empirically validated, and the aggregate accuracy claims lack a train/test split. The authors may be able to address these points with additional experiments, but as submitted the evidence does not support the central contribution. I would also note that the manuscript contains a NeurIPS workshop footnote and a future arXiv date; the submission history should be clarified to the editor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fairly simple idea: fit a piecewise-linear OLS model over token counts, context, squared tokens, and batch size, with separate coefficient sets for prefill/decode and memory/compute segments. The additive structure gives closed-form per-request shares, which is convenient. The paper is right that VTC's token-only linear model misses context and batch effects, and the reported R2 improvements over VTC are directionally plausible. But the evidence as written does not support the central claims.\n\nThe biggest issue is the train/test ambiguity. Coefficients are fit on short warm-up profiling runs, but the evaluation never says the reported numbers come from held-out batches. If Tables 1 and 2 are in-sample, the near-perfect R2 values only say the linear form fits the training data; the 2.5-4.4x error reductions are not established. The authors need to state and enforce a clean split, or release data and code so reviewers can check.\n\nSecond, the headline contribution is per-request attribution, yet no per-request ground truth is defined or compared. An accurate aggregate batch fit does not identify a unique decomposition; many allocations sum to the same batch latency. The batch-level term a4 * sum_i |B| is especially telling: it is a batch feature, not a per-request one, so distributing it requires an unstated convention. If that convention is wrong, per-tenant accounting and admission control inherit a systematic bias. The paper never validates f(i|B) against any measured t_i, and it doesn't say how t_i would be measured inside a co-batched GPU step. This is the load-bearing gap.\n\nMinor but real: the paper excludes 'blue-marked outliers' from averages without a criterion; the batch-size term is labeled 'utilization gain' but sums to a4 B^2; no error bars; no code, data, or fitted coefficients. These are fixable but should be addressed.\n\nWhat is genuinely here: a clean, interpretable model that encodes quadratic attention, KV-cache traffic, and batch effects in a way that admits additive attribution by construction. For a workshop paper with a working vLLM prototype, this is a reasonable starting point. But as written, the evaluation overclaims.\n\nI'd send this to peer review because the question is real and the model is worth examining, but I'd expect major revisions: out-of-sample evaluation, an explicit per-request validation protocol or acknowledgment that shares are a modeling convention, and artifact release. A reader going straight to the tables should not walk away thinking per-request attribution has been measured.","headline":"Plausible, interpretable latency attribution model that is undermined by an evaluation that cannot distinguish fit from prediction and never validates the per-request shares it advertises.","tokens_in":7544,"tokens_out":2838,"would_cite":false,"duration_ms":26336,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMVisor claims that batched LLM inference latency can be decomposed into additive per-request shares in real time by a short roofline-guided piecewise-linear formula.","keywords":["LLM serving","latency attribution","multi-tenant scheduling","roofline model","piecewise-linear regression","prefill and decode","GPU inference","resource accounting"],"falsifier":"A direct check is to hold out entire workload mixes (different concurrency, length skew, and token-to-capacity ratios) from the coefficient-fitting runs and then measure p99 relative error on those held-out mixes; if error returns to the token-count baseline, the model has not generalised.","tokens_in":6484,"feed_emoji":"⏱️","tokens_out":8993,"duration_ms":88330,"temperature":0.7,"pith_summary":"LLMVisor claims that the end-to-end latency of a batch of LLM requests can be decomposed into additive per-request shares by a concise piecewise-linear formula fitted from profiling data. The formula captures compute-bound prefill and memory-bound decode through features proportional to FLOPs and memory I/O traffic, including quadratic self-attention and KV-cache loads. If this holds, multi-tenant GPU serving can move from opaque co-batching toward fractional sharing, where schedulers enforce per-tenant GPU-time quotas, admission control, and transparent accounting without leaving the scheduling loop. Across Llama 3.1-8B and Qwen 2.5-14B/32B on A100 and H100 GPUs, the paper reports near-perfect $R^2$ and p90/p99 relative errors 2.5x to 4.4x lower than a token-count baseline.","feed_headline":"LLMVisor splits LLM latency into per-tenant shares","feed_subtitle":"A microsecond-scale formula lets schedulers enforce fairness and SLOs without slowing batching.","key_machinery":"The mechanism is a roofline-guided piecewise-linear latency model: Eq. (1), with coefficients fit by ordinary least squares from short warm-up profiling runs. The features $p_i$, $c_i$, $p_i^2$, and $|B|$ stand in for MLP and causal-attention FLOPs, KV-cache memory traffic, quadratic self-attention cost, and batch-size utilization, respectively. Two-segment coefficient sets distinguish the compute-bound prefill phase from the memory-bound decode phase. The load-bearing property is additivity: because batch latency is linear in sums of per-request features, per-request attribution and per-tenant aggregation follow in closed form, which is what makes microsecond-scale what-if queries possible.","core_discovery":"The central discovery is that a single additive regression form, $T_B = \\beta + a_1 \\sum_i p_i + a_2 \\sum_i c_i + a_3 \\sum_i p_i^2 + a_4 |B|$, models both prefill and decode step latency accurately enough for real-time attribution, with one coefficient set per phase. Here $p_i$ is the number of tokens request $i$ processes in the step, $c_i$ is the context length whose KV cache must be loaded, and $|B|$ is the batch size. Because the formula is linear in sums over requests, each request receives a closed-form additive share, and per-tenant usage aggregates by summing those shares. The paper argues that this is the missing primitive for fair multi-tenant LLM serving and shows that the computation runs at microsecond scale, more than 100 times faster than ML predictors such as Random Forest.","pith_inferences":["Extending beyond the paper, a direct stress test would hold out entire workload mixes, not just scattered batches, from the profiling fit; the claim stands or falls on whether p99 error stays low there.","The same additive feature set could transfer to disaggregated prefill and decode serving by fitting one coefficient set per engine stage.","If the shares are used for billing, they give a cost basis but not a fairness policy; how to prioritise tenants whose attributed shares exceed budgets is a separate decision.","The paper does not study whether the fitted coefficients stay stable under GPU clock changes or thermal throttling; that would determine whether the model needs online refitting."],"forward_implications":["A scheduler can sum per-request shares by tenant each step to enforce GPU-time quotas, a direct software analogue of CPU accounting in containers.","Admission control can reject or defer a request whenever its attributed share would push its tenant beyond an SLO latency budget.","The same formula supports online 'what-if' planning: changing batch size or request mix changes only the four aggregate sums, so the cost estimate stays microsecond-scale.","Because the coefficient sets are re-fit per phase, model, tensor-parallel size, and GPU, the method is not tied to one architecture and transfers by short profiling runs.","Running inside the scheduler does not disturb batching, since the model is over 100x faster than ML predictors such as Random Forest."],"supporting_citations":[{"why":"The serving engine and internal per-step profiler used to obtain ground-truth latencies for Tables 1 and 2.","marker":"[11]"},{"why":"VTC is the token-count baseline; the measured error comparisons against it carry the accuracy claim.","marker":"[17]"},{"why":"Supplies the roofline analysis of compute-versus memory-bound LLM inference that LLMVisor turns into a fitted piecewise-linear form.","marker":"[22]"},{"why":"The ML latency predictor that motivates the speed comparison; LLMVisor must beat it on both attribution and runtime.","marker":"[6]"},{"why":"Grounds the claim that KV-cache load and compute vary with token position, which justifies using context length as a feature.","marker":"[9]"}],"fun_headline_variants":["LLMVisor attributes LLM latency per tenant in microseconds","LLMVisor: one formula, 4.4x better p99 latency attribution","Microsecond linear model splits LLM batch cost per request","LLMVisor: additive shares enable fair multi-tenant LLM serving","LLMVisor beats token-count baseline by 4.4x at p99"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy claim depends on the reported evaluation using held-out batches that were not part of the warm-up profiling runs used to fit the coefficients, and the paper never states that this is the case.","fun_headline_variants_meta":{"raw":{"variants":["LLMVisor attributes LLM latency per tenant in microseconds","LLMVisor: one formula, 4.4x better p99 latency attribution","Microsecond linear model splits LLM batch cost per request","LLMVisor: additive shares enable fair multi-tenant LLM serving","LLMVisor beats token-count baseline by 4.4x at p99"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1723,"prompt_tokens":954,"completion_tokens":769,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":670}},"tokens_in":570,"tokens_out":769,"duration_ms":7706,"temperature":1.0,"reasoning_tokens":670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:37:11.948230+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check is to hold out entire workload mixes (different concurrency, length skew, and token-to-capacity ratios) from the coefficient-fitting runs and then measure p99 relative error on those held-out mixes; if error returns to the token-count baseline, the model has not generalised.","supporting_citations":[{"cited_title":"Fairness in serving large language models","cited_arxiv_id":null,"evidence_quote":"VTC is the token-count baseline; the measured error comparisons against it carry the accuracy claim."},{"cited_title":"Predicting llm inference latency: A roofline-driven ml method","cited_arxiv_id":null,"evidence_quote":"The ML latency predictor that motivates the speed comparison; LLMVisor must beat it on both attribution and runtime."}],"review_version":1}