LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

Jiaxin Shan; Le Xu; Liguang Xie; Shuowei Jin; Tieying Zhang; Xueshen Liu; Z. Morley Mao

REVIEW 4 major objections 4 minor 23 references

LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMVisor claims that batched LLM inference latency can be decomposed into additive per-request shares in real time by a short roofline-guided piecewise-linear formula.

desk verdict Plausible, interpretable latency attribution model that is undermined by an evaluation that cannot distinguish fit from prediction and never validates the per-request shares it advertises. read the letter →

arxiv 2608.08382 v1 pith:NFXWJCQP submitted 2026-08-09 cs.AI cs.DC

Shuowei Jin , Xueshen Liu , Jiaxin Shan , Le Xu , Tieying Zhang , Liguang Xie , Z. Morley Mao This is my paper · ORCID

classification cs.AIcs.DC

keywords LLMservinglatencyattributionmulti-tenantschedulingrooflinemodelpiecewise-linearregressionprefillanddecodeGPUinferenceresourceaccounting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMVisor claims that the end-to-end latency of a batch of LLM requests can be decomposed into additive per-request shares by a concise piecewise-linear formula fitted from profiling data. The formula captures compute-bound prefill and memory-bound decode through features proportional to FLOPs and memory I/O traffic, including quadratic self-attention and KV-cache loads. If this holds, multi-tenant GPU serving can move from opaque co-batching toward fractional sharing, where schedulers enforce per-tenant GPU-time quotas, admission control, and transparent accounting without leaving the scheduling loop. Across Llama 3.1-8B and Qwen 2.5-14B/32B on A100 and H100 GPUs, the paper reports near-perfect $R^2$ and p90/p99 relative errors 2.5x to 4.4x lower than a token-count baseline.

What carries the argument

The mechanism is a roofline-guided piecewise-linear latency model: Eq. (1), with coefficients fit by ordinary least squares from short warm-up profiling runs. The features $p_i$, $c_i$, $p_i^2$, and $|B|$ stand in for MLP and causal-attention FLOPs, KV-cache memory traffic, quadratic self-attention cost, and batch-size utilization, respectively. Two-segment coefficient sets distinguish the compute-bound prefill phase from the memory-bound decode phase. The load-bearing property is additivity: because batch latency is linear in sums of per-request features, per-request attribution and per-tenant aggregation follow in closed form, which is what makes microsecond-scale what-if queries possible.

What would settle it

A direct check is to hold out entire workload mixes (different concurrency, length skew, and token-to-capacity ratios) from the coefficient-fitting runs and then measure p99 relative error on those held-out mixes; if error returns to the token-count baseline, the model has not generalised.

Watch

Extended reading notes

Core claim

The central discovery is that a single additive regression form, $T_B = \beta + a_1 \sum_i p_i + a_2 \sum_i c_i + a_3 \sum_i p_i^2 + a_4 |B|$, models both prefill and decode step latency accurately enough for real-time attribution, with one coefficient set per phase. Here $p_i$ is the number of tokens request $i$ processes in the step, $c_i$ is the context length whose KV cache must be loaded, and $|B|$ is the batch size. Because the formula is linear in sums over requests, each request receives a closed-form additive share, and per-tenant usage aggregates by summing those shares. The paper argues that this is the missing primitive for fair multi-tenant LLM serving and shows that the computation runs at microsecond scale, more than 100 times faster than ML predictors such as Random Forest.

Load-bearing premise

The accuracy claim depends on the reported evaluation using held-out batches that were not part of the warm-up profiling runs used to fit the coefficients, and the paper never states that this is the case.

Editorial extensions

If this is right

A scheduler can sum per-request shares by tenant each step to enforce GPU-time quotas, a direct software analogue of CPU accounting in containers.
Admission control can reject or defer a request whenever its attributed share would push its tenant beyond an SLO latency budget.
The same formula supports online 'what-if' planning: changing batch size or request mix changes only the four aggregate sums, so the cost estimate stays microsecond-scale.
Because the coefficient sets are re-fit per phase, model, tensor-parallel size, and GPU, the method is not tied to one architecture and transfers by short profiling runs.
Running inside the scheduler does not disturb batching, since the model is over 100x faster than ML predictors such as Random Forest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extending beyond the paper, a direct stress test would hold out entire workload mixes, not just scattered batches, from the profiling fit; the claim stands or falls on whether p99 error stays low there.
The same additive feature set could transfer to disaggregated prefill and decode serving by fitting one coefficient set per engine stage.
If the shares are used for billing, they give a cost basis but not a fairness policy; how to prioritise tenants whose attributed shares exceed budgets is a separate decision.
The paper does not study whether the fitted coefficients stay stable under GPU clock changes or thermal throttling; that would determine whether the model needs online refitting.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Fairly simple idea: fit a piecewise-linear OLS model over token counts, context, squared tokens, and batch size, with separate coefficient sets for prefill/decode and memory/compute segments. The additive structure gives closed-form per-request shares, which is convenient. The paper is right that VTC's token-only linear model misses context and batch effects, and the reported R2 improvements over VTC are directionally plausible. But the evidence as written does not support the central claims.

The biggest issue is the train/test ambiguity. Coefficients are fit on short warm-up profiling runs, but the evaluation never says the reported numbers come from held-out batches. If Tables 1 and 2 are in-sample, the near-perfect R2 values only say the linear form fits the training data; the 2.5-4.4x error reductions are not established. The authors need to state and enforce a clean split, or release data and code so reviewers can check.

Second, the headline contribution is per-request attribution, yet no per-request ground truth is defined or compared. An accurate aggregate batch fit does not identify a unique decomposition; many allocations sum to the same batch latency. The batch-level term a4 * sum_i |B| is especially telling: it is a batch feature, not a per-request one, so distributing it requires an unstated convention. If that convention is wrong, per-tenant accounting and admission control inherit a systematic bias. The paper never validates f(i|B) against any measured t_i, and it doesn't say how t_i would be measured inside a co-batched GPU step. This is the load-bearing gap.

Minor but real: the paper excludes 'blue-marked outliers' from averages without a criterion; the batch-size term is labeled 'utilization gain' but sums to a4 B^2; no error bars; no code, data, or fitted coefficients. These are fixable but should be addressed.

What is genuinely here: a clean, interpretable model that encodes quadratic attention, KV-cache traffic, and batch effects in a way that admits additive attribution by construction. For a workshop paper with a working vLLM prototype, this is a reasonable starting point. But as written, the evaluation overclaims.

I'd send this to peer review because the question is real and the model is worth examining, but I'd expect major revisions: out-of-sample evaluation, an explicit per-request validation protocol or acknowledgment that shares are a modeling convention, and artifact release. A reader going straight to the tables should not walk away thinking per-request attribution has been measured.

Referee Report

4 major / 4 minor

Summary. The paper presents LLMVisor, a piecewise-linear latency model for LLM serving that predicts end-to-end batch latency and, by linear additivity, assigns per-request latency shares. The model uses four terms: per-request tokens, context tokens, squared tokens, and a batch-size term, with coefficients fit by OLS on short warm-up profiling runs. The authors evaluate prefill and decode latency across Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 GPUs with varying tensor parallelism, comparing against VTC and reporting near-perfect R2 and reduced p90/p99 relative errors. The paper claims microsecond-scale runtime, making the model suitable for in-scheduler use.

Significance. If the central claim were fully established, LLMVisor would be a useful primitive for multi-tenant LLM serving: an interpretable, additive latency model that runs at microsecond scale and can support per-tenant accounting and admission control. The roofline-guided feature choice and the decision to fit coefficients from profiling data are reasonable, and the aggregate batch-latency evaluation shows a plausible improvement over a token-count baseline. However, the paper's advertised contribution is per-request attribution, and that contribution is never directly validated. The reported metrics are all aggregate batch-level fits, and the paper does not show that the per-request shares f(i|B) correspond to any measurable per-request latency. The absence of a train/test split and the arbitrary exclusion of outliers further weaken the empirical claims. The central idea is defensible and potentially fixable, but the evidence as presented does not support the headline attribution claim.

major comments (4)

[§2.1, Eq. (1)] The paper's central contribution is per-request latency attribution, but every reported metric (R2, p90/p99 relative error) is computed on the aggregate batch-latency prediction. The per-request shares f(i|B) are never compared with any ground-truth per-request latency t_i, and the vLLM internal profiler described in §3.1 measures per-step batch latency, not per-request latency inside a co-batched GPU step. An aggregate fit does not identify a unique per-request decomposition: many different allocations sum to the same batch latency, so a near-perfect batch-level fit can coexist with arbitrary or incorrect per-request shares. The paper needs to validate attribution explicitly, for example by comparing LLMVisor's shares against isolated single-request latencies or against a controlled abatement study where requests are removed one at a time.
[§3.1 and Tables 1–2] The paper fits coefficients using short warm-up profiling runs but never states that the evaluation data in Tables 1 and 2 are held out from those runs. If the evaluation batches overlap the fitting data, the near-perfect R2 and the 2.5x–4.4x error improvements are statements about in-sample fit quality, not prediction accuracy. The authors should specify the exact split between profiling and evaluation data, and report test-set metrics with confidence intervals or error bars. Without this, the claimed accuracy improvements over VTC are not established.
[§2.1, Eq. (1)] The batch-level term a4 * sum_i |B| equals a4 * |B|^2 and is not a per-request feature. Distributing this term across requests requires an unstated convention (for example, crediting a4 * |B| to each request), and the choice of convention directly affects per-tenant accounting and admission decisions. Moreover, the design requirement in §2 that shares be nonnegative is not enforced by the unconstrained OLS fit; some coefficients can be negative, leading to negative attributed latency for some requests. The paper should state the attribution rule for the batch term and discuss how nonnegativity is guaranteed or handled.
[§3.2] The paper reports average improvements 'excluding the two blue-marked outliers' but Table 1 does not visibly mark any entries in blue, and no criterion is given for identifying an outlier. This makes the averaged improvement claims non-reproducible. The authors should either include all configurations in the average or provide an explicit, pre-specified outlier-removal rule.

minor comments (4)

[§2.1] The notation sum_i |B| is confusing: since |B| does not depend on i, the term should be written as a4 * |B|^2 or a4 * B^2, and the phrase 'captures utilization gain' should be reconciled with the quadratic dependence on batch size.
[§2.1] The paper says the model yields 'four sets of parameters in total' but then reports results across multiple models, GPU types, and tensor-parallel sizes. It should clarify whether the coefficients are refit per configuration or shared across configurations; the latter would make the total parameter count ambiguous.
[§3.1] The evaluation section does not report the number of profiling steps, the number of evaluation batches, or the variance of the latency measurements. Adding this information would help assess the stability of the reported R2 and percentile errors.
[§3.2–3.3] The phrase 'up to 3.5x and 4.4x improvement' is imprecise because the improvements are average improvements computed after excluding outliers. The text should specify whether these are maximum or average improvements across configurations.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity; the additive attribution is an explicit design property and the reported accuracy is not shown to reduce to the fit.

full rationale

LLMVisor's derivation chain is self-contained as a modeling proposal. Equation (1) is explicitly introduced as a linear ansatz with coefficients fitted by OLS from profiling data; the paper does not claim these coefficients are derived from first principles or from the per-request shares it later outputs. The additive per-request attribution is stated as a design property ('The model is additive by design, yielding closed-form per-request latency shares'), and the shares are obtained by reading terms out of the same equation; this is a definitional property of the linear model, not a circular reduction of a prediction to its input. The empirical accuracy claims (R2, p90/p99 relative error) are about batch-latency prediction and are in principle falsifiable against measured step latencies. The main caveat is that Section 3.1 does not explicitly state that Tables 1 and 2 evaluate held-out steps disjoint from the warm-up profiling runs used to fit the coefficients; if the evaluation overlapped the fitting data, the near-perfect R2 would be an in-sample fit property rather than an independent prediction. But the text does not assert such overlap, and no equation or fitted parameter is demonstrably renamed as a prediction. Author self-citations ([8,9,20,21,23]) appear only as related-work and background references, not as load-bearing uniqueness theorems or ansatz justifications. Therefore no circular step can be exhibited; the lack of an explicit train/test split is a reporting/validation gap, not demonstrated circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The model is an empirical regression. Its accuracy and attribution claims rest on 20 fitted coefficients (beta and a1-a4 for four regime segments), unspecified segment boundaries, an additive-linear form, and the unstated premise that evaluation is out-of-sample. No coefficient values, code, or data are provided. No new physical entities are introduced.

free parameters (2)

OLS coefficients (beta, a1, a2, a3, a4) for each of four regime-segment combinations = not reported
The central latency equation is determined entirely by these fitted coefficients; their values, ranges, and the fitting data are not given, so the claimed accuracy cannot be reproduced.
Phase-segment boundary (memory-bound versus compute-bound) for prefill and decode = not reported
The paper says the model is two-segment but does not define how the boundary is chosen or estimated; it is a free modeling choice that affects the fit.

assumptions (4)

domain assumption Batch latency is an additive linear function of the chosen features within each phase segment.
Equation (1) assumes no interaction terms and no nonlinearity beyond p_i^2; this is what makes per-request shares additive, but it is asserted, not derived.
domain assumption The feature sums map to physical costs: sum_i p_i to MLP and attention compute, sum_i c_i to KV-cache I/O, sum_i p_i^2 to quadratic attention, and sum_i |B| to batch effects.
Section 2.1 states these mappings without derivation; the batch-size term in particular has no clear physical basis and equals B squared.
ad hoc to paper The evaluation data are out-of-sample relative to the warm-up profiling runs used to fit the coefficients.
The paper never states this, and if the data overlap, the reported accuracy is just a fit statistic rather than a prediction.
ad hoc to paper The two-segment piecewise split has a known or learnable boundary.
The paper says the model has two segments per phase but never specifies how the boundary is determined; this is an unstated modeling choice.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving." pith.science (2026). https://pith.science/paper/NFXWJCQP

@misc{pith2026260808382,
  author       = {Pith},
  title        = {Pith review of: LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NFXWJCQP}},
  note         = {Machine review of arXiv:2608.08382}
}

read the original abstract

As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control. Enabling fractional sharing of the inference engine requires a real-time, per-request attribution primitive that is accurate and light enough to run inside the scheduling loop. We present LLMVisor, a roofline-guided latency attribution model that captures the memory-bound and compute-bound phases via a concise piecewise-linear form over features proportional to FLOPs and memory I/O traffic. LLMVisor decomposes batch latency into additive, per-request shares and runs efficiently at microsecond scale. We evaluate LLMVisor across Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 GPUs under varying tensor parallelism and workload mixes. Compared to a token-count baseline, LLMVisor attains near-perfect R-squared and reduces relative error by up to 2.5x and 3.3x at p90 and p99, respectively, for prefill, and by up to 3.5x and 4.4x for decode, despite batching variability and sequence divergence.

Figures

Figures reproduced from arXiv: 2608.08382 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages

[1]

Amazon EC2 Instance Types – Burstable Performance Instances (T3)

Amazon Web Services. Amazon EC2 Instance Types – Burstable Performance Instances (T3). Online, 2023

work page 2023
[2]

Anthropic API.https://www.anthropic.com/api, 2025

Anthropic. Anthropic API.https://www.anthropic.com/api, 2025. [Online]

work page 2025
[3]

Xen and the art of virtualization.ACM SIGOPS operating systems review, 37(5):164–177, 2003

Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. Xen and the art of virtualization.ACM SIGOPS operating systems review, 37(5):164–177, 2003

work page 2003
[4]

On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

arXiv 2021
[5]

Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

arXiv 2024
[6]

Predicting llm inference latency: A roofline-driven ml method

Saki Imai, Rina Nakazawa, Marcelo Amaral, Sunyanan Choochotkaew, and Tatsuhiro Chiba. Predicting llm inference latency: A roofline-driven ml method. InAnnual Conference on Neural Information Processing Systems, 2024

work page 2024
[7]

Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

arXiv 2025
[8]

Plato: Plan to efficient decode for large language model inference

Shuowei Jin, Xueshen Liu, Yongji Wu, Haizhong Zheng, Qingzhao Zhang, Atul Prakash, Matthew Lentz, Danyang Zhuo, Feng Qian, and Zhuoqing Mao. Plato: Plan to efficient decode for large language model inference. InSecond Conference on Language Modeling, 2025

work page 2025

Show all 23 references

[9]

Compute or load kv cache? why not both?arXiv preprint arXiv:2410.03065, 2024

Shuowei Jin, Xueshen Liu, Qingzhao Zhang, and Z Morley Mao. Compute or load kv cache? why not both?arXiv preprint arXiv:2410.03065, 2024

2024 arXiv
[10]

s3: Increasing gpu utilization during generative inference for higher throughput.Advances in Neural Information Processing Systems, 36:18015–18027, 2023

Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. s3: Increasing gpu utilization during generative inference for higher throughput.Advances in Neural Information Processing Systems, 36:18015–18027, 2023

2023
[11]

Gonzalez, Hao Zhang, and Ion Stoica

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th ACM Symposium on Operating Systems Princip...

2023 arXiv
[12]

Cgroups.Available on-line at: http://www

Paul Menage, Paul Jackson, and Christoph Lameter. Cgroups.Available on-line at: http://www. mjmwired. net/kernel/Documentation/cgroups. txt, 2008

2008
[13]

Docker: lightweight linux containers for consistent development and deployment.Linux j, 239(2):2, 2014

Dirk Merkel et al. Docker: lightweight linux containers for consistent development and deployment.Linux j, 239(2):2, 2014

2014
[14]

Introducing llama 3.1: The next generation of open models

Meta. Introducing llama 3.1: The next generation of open models. https://ai.meta.com/ blog/meta-llama-3-1/, July 2024. Accessed: 2025-03-29

2024
[15]

NVIDIA Multi-Instance GPU (MIG) User Guide

NVIDIA Corporation. NVIDIA Multi-Instance GPU (MIG) User Guide. https://docs.nvi dia.com/datacenter/tesla/mig-user-guide/, 2025. [Online]

2025
[16]

OpenAI API.https://openai.com/api/, 2025

OpenAI. OpenAI API.https://openai.com/api/, 2025. [Online]

2025
[17]

Fairness in serving large language models

Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E Gonzalez, and Ion Stoica. Fairness in serving large language models. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 965–988, 2024

2024
[18]

Qwen2.5: A party of foundation models, September 2024

Qwen Team. Qwen2.5: A party of foundation models, September 2024. 5

2024
[19]

Aibrix: Towards scalable, cost-effective large language model inference infrastructure.arXiv preprint arXiv:2504.03648, 2025

The AIBrix Team, Jiaxin Shan, Varun Gupta, Le Xu, Haiyang Shi, Jingyuan Zhang, Ning Wang, Linhui Xu, Rong Kang, Tongping Liu, et al. Aibrix: Towards scalable, cost-effective large language model inference infrastructure.arXiv preprint arXiv:2504.03648, 2025

2025 arXiv
[20]

Hetermoe: Efficient training of mixture-of-experts models on heterogeneous gpus.arXiv preprint arXiv:2504.03871, 2025

Yongji Wu, Xueshen Liu, Shuowei Jin, Ceyu Xu, Feng Qian, Z Morley Mao, Matthew Lentz, Danyang Zhuo, and Ion Stoica. Hetermoe: Efficient training of mixture-of-experts models on heterogeneous gpus.arXiv preprint arXiv:2504.03871, 2025

2025 arXiv
[21]

Rlboost: Harvesting preemptible resources for cost-efficient reinforcement learning on llms.arXiv preprint arXiv:2510.19225, 2025

Yongji Wu, Xueshen Liu, Haizhong Zheng, Juncheng Gu, Beidi Chen, Z Morley Mao, Arvind Krishnamurthy, and Ion Stoica. Rlboost: Harvesting preemptible resources for cost-efficient reinforcement learning on llms.arXiv preprint arXiv:2510.19225, 2025

arXiv
[22]

Llm inference unveiled: Survey and roofline model insights.arXiv preprint arXiv:2402.16363, 2024

Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, et al. Llm inference unveiled: Survey and roofline model insights.arXiv preprint arXiv:2402.16363, 2024

2024 arXiv
[23]

Eagle: Efficient training-free router for multi-llm inference.arXiv preprint arXiv:2409.15518, 2024

Zesen Zhao, Shuowei Jin, and Z Morley Mao. Eagle: Efficient training-free router for multi-llm inference.arXiv preprint arXiv:2409.15518, 2024. 6 A Evaluation Setup Details Our evaluation framework is built on vLLM (v0.7.3) [11], using its internal profiler to capture ground t...

2024 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Amazon EC2 Instance Types – Burstable Performance Instances (T3)

Amazon Web Services. Amazon EC2 Instance Types – Burstable Performance Instances (T3). Online, 2023

work page 2023

[2] [2]

Anthropic API.https://www.anthropic.com/api, 2025

Anthropic. Anthropic API.https://www.anthropic.com/api, 2025. [Online]

work page 2025

[3] [3]

Xen and the art of virtualization.ACM SIGOPS operating systems review, 37(5):164–177, 2003

Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. Xen and the art of virtualization.ACM SIGOPS operating systems review, 37(5):164–177, 2003

work page 2003

[4] [4]

On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

arXiv 2021

[5] [5]

Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

arXiv 2024

[6] [6]

Predicting llm inference latency: A roofline-driven ml method

Saki Imai, Rina Nakazawa, Marcelo Amaral, Sunyanan Choochotkaew, and Tatsuhiro Chiba. Predicting llm inference latency: A roofline-driven ml method. InAnnual Conference on Neural Information Processing Systems, 2024

work page 2024

[7] [7]

Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

arXiv 2025

[8] [8]

Plato: Plan to efficient decode for large language model inference

Shuowei Jin, Xueshen Liu, Yongji Wu, Haizhong Zheng, Qingzhao Zhang, Atul Prakash, Matthew Lentz, Danyang Zhuo, Feng Qian, and Zhuoqing Mao. Plato: Plan to efficient decode for large language model inference. InSecond Conference on Language Modeling, 2025

work page 2025