Pith. sign in

REVIEW 3 major objections 5 minor 39 references

The paper claims that LLM serving throughput plateaus because decode-phase attention kernels saturate DRAM bandwidth, not because compute is exhausted.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:15 UTC pith:USFPRWVH

load-bearing objection Kernel-level evidence for attention DRAM-bandwidth saturation is the real contribution; SLIM is a useful engineering model, but the 'no-refit' generalization claim is stronger than the paper's own data support. the 3 major comments →

arxiv 2607.29575 v1 pith:USFPRWVH submitted 2026-07-31 cs.DC cs.PF

SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving

Pol G.Recasens , Ferran Agullo , Yue Zhu , Chen Wang , Jordi Torres , Josep Ll. Berral This is my paper
classification cs.DC cs.PF
keywords LLM servingthroughput plateauattention kernelsDRAM bandwidtharithmetic intensityKV cacheperformance modelingbatching
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to explain why LLM serving throughput stops improving as batch size grows, and turns the usual hand-wavy 'memory bandwidth' explanation into a kernel-level mechanism. Kernel profiling shows the decode-phase attention kernels keep an effectively constant arithmetic intensity of about 3.9 FLOP/byte as batch size, input length, or output length grows; their KV-cache reads therefore scale with the active context until DRAM read bandwidth saturates, while tensor cores sit mostly idle. From that, the paper builds SLIM, a lightweight semi-analytical model with an explicit KV-scan term, plus a batch-cap advisor that frees up to 55 GB of GPU memory for small models. If right, it gives operators a cheap way to find the efficient operating point without exhaustively profiling.

Core claim

The paper's central claim is that DRAM-bandwidth saturation in attention kernels—not compute or batch size itself—is the primary cause of the throughput plateau in large active-context LLM serving. The supporting evidence comes from low-level profiling: across sweeps of batch size, input length, and output length, the attention kernel's arithmetic intensity stays between 3.7 and 4.0 FLOP/byte, its DRAM read utilization approaches 80–100% while tensor-core activity stays below 30%, L1/TEX and L2 hit rates remain under 1% and 7%, and long scoreboard stalls exceed 50% of active warp cycles. Because attention dominates decode time and decode dominates end-to-end time at large active contexts, th

What carries the argument

The load-bearing invariant is the measured arithmetic intensity of decode attention: roughly 3.9 FLOP per byte of DRAM traffic, nearly independent of batch size, input length, or output length, which pins the kernel to the memory-bound side of the roofline and makes its traffic grow with the active context. The paper's model, SLIM, encodes this as an explicit KV-cache scan term, M_dec = 2·L·D_kv·b_kv·C(IL,OL) with C(IL,OL)=OL(IL+1)+OL(OL−1)/2, so traffic scales linearly in input length and quadratically in output length, reduced by grouped-query attention through D_kv and by KV-cache formats through b_kv. Around this, SLIM layers an analytical prefill cost (calibrated tensor-core efficiency)

Load-bearing premise

The load-bearing premise is that one set of calibration constants fitted on small single-GPU models transfers to larger models, grouped-query attention, and multi-GPU deployments through the analytical scalings, although the paper's own tensor-parallel results show this transfer already bends and its model-transfer table records a 6.9% SLO violation.

What would settle it

Serve the same decode workload with an int8 KV cache (halving bytes per token) on the same GPU: SLIM's KV-traffic term predicts the saturation knee shifts to roughly double the active context. If the measured plateau appears at the same batch size and context as with the 16-bit KV cache, then DRAM-bandwidth saturation is not the dominant mechanism the paper claims.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Operating a serving system at the batch cap SLIM identifies keeps throughput near its peak while bounding latency; for the small models evaluated, the advisor frees up to 55 GB of GPU KV-cache reservation.
  • SLIM predicts unseen input and output lengths with under 6% throughput MAPE and an unseen model with 24.3%, from sparse calibration, so the most expensive profiling runs (large models, long generations) can be skipped.
  • Because KV traffic is proportional to per-token bytes (b_kv) and KV width (D_kv), grouped/multi-query attention and KV-cache quantization directly postpone the saturation knee, making them quantitative design levers.
  • The plateau mechanism is architectural, not a quirk of one serving stack, so it should persist on future accelerators as long as compute grows faster than DRAM bandwidth—the paper argues this trend will continue.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the ~3.9 FLOP/byte invariant is set by the attention structure itself, then much larger models should plateau for the same reason; measuring arithmetic intensity of attention kernels across a wider range of model scales and context lengths would be a direct test of that extrapolation.
  • Editorial inference: the paper's tensor-parallel results show the transfer assumption already bends—the model overestimates the throughput gap between the two largest tested models because it omits sharding—so a natural, untested repair is dividing M_dec by the shard count.
  • Editorial inference: the 55 GB of 'freed' memory is a reservation reduction, not reclaimed capacity; realizing it requires scheduling co-located workloads into that space, a step the paper leaves to the runtime.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper makes two main claims. First, using Nsight profiling on H100 GPUs, it argues that the throughput plateau in LLM serving under large active contexts is caused by DRAM-bandwidth saturation in decode-phase attention kernels: attention has nearly constant arithmetic intensity as batch size, input length, or output length grows, so KV-cache traffic scales with computation until memory bandwidth saturates while tensor-core utilization remains low. Second, it introduces SLIM, a semi-analytical performance model with a small set of calibrated constants, and BCA, a batching-configuration advisor built on SLIM. SLIM is evaluated on held-out batch sizes, sequence lengths, and model scales, and is compared with two baselines; BCA is evaluated for recommendation quality, profiling-cost savings, and KV-cache memory savings.

Significance. The kernel-level characterization in Section V is the strongest contribution. If it holds, it converts a previously conceptual bandwidth-saturation explanation into a mechanism supported by low-level metrics: nearly constant arithmetic intensity, low cache hit rates, increasing long-scoreboard stalls, and DRAM utilization near 90% with tensor-core activity below 30%. The ablation in Table IV also gives clear evidence that the KV-scan term in SLIM is doing real work: removing M_dec degrades output-length generalization MAPE from 5.6% to 33.0%. The SLIM model is interpretable and the comparison against LLMVisor-Agg and IMAI OOD-LR shows substantial MAPE reductions. However, the paper's headline generalization claim—'successfully generalizing to previously unseen operating conditions' and 'without refitting'—is only partially supported. Model transfer is evaluated at 24.3% throughput MAPE, tensor-parallel deployments are explicitly not captured, and no code, data, or fitted parameter values are provided. The central mechanism is credible, but the modeling contribution needs either a restricted scope statement or additional evidence before the claims as written can be accepted.

major comments (3)
  1. [Section VI-F3 and Eqs. (8)-(11)] The generalization claim is not supported for tensor-parallel deployments. Equations 8-11 contain no tensor-parallel sharding term: the model scale D and KV width D_kv are treated as monolithic values. Under tensor parallelism the per-GPU workload is approximately D/tp and D_kv/tp, so Qwen-72B on 4 GPUs should be compared to Qwen-32B on 2 GPUs at similar per-GPU scale. Figure 11 shows exactly this: SLIM overestimates the throughput gap between the two models because it treats Qwen-72B as a uniformly larger monolithic model. Since the abstract and introduction claim generalization to unseen deployments, this is a load-bearing gap. The authors should either extend the model with a sharding term or explicitly restrict the generalization claim to single-GPU serving.
  2. [Section VI-F1, Table III, Section VII-B1] The 'no-refit generalization' claim is quantitatively weak. In the model-transfer setting, SLIM achieves 24.3% throughput MAPE and 16.5% E2E-latency MAPE on OPT-6.7B after calibration on OPT-125M through OPT-2.7B (Table III). Table V further shows that when calibrating on OPT-125M alone the BCA decision has a 28.4% throughput discrepancy and a 6.9% SLO violation; adding more training curves leaves a 15.1% discrepancy. The paper itself calls the SLO violation a residual risk in Section VII-B1. No error bars, confidence intervals, or repeated calibrations are reported, and the fitted constants (eta_c, eta_m, B_0, alpha, gamma, t_fixed) are never listed. Without these, the reader cannot distinguish a stable analytical model from a curve fit to the training set. The authors should report the fitted parameter values, show sensitivity to the training subset, and define a target accuracy thresh
  3. [General (reproducibility)] The paper does not ship code, data, or fitted parameter values. The central claims of SLIM and BCA are therefore not independently reproducible. Given that the contribution is a performance model with calibrated constants, the numerical values of those constants are essential scientific output. The authors should provide an artifact or at minimum a table of all fitted parameters and the profiled measurements used for calibration.
minor comments (5)
  1. [Eq. (11)] Equation 11 describes M_dec as 'the total memory traffic generated by KV-cache reads during decoding,' but the formula is per-request; the batch dependence is introduced later through the (B+B_0) factor in Eq. (9). The wording should be adjusted to avoid confusion.
  2. [Figure 5] The x-axis labels '5012' appear twice; this is likely a typo for '512' or '5012' should be consistent with the values used in the text. Please clarify.
  3. [Table I] Table I reports arithmetic intensity to two decimal places and DRAM utilization to one decimal place, but no information is given about the number of profiling runs or run-to-run variance. For a hardware characterization, at least a note on measurement variability is needed.
  4. [Section IV] The hardware is described as 'H100 (64GB HBM2)'. The H100 64GB is typically equipped with HBM3, not HBM2. Please verify and correct.
  5. [Table V] The column 'N' is not explained in the caption. It presumably denotes the number of profiled configurations used for calibration; please state this explicitly.

Circularity Check

0 steps flagged

No significant circularity: SLIM's kernel characterization and held-out generalization are self-contained; the only fitted constants are explicitly calibrated, not renamed predictions.

full rationale

The load-bearing derivation (Section V) is an empirical kernel-level characterization: attention arithmetic intensity remains ~3.9 FLOP/byte across batch/IL/OL sweeps (Table I), DRAM read utilization approaches 90-100% while tensor-core activity stays below 30% (Figure 4), and the roofline trajectory (Figure 6) independently establishes the bandwidth-saturation mechanism. SLIM's equations (7)-(12) combine standard transformer FLOP counts with an analytical KV-traffic term M_dec = 2 L D_kv b_kv C(IL,OL); the saturation shape of Eq. 6 is not a renamed fit because M_dec is derived, not calibrated. The calibrated constants (eta_c, eta_m, B_0, alpha, gamma, t_fixed) are explicitly called calibrated; they are fitted on small OPT/short-OL configurations and tested on held-out OPT-6.7B, OL=512, IL=1024, and Mistral-7B (Tables II-III, Figure 9), so they are not inputs renamed as predictions. The only self-citation, [9] in Section III-C, is a provenance statement for the conference precursor and is not load-bearing. The paper's own limitations (Section VI-F3 tensor-parallel sharding omission; Section VII-B1 residual 6.9% SLO violation; Section VIII single-GPU profiling scope) concern generalization and reproducibility, not circularity, and they are disclosed rather than hidden.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

SLIM and BCA are methods, not new physical entities. The model deploys six calibrated constants not reported in the paper; the analytical skeleton (KV traffic, FLOP counts) is derived, but the calibration constants carry the deployment-specific load.

free parameters (6)
  • η_c (prefill compute efficiency) = not reported
    Calibrated factor multiplying peak tensor throughput in Eq. (7); fit by nonlinear least squares.
  • η_m (decode efficiency multiplier) = not reported
    Calibrated multiplier on the whole decode-time expression in Eq. (9); absorbs runtime overheads.
  • B₀ (small-batch offset) = not reported
    Calibrated offset in Eq. (9) to capture nonlinear decode scaling at small batch sizes.
  • α (per-output-token overhead coefficient) = not reported
    Calibrated coefficient in Eq. (10) capturing decode-step scheduling and lightweight operations.
  • γ (dense decode cost coefficient) = not reported
    Calibrated coefficient multiplying L(D/1024)²OL in Eq. (10); absorbs dense projection/FFN overheads.
  • t_fixed (system overhead constant) = not reported
    Calibrated constant in Eq. (5) capturing scheduling and runtime management.
axioms (5)
  • domain assumption Prefill is compute-bound and can be modeled as F_pre / (η_c F_max).
    Section VI-C, Eq. (7); used to set T_pre without explicit memory traffic.
  • domain assumption The workload can be represented by fixed average IL/OL/B with all requests finishing simultaneously.
    Section VI-A, Eqs. (5)-(6); end-to-end latency formula assumes synchronized batch completion.
  • domain assumption Attention arithmetic intensity is essentially constant across active contexts, so KV traffic scales proportionally with C(IL,OL).
    Measured in Section V; encoded in Eqs. (11)-(12) as the saturation mechanism.
  • domain assumption One set of calibration constants transfers to unseen models through architectural scalings L(D/1024)² and D_kv.
    Section VI-F3; partially contradicted by the Qwen-72B tensor-parallel overestimate and the Table V model-transfer variation.
  • domain assumption H100 peak tensor throughput (989 TFLOP/s) and peak DRAM bandwidth (1.62 TB/s) are the correct hardware ceilings.
    Section IV; used for rooflines and the BW_max/F_max denominators.

pith-pipeline@v1.3.0-daily-deepseek · 21136 in / 14262 out tokens · 153500 ms · 2026-08-03T04:15:54.690443+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM) serving commonly increases batch size to improve throughput, but performance eventually reaches a deployment-dependent plateau beyond which larger batches provide marginal gains while increasing latency and GPU memory consumption. Previous studies have attributed this behavior to HBM/DRAM bandwidth limitations, but the underlying causes have primarily been supported by conceptual arguments or high-level performance observations. As our first contribution, we present a detailed GPU characterization using hardware profiling techniques, demonstrating that throughput saturation originates in the attention kernels during the decode phase. Specifically, we show that their nearly constant arithmetic intensity as active-context lengths increases -not merely larger batch sizes- drives DRAM-bandwidth saturation, while the achieved compute throughput remains far below the hardware limit. Building on this analysis, we present the Batching Configuration Advisor (BCA), which selects the highest-throughput batching configuration satisfying a target latency constraint and identifies up to 55 GB of GPU memory allocation that can be avoided for the evaluated OPT models with minimal throughput loss. To enable these recommendations, we introduce SLIM (Saturation-Aware Lightweight Performance Model), a semi-analytical model that predicts LLM inference throughput and latency from analytical formulations of Transformer computation and memory traffic. Across the evaluated scenarios, SLIM outperforms representative performance-modeling baselines while successfully generalizing to previously unseen operating conditions.

Figures

Figures reproduced from arXiv: 2607.29575 by Chen Wang, Ferran Agullo, Jordi Torres, Josep Ll. Berral, Pol G.Recasens, Yue Zhu.

Figure 1
Figure 1. Figure 1: Throughput (tokens/s) and latency (ms) evolution when [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Throughput evolution under varying input lengths (left) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Whereas prior analyses have primarily examined this [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Breakdown of total inference time into prefill and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Contribution of GPU kernels to the execution time of [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Evolution of average Tensor Core activity, SM [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: L1/TEX and L2 cache hit rates of the attention [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: Roofline characterization of the attention kernels ex [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Percentage of Occupancy and Long Scoreboard Stalls for the attention kernels executed by Mistral-7B during the final decode step under increasing batch size, input length, and output length. E. Concluding Remarks Collectively, these results show that the throughput plateau under large active-context workloads is driven by DRAM￾bandwidth saturation in attention kernels. As the active context increases, thei… view at source ↗
Figure 9
Figure 9. Figure 9: Performance prediction accuracy when evaluating unseen input length, unseen output length, unseen model, and unseen [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Held-out prediction for an unseen model. The esti [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Predicted and measured throughput and end-to-end [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Sensitivity of SLIM-BCA to the throughput-tolerance [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Memory usage distribution for each model size in our [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 12 linked inside Pith

  1. [1]

    The rapid adoption of generative ai,

    A. Bick, A. Blandin, and D. J. Deming, “The rapid adoption of generative ai,” National Bureau of Economic Research, NBER Working Paper 32966, Feb. 2025, revised February 2025. [Online]. Available: https://www.nber.org/papers/w32966

    2025

  2. [2]

    The adoption of chatgpt,

    A. Humlum and E. Vestergaard, “The adoption of chatgpt,” Becker Friedman Institute for Economics, University of Chicago, Working Paper 2024-50, Apr. 2024, available at SSRN. [Online]. Available: https://ssrn.com/abstract=4807516

    2024

  3. [3]

    Quantifying large language model usage in scientific papers,

    W. Liang, Y . Zhang, Z. Wu, H. Lepp, W. Ji, X. Zhao, H. Cao, S. Liu, S. He, Z. Huang, D. Yang, C. Potts, C. D. Manning, and J. Zou, “Quantifying large language model usage in scientific papers,”Nature Human Behaviour, vol. 9, no. 12, pp. 2599–2609, Dec 2025. [Online]. Available: https://doi.org/10.1038/s41562-025-02273-8

    doi:10.1038/s41562-025-02273-8 2025

  4. [4]

    Llumnix: Dynamic scheduling for large language model serving,

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y . Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 173–191. [Online]. Available: https: //www.usenix.org/conference/osdi24/presentation/sun-biao

    2024

  5. [5]

    Sageserve: Optimizing llm serving on cloud data centers with forecast aware auto-scaling,

    S. Jaiswal, K. Jain, Y . Simmhan, A. Parayil, A. Mallick, R. Wang, R. S. Amant, C. Bansal, V . Ruhle, A. Kulkarniet al., “Sageserve: Optimizing llm serving on cloud data centers with forecast aware auto-scaling,” Proceedings of the ACM on Measurement and Analysis of Computing Systems, vol. 9, no. 3, pp. 1–24, 2025

    2025

  6. [6]

    Large language model inference acceleration: A comprehensive hardware perspective,

    J. Li, J. Xu, S. Huang, Y . Chen, W. Li, J. Liu, Y . Lian, J. Pan, L. Ding, H. Zhouet al., “Large language model inference acceleration: A comprehensive hardware perspective,”arXiv preprint arXiv:2410.04466, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Efficiently scaling transformer inference,

    R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,”Proceedings of Machine Learning and Systems, vol. 5, 2023

    2023

  8. [8]

    Efficient llm inference: Bandwidth, compute, synchronization, and capacity are all you need,

    M. Davies, N. Crago, K. Sankaralingam, and C. Kozyrakis, “Efficient llm inference: Bandwidth, compute, synchronization, and capacity are all you need,”arXiv e-prints, pp. arXiv–2507, 2025

    2025

  9. [9]

    Mind the memory gap: Unveiling gpu bot- tlenecks in large-batch llm inference,

    P. G. Recasens, F. Agullo, Y . Zhu, C. Wang, E. K. Lee, O. Tardieu, J. Torres, and J. L. Berral, “Mind the memory gap: Unveiling gpu bot- tlenecks in large-batch llm inference,” in2025 IEEE 18th International Conference on Cloud Computing (CLOUD). IEEE, 2025, pp. 277–287

    2025

  10. [10]

    Llmvisor: A real-time latency attribution model for multi-tenant llm serving,

    S. Jin, X. Liu, J. Shan, L. Xu, T. Zhang, L. Xie, and Z. M. Mao, “Llmvisor: A real-time latency attribution model for multi-tenant llm serving,” 2025

    2025

  11. [11]

    Predicting llm inference latency: A roofline-driven ml method,

    S. Imai, R. Nakazawa, M. Amaral, S. Choochotkaew, and T. Chiba, “Predicting llm inference latency: A roofline-driven ml method,” in Annual Conference on Neural Information Processing Systems, 2024

    2024

  12. [13]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  13. [14]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  14. [15]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  15. [16]

    Orca: A distributed serving system for transformer-based generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implemen- tation (OSDI 22), 2022, pp. 521–538

    2022

  16. [17]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 611–626

    2023

  17. [18]

    TensorRT-LLM,

    NVIDIA, “TensorRT-LLM,” GitHub repository, 2023–2025. [Online]. Available: https://github.com/NVIDIA/TensorRT-LLM

    2023

  18. [19]

    DeepSpeed-MII,

    Microsoft, “DeepSpeed-MII,” GitHub repository, 2022–2025. [Online]. Available: https://github.com/deepspeedai/DeepSpeed-MII

    2022

  19. [20]

    Slora: Scalable serving of thousands of lora adapters,

    Y . Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzeret al., “Slora: Scalable serving of thousands of lora adapters,”Proceedings of Machine Learning and Systems, vol. 6, pp. 296–311, 2024

    2024

  20. [21]

    Llm inference unveiled: Survey and roofline model insights,

    Z. Yuan, Y . Shang, Y . Zhou, Z. Dong, Z. Zhou, C. Xue, B. Wu, Z. Li, Q. Gu, Y . J. Leeet al., “Llm inference unveiled: Survey and roofline model insights,”arXiv preprint arXiv:2402.16363, 2024

    Pith/arXiv arXiv 2024

  21. [22]

    Taming{Throughput-Latency}tradeoff in{LLM}inference with{Sarathi-Serve},

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming{Throughput-Latency}tradeoff in{LLM}inference with{Sarathi-Serve},” in18th USENIX symposium on operating systems design and implementation (OSDI 24), 2024, pp. 117–134

    2024

  22. [23]

    Fast inference from transform- ers via speculative decoding,

    Y . Leviathan, M. Kalman, and Y . Matias, “Fast inference from transform- ers via speculative decoding,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 19 274–19 286

    2023

  23. [24]

    Accelerating large language model decoding with speculative sam- pling,

    C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sam- pling,”arXiv preprint arXiv:2302.01318, 2023

    Pith/arXiv arXiv 2023

  24. [25]

    Fast transformer decoding: One write-head is all you need,

    N. Shazeer, “Fast transformer decoding: One write-head is all you need,” arXiv preprint arXiv:1911.02150, 2019

    Pith/arXiv arXiv 1911

  25. [26]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints,

    J. Ainslie, J. Lee-Thorp, M. de Jong, Y . Zemlyanskiy, F. Lebr ´on, and S. Sanghai, “Gqa: Training generalized multi-query transformer models from multi-head checkpoints,”arXiv preprint arXiv:2305.13245, 2023

    Pith/arXiv arXiv 2023

  26. [27]

    Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,”Proceedings of machine learning and systems, vol. 6, pp. 87–100, 2024

    2024

  27. [28]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,”arXiv preprint arXiv:2210.17323, 2022

    Pith/arXiv arXiv 2022

  28. [29]

    Vidur: A large-scale simulation framework for llm inference,

    A. Agrawal, N. Kedia, J. Mohan, A. Panwar, N. Kwatra, B. S. Gula- vani, R. Ramjee, and A. Tumanov, “Vidur: A large-scale simulation framework for llm inference,” inProceedings of Machine Learning and Systems, 2024

    2024

  29. [30]

    Demystifying ai platform design for distributed inference of next-generation llm models,

    A. Bambhaniya, R. Raj, G. Jeong, S. Kundu, S. Srinivasan, S. Subra- manian, M. Elavazhagan, M. Kumar, and T. Krishna, “Demystifying ai platform design for distributed inference of next-generation llm models,” arXiv preprint arXiv:2406.01698, 2024

    Pith/arXiv arXiv 2024

  30. [31]

    Llmcompass: Enabling efficient hardware design for large language model inference,

    H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff, “Llmcompass: Enabling efficient hardware design for large language model inference,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 1080–1096

    2024

  31. [32]

    Amali: An analytical model for accurately modeling llm inference on modern gpus,

    S. Cao, J. Wu, J. Chen, H. An, and Z. Yu, “Amali: An analytical model for accurately modeling llm inference on modern gpus,” inProceedings of the 52nd Annual International Symposium on Computer Architecture, 2025, pp. 1495–1508

    2025

  32. [33]

    Fairness in serving large language models,

    Y . Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Fairness in serving large language models,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 965–988

    2024

  33. [34]

    Clean sharegpt dataset,

    anon8231489123, “Clean sharegpt dataset,” 2023. [Online]. Available: https://huggingface.co/datasets/anon8231489123/ShareGPT Vicuna unfiltered

    2023

  34. [35]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06825

    Pith/arXiv arXiv 2023

  35. [36]

    Granite code models: A family of open foundation models for code intelligence,

    M. Mishra, M. Stallone, G. Zhang, Y . Shen, A. Prasad, A. M. Soria, M. Merler, P. Selvam, S. Surendran, S. Singhet al., “Granite code models: A family of open foundation models for code intelligence,” arXiv preprint arXiv:2405.04324, 2024

    Pith/arXiv arXiv 2024

  36. [37]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,”

  37. [38]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W....

    Pith/arXiv arXiv 2024

  38. [39]

    Ai and memory wall,

    A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer, “Ai and memory wall,”IEEE Micro, vol. 44, no. 3, pp. 33–39, 2024

    2024

  39. [2022]

    Available: https://arxiv.org/abs/2205.01068

    [Online]. Available: https://arxiv.org/abs/2205.01068

    Pith/arXiv arXiv