Pith. sign in

REVIEW 4 major objections 5 minor 40 references

AFlex shows that assigning Attention and FFN separate GPU pools with independent clock speeds cuts LLM serving energy per token by up to 49% while keeping P90 TTFT and TPOT within SLOs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 18:44 UTC pith:5OTQOWUY

load-bearing objection Real systems contribution with genuine novelty, but the headline energy savings are measured only against P90 SLOs and need P99 sensitivity before I'd sign off. the 4 major comments →

arxiv 2608.01891 v1 pith:5OTQOWUY submitted 2026-08-03 cs.DC cs.AI

Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling

Cunchen Hu , Liangliang Xu , Tian Liu , Min Lyu , Yongkun Li , Sa Wang , Shuo Quan , Yanan Yang
show 4 more authors
Wenda Tang Yiduo Wang Fu Yu Jie Wu
This is my paper
classification cs.DC cs.AI
keywords LLM servingenergy efficiencyDVFSAttention-FFN disaggregationGPU frequency scalingSLO-aware schedulinginterleaved pipelinedynamic microbatch depth
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that running every GPU operator at a single shared frequency—or even one frequency per inference phase—wastes energy, because Attention and feed-forward network (FFN) operators respond very differently to clock speed. To prove the point, the authors build AFlex, a serving framework that splits Attention and FFN onto separate GPU pools, gives each pool its own dynamic frequency, and schedules resources with a global optimizer. Across production workloads on dense and mixture-of-experts models, AFlex reduces energy per token by up to 49% relative to state-of-the-art disaggregated serving and 48% relative to frequency-scaling systems, while keeping P90 time-to-first-token and per-output-token latencies within their SLOs. The practical stake is that datacenter LLM serving could cut a large fraction of its GPU energy bill without asking users to accept slower responses.

Core claim

The central discovery is that Attention and FFN operators have distinct, workload-dependent frequency sensitivities: over the profiled configurations, Attention reaches its energy-optimal frequency on average 429 MHz below FFN, and a single shared frequency cannot match both operators' optima. AFlex exploits this by formalizing four operator-phase pools (prefill Attention, prefill FFN, decode Attention, decode FFN), each with its own GPU allocation, tensor-parallel degree, and frequency. A global scheduler solves an integer linear program to select these configurations under TTFT and TPOT constraints, while a local controller adapts frequencies at runtime, and an interleaved A/F pipeline wit

What carries the argument

The load-bearing mechanism is a two-level control plane plus a pipelined data plane. The global scheduler uses an offline latency/energy predictor and an ILP to allocate GPU counts, tensor-parallel degrees, and baseline frequencies to the four operator-phase pools; the local DVFS controller refines frequencies at runtime, using window-based amortization for decode frequency switches. The data plane runs an interleaved Attention–FFN pipeline that overlaps hidden-state transfers with computation and tunes microbatch depth and request batch size to reduce bubbles. Together these let AFlex lower the frequency of the less latency-sensitive operator (typically Attention) without violating per-requ

Load-bearing premise

The evaluation defines SLO compliance at the P90 percentile of TTFT and TPOT; if the SLO must be met at a stricter tail such as P99, the latency slack that AFlex trades for energy savings narrows, so the claimed energy reductions may not hold.

What would settle it

Re-run the same workloads and baselines on the same hardware with P99 TTFT and TPOT SLOs (instead of P90). If AFlex's energy-per-token advantage over the disaggregated and frequency-scaling baselines drops below roughly 10% (or to zero), the central claim that operator-level DVFS yields large SLO-safe savings is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Energy per token can drop by up to 49% relative to state-of-the-art disaggregated serving and 48% relative to frequency-scaling systems while keeping P90 TTFT and TPOT within SLOs.
  • Operator-level disaggregation and DVFS are complementary: the global scheduler alone (without local DVFS) accounts for a large share of savings, and local DVFS adds up to a further 20–47% depending on workload.
  • The interleaved A/F pipeline with dynamic microbatch depth and adaptive batching reduces pipeline bubbles by 8–65% and per-layer latency by up to roughly 38%, which is what makes the frequency savings SLO-safe.
  • The approach generalizes across dense and mixture-of-experts models and up to 32 GPUs, with energy savings over the best baseline remaining at least about 20%.
  • The system adds negligible planning overhead (ILP solves in under 0.6 seconds, less than 0.2% of the scheduling window) and reduces TP reconfiguration time by 52–72% via incremental weight resharding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported savings are anchored to P90 latency SLOs; if operators require P99 or stricter tail guarantees, the available frequency slack shrinks and the 49% and 48% numbers are likely optimistic.
  • The operator-split idea may extend to finer-grained execution units (e.g., expert layers in MoE models) wherever frequency sensitivity differs, potentially yielding additional savings on models like Mixtral.
  • The 5-minute global replanning window means workloads that shift faster than that would rely on the local DVFS controller alone; a learned or online planner could close that gap.
  • The quantitative results are measured on A800 GPUs; on hardware with a narrower DVFS range or a different compute/memory balance, the magnitude of savings would change, though the qualitative principle—match frequency to the bottleneck operator—should carry over.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents AFlex, a framework for energy-efficient LLM serving that disaggregates attention (A) and feed-forward network (F) execution into separate GPU pools and independently scales GPU frequencies for the two operators across prefill and decode. A global resource controller solves an ILP (Eq. 1) to allocate A/F pairs, TP degrees, and baseline frequencies; a local DVFS controller (Alg. 1) adjusts frequencies at runtime; an interleaved A/F pipeline with dynamic microbatch depth and adaptive batching reduces bubbles. The system is implemented in SGLang and evaluated on A800 GPUs with Qwen3-32B and Mixtral-8×7B under Azure Conversation/Coding traces and controlled workloads. The reported results show up to 49% energy-per-token reduction over DistServe and 48% over DynamoLLM while meeting P90 TTFT (400 ms) and TPOT (120 ms) SLOs.

Significance. If the quantitative claims hold, AFlex is a meaningful advance over phase-level DVFS for LLM serving: it exposes operator-level frequency sensitivity and demonstrates measured, end-to-end energy savings on real hardware. The strengths of the paper include the full system implementation, the clear ILP formulation, the ablation isolating global scheduling and local DVFS, the low measured overheads, and the high predictor accuracy (R²≥0.97). The main caveats are that SLO compliance is evaluated at P90 rather than a stricter tail percentile, that two recent frequency-scaling baselines are omitted, and that all measurements are single runs without variance. These issues are fixable and do not invalidate the design, but they currently limit the strength of the headline claims.

major comments (4)
  1. [§VI-A, §IV-D] SLO compliance is defined at the P90 percentile (400 ms TTFT, 120 ms TPOT), and the local DVFS controller uses exactly these targets to select the lowest-energy feasible frequency pair: the prefill budget is s = min_r(TTFTSLO(r) - Queue(r)) and Alg. 1 constructs F_τ = {f' : t̂(D,M,f') ≤ τ}. The reported P90 headroom is about 25% for TTFT (≤302 vs 400 ms) and 19% for TPOT (≤97 vs 120 ms). Since the energy savings come from exploiting this slack, tightening the SLO percentile to P99 or P999 will shrink F_τ and likely reduce the savings. The paper does not report P99/P999 latencies or energy-per-token under stricter SLOs. This is load-bearing: the abstract's 'stringent SLOs' and the SLO-preserving claim depend on the percentile choice. Please add a tail analysis, e.g., report P99/P999 latencies and re-run under P99 SLOs, or explicitly justify P90 for the target workloads.
  2. [§VI-A] The baseline set omits throttLL'eM [14] and GreenLLM [16], both of which are SLO-aware DVFS systems for LLM serving. The abstract and conclusion claim up to 48% reduction 'over frequency-scaling systems,' but the evaluation only includes DynamoLLM and BiScale. This overstates the state of the art. Please add these baselines or narrow the claim to 'over the evaluated baselines.'
  3. [§VI] All quantitative results are single point estimates; no error bars, confidence intervals, or repetition counts are reported. NVML energy measurements are subject to run-to-run variability (DVFS transitions, temperature, system noise). To support the quantitative margins (e.g., 19.6%–46.8% over the best baseline), the authors should run each configuration multiple times and report mean±std (or box plots). Without this, the headline energy savings may not be statistically robust.
  4. [§IV-D, §VI] The design introduces several tunable parameters that are not studied: capacity margin ε (Eq. 1f), batch-size change threshold δ_b and window constants κ/W_min (Alg. 1), and the workload-monitor thresholds (§IV-B). The decode DVFS controller's switching frequency is directly governed by δ_b and κ, and these values are not justified or ablated. A sensitivity analysis over these parameters is needed to show the results are not tuned to a single set of constants.
minor comments (5)
  1. [§IV-C, Eq. (2)] The overlap model assumes communication is perfectly amortized as t_comm/M for M≥2. Please provide a derivation or measurement of overlap efficiency, particularly for cross-node A/F transfers.
  2. [Figures 4, 5, 12] Several panels need better captions. Fig. 5 combines two different units (s and ms) in one panel; Fig. 12's red dashed line is not explained in the caption; Fig. 4's relative-range color scales are not labeled.
  3. [§VI-B] The sentence 'AFlex reduces energy per token by 19.6%–46.8% relative to the best-performing baseline' should be tied to the specific workloads/RPS and reconciled with the 49%/48% numbers reported for individual baselines.
  4. [§IV] The 'first' claim for combining operator-disaggregated execution with independent frequency scaling would benefit from a brief distinction from operator-disaggregation systems [34], [40] that do not control frequency.
  5. [References] Reference [3] is a generic GitHub URL; please cite a specific version or dataset identifier so the traces are reproducible.

Circularity Check

0 steps flagged

No significant circularity: AFlex's energy savings are measured, and its DVFS decisions are validated on held-out configurations.

full rationale

The paper's central claim—up to 49% energy-per-token reduction under TTFT/TPOT SLOs—is supported by direct measurements on NVIDIA A800 GPUs using NVML, not by re-deriving the headline number from the paper's own predictor. The offline profiling and gradient-boosted predictors (§IV-A) are used to guide the ILP and Local DVFS Controller, but the final energy figures in §VI-B are measured end-to-end. The predictors themselves are tested under 90/10 and 10/90 train/test splits with R² ≥ 0.97 (§VI-D, 'Predictor Accuracy'), so the optimizers are not simply re-fitting the exact evaluation outcomes. The ILP objective (Eq. 1) minimizes profiled energy constants; Algorithm 1 selects from hardware-supported frequency pairs using predicted latency/energy, but does not define the reported savings by construction. The self-citations [12], [13] to the authors' prior work on disaggregation/context caching are background context and are not load-bearing for the energy-efficiency claim. The P90 SLO choice noted in the skeptic analysis is a legitimate evaluation-validity concern—stricter tail percentiles might reduce the reported savings—but it is not a circularity issue under the review rubric; it concerns external validity, not the derivation reducing to its own inputs. No self-definitional, fitted-input-called-prediction, uniqueness-imported, ansatz-smuggled, or renaming patterns are present. The evaluation is self-contained and compares against external baselines, so the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim relies on a set of empirically fitted predictors and hand-chosen thresholds. The predictors are trained on the same hardware used in evaluation, which is standard for systems papers but limits external validity. No new physical or theoretical entities are introduced.

free parameters (5)
  • Capacity margin epsilon
    In Eq. (1f), the throughput constraint requires sustainable throughput >= (1+epsilon)*lambda_c. The value of epsilon is not reported; it is a hand-chosen slack parameter that affects how much overprovisioning the ILP requires.
  • Batch-size change threshold delta_b
    In Algorithm 1, the decode DVFS controller only triggers a new frequency search when the relative batch-size change exceeds delta_b. The paper does not report its value.
  • DVFS window constants kappa and W_min
    The decode DVFS window size is W = max(W_min, round(kappa*T_switch / t_iter)). The values of kappa and W_min are not reported.
  • Workload monitor thresholds (KL divergence, utilization gaps)
    The Workload Monitor triggers reconfiguration on A/F imbalance, P/D skew, or workload shift when thresholds are exceeded; these threshold values are not specified.
  • SLO percentiles and targets = P90 TTFT=400ms, TPOT=120ms
    The evaluation uses P90 latency SLOs chosen by the authors; changing to P99 would likely reduce the achievable frequency slack and energy savings.
axioms (5)
  • domain assumption Offline profiles on the target A800 GPUs accurately predict runtime operator latency and energy under dynamic batching and sequence lengths.
    The Global Scheduler and Local DVFS Controller rely on gradient-boosted predictors trained from offline sweeps (Section IV-A). If profiling conditions differ from live serving (e.g., thermal states, contention), the selected frequencies could miss SLOs or waste energy.
  • domain assumption NVML-reported GPU energy accurately represents total energy consumption.
    Energy per token is computed from NVML counter readings (Section VI-A); the paper does not validate NVML against wall-power meters.
  • domain assumption The Azure public traces are representative of production LLM serving workloads.
    End-to-end claims rely on 'Conversation' and 'Coding' traces from the Azure Public Dataset [3]. The paper does not characterize the traces or justify their representativeness.
  • domain assumption P90 is the appropriate latency SLO metric.
    SLOs are defined at the P90 percentile (Section VI-A); this is a design choice that affects how much frequency slack can be exploited.
  • domain assumption The pipeline timing model t_layer = max(t_A, t_F) + t_comm/M for M>=2 is accurate under varying microbatch depth.
    Eq. (2) is used by the DVFS controller to predict layer latency; it assumes perfect overlap between Attention and FFN stages and communication amortization proportional to 1/M.

pith-pipeline@v1.3.0-daily-deepseek · 19213 in / 15245 out tokens · 152400 ms · 2026-08-04T18:44:30.939666+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling." pith.science (2026). https://pith.science/paper/5OTQOWUY

@misc{pith2026260801891,
  author       = {Pith},
  title        = {Pith review of: Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5OTQOWUY}},
  note         = {Machine review of arXiv:2608.01891}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language model (LLM) serving spans diverse applications with stringent service-level objectives (SLOs), often requiring GPUs to run at maximum frequencies and increasing energy consumption. Existing energy-management approaches adapt GPU frequencies only at the request or inference-phase level, overlooking operator-level differences in frequency sensitivity between Attention and feed-forward networks (FFNs). We find that the energy-optimal frequencies of Attention and FFN (A/F) differ and vary with the inference phase, workload, and system configurations. However, runtime variability and independent A/F frequency control create a large search space and high communication overhead. To address these challenges, we present AFlex, a framework that jointly optimizes resource provisioning and GPU frequency scaling for disaggregated A/F serving. AFlex introduces a global scheduler and a local operator-level dynamic voltage and frequency scaling (DVFS) controller to determine A/F resource allocations and frequencies. It further introduces an interleaved A/F pipeline with dynamic microbatch depth and adaptive request batching to reduce pipeline bubbles. We implement AFlex in SGLang and evaluate it on NVIDIA A800 GPUs using Qwen3-32B and Mixtral-8$\times$7B under production Conversation and Coding traces. \AFlex reduces energy per token by up to 49\% over state-of-the-art disaggregated serving and 48\% over frequency-scaling systems while satisfying TTFT and TPOT SLOs.

Figures

Figures reproduced from arXiv: 2608.01891 by Cunchen Hu, Fu Yu, Jie Wu, Liangliang Xu, Min Lyu, Sa Wang, Shuo Quan, Tian Liu, Wenda Tang, Yanan Yang, Yiduo Wang, Yongkun Li.

Figure 2
Figure 2. Figure 2: A/F frequency sensitivity: (a) Normalized latency and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Frequency sensitivity across decode batch sizes and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: A/F configuration overhead. (a) Instance startup over [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Independent A/F TP reconfiguration with incremental [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: The architecture of AFlex. First, the profiling and prediction module performs offline profiling and generates an energy table and a performance predictor. Second, the global resource controller monitors workload changes and adjusts resource provisioning based on these models. It periodically solves a joint ILP to determine the GPU allocation, TP degree, and baseline frequencies for each operator pool unde… view at source ↗
Figure 8
Figure 8. Figure 8: Placement-aware hidden-state transfer in AFlex: (a) [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The interleaved A/F pipeline in AFlex. AFlex adapts [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: End-to-end performance under the Azure Conversa [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: GPU frequency timelines for Coding trace on 16 [PITH_FULL_IMAGE:figures/full_fig_p009_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: Energy per token scaling from one to four nodes at [PITH_FULL_IMAGE:figures/full_fig_p010_14.png] view at source ↗
Figure 16
Figure 16. Figure 16: Benefits of Global Scheduler and Local DVFS. [PITH_FULL_IMAGE:figures/full_fig_p011_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 9 linked inside Pith

  1. [1]

    Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve,” in18th USENIX symposium on operating systems design and implementation (OSDI 24), 2024, pp. 117– 134

    2024

  2. [2]

    SARATHI: Efficient LLM inference by piggybacking decodes with chunked prefills,

    A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee, “SARATHI: Efficient LLM inference by piggybacking decodes with chunked prefills,”arXiv preprint arXiv:2308.16369, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Azure Public Dataset,

    Azure, “Azure Public Dataset,” https://github.com/Azure/ AzurePublicDataset, 2025

    2025

  4. [4]

    Biscale: Energy-efficient disaggregated llm serving via phase-aware placement and dvfs,

    O. Basit, Y . Liu, Z. J. Kong, and Y . C. Hu, “Biscale: Energy-efficient disaggregated llm serving via phase-aware placement and dvfs,”arXiv e-prints, pp. arXiv–2602, 2026

    2026

  5. [5]

    Reducing the carbon impact of generative ai inference (today and in 2035),

    A. A. Chien, L. Lin, H. Nguyen, V . Rao, T. Sharma, and R. Wijayawar- dana, “Reducing the carbon impact of generative ai inference (today and in 2035),” inProceedings of the 2nd workshop on sustainable computer systems, 2023, pp. 1–7

    2035

  6. [6]

    Copilot,

    Copilot, “Copilot,” https://github.com/features/, 2023

    2023

  7. [7]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” inInternational Conference on Learning Represen- tations, vol. 2024, 2024, pp. 35 549–35 562

    2024

  8. [8]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” inAdvances in neural information processing systems, 2022

    2022

  9. [9]

    Context Caching,

    Google, “Context Caching,” https://ai.google.dev/gemini- api/docs/caching?lang=python, 2024

    2024

  10. [10]

    Flashdecoding++: Faster large language model inference on gpus,

    K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, K. Chen, H. Dong, and Y . Wang, “Flashdecoding++: Faster large language model inference on gpus,”arXiv preprint arXiv:2311.01282, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Smoothoperator: Reducing power fragmentation and improving power utilization in large-scale dat- acenters,

    C.-H. Hsu, Q. Deng, J. Mars, and L. Tang, “Smoothoperator: Reducing power fragmentation and improving power utilization in large-scale dat- acenters,” inProceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, 2018, pp. 535–548

    2018

  12. [12]

    Memserve: Context caching for disaggregated llm serving with elastic memory pool,

    C. Hu, H. Huang, J. Hu, J. Xu, X. Chen, T. Xie, C. Wang, S. Wang, Y . Bao, N. Sunet al., “Memserve: Context caching for disaggregated llm serving with elastic memory pool,”arXiv preprint arXiv:2406.17565, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Inference without interference: Disaggre- gate llm inference for mixed downstream workloads,

    C. Hu, H. Huang, L. Xu, X. Chen, J. Xu, S. Chen, H. Feng, C. Wang, S. Wang, Y . Baoet al., “Inference without interference: Disaggre- gate llm inference for mixed downstream workloads,”arXiv preprint arXiv:2401.11181, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    throttll’em: Predictive gpu throttling for energy efficient llm inference serving,

    A. K. Kakolyris, D. Masouros, P. Vavaroutsos, S. Xydis, and D. Soudris, “throttll’em: Predictive gpu throttling for energy efficient llm inference serving,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1363–1378

    2025

  15. [15]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, 2023

    2023

  16. [16]

    Greenllm: Slo-aware dynamic frequency scaling for energy-efficient llm serving,

    Q. Liu, D. Huang, M. Zapater, and D. Atienza, “Greenllm: Slo-aware dynamic frequency scaling for energy-efficient llm serving,” inProceed- ings of the 63rd ACM/IEEE Design Automation Conference (DAC ’26). New York, NY , USA: ACM, 2026, p. 7

    2026

  17. [17]

    Lmcache: An efficient kv cache layer for enterprise-scale llm inference,

    Y . Liu, Y . Cheng, J. Yao, Y . An, X. Chen, S. Feng, Y . Huang, S. Shen, R. Zhang, K. Duet al., “Lmcache: An efficient kv cache layer for enterprise-scale llm inference,”arXiv preprint arXiv:2510.09665, 2025

    arXiv 2025

  18. [18]

    The Mixtral-8x7B Large Language Model,

    Mistral AI, “The Mixtral-8x7B Large Language Model,” https:// huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1, 2024

    2024

  19. [19]

    NVIDIA NVML API,

    NVIDIA, “NVIDIA NVML API,” https://docs.nvidia.com/deploy/nvml- api/nvml-api-reference.html, 2011

    2011

  20. [20]

    TensorRT-LLM,

    NVIDIA, “TensorRT-LLM,” https://github.com/NVIDIA/TensorRT- LLM, 2023

    2023

  21. [21]

    Chatgpt,

    OpenAI, “Chatgpt,” Large language model, 2023, mar 14 version. [Online]. Available: https://chat.openai.com

    2023

  22. [22]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, ´I. Goiri, A. Shah, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,”arXiv preprint arXiv:2311.18677, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Characterizing power management opportunities for llms in the cloud,

    P. Patel, E. Choukse, C. Zhang, ´I. Goiri, B. Warrier, N. Mahalingam, and R. Bianchini, “Characterizing power management opportunities for llms in the cloud,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, 2024, pp. 207–222

    2024

  24. [24]

    Scikit-learn: Machine learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay, “Scikit-learn: Machine learning in Python,”Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011

    2011

  25. [25]

    A study of generative large language model for medical research and healthcare,

    C. Peng, X. Yang, A. Chen, K. E. Smith, N. PourNejatian, A. B. Costa, C. Martin, M. G. Flores, Y . Zhang, T. Magocet al., “A study of generative large language model for medical research and healthcare,” NPJ digital medicine, vol. 6, no. 1, p. 210, 2023

    2023

  26. [26]

    Mooncake: A kvcache-centric disaggregated architecture for llm serving,

    R. Qin, Z. Li, W. He, J. Cui, H. Tang, F. Ren, T. Ma, S. Cai, Y . Zhang, M. Zhanget al., “Mooncake: A kvcache-centric disaggregated architecture for llm serving,”ACM Transactions on Storage, 2024

    2024

  27. [27]

    Accelerating retrieval-augmented generation,

    D. Quinn, M. Nouri, N. Patel, J. Salihu, A. Salemi, S. Lee, H. Za- mani, and M. Alian, “Accelerating retrieval-augmented generation,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, 2025, pp. 15–32

    2025

  28. [28]

    Flexgen: High-throughput generative inference of large language models with a single gpu,

    Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” inInternational Conference on Machine Learning, 2023

    2023

  29. [29]

    Towards greener llms: Bringing energy-efficiency to the forefront of llm infer- ence,

    J. Stojkovic, E. Choukse, C. Zhang, I. Goiri, and J. Torrellas, “Towards greener llms: Bringing energy-efficiency to the forefront of llm infer- ence,”arXiv preprint arXiv:2403.20306, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Dy- namollm: Designing llm inference clusters for performance and energy efficiency,

    J. Stojkovic, C. Zhang, ´I. Goiri, J. Torrellas, and E. Choukse, “Dy- namollm: Designing llm inference clusters for performance and energy efficiency,” in2025 IEEE International Symposium on High Perfor- mance Computer Architecture (HPCA). IEEE, 2025, pp. 1348–1362

    2025

  31. [31]

    Qwen3 technical report,

    Q. Team, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  32. [32]

    UCX, https://github.com/rapidsai/ucx-py, 2015

    2015

  33. [33]

    Fast distributed inference serving for large language models,

    B. Wu, Y . Zhong, Z. Zhang, G. Huang, X. Liu, and X. Jin, “Fast distributed inference serving for large language models,”arXiv preprint arXiv:2305.05920, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    xdeepserve: Model-as-a-service on huawei cloudmatrix384,

    A. Xiao, B. He, B. Zhang, B. Huai, B. Wang, B. Wang, B. Xu, B. Hou, C. Yang, C. Liuet al., “xdeepserve: Model-as-a-service on huawei cloudmatrix384,”arXiv preprint arXiv:2508.02520, 2025

    arXiv 2025

  35. [35]

    Orca: A distributed serving system for{Transformer-Based}generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for{Transformer-Based}generative models,” in16th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 22), 2022

    2022

  36. [36]

    Flashattention-4: Algorithm and kernel pipelining co-design for asym- metric hardware scaling,

    T. Zadouri, M. Hoehnerbach, J. Shah, T. Liu, V . Thakkar, and T. Dao, “Flashattention-4: Algorithm and kernel pipelining co-design for asym- metric hardware scaling,”arXiv preprint arXiv:2603.05451, 2026

    arXiv 2026

  37. [37]

    ZeroMQ API,

    ZeroMQ, “ZeroMQ API,” https://zeromq.org, 2024

    2024

  38. [38]

    Efficiently programming large language models using sglang,

    L. Zheng, L. Yin, Z. Xie, J. Huang, C. Sun, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalezet al., “Efficiently programming large language models using sglang,”arXiv preprint arXiv:2312.07104, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,” 2024

    2024

  40. [40]

    Megascale-infer: Efficient mixture- of-experts model serving with disaggregated expert parallelism,

    R. Zhu, Z. Jiang, C. Jin, P. Wu, C. A. Stuardo, D. Wang, X. Zhang, H. Zhou, H. Wei, Y . Chenget al., “Megascale-infer: Efficient mixture- of-experts model serving with disaggregated expert parallelism,” in Proceedings of the ACM SIGCOMM 2025 Conference, 2025, pp. 592– 608. 12

    2025