Pith. sign in

REVIEW 2 major objections 5 minor 101 references

Match LLM requests to the best NPU and cut serving cost 1.43x

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:48 UTC pith:DUOLM5GF

load-bearing objection Solid systems paper with a real TPU prototype backing the cost/SLO claims, but the headline 1.13x energy gain rides on an inferred, partly self-referential simulator power model that needs external validation or honest uncertainty bounds. the 2 major comments →

arxiv 2607.16488 v1 pith:DUOLM5GF submitted 2026-07-17 cs.AR

Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving

Yuqi Xue , Jichuan Chang , Jian Huang This is my paper
classification cs.AR
keywords heterogeneous NPUsLLM servingauto-scalingroofline modelvPod abstractionenergy efficiencycost efficiencySLO satisfaction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the heterogeneous mix of NPU generations accumulating in cloud data centers is an opportunity, not just a headache: no single NPU version is best for all LLM requests, and the best fit depends on whether a request is compute-bound or memory-bound and on whether the operator optimizes energy or money. To exploit this, it proposes NeuScale, an auto-scaling system built on a 'vPod' abstraction that hides the differences between NPU versions behind a small set of hardware parameters. NeuScale uses a lightweight roofline analysis to predict, for each sequence-length bucket, which vPod configuration is Pareto-optimal for latency versus energy/cost, then adjusts both the size of each vPod and the number of vPods at runtime, with fail-over to older chips when the best-fit generation is unavailable. The paper reports that this improves energy efficiency by 1.13x, monetary cost efficiency by 1.43x, and SLO satisfaction by 1.36x over a state-of-the-art auto-scaling baseline. A sympathetic reader would care because the claim, if right, turns the hardware generation gap into a direct lever for cutting LLM serving cost and carbon.

Core claim

The central claim is that energy/cost efficiency of LLM serving is governed by a request's bottleneck resource, and that bottleneck can be read off its arithmetic intensity (FLOPs per byte fetched from HBM). By extending a roofline model to treat 'cost' as energy or money as well as time, NeuScale predicts performance and efficiency for any candidate (NPU version, pod shape, parallelism, batch size) without profiling, keeps the Pareto-optimal allocations for each sequence length in a small database, and at runtime creates, coalesces, and resizes vPods to match demand, migrating requests between vPod groups as their sequence length grows. If this is right, the practical consequence is that ol

What carries the argument

The carrying mechanism is the vPod, a virtualized slice of an NPU pod exposing only its chip count, topology, and per-chip compute, memory, and link parameters. Alongside it, the paper redefines the roofline model as work-done-per-cost, so a chip has three rooflines: FLOP/sec, FLOP/Joule, and FLOP/$. A request's arithmetic intensity places it on those curves, and the intersection predicts which allocations lie on the latency/efficiency Pareto frontier. The roofline's job is to reduce the enormous search space of NPU version x pod shape x parallelism x batch size to a short list of promising candidates; the vPod's job is to make those candidates deployable and dynamically resizable.

Load-bearing premise

The headline efficiency gains depend on the simulated energy model correctly ordering chips by energy efficiency, and several key chip efficiency values are inferred from public data rather than directly measured; the real-hardware validation reports only monetary cost because power is not exposed.

What would settle it

Run the same LLM workloads on the four physical NPU generations and compare per-token energy to the roofline predictions; if the ranking of chips by energy efficiency for prefill or decode differs from the predictions, the allocation logic's premise fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Mapping each request to its best-fit NPU generation improves energy efficiency by 1.13x and monetary cost efficiency by 1.43x on average, while raising SLO satisfaction by 1.36x.
  • Requests of different sequence lengths can be served concurrently by different vPod groups, so fine-grained, sequence-length-aware grouping beats a one-size-fits-all homogeneous pool.
  • When the best-fit NPU generation is unavailable, automated fail-over to an older generation preserves near-100% SLO satisfaction under high load.
  • Reusing older NPUs for 3 to 6 years instead of retiring them every 1 to 2 years cuts lifetime carbon emissions by amortizing embodied carbon.
  • The framework can absorb new NPU generations by registering their parameters, and prefill and decode phases can be pinned to differently specialized chips.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the roofline's rank accuracy (Spearman correlation above 0.97) holds for future hardware, the same vChipConfig-style parameterization could plausibly be applied to GPUs and other accelerators, making the vPod a general interface for heterogeneous AI chips.
  • The cost model assumes flat chip-hour pricing; if cloud pricing becomes dynamic or carbon-aware, the same allocation machinery could optimize directly for those price signals, potentially widening the reported gains.
  • The framework's value scales with output-sequence-length prediction accuracy: the paper shows savings grow from 12% at 60% accuracy to 33% at perfect accuracy, so better predictors would compound the benefit.
  • Because the real-chip validation measures only monetary cost, the 1.13x energy claim awaits direct power measurement on physical NPUs; that measurement is a natural next step for cloud providers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents NeuScale, an auto-scaling framework for heterogeneous NPU clusters that serve LLM inference requests. It introduces a vPod abstraction over NPU pod configurations, an offline design-space exploration that derives arithmetic intensity from the ML compiler and applies roofline models to predict latency, energy, and monetary cost, and a runtime controller that creates, resizes, coalesces, and fails over vPod groups by sequence length. The evaluation combines a custom cluster simulator with a real TPU prototype (TPUv4/v5e/v6e) and claims average improvements over a homogeneous auto-scaling baseline of 1.13x energy efficiency, 1.43x monetary cost efficiency, and 1.36x SLO satisfaction on Azure/LVEval/OpenThoughts traces with Llama and DeepSeek models. The execution-time and monetary-cost validation are the most grounded parts of the evaluation; the energy-efficiency claims are primarily simulation-based and are the main concern.

Significance. If the energy and cost claims hold, NeuScale is a useful systems contribution: it is, to the authors' knowledge, the first auto-scaling framework designed for heterogeneous NPUs, and its roofline-based allocation is a lightweight alternative to profiling-heavy approaches. The vPod abstraction is clean and appears compatible with existing ML frameworks, and the paper provides valuable real-TPU measurements of vPod operations, DCN transfer, operator/request latency, and a small real-cluster cost/SLO study. The plan to open-source the simulator is also a strength. However, the headline energy-efficiency benefit is the least secure part of the paper: the energy model depends on inferred chip parameters and the 'actual' energy values used for validation come from the same simulator. The paper's central contribution would be substantially strengthened by concrete external energy validation or by re-scoping the claims.

major comments (2)
  1. [Table 2, §4, Figure 9, §5.7] The headline energy-efficiency claims lack external validation in the manuscript. Table 2 marks Max TFLOP/Joule and Max HBM GB/Joule as inferred; §4 builds the simulator power model from RTL synthesis and public data, saying only in one sentence that it was validated against published TPU data, with no results shown. Figure 9's 'Actual FLOP/Joule' values are produced by the same simulator (per §4), so the >0.97 rank correlation demonstrates self-consistency, not correctness. §5.7 confirms that real TPUs expose no power API, and the real-system evaluation reports only monetary cost. Since the average energy gain over Base is only 1.13x, a moderate systematic error in the inferred per-chip energy efficiencies could reorder vPod allocations and erase the claimed benefit. Please report concrete external validation numbers, add a sensitivity sweep over the Table 2 inferred parameters showing
  2. [§4, Figure 18] DeepSeekV3-671B, the model carrying the main end-to-end results, is simulated with a single fixed MoE load-imbalance factor (the most loaded expert receives 7x the average expert's tokens), taken from an external empirical study. No sensitivity analysis is presented, and the simulator validations in Figures 14–16 are for dense-model operators and Llama3-70B, not for MoE routing. An inaccurate imbalance factor would directly change the latency, energy, and SLO numbers for DeepSeek workloads. Please validate against a real MoE workload or sweep this factor and show that the reported improvements are robust.
minor comments (5)
  1. [Abstract, §1, Table 2] The paper says it 'conducts a characterization study of various generations of real NPU chips,' but the energy parameters in Table 2 are inferred and not directly measured. Please qualify this wording, and add a caption note that Figure 2's energy efficiencies are simulator predictions rather than direct measurements.
  2. [§5.1, Figures 9–10] The term 'Actual' in Figures 9 and 10 is misleading because the values come from the simulator, not from real hardware energy measurements. Consider renaming the axis to 'Simulator' or 'Simulated' to avoid implying external validation.
  3. [§5.2 vs. Abstract] The abstract reports averages of 1.13x/1.43x/1.36x, while §5.2 reports 1.37x/1.73x/1.31x for DeepSeekV3-671B on Azure. State explicitly that the abstract numbers are averaged over all models/traces in §5.3, so readers are not confused by the different figures.
  4. [Table 3, Figure 13] Units are inconsistent: Table 3 uses 'Gbps/chip' for DCN bandwidth while Figure 13 uses 'GB/s', and Table 2 lists DCN BW/chip in Gbps while HBM bandwidth is in GB/s. Please make the bit/byte distinction explicit and uniform.
  5. [§2.4, §5.1] The SLO is set to 5x the single-request latency, which is a loose target. A sentence justifying this choice or a sensitivity experiment with tighter SLO multipliers would help assess how representative the SLO-satisfaction results are.

Circularity Check

1 steps flagged

Energy-efficiency validation is partially self-referential: Table 2's inferred chip energy efficiencies feed both the roofline 'prediction' and the simulator 'actual', so the headline 1.13x energy claim lacks fully independent whole-chip power verification; the cost/SLO claims are independently grounded.

specific steps
  1. fitted input called prediction [Table 2; §3.3 'Roofline prediction accuracy' (Figure 9); §4 simulator power/energy model; §5.7 real-TPU power limitation]
    "Max. TFLOP/Joule (bf16) 1.07* 0.72* 1.16* 3.50* ... '*' means the parameter is not officially disclosed, and it is inferred from public data and our experiments. ... For each request, a backend production-level NPU chip simulator is invoked to get its latency and energy. ... Each point represents the predicted/actual value of an allocation. ... As the cloud TPU stack does not expose a public API for measuring power, we report monetary cost."

    The roofline's energy prediction is built from the same Table 2 per-chip energy efficiencies that are marked '* inferred from public data and our experiments,' while the 'Actual FLOP/Joule' used in Figure 9 comes from the backend simulator whose power/energy model is constructed from those same inferred/public per-chip parameters. Thus the Figure 9 validation checks the roofline against a simulator that already encodes the roofline's own inputs; the high Spearman correlation is partly a self-consistency result rather than an independent confirmation of the energy model. §5.7 confirms that no real-TPU power measurement was available, so the reported 1.13x energy improvement rests on this partially self-referential loop. It is not purely by construction because the simulator adds RTL-derived

full rationale

The paper's central mechanism is not circular in the strict sense: the vPod abstraction, roofline-based allocation, and auto-scaling controller are not defined in terms of the measured outcomes, and the cost-efficiency and SLO claims have independent real-TPU grounding (§5.7, Figures 14-16). The only load-bearing weakness is on the energy side. Table 2's Max TFLOP/Joule and Max HBM GB/Joule are inferred parameters, and the 'actual' energy values against which the roofline is validated in Figure 9 are produced by the same simulator family whose power model is built from those inferred parameters. Since §5.7 states the cloud TPU stack does not expose a power API, the reported 1.13x energy improvement cannot be externally confirmed and could shift if the inferred per-chip energy efficiencies are systematically wrong. This is a partial validation circularity, not a tautology: the simulator contains independent RTL-based component modeling and the latency model is externally validated. No load-bearing self-citation chain or imported uniqueness theorem is present; the co-authored reference [36] supplies only an empirical MoE load-skew heuristic. Accordingly, a moderate score of 4 is appropriate: the energy claim is partly self-referential, while the cost/SLO claims retain independent content.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The central efficiency claims rest on the vPod abstraction (implemented, with measured overheads), the roofline DSE (which depends on HLO cost analysis and inferred NPU energy parameters), and the simulator power model. The main load-bearing free choices are the inferred chip energy efficiencies, the MoE imbalance factor, and several auto-scaling policy constants. No new physical entity is postulated beyond the software vPod abstraction.

free parameters (4)
  • Inferred NPU energy parameters (TFLOP/Joule, HBM GB/Joule) = NPU-A 1.07/11.06; NPU-B 0.72/6.94; NPU-C 1.16/18.51; NPU-D 3.50/13.52
    Marked * in Table 2 as inferred from public data and experiments. Both the energy roofline and the simulator's power model use these parameters, so energy-efficiency predictions are not independent of them.
  • MoE expert-load imbalance factor = 7× tokens on the most loaded expert
    §4: 'based on empirical studies[36], we adjust the factor such that the most loaded expert receives 7× as many tokens as an average expert' — a modeling choice that affects DeepSeek evaluation results.
  • Auto-scaling policy thresholds = coalescing threshold 0.5, monitoring window 30 min, epoch 5 min, max 2× vPod growth per epoch, 30-min halving
    User-configurable constants in §3.4 chosen by hand; they directly affect the efficiency/SLO trade-offs reported in Figures 18–23.
  • SLO multiplier for evaluation = 5× single-request latency on the minimal NPU-C or NPU-D configuration
    Defined in §2.4 and used for all experiments; results are relative to this normalization and would change under different SLO definitions.
axioms (5)
  • domain assumption Roofline model: achievable performance/energy/cost is determined by peak FLOP rate and HBM bandwidth (with ICI where relevant) for each NPU version.
    Used throughout §3.3 to predict allocation efficiency from arithmetic intensity. Ignores microarchitectural effects beyond rooflines.
  • domain assumption XLA HLO cost analysis gives accurate per-operator FLOP and HBM traffic counts for every candidate parallelism/batch configuration.
    The arithmetic-intensity analysis pass in §3.3 depends on this; errors in HLO counts would shift the predicted best-fit allocations.
  • ad hoc to paper Restricting DSE to pod sizes ≤1024 chips, power-of-two topologies, and specific batch-size buckets is sufficient to find optimal or representative allocations.
    Explicit search-space pruning in §3.3 with no proof that excluded configurations are never optimal.
  • domain assumption LLM execution energy is the sum of independently modeled component energies (SA, VU, SRAM, HBM, ICI).
    Power model described in §4; used to produce the headline energy improvements, but not validated against real power measurements.
  • domain assumption Different NPU versions cannot be mixed in one physical pod because their ICI links are incompatible.
    Stated in §3.2; justifies vPod homogeneity and DCN-only communication across NPU versions.
invented entities (1)
  • vPod (virtualized NPU pod) and vChipConfig independent evidence
    purpose: Unified hardware abstraction for allocating and scheduling heterogeneous NPU chips while staying compatible with existing ML frameworks.
    Implemented in the real TPU prototype; vPod creation/deletion and routing overheads are measured on real TPU instances (§4, Table 3).

pith-pipeline@v1.3.0-alltime-deepseek · 31319 in / 13329 out tokens · 142187 ms · 2026-08-01T20:48:33.714370+00:00 · methodology

0 comments
read the original abstract

To meet the ever-increasing computing demands of large language model (LLM) services, modern cloud platforms have widely deployed neural processing units (NPUs). These NPU chips have been developed and evolved at an incredibly fast pace, this inevitably produces heterogeneous compute pools backed by different versions of NPU chips. Unfortunately, due to the lack of system and architecture support for managing NPU heterogeneity in the cloud, it is unclear how to best utilize heterogeneous NPUs to maximize the energy and cost efficiency for LLM services. In this paper, we first conduct a characterization study of various generations of real NPU chips to demonstrate the potential benefits on energy/cost efficiency and performance by utilizing heterogeneous NPU chips. To realize these benefits, we present NeuScale, an auto-scaling framework to automatically exploit heterogeneous NPUs for cloud platforms. NeuScale manages heterogeneous NPU resources with a new vPod abstraction, which abstracts the core hardware parameters of different NPU versions and provides compatibility with existing ML frameworks. It makes the best-fit vPod allocations for different LLM inference requests using an intuitive and lightweight roofline-based analysis. It supports fine-grained dynamic NPU resource provisioning by adjusting both the vPod configuration (i.e., scaling up/down) and the number of vPods (e.g., scaling in/out). To validate the benefits of NeuScale at scale, we implement it with a production-level NPU simulator. Our evaluation with popular LLMs shows that NeuScale can significantly improve cost efficiency and service-level objective (SLO) satisfaction rate by best utilizing heterogeneous NPU resources.

Figures

Figures reproduced from arXiv: 2607.16488 by Jian Huang, Jichuan Chang, Yuqi Xue.

Figure 1
Figure 1. Figure 1: The NPU pod architecture. We draw 2D NPU pods [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Energy and economic cost efficiency of different NPU versions. The legend labels are [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The optimal energy efficiency of LLM requests with [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Optimal energy efficiency vs. latency across NPU [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: System architecture of NeuScale. NPU-C_1x2x2 is an 1x2x2 NPU-C pod. NPU-C_1x1 is a single NPU-C chip. NPU-D_1x2 represents two interconnected NPU-D chips. struct vChipConfig { int flops_per_sec; // peak FLOPS int hbm_bw_GBps; // peak HBM bandwidth int ici_bw_GBps; // peak ICI bandwidth int HBM_capacity_GB; // per-chip HBM capacity int sram_capacity_MB; // per-chip SRAM capacity }; struct vPodConfig { NPUVe… view at source ↗
Figure 6
Figure 6. Figure 6: vPod configuration parameters. 3.1 NeuScale Overview [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Mapping of vPods to different slices of physical NPU [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cost-efficiency roofline models with different cost [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Roofline prediction accuracy for Llama3.1-405B and [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 12
Figure 12. Figure 12: vPod auto-scaling process in NeuScale. 2 Scaling in/out a vPod group. NeuScale determines the vPod count in a group (𝑁𝑝𝑜𝑑 ) based on the peak request rate (𝑅𝑝𝑒𝑎𝑘 ) in the last time window that would be served by this group and the maximum throughput of each vPod (𝑇𝑝𝑜𝑑 ): 𝑁𝑝𝑜𝑑 =  𝑅𝑝𝑒𝑎𝑘 /𝑇𝑝𝑜𝑑  . NeuScale evicts a vPod group if 𝑁𝑝𝑜𝑑 is scaled down to 0. To avoid thrashing, the vPod count can at most double… view at source ↗
Figure 13
Figure 13. Figure 13: P2P data transfer latency and bandwidth between [PITH_FULL_IMAGE:figures/full_fig_p008_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Simulated vs. profiled execution time of represen [PITH_FULL_IMAGE:figures/full_fig_p009_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Simulated vs. profiled latency of a single LLM [PITH_FULL_IMAGE:figures/full_fig_p009_15.png] view at source ↗
Figure 17
Figure 17. Figure 17: Sequence length and request rate of traces. [PITH_FULL_IMAGE:figures/full_fig_p010_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: End-to-end energy/cost efficiency and SLO satis [PITH_FULL_IMAGE:figures/full_fig_p010_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Left/middle columns: energy consumption and SLO satisfaction rate over time. Right column: energy/latency of each [PITH_FULL_IMAGE:figures/full_fig_p011_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Number of vPods in each vPod group over time. [PITH_FULL_IMAGE:figures/full_fig_p011_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Energy efficiency vs. SLO satisfaction rate. Differ [PITH_FULL_IMAGE:figures/full_fig_p011_21.png] view at source ↗
Figure 25
Figure 25. Figure 25: Energy efficiency (normalized to Unlimited) and [PITH_FULL_IMAGE:figures/full_fig_p012_25.png] view at source ↗
Figure 24
Figure 24. Figure 24: Energy efficiency (normalized to “Het”) and SLO [PITH_FULL_IMAGE:figures/full_fig_p012_24.png] view at source ↗
Figure 27
Figure 27. Figure 27: Energy efficiency and SLO satisfaction rate of de [PITH_FULL_IMAGE:figures/full_fig_p013_27.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

101 extracted references · 18 linked inside Pith

  1. [1]

    vllm-project/router,

    “vllm-project/router, ” accessed: 2026-06. [Online]. Available: https://github. com/vllm-project/router

    2026

  2. [2]

    Ahead: A tool for projecting next-generation hardware enhancements on gpu-accelerated sys- tems,

    H. A. Abdelhafez, C. Zimmer, S. S. Vazhkudai, and M. Ripeanu, “Ahead: A tool for projecting next-generation hardware enhancements on gpu-accelerated sys- tems, ” in2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2019, pp. 583–592

    2019

  3. [3]

    Taming throughput-latency tradeoff in llm inference with sarathi-serve,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in llm inference with sarathi-serve, ” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 117–134. [Online]. Available: https://www.usenix.org/c...

    2024

  4. [4]

    Loggp: incorporating long messages into the logp model—one step closer towards a realistic model for parallel computation,

    A. Alexandrov, M. F. Ionescu, K. E. Schauser, and C. Scheiman, “Loggp: incorporating long messages into the logp model—one step closer towards a realistic model for parallel computation, ” inProceedings of the Seventh Annual ACM Symposium on Parallel Algorithms and Architectures, ser. SPAA ’95. New York, NY, USA: Association for Computing Machinery, 1995,...

    arXiv 1995

  5. [5]

    Aws auto scaling,

    Amazon Web Services, Inc., “Aws auto scaling, ” nov 2025. [Online]. Available: https://aws.amazon.com/autoscaling/

    2025

  6. [6]

    Axi high bandwidth memory controller logicore ip product guide (pg276)

    AMD, “Axi high bandwidth memory controller logicore ip product guide (pg276). ” [Online]. Available: https://docs.amd.com/r/en-US/pg276-axi-hbm

  7. [7]

    Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compila- tion,

    J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski, G. Chauhan, A. Chourdia, W. Constable, A. Desmai- son, Z. DeVito, E. Ellison, W. Feng, J. Gong, M. Gschwind, B. Hirsh, S. Huang, K. Kalambarkar, L. Kirsch, M. Lazos, M. Lezcano, Y. Liang, J. Liang, Y. Lu, C. K. Luk, B. Maher, Y. Pan, C. Puhrsch, M. R...

    2024

  8. [8]

    Claude,

    Anthropic, “Claude, ” https://claude.ai, 2024

    2024

  9. [9]

    Amazon ec2 trn1/trn1n architecture,

    AWS, “Amazon ec2 trn1/trn1n architecture, ” nov 2025. [Online]. Available: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about- neuron/arch/neuron-hardware/trn1-arch.html 14 Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving

    2025

  10. [10]

    Disaggregated inference: 18 months later,

    J. Chen, Y. Zhuang, and H. Zhang, “Disaggregated inference: 18 months later, ” nov 2025. [Online]. Available: https://hao-ai-lab.github.io/blogs/distserve-retro/

    2025

  11. [11]

    Slice-level scheduling for high throughput and load balanced llm serving,

    K. Cheng, W. Hu, Z. Wang, H. Peng, J. Li, and S. Zhang, “Slice-level scheduling for high throughput and load balanced llm serving, ”arXiv preprint arXiv:2406.13511, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Asap7: A 7-nm finfet predictive process design kit,

    L. T. Clark, V. Vashishtha, L. Shifren, A. Gujja, S. Sinha, B. Cline, C. Ramamurthy, and G. Yeric, “Asap7: A 7-nm finfet predictive process design kit, ”Microelectronics Journal, vol. 53, pp. 105–115, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S002626921630026X

    2016

  13. [13]

    Deepseek-v3 technical report,

    DeepSeek-AI, “Deepseek-v3 technical report, ” 2025. [Online]. Available: https://arxiv.org/abs/2412.19437

    Pith/arXiv arXiv 2025

  14. [14]

    Design considerations for high bandwidth mem- ory controller

    A. Dhamba and A. V. Kulkarni, “Design considerations for high bandwidth mem- ory controller. ” [Online]. Available: https://www.design-reuse.com/articles/ 41186/design-considerations-for-high-bandwidth-memory-controller.html

  15. [15]

    Nvidia dynamo adds gpu autoscaling, kubernetes automation, and networking optimizations,

    A. Elmeleegy, H. Kim, H. Zhou, I. Dhanani, M. Khadkevich, O. Kahalon, and V. S. Mailthody, “Nvidia dynamo adds gpu autoscaling, kubernetes automation, and networking optimizations, ” may 2025. [Online]. Avail- able: https://developer.nvidia.com/blog/nvidia-dynamo-adds-gpu-autoscaling- kubernetes-automation-and-networking-optimizations/

    2025

  16. [16]

    Switch transformers: Scaling to trillion pa- rameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion pa- rameter models with simple and efficient sparsity, ”Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022

    2022

  17. [17]

    Autoscaling web applications in heterogeneous cloud infrastructures,

    H. Fernandez, G. Pierre, and T. Kielmann, “Autoscaling web applications in heterogeneous cloud infrastructures, ” in2014 IEEE International Conference on Cloud Engineering, 2014, pp. 195–204

    2014

  18. [18]

    {ServerlessLLM}:{Low-Latency} serverless inference for large language mod- els,

    Y. Fu, L. Xue, Y. Huang, A.-O. Brabete, D. Ustiugov, Y. Patel, and L. Mai, “{ServerlessLLM}:{Low-Latency} serverless inference for large language mod- els, ” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 135–153

    2024

  19. [19]

    Github copilot,

    GitHub, “Github copilot, ” https://github.com/features/copilot, 2024

    2024

  20. [20]

    Jetstream

    Google, “Jetstream. ” [Online]. Available: https://github.com/AI-Hypercomputer/ JetStream

  21. [21]

    System architecture - cloud TPU,

    ——, “System architecture - cloud TPU, ” 2022. [Online]. Available: https: //cloud.google.com/tpu/docs/system-architecture-tpu-vm

    2022

  22. [22]

    XLA: Optimizing Compiler for Machine Learning,

    ——, “XLA: Optimizing Compiler for Machine Learning, ” 2023. [Online]. Available: https://www.tensorflow.org/xla

    2023

  23. [23]

    2024 environmental report,

    ——, “2024 environmental report, ” 2024. [Online]. Available: https://www. gstatic.com/gumdrop/sustainability/google-2024-environmental-report.pdf

    2024

  24. [24]

    Configure autoscaling for llm workloads on gpus with google kubernetes engine (gke),

    ——, “Configure autoscaling for llm workloads on gpus with google kubernetes engine (gke), ” nov 2025. [Online]. Available: https://docs.cloud.google.com/ kubernetes-engine/docs/how-to/machine-learning/inference/autoscaling

    2025

  25. [25]

    Configure autoscaling for llm workloads on tpus,

    ——, “Configure autoscaling for llm workloads on tpus, ” aug 2025. [Online]. Available: https://cloud.google.com/kubernetes-engine/docs/how-to/machine- learning/inference/autoscaling-tpu

    2025

  26. [26]

    Google 2025 environmental report,

    Google, “Google 2025 environmental report, ” https://sustainability.google/ reports/google-2025-environmental-report/, 2025, accessed: 2026-02-28

    2025

  27. [27]

    Mlops: Continuous delivery and automation pipelines in machine learning,

    Google, “Mlops: Continuous delivery and automation pipelines in machine learning, ” nov 2025. [Online]. Avail- able: https://docs.cloud.google.com/architecture/mlops-continuous-delivery- and-automation-pipelines-in-machine-learning

    2025

  28. [28]

    Vertex AI platform,

    ——, “Vertex AI platform, ” may 2025. [Online]. Available: https://cloud.google. com/vertex-ai

    2025

  29. [29]

    Cloud TPU pricing,

    ——, “Cloud TPU pricing, ” https://cloud.google.com/tpu/pricing?hl=en, 2026

    2026

  30. [30]

    Gemini,

    Google DeepMind, “Gemini, ” https://gemini.google.com, 2024

    2024

  31. [31]

    Mélange: Cost efficient large language model serving by exploiting gpu heterogeneity,

    T. Griggs, X. Liu, J. Yu, D. Kim, W.-L. Chiang, A. Cheung, and I. Stoica, “Mélange: Cost efficient large language model serving by exploiting gpu heterogeneity, ” 2024. [Online]. Available: https://arxiv.org/abs/2404.14527

    Pith/arXiv arXiv 2024

  32. [32]

    Has-gpu: Efficient hybrid auto-scaling with fine-grained gpu allocation for slo-aware serverless inferences,

    J. Gu, P. Wang, I. D. N. Araya, K. Huang, and M. Gerndt, “Has-gpu: Efficient hybrid auto-scaling with fine-grained gpu allocation for slo-aware serverless inferences, ” 2025. [Online]. Available: https://arxiv.org/abs/2505.01968

    Pith/arXiv arXiv 2025

  33. [33]

    Openthoughts: Data recipes for reasoning models,

    E. Guha, R. Marten, S. Keh, N. Raoof, G. Smyrnis, H. Bansal, M. Nezhurina, J. Mercat, T. Vu, Z. Sprague, A. Suvarna, B. Feuer, L. Chen, Z. Khan, E. Frankel, S. Grover, C. Choi, N. Muennighoff, S. Su, W. Zhao, J. Yang, S. Pimpalgaonkar, K. Sharma, C. C.-J. Ji, Y. Deng, S. Pratt, V. Ramanujan, J. Saad-Falcon, J. Li, A. Dave, A. Albalak, K. Arora, B. Wulfe, ...

    Pith/arXiv arXiv 2025

  34. [34]

    Tenstorrent scales ai performance: New multicore architecture leads in data-center power efficiency,

    L. Gwennap, “Tenstorrent scales ai performance: New multicore architecture leads in data-center power efficiency, ” 2020. [Online]. Available: https: //www.linleygroup.com/mpr/article.php?id=12287

    2020

  35. [35]

    Pipedream: Fast and efficient pipeline parallel dnn training,

    A. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. Devanur, G. Ganger, and P. Gibbons, “Pipedream: Fast and efficient pipeline parallel dnn training, ” arXiv preprint arXiv:1806.03377, 2018

    Pith/arXiv arXiv 2018

  36. [36]

    Capacity-aware inference: Mitigating the straggler effect in mixture of experts,

    S. He, W. Cai, J. Huang, and A. Li, “Capacity-aware inference: Mitigating the straggler effect in mixture of experts, ” 2026. [Online]. Available: https://arxiv.org/abs/2503.05066

    Pith/arXiv arXiv 2026

  37. [37]

    HeteCCL: Synthesizing Near-Optimal collective communication schedules for heterogeneous GPU clusters,

    C. Hei, J. Li, J. Cao, C. Gao, X. Sha, T. Liu, D. Zhang, E. Zhai, and X. Wang, “HeteCCL: Synthesizing Near-Optimal collective communication schedules for heterogeneous GPU clusters, ” in23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). Renton, WA: USENIX Association, May 2026, pp. 2533–2551. [Online]. Available: https://www.u...

    2026

  38. [38]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism,

    Y. Huang, Y. Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen, “Gpipe: Efficient training of giant neural networks using pipeline parallelism, ”Advances in neural information processing systems, vol. 32, 2019

    2019

  39. [39]

    Whale: Efficient giant model training over heterogeneous GPUs,

    X. Jia, L. Jiang, A. Wang, W. Xiao, Z. Shi, J. Zhang, X. Li, L. Chen, Y. Li, Z. Zheng, X. Liu, and W. Lin, “Whale: Efficient giant model training over heterogeneous GPUs, ” in2022 USENIX Annual Technical Conference (USENIX ATC 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp. 673–688. [Online]. Available: https://www.usenix.org/conference/atc22/presen...

    2022

  40. [40]

    Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings,

    N. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles, C. Young, X. Zhou, Z. Zhou, and D. A. Patterson, “Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings, ” inProceedings of the 50th Annual International Symposium on Computer Architecture (ISCA’23)...

  41. [41]

    Ten lessons from three generations shaped google’s tpuv4i : Industrial product,

    N. P. Jouppi, D. Hyun Yoon, M. Ashcraft, M. Gottscho, T. B. Jablin, G. Kurian, J. Laudon, S. Li, P. Ma, X. Ma, T. Norrie, N. Patil, S. Prasad, C. Young, Z. Zhou, and D. Patterson, “Ten lessons from three generations shaped google’s tpuv4i : Industrial product, ” in2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA’21), Virtua...

    2021

  42. [42]

    A domain-specific supercomputer for training deep neural networks,

    N. P. Jouppi, D. H. Yoon, G. Kurian, S. Li, N. Patil, J. Laudon, C. Young, and D. Pat- terson, “A domain-specific supercomputer for training deep neural networks, ” Commun. ACM, vol. 63, no. 7, June 2020

    2020

  43. [43]

    Reducing activation recomputation in large transformer models,

    V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models, ” Proceedings of Machine Learning and Systems, vol. 5, pp. 341–353, 2023

    2023

  44. [44]

    Is the gpu half-empty or half-full? practical scheduling techniques for llms,

    F. Kossmann, B. Fontaine, D. Khudia, M. Cafarella, and S. Madden, “Is the gpu half-empty or half-full? practical scheduling techniques for llms, ”arXiv preprint arXiv:2410.17840, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention, ” inProceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 611–626

    2023

  46. [46]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding, ” 2020. [Online]. Available: https://arxiv.org/abs/2006.16668

    Pith/arXiv arXiv 2020

  47. [47]

    Sequence parallelism: Long sequence training from system perspective,

    S. Li, F. Xue, C. Baranwal, Y. Li, and Y. You, “Sequence parallelism: Long sequence training from system perspective, ” 2022. [Online]. Available: https://arxiv.org/abs/2105.13120

    Pith/arXiv arXiv 2022

  48. [48]

    Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache,

    B. Lin, C. Zhang, T. Peng, H. Zhao, W. Xiao, M. Sun, A. Liu, Z. Zhang, L. Li, X. Qiu, S. Li, Z. Ji, T. Xie, Y. Li, and W. Lin, “Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache, ”arXiv preprint arXiv:2401.02669, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    The llama 3 herd of models,

    Llama Team, “The llama 3 herd of models, ” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  50. [50]

    A hypervisor for shared-memory fpga platforms,

    J. Ma, G. Zuo, K. Loughlin, X. Cheng, Y. Liu, A. M. Eneyew, Z. Qi, and B. Kasikci, “A hypervisor for shared-memory fpga platforms, ” inProceedings of the 25th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’20), Lausanne, Switzerland, 2020

    2020

  51. [51]

    Helix: Serving large language models over heterogeneous gpus and network via max-flow,

    Y. Mei, Y. Zhuang, X. Miao, J. Yang, Z. Jia, and R. Vinayak, “Helix: Serving large language models over heterogeneous gpus and network via max-flow, ” ser. ASPLOS ’25. New York, NY, USA: Association for Computing Machinery, 2025, p. 586–602

    2025

  52. [52]

    Spotserve: Serving generative large language models on preemptible instances,

    X. Miao, C. Shi, J. Duan, X. Xi, D. Lin, B. Cui, and Z. Jia, “Spotserve: Serving generative large language models on preemptible instances, ” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’24. New York, NY, USA: Association for Computing Machinery, 202...

    arXiv 2024

  53. [53]

    Autoscale in azure monitor,

    Microsoft, “Autoscale in azure monitor, ” nov 2025. [Online]. Available: https://learn.microsoft.com/en-us/azure/azure-monitor/autoscale/ autoscale-overview

    2025

  54. [54]

    Azure public dataset: Azure LLM inference trace 2023,

    Microsoft Azure, “Azure public dataset: Azure LLM inference trace 2023, ” GitHub, 2023, accessed: 2026-06-17. [Online]. Available: https://github.com/ Azure/AzurePublicDataset/blob/master/AzureLLMInferenceDataset2023.md

    2023

  55. [55]

    Hetis: Serving llms in heterogeneous gpu clusters with fine-grained and dynamic parallelism,

    Z. Mo, J. Liao, H. Xu, Z. Zhou, and C. Xu, “Hetis: Serving llms in heterogeneous gpu clusters with fine-grained and dynamic parallelism, ” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’25. ACM, Nov. 2025, p. 1710–1724. [Online]. Available: http://dx.doi.org/10.1145/3712285.3759784 ...

    arXiv 2025

  56. [56]

    Heet: Accelerating elastic training in heterogeneous deep learning clusters,

    Z. Mo, H. Xu, and C. Xu, “Heet: Accelerating elastic training in heterogeneous deep learning clusters, ” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2024, pp. 499–513

    2024

  57. [57]

    Heterogeneity-Aware cluster scheduling policies for deep learning workloads,

    D. Narayanan, K. Santhanam, F. Kazhamiaka, A. Phanishayee, and M. Zaharia, “Heterogeneity-Aware cluster scheduling policies for deep learning workloads, ” in14th USENIX Symposium on Operating Systems Design and Implementation (OSDI’20), Nov. 2020, pp. 481–498. [Online]. Available: https://www.usenix.org/ conference/osdi20/presentation/narayanan-deepak

    2020

  58. [58]

    Chatgpt,

    OpenAI, “Chatgpt, ” https://chat.openai.com, 2024

    2024

  59. [59]

    Hlo passes,

    OpenXLA Authors, “Hlo passes, ” nov 2025. [Online]. Available: https: //openxla.org/xla/hlo_passes#analysis_passes

    2025

  60. [60]

    Xla tooling,

    ——, “Xla tooling, ” nov 2025. [Online]. Available: https://openxla.org/xla/tools

    2025

  61. [61]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting, ” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118–132

    2024

  62. [62]

    Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters,

    Y. Peng, Y. Bao, Y. Chen, C. Wu, and C. Guo, “Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters, ” inProceedings of the 13th European Conference on Computer Systems (EuroSys’18), Porto, Portugal, Apr. 2018

    2018

  63. [63]

    When will the tokens end? graph-based forecasting for LLMs output length,

    G. Piotrowski, M. Bystroński, M. Hołysz, J. Binkowski, G. Chodak, and T. J. Kajdanowicz, “When will the tokens end? graph-based forecasting for LLMs output length, ” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), J. Zhao, M. Wang, and Z. Liu, Eds. Vienna, Austria: Associatio...

    2025

  64. [64]

    FIRM: An intelligent fine-grained resource management framework for SLO-Oriented microservices,

    H. Qiu, S. S. Banerjee, S. Jha, Z. T. Kalbarczyk, and R. K. Iyer, “FIRM: An intelligent fine-grained resource management framework for SLO-Oriented microservices, ” in14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, Nov. 2020, pp. 805–825. [Online]. Available: https://www.usenix.org/conference/osdi20/pres...

    2020

  65. [65]

    Efficient interactive llm serving with proxy model-based sequence length prediction,

    H. Qiu, W. Mao, A. Patke, S. Cui, S. Jha, C. Wang, H. Franke, Z. T. Kalbarczyk, T. Başar, and R. K. Iyer, “Efficient interactive llm serving with proxy model-based sequence length prediction, ” 2024. [Online]. Available: https://arxiv.org/abs/2404.08509

    Pith/arXiv arXiv 2024

  66. [66]

    AWARE: Automate workload autoscaling with reinforcement learning in production cloud systems,

    H. Qiu, W. Mao, C. Wang, H. Franke, A. Youssef, Z. T. Kalbarczyk, T. Başar, and R. K. Iyer, “AWARE: Automate workload autoscaling with reinforcement learning in production cloud systems, ” inProccedings of 2023 USENIX Annual Technical Conference (USENIX ATC 23). Boston, MA: USENIX Association, Jul. 2023, pp. 387–402. [Online]. Available: https://www.useni...

    2023

  67. [67]

    Hbm3e / hbm3 controller ip

    Rambus, “Hbm3e / hbm3 controller ip. ” [Online]. Available: https://www. rambus.com/interface-ip/hbm/hbm3-controller/

  68. [68]

    A distributed self-learning approach for elastic provisioning of virtualized cloud resources,

    J. Rao, X. Bu, C.-Z. Xu, and K. Wang, “A distributed self-learning approach for elastic provisioning of virtualized cloud resources, ” in2011 IEEE 19th Annual International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, 2011, pp. 45–54

    2011

  69. [69]

    Google TPU Architecture and Performance Best Practices,

    RUN:AI, “Google TPU Architecture and Performance Best Practices, ” 2022. [Online]. Available: https://www.run.ai/guides/cloud-deep-learning/google- tpu

    2022

  70. [70]

    Autopilot: workload autoscal- ing at google,

    K. Rzadca, P. Findeisen, J. Swiderski, P. Zych, P. Broniek, J. Kusmierek, P. Nowak, B. Strack, P. Witusowski, S. Hand, and J. Wilkes, “Autopilot: workload autoscal- ing at google, ” inProceedings of the Fifteenth European Conference on Computer Systems (EuroSys’20), Heraklion, Greece, 2020

    2020

  71. [71]

    Traceupscaler: Upscaling traces to evaluate systems at high load,

    S. M. Sajal, T. Zhu, B. Urgaonkar, and S. Sen, “Traceupscaler: Upscaling traces to evaluate systems at high load, ” inProceedings of the Nineteenth European Confer- ence on Computer Systems, ser. EuroSys ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 942–961

    2024

  72. [72]

    Life-cycle emissions of ai hardware: A cradle-to- grave approach and generational trends,

    I. Schneider, H. Xu, S. Benecke, D. Patterson, K. Huang, P. Ranganathan, and C. Elsworth, “Life-cycle emissions of ai hardware: A cradle-to- grave approach and generational trends, ” 2025. [Online]. Available: https: //arxiv.org/abs/2502.01671

    Pith/arXiv arXiv 2025

  73. [73]

    XSched: Preemptive scheduling for diverse XPUs,

    W. Shen, M. Han, J. Liu, R. Chen, and H. Chen, “XSched: Preemptive scheduling for diverse XPUs, ” in19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). Boston, MA: USENIX Association, Jul. 2025, pp. 671–692. [Online]. Available: https://www.usenix.org/conference/osdi25/ presentation/shen-weihang

    2025

  74. [74]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism, ” 2020. [Online]. Available: https://arxiv.org/abs/1909.08053

    Pith/arXiv arXiv 2020

  75. [75]

    Jupiter rising: A decade of clos topologies and centralized control in google’s datacenter network,

    A. Singh, J. Ong, A. Agarwal, G. Anderson, A. Armistead, R. Bannon, S. Bov- ing, G. Desai, B. Felderman, P. Germano, A. Kanagala, J. Provost, J. Simmons, E. Tanda, J. Wanderer, U. Hölzle, S. Stuart, and A. Vahdat, “Jupiter rising: A decade of clos topologies and centralized control in google’s datacenter network, ” inSigcomm ’15, 2015

    2015

  76. [76]

    DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency ,

    J. Stojkovic, C. Zhang, I. Goiri, J. Torrellas, and E. Choukse, “ DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency , ” in 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). Los Alamitos, CA, USA: IEEE Computer Society, Mar. 2025, pp. 1348–1362. [Online]. Available: https://doi.ieeecomputers...

    arXiv 2025

  77. [77]

    Sailor: Automating distributed training over dynamic, heterogeneous, and geo-distributed clusters,

    F. Strati, Z. Zhang, G. Manos, I. S. Périz, Q. Hu, T. Chen, B. Buzcu, S. Han, P. Delgado, and A. Klimovic, “Sailor: Automating distributed training over dynamic, heterogeneous, and geo-distributed clusters, ” inProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, ser. SOSP ’25. New York, NY, USA: Association for Computing Machiner...

    2025

  78. [78]

    Llumnix: Dynamic scheduling for large language model serving,

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving, ” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 173–191. [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/sun-biao

    2024

  79. [79]

    Neurometer: An integrated power, area, and timing modeling framework for machine learning accelerators indus- try track paper,

    T. Tang, S. Li, L. Nai, N. Jouppi, and Y. Xie, “Neurometer: An integrated power, area, and timing modeling framework for machine learning accelerators indus- try track paper, ” in2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2021, pp. 841–853

    2021

  80. [80]

    Prompt- aware scheduling for low-latency llm serving,

    Y. Tao, Y. Zhang, M. T. Dearing, X. Wang, Y. Fan, and Z. Lan, “Prompt- aware scheduling for low-latency llm serving, ” 2025. [Online]. Available: https://arxiv.org/abs/2510.03243

    Pith/arXiv arXiv 2025

Showing first 80 references.