Pith. sign in

REVIEW 2 major objections 5 minor 117 references

MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systems

T0 review · 2 major / 5 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ML systems can be reasoned about as 22 physics walls solved in under a second, without hardware in hand.

desk verdict Solid engineering artifact: unifies known first-principles models into a dimensionally strict, registry-backed engine with transparent first-order validation; useful for education and early design, not new physics. read the letter →

arxiv 2607.02558 v1 pith:T3ITN2E6 submitted 2026-06-28 cs.DC cs.LG

Vijay Janapa Reddi This is my paper
classification cs.DCcs.LG
keywords MLsystemsmodelingRooflineanalysisdemand-supplyseparationdimensionalstrictnessbindingconstraintshardware-softwareco-designsustainabilityaccountinganalyticalsimulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Machine-learning systems now span milliwatt chips to multi-gigawatt fleets, yet the usual tools force a bad trade-off: you either need the real hardware or you wait hours for cycle-accurate simulation. This paper argues that most of the useful design questions can be answered by treating the stack as a set of first-principles constraints—memory bandwidth, communication cost, queueing, carbon, TCO—and solving them analytically. It packages that idea as MLSYSIM: a pure-Python engine that separates what a model demands from what silicon and infrastructure supply, enforces SI units at runtime so unit mistakes cannot hide, and stores every constant with a documented source. Twenty-two named “Systems Walls” are each resolved by dedicated models or solvers that compose into a three-level scorecard (does it fit, how fast is it, what does it cost and emit). The same run that flags the binding bottleneck can also invert the equations to synthesize the minimum hardware that meets a latency SLA. Validation against published training, inference, scaling-law and carbon figures stays inside roughly seven percent while more than a thousand configurations finish in under a second on a laptop. The practical claim is that architectural intuition and early capacity planning no longer require a GPU cluster.

What carries the argument

The 5-layer demand–supply stack and the 22 Systems Walls: workload demand is lowered once into a hardware-agnostic computation graph; silicon, grid and topology supply remain separate; 28 pure resolvers (models, solvers, optimizers) compose through typed, pint-enforced quantities to produce a Feasibility–Performance–Macro scorecard and an inverse-Roofline synthesis path.

What would settle it

Take a published multi-node training or serving result whose binding constraint is known from profiling; if MLSYSIM’s predicted binding wall, MFU or carbon figure systematically disagrees beyond the claimed envelope after the stated efficiency defaults are applied, the central claim fails.

Watch

Extended reading notes

Core claim

A demand–supply architecture plus a taxonomy of 22 Systems Walls, resolved by 28 composable, dimensionally strict resolvers, is sufficient to identify the binding constraint and to synthesize ideal hardware specifications for full-stack ML systems in sub-second time, matching published empirical anchors within roughly seven percent.

Load-bearing premise

A single efficiency number (and a few related overlap defaults) can soak up all the messy micro-architectural, runtime and congestion effects so that simple first-principles equations still correctly name which wall is binding.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. MLSYS·IM is a pure-Python analytical framework for first-principles modeling of ML systems infrastructure. It separates computational demand from silicon supply and environmental context via a five-layer architecture and a hardware-agnostic ComputationGraph IR, enforces SI units at runtime with pint, and draws every numeric input from typed, provenance-tracked registries (the MLSys Zoo). On this engine the authors codify 22 Systems Walls across six domains (Node, Data, Algorithm, Fleet, Operations, Analysis), resolved by 28 composable models, solvers, and optimizers that produce a three-level SystemEvaluation (Feasibility, Performance, Macro) in sub-second time. The framework identifies binding constraints, synthesizes minimum hardware from SLAs (inverse Roofline), and is validated against seven published anchors spanning training, inference, scaling laws, carbon, and parallelism search, with design-space sweeps of >1000 configurations in under one second on a laptop.

Significance. If the engineering claim holds, the paper fills a real gap between cycle-accurate simulators (hours per config) and ad-hoc spreadsheets (no dimensional safety, narrow scope). The combination of demand–supply separation, runtime unit enforcement, provenance-tracked registries, and a full-stack wall taxonomy is a genuine contribution for education and early co-design. Explicit strengths include open pure-Python execution, machine-checked structural invariants (862-test suite), seven independent empirical anchors (Table 6), and clear pedagogical integration with the companion textbook. The work is useful even when absolute latency is only first-order accurate, because the ordinal ranking of binding walls is the actionable output.

major comments (2)
  1. §6.1 Anchors 1–4 and Table 6: several headline matches (ResNet-50 throughput, Llama-3 MFU 39.1% vs 38–43%, PaLM MFU 43% vs 46.2%) depend on choosing η (and η_overlap) from the same literature that supplies the target. The paper is transparent that η is an exposed free parameter (Eqs. 1, 6; §6.3), and two anchors need no η, but the manuscript should state more clearly which anchors are pure first-principles predictions versus literature-calibrated regime checks. Without that distinction, the “within ~7%” claim can be read as stronger than the evidence supports for uncalibrated what-if use.
  2. §4.1 Wall 4 (Serving) and §4.4 Wall 14 (Communication): modern serving features (chunked prefill, speculative decoding, hierarchical AllReduce with overlap) are modeled with coarse analytical proxies (e.g., decode_stall_bound, η_overlap default 0.85). These are reasonable for first-order ranking, but the paper should quantify sensitivity of the binding-wall label to these defaults on at least one multi-wall case (e.g., the LLaMA-70B 512-GPU audit in §7.3.3), so readers know when the ordinal ranking is robust versus when it flips under plausible η_overlap or service_time_cv variation.
minor comments (5)
  1. Abstract and title use both MLSYSIM and MLSYS·IM; pick one spelling consistently (including the arXiv identifier line).
  2. Table 1 lists 22 walls but the prose sometimes says “20 walls + 2 diagnostics”; align the wording with the table so the count is unambiguous.
  3. Figure 1 panel (d) and §7.3.3 report concrete TCO/carbon numbers; ensure the registry snapshot (Appendix B) and provenance catalog make those exact inputs reconstructible without private defaults.
  4. §2.3 comparison to Calculon, Lumos, Vidur, GenZ is fair but could briefly note which of their equations are reused versus re-derived, to help readers map the taxonomy onto prior analytical tools.
  5. Notation Table 5 is helpful; a few symbols (e.g., β_opt, γ routing imbalance) appear later without a forward pointer from the table—add them or cite the table at first use.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: first-principles equations plus explicit free parameters are checked against independent published anchors, not re-derived from the targets.

full rationale

MLSYSIM's central claim is an engineering artifact (demand–supply separation, dimensional strictness, 22-wall taxonomy, 28 resolvers) that composes published first-principles formulas (Roofline, Ring AllReduce, Chinchilla C=6PD, Erlang-C, Young–Daly, TCO, operational carbon). Validation Anchors 1–7 compare outputs to external sources (MLPerf, vLLM/Kwon, Meta Llama-3 report, PaLM, Hoffmann Chinchilla, Patterson carbon). Two anchors need no efficiency parameter at all. Where η or overlap coefficients appear, the paper treats them as explicit free parameters taken from the same literature or stated defaults, not as quantities fitted inside a closed loop that re-predicts the fitted target. Self-citations (companion textbook, TinyTorch) are pedagogical companions, not uniqueness theorems that force the modeling results. The taxonomy and engine are new software organization, not a renaming of a single known equation set equal to itself. Score 1 reflects only the ordinary, non-load-bearing use of literature-sourced η ranges; the derivation chain does not reduce by construction to its inputs.

Assumptions & free parameters 4 free parameters · 8 assumptions · 3 invented entities

The engine rests on classical systems equations plus one dominant free efficiency parameter and a set of domain modeling choices. No new physical entities are postulated; the “walls” and “demand–supply layers” are organizational inventions.

free parameters (4)
  • η (hardware utilization / MFU)
    Single scalar that absorbs kernel, framework, straggler and micro-arch effects; set per workload class from literature ranges (0.19 ResNet, 0.42 Llama-3, etc.). Central to every Roofline and serving prediction.
  • η_overlap (compute–comm overlap)
    Default 0.85 used in DistributedModel; taken from typical Megatron-LM behavior rather than measured per run.
  • service_time_cv / Kingman correction
    Optional coefficient-of-variation for non-exponential service times in TailLatencyModel; defaults to 1 (pure M/M/c).
  • β_opt (optimizer-state multiplier)
    ≈7 for mixed-precision Adam checkpoint size; conventional but architecture-dependent.
assumptions (8)
  • domain assumption Classical Roofline: T = max(OPs/(Peak·η), |W|/BW)
    Williams et al. 2009; used for Walls 1–2 and all single-node analysis.
  • domain assumption Chinchilla scaling C = 6PD, D* ≈ 20P
    Hoffmann et al. 2022; Wall 11.
  • domain assumption Erlang-C / M/M/c for P99 tail latency
    Dean & Barroso 2013; Wall 7; paper notes it is optimistic.
  • domain assumption Young–Daly optimal checkpoint interval τ_opt = √(2δ·MTBF)
    Daly 2006; Wall 15/19.
  • domain assumption Independent exponential failures → MTBF_cluster = MTBF_node / N
    Standard reliability assumption; Wall 15.
  • domain assumption Static bisection bandwidth fraction β by topology (Fat-Tree=1, Dragonfly≈0.85, Torus≈0.67)
    Leiserson 1985; Wall 10; ignores dynamic congestion.
  • standard math Dimensional quantities must be closed under pint arithmetic; FLOP is a base dimension
    Runtime invariant of the engine (Section 3.1.2).
  • domain assumption Linear CapEx amortization over 3–5 years; energy-proportional power (idle = 30 % TDP)
    Barroso et al.; Walls 17–18.
invented entities (3)
  • 22 Systems Walls taxonomy
    purpose: Organizes previously scattered constraints into six domains with one resolver each
    Conceptual invention; each wall maps to a published equation but the complete catalog and numbering are new to this paper.
  • 5-layer demand–supply architecture + ComputationGraph IR
    purpose: Decouples workload demand from hardware supply and environmental context so the same model can be swept across devices
    Software architecture contribution; not a physical entity.
  • MLSys Zoo (8 typed, provenance-tracked registries)
    purpose: Ensure every numeric input has a documented source and unit
    Engineering artifact enabling reproducibility claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systems." pith.science (2026). https://pith.science/paper/T3ITN2E6

@misc{pith2026260702558,
  author       = {Pith},
  title        = {Pith review of: MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T3ITN2E6}},
  note         = {Machine review of arXiv:2607.02558}
}
read the original abstract

As machine learning shifts from laboratory curiosity to critical infrastructure, the systems that sustain it span an extraordinary range, from sub-milliwatt microcontrollers to multi-gigawatt datacenter fleets. Reasoning across this range is hard: empirical profiling requires the target hardware in hand, while cycle-accurate simulation costs hours per configuration, leaving no tool for rapid, full-stack architectural reasoning. We present MLSYSIM (Machine Learning Systems Infrastructure Modeling), a first-principles analytical framework that formalizes the "physics of systems" into a dimensionally-strict Python engine. MLSysim is built on a demand-supply abstraction that decouples computational demand from silicon supply and environmental context, and it enforces unit integrity at runtime so the silent conversion errors that plague ad-hoc modeling cannot occur. Every input is drawn from a typed, provenance-tracked registry, so no number enters an analysis without a documented source. On this engine we codify a taxonomy of 22 "Systems Walls" resolved by 28 composable models and solvers, enabling sub-second design-space exploration that identifies binding constraints and synthesizes ideal hardware specifications across the entire ML systems lifecycle.

Figures

Figures reproduced from arXiv: 2607.02558 by the authors.

Figure 1
Figure 1. MLSYS · IM Framework Overview. (a) Demand–supply separation decouples workload specifications from hardware capabilities and environmental context. (b) All 22 Systems Walls organized into six domains (Node, Data, Algorithm, Fleet, Operations, Analysis), each grounded in a published equation. (c) Stateless resolver composition chains 28 resolvers to identify binding constraints through a three-level evaluation. (d) E… view at source ↗
Figure 2
Figure 2. The MLSYS · IM 5-Layer Architecture. Layers A–D provide typed inputs: workload demand, hardware supply, infrastructure context, and network topology. The single lowering step occurs in Layer A, where lower() produces a hardware-agnostic Computation Graph (an intermediate representation of total FLOPs, weight bytes, and arithmetic intensity). Layer E’s 28 stateless resolvers consume these inputs and produce a three-l… view at source ↗
Figure 3
Figure 3. Registry-Backed System Composition. MLSYS · IM stores workload, hardware, systems, infrastructure, operational, and literature facts once, with typed schemas and provenance records. A scenario composes those reusable facts into a local infrastructure question—for example, training a 70B model on a 256-GPU H100 fleet in Québec or Iowa—and hands dimensioned objects to the resolver chain. The resulting SystemEvaluation… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: System Anatomy. (a) Package architecture: eight registries (six shown) provide typed, provenance-tracked specifications, nine physics modules supply domain-aligned formulas, and five invariant checks enforce design decisions as the codebase evolves. (b) Analysis pipeli…
Figure 5
Figure 5. Figure 5: Resolver Composition. Three input layers feed four resolvers. Each resolver is a pure function: typed inputs in, dimensionally correct outputs out. The scorecard aggregates three evaluation levels: Feasibility, Performance, and Macro (economics, sustainability, safety)…
Figure 6
Figure 6. Figure 6: Design-Space Exploration: Bottleneck Regime Map. Each cell shows the binding constraint (memory or compute) for a given model size and HBM bandwidth combination under FP16 training at batch size 256. Larger models with lower bandwidth are memory-bound; smaller models w…
Figure 7
Figure 7. Figure 7: Roofline Crossover: ResNet-50 Batch Size Sweep on H100. Increasing batch size moves the oper￾ating point rightward along the Roofline, transitioning from memory-bound (purple) to compute-bound (cyan) between batch 1 and batch 8. The peak ridge point is 989/3.35 ≈ 295 F…
Figure 8
Figure 8. Figure 8: Geography as a Systems Variable: Iowa vs. Québec. An identical 256-GPU cluster training a 70B model for 30 days produces 36× more CO2 in Iowa (coal/gas grid circa 2020, 680 gCO2/kWh, PUE 1.12) than in Québec (95% hydroelectric, 20 gCO2/kWh, PUE 1.06): 58.4 vs. 1.6 tonn…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

117 extracted references · 7 canonical work pages

  1. [2]

    and Tumanov, Alexey and Ramjee, Ramachandran , year =

    Agrawal, Amey and Kedia, Nitin and Panwar, Ashish and Mohan, Jayashree and Kwatra, Nipun and Gulavani, Bhargav S. and Tumanov, Alexey and Ramjee, Ramachandran , year =. Taming Throughput-Latency Tradeoff in. Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , publisher =

  2. [3]

    and Tumanov, Alexey and Ramjee, Ramachandran , year =

    Agrawal, Amey and Kedia, Nitin and Panwar, Ashish and Mohan, Jayashree and Kwatra, Nipun and Gulavani, Bhargav S. and Tumanov, Alexey and Ramjee, Ramachandran , year =. Vidur: A Large-Scale Simulation Framework For. Proceedings of Machine Learning and Systems (MLSys) , publisher =

  3. [4]

    OpenAI Blog , publisher =

    Amodei, Dario and Hernandez, Danny , year =. OpenAI Blog , publisher =

  4. [7]

    2018 , publisher =

    The Datacenter as a Computer: Designing Warehouse-Scale Machines , author =. 2018 , publisher =

    2018

  5. [11]

    Journal of Machine Learning Research , volume =

    Chowdhery, Aakanksha and Narang, Sharan and Devlin, Jacob and others , year =. Journal of Machine Learning Research , volume =

  6. [12]

    Frans and Morris, Robert , year =

    Cox, Russ and Kaashoek, M. Frans and Morris, Robert , year =. xv6: A Simple,

  7. [14]

    and Ermon, Stefano and Rudra, Atri and R

    Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R. 2022 , booktitle =

    2022

  8. [15]

    2012 , journal =

    Large Scale Distributed Deep Networks , author =. 2012 , journal =

    2012

Show all 117 references
  1. [18]

    2022 , booktitle =

    Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models , author =. 2022 , booktitle =

    2022

  2. [19]

    Proceedings of the 12th International Conference on Learning Representations (ICLR) , publisher =

    Faiz, Ahmad and Kannan, Sotaro and Rishi, James and Chuber, James and Addanki, Raviteja and Noman, Abdulrahman , year =. Proceedings of the 12th International Conference on Learning Representations (ICLR) , publisher =

  3. [20]

    Proceedings of the 11th International Conference on Learning Representations (ICLR) , publisher =

    Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan , year =. Proceedings of the 11th International Conference on Learning Representations (ICLR) , publisher =

  4. [21]

    2021 , journal =

    A Survey of Quantization Methods for Efficient Neural Network Inference , author =. 2021 , journal =

    2021

  5. [22]

    and Brooks, David and Wu, Carole-Jean , year =

    Gupta, Udit and Elgamal, Mariam and Hills, Gage and Wei, Gu-Yeon and Lee, Hsien-Hsin S. and Brooks, David and Wu, Carole-Jean , year =. Proceedings of the 49th Annual International Symposium on Computer Architecture (ISCA) , publisher =. doi:10.1145/3470496.3527408 , x-verified =

    doi

  6. [23]

    2022 , booktitle =

    Chasing Carbon: The Elusive Environmental Footprint of Computing , author =. 2022 , booktitle =. doi:10.1109/HPCA53966.2022.00076 , x-verified =

    2022 doi

  7. [24]

    2016 , booktitle =

    Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding , author =. 2016 , booktitle =

    2016

  8. [25]

    2024 , publisher =

    Computer Architecture: A Quantitative Approach , author =. 2024 , publisher =

    2024

  9. [26]

    2022 , booktitle =

    Training Compute-Optimal Large Language Models , author =. 2022 , booktitle =

    2022

  10. [27]

    2019 , booktitle =

    Beyond Data and Model Parallelism for Deep Neural Networks , author =. 2019 , booktitle =

    2019

  11. [28]

    2017 , booktitle =

    In-Datacenter Performance Analysis of a Tensor Processing Unit , author =. 2017 , booktitle =. doi:10.1145/3079856.3080246 , x-verified =

    2017 doi

  12. [29]

    2020 , journal =

    Scaling Laws for Neural Language Models , author =. 2020 , journal =

    2020

  13. [30]

    2023 , note =

    llm-analysis: Latency and Memory Analysis of Transformer Models , author =. 2023 , note =

    2023

  14. [32]

    1985 , journal =

    Fat-Trees: Universal Networks for Hardware-Efficient Supercomputing , author =. 1985 , journal =

    1985

  15. [33]

    2023 , booktitle =

    Fast Inference from Transformers via Speculative Decoding , author =. 2023 , booktitle =

    2023

  16. [34]

    Lumos: Efficient Performance Modeling and Estimation for Large-scale

    Liang, Mingyu and Kassa, Hiwot Tadese and Fu, Wenyin and Coutinho, Brian and Feng, Louis and Delimitrou, Christina , year =. Lumos: Efficient Performance Modeling and Estimation for Large-scale. Proceedings of Machine Learning and Systems (MLSys) , publisher =

  17. [35]

    Cerebras Architecture Deep Dive: First Look Inside the

    Lie, Sean , year =. Cerebras Architecture Deep Dive: First Look Inside the. IEEE Hot Chips 34 Symposium , publisher =

  18. [36]

    Proceedings of Machine Learning and Systems (MLSys) , publisher =

    Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Chen, Wei-Ming and Wang, Wei-Chen and Xiao, Guangxuan and Dang, Xingyu and Gan, Chuang and Han, Song , year =. Proceedings of Machine Learning and Systems (MLSys) , publisher =

  19. [38]

    2024 , journal =

    The. 2024 , journal =

    2024

  20. [39]

    2019 , howpublished =

    Energy Usage Reports: Environmental Awareness as Part of Algorithmic Accountability , author =. 2019 , howpublished =

    2019

  21. [41]

    2025 , publisher =

    Machine Learning Systems: Principles and Practices of Engineering Artificially Intelligent Systems , author =. 2025 , publisher =

    2025

  22. [42]

    Analyzing and Mitigating Data Stalls in

    Mohan, Jayashree and Phanishayee, Amar and Raniwala, Ashish and Chidambaram, Vijay , year =. Analyzing and Mitigating Data Stalls in. Proceedings of the VLDB Endowment , volume =

  23. [43]

    2021 , booktitle =

    tf.data: A Machine Learning Data Processing Framework , author =. 2021 , booktitle =

    2021

  24. [44]

    2021 , journal =

    Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM , author =. 2021 , journal =

    2021

  25. [47]

    2014 , publisher =

    Computer Organization and Design: The Hardware/Software Interface , author =. 2014 , publisher =

    2014

  26. [48]

    2021 , journal =

    Carbon Emissions and Large Neural Network Training , author =. 2021 , journal =

    2021

  27. [49]

    2023 , journal =

    Efficiently Scaling Transformer Inference , author =. 2023 , journal =

    2023

  28. [50]

    and Talwalkar, Ameet , year =

    Qi, Hang and Sparks, Evan R. and Talwalkar, Ameet , year =. Proceedings of the 5th International Conference on Learning Representations (ICLR) , publisher =

  29. [52]

    ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , publisher =

    Rasley, Jeff and Rajbhandari, Samyam and Ruwase, Olatunji and He, Yuxiong , year =. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , publisher =. doi:10.1145/3394486.3406703 , x-verified =

    doi

  30. [53]

    2017 , booktitle =

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author =. 2017 , booktitle =

    2017

  31. [55]

    Snell, Charlie and Lee, Jaehoon and Xu, Kelvin and Kumar, Aviral , year =. Scaling. Proceedings of the 13th International Conference on Learning Representations (ICLR) , publisher =

  32. [56]

    and LaPiana, Lia S

    Stephenson, Arthur G. and LaPiana, Lia S. and Mulville, Daniel R. and Rutledge, Peter J. and Bauer, Frank H. and Folta, David and Dukeman, Greg A. and Sackheim, Robert and Norvig, Peter , year =

  33. [57]

    2019 , note =

    The Bitter Lesson , author =. 2019 , note =

    2019

  34. [58]

    2006 , publisher =

    Operating Systems: Design and Implementation , author =. 2006 , publisher =

    2006

  35. [59]

    Reddi, Vijay Janapa and others , year =

  36. [60]

    Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI) , publisher =

    Wang, Zhuo and Zheng, Weicheng and Liu, Chengwei and others , year =. Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI) , publisher =

  37. [63]

    Machine Learning Fleet Efficiency: Analyzing and Optimizing Large-Scale

    Wongpanich, Arissa and Oguntebi, Tayo and Baiocchi Paredes, Jose and Wang, Yu Emma and Phothilimthana, Phitchaya Mangpo and Mitra, Ritwika and Zhou, Zongwei and Kumar, Naveen and Reddi, Vijay Janapa , year =. Machine Learning Fleet Efficiency: Analyzing and Optimizing Large-Scale

  38. [66]

    2021 , booktitle =

    Habitat: A Runtime-Based Computational Performance Predictor for Deep Neural Network Training , author =. 2021 , booktitle =

    2021

  39. [69]

    arXiv preprint arXiv:2312.07104 , x-verified =

    Zheng, Lianmin and Yin, Liangsheng and Xie, Zhiqiang and others , year =. arXiv preprint arXiv:2312.07104 , x-verified =

    arXiv

  40. [70]

    Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , publisher =

    Zhong, Yinmin and Liu, Shengyu and Chen, Junda and Hu, Jianbo and Zhu, Yibo and Liu, Xuanzhe and Jin, Xin and Zhang, Hao , year =. Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI) , publisher =

  41. [71]

    Deep learning with differential privacy

    Martin Abadi, Andy Chu, Ian Goodfellow, et al. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308--318. ACM, 2016. doi:10.1145/2976749.2978318. URL https://doi.org/10.1145/2976749.2978318

    2016 doi

  42. [72]

    Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve . In 18th USENIX Symposium on Operating Systems Design and Implementation ...

    2024

  43. [73]

    Gulavani, Alexey Tumanov, and Ramachandran Ramjee

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Vidur: A large-scale simulation framework for LLM inference. In Proceedings of Machine Learning and Systems (MLSys). mlsys.org, 2024 b . URL h...

    2024 arXiv

  44. [74]

    Demystifying platform requirements for diverse LLM inference use cases

    Abhimanyu Bambhaniya, Ritik Raj, Geonhwa Jeong, Souvik Kundu, Sudarshan Srinivasan, Midhilesh Elavazhagan, Madhu Kumar, and Tushar Krishna. Demystifying platform requirements for diverse LLM inference use cases. arXiv preprint arXiv:2406.01698, 2024

    2024 arXiv

  45. [75]

    The case for energy-proportional computing

    Luiz Andr \'e Barroso and Urs H \"o lzle. The case for energy-proportional computing. Computer, 40 0 (12): 0 33--37, 2007. ISSN 0018-9162. doi:10.1109/mc.2007.443. URL https://doi.org/10.1109/mc.2007.443

    2007 doi

  46. [76]

    The Datacenter as a Computer

    Luiz Andr \'e Barroso, Urs H \"o lzle, and Parthasarathy Ranganathan. The Datacenter as a Computer. Synthesis Lectures on Computer Architecture. Springer International Publishing, 3rd edition, 2018. ISBN 9783031006333, 9783031017612. doi:10.1007/978-3-031-01761-2. URL https://...

    2018 doi

  47. [77]

    The gem5 simulator

    Nathan Binkert, Bradford Beckmann, Gabriel Black, et al. The gem5 simulator. ACM SIGARCH Computer Architecture News, 39 0 (2): 0 1--7, 2011. ISSN 0163-5964. doi:10.1145/2024716.2024718. URL https://doi.org/10.1145/2024716.2024718

    2011 doi

  48. [78]

    George E. P. Box. Science and statistics. J. Am. Stat. Assoc., 71 0 (356): 0 791--799, None 1976. ISSN 0162-1459, 1537-274X. doi:10.1080/01621459.1976.10480949. URL https://doi.org/10.1080/01621459.1976.10480949

    1976 doi

  49. [79]

    PaLM : Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, et al. PaLM : Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  50. [80]

    Frans Kaashoek, and Robert Morris

    Russ Cox, M. Frans Kaashoek, and Robert Morris. xv6: A simple, Unix -like teaching operating system. MIT PDOS, 2011. URL https://pdos.csail.mit.edu/6.828/xv6

    2011

  51. [81]

    John T. Daly. A higher order estimate of the optimum checkpoint interval for restart dumps. Future Gener. Comput. Syst., 22 0 (3): 0 303--312, 2006. ISSN 0167-739X. doi:10.1016/j.future.2004.11.016. URL https://doi.org/10.1016/j.future.2004.11.016

    2006 doi

  52. [82]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems 35, volume 35, pages 16344--16359. Neural Information Processing Systems ...

    2022 doi

  53. [83]

    The tail at scale

    Jeffrey Dean and Luiz Andr \'e Barroso. The tail at scale. Communications of the ACM, 56 0 (2): 0 74--80, 2013. ISSN 0001-0782, 1557-7317. doi:10.1145/2408776.2408794. URL https://doi.org/10.1145/2408776.2408794

    2013 doi

  54. [84]

    Corrado, Rajat Monga, et al

    Jeffrey Dean, Greg S. Corrado, Rajat Monga, et al. Large scale distributed deep networks. Advances in Neural Information Processing Systems, 25: 0 1223--1231, 2012

    2012

  55. [85]

    Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures

    DeepSeek-AI . Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pages 1731--1745. ACM, 2025. doi:10.1145/3695053.3731412. URL https://doi.org/10.11...

    2025 doi

  56. [86]

    Check-n-run: a checkpointing system for training deep learning recommendation models

    Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Krishnakumar Nair, Misha Smelyanskiy, and Murali Annavaram. Check-n-run: a checkpointing system for training deep learning recommendation models. In Proceedings of the 19th USENIX ...

    2022

  57. [87]

    LLMCarbon : Modeling the end-to-end carbon footprint of large language models

    Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang. LLMCarbon : Modeling the end-to-end carbon footprint of large language models. In Proceedings of the 12th International Conference on Learning Representations (ICLR). OpenReview.net, 2024

    2024

  58. [88]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022. URL https://www.jmlr.org/papers/v23/21-0998.html

    2022

  59. [89]

    GPTQ : Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ : Accurate post-training quantization for generative pre-trained transformers. In Proceedings of the 11th International Conference on Learning Representations (ICLR). OpenReview.net, 2023

    2023

  60. [90]

    A Survey of Quantization Methods for Efficient Neural Network Inference, pages 291--326

    Amir Gholami, Sehoon Kim, Zhen Dong, et al. A Survey of Quantization Methods for Efficient Neural Network Inference, pages 291--326. Chapman and Hall/CRC, 2021. ISBN 9781003162810. doi:10.1201/9781003162810-13. URL https://doi.org/10.1201/9781003162810-13

    2021 doi

  61. [91]

    Chasing carbon: The elusive environmental footprint of computing

    Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. Chasing carbon: The elusive environmental footprint of computing. In IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 85--99...

    2022 doi

  62. [92]

    Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In Proceedings of the 4th International Conference on Learning Representations (ICLR). OpenReview.net, 2016

    2016

  63. [93]

    Hennessy, David A

    John L. Hennessy, David A. Patterson, and Christos Kozyrakis. Computer Architecture: A Quantitative Approach. Morgan Kaufmann, 7th edition, 2024. ISBN 978-0443154065

    2024

  64. [94]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems 35, volume 35, pages 30016--30030. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022. doi:10.522...

    2022 doi

  65. [95]

    Calculon: A methodology and tool for high-level co-design of systems and large language models

    Mikhail Isaev, Nic McDonald, Larry Dennison, and Richard Vuduc. Calculon: A methodology and tool for high-level co-design of systems and large language models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, page...

    2023 doi

  66. [96]

    Beyond data and model parallelism for deep neural networks

    Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond data and model parallelism for deep neural networks. In Proceedings of Machine Learning and Systems (MLSys). mlsys.org, 2019. URL https://arxiv.org/abs/1807.05358

    2019 arXiv

  67. [97]

    Reducing activation recomputation in large transformer models

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023. URL https://proceedings.mlsys.org/...

    2023

  68. [98]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611--626. ACM, 2023. doi:10.1145/3600006.3613165. URL https://doi.org/10...

    2023 doi

  69. [99]

    Leiserson

    Charles E. Leiserson. Fat-trees: Universal networks for hardware-efficient supercomputing. IEEE Transactions on Computers, C-34 0 (10): 0 892--901, 1985. ISSN 0018-9340. doi:10.1109/tc.1985.6312192. URL https://doi.org/10.1109/tc.1985.6312192

    1985 doi

  70. [100]

    GShard : Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard : Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020. URL https://arxiv.org/...

    2006 arXiv

  71. [101]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML). PMLR, 2023

    2023

  72. [102]

    llm-analysis: Latency and memory analysis of transformer models

    Cheng Li. llm-analysis: Latency and memory analysis of transformer models. https://github.com/cli99/llm-analysis, 2023. Accessed: 2025-01-15

    2023

  73. [103]

    Lumos: Efficient performance modeling and estimation for large-scale LLM training

    Mingyu Liang, Hiwot Tadese Kassa, Wenyin Fu, Brian Coutinho, Louis Feng, and Christina Delimitrou. Lumos: Efficient performance modeling and estimation for large-scale LLM training. In Proceedings of Machine Learning and Systems (MLSys). mlsys.org, 2025

    2025

  74. [104]

    Cerebras architecture deep dive: First look inside the HW/SW co-design for deep learning

    Sean Lie. Cerebras architecture deep dive: First look inside the HW/SW co-design for deep learning. In IEEE Hot Chips 34 Symposium. IEEE, 2022

    2022

  75. [105]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. GetMobile: Mobile Computing and Communications, 28 0 (...

    2024 doi

  76. [106]

    John D. C. Little. A proof for the queuing formula: <i>l</i> = <i>w</i>. Oper. Res., 9 0 (3): 0 383--387, 1961. ISSN 0030-364X, 1526-5463. doi:10.1287/opre.9.3.383. URL https://doi.org/10.1287/opre.9.3.383

    1961 doi

  77. [107]

    The Llama 3 herd of models

    Llama Team, AI \@ Meta . The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  78. [108]

    Friedler, and Jonathan P

    Kadan Lottick, Silvia Susai, Sorelle A. Friedler, and Jonathan P. Wilson. Energy usage reports: Environmental awareness as part of algorithmic accountability. arXiv preprint arXiv:1911.08354, 2019

    1911 arXiv

  79. [109]

    Mlperf: An industry standard benchmark suite for machine learning performance

    Peter Mattson, Christine Cheng, Gregory Diamos, et al. Mlperf: An industry standard benchmark suite for machine learning performance. IEEE Micro, 40 0 (2): 0 8--16, 2020. ISSN 0272-1732, 1937-4143. doi:10.1109/mm.2020.2974843. URL https://doi.org/10.1109/mm.2020.2974843

    2020 doi

  80. [110]

    Analyzing and mitigating data stalls in dnn training

    Jayashree Mohan, Amar Phanishayee, Ashish Raniwala, and Vijay Chidambaram. Analyzing and mitigating data stalls in dnn training. Proc. VLDB Endow., 14 0 (5): 0 771--784, 2021. ISSN 2150-8097. doi:10.14778/3446095.3446100. URL https://doi.org/10.14778/3446095.3446100

    2021 doi

  81. [111]

    Murray, Jiri Simsa, Ana Klimovic, and Ihor Indyk

    Derek G. Murray, Jiri Simsa, Ana Klimovic, and Ihor Indyk. tf.data: A machine learning data processing framework. Proceedings of the VLDB Endowment, 14 0 (12): 0 2945--2958, 2021

    2021

  82. [112]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--15. ACM, 2021. do...

    2021 doi

  83. [113]

    NVIDIA H100 Tensor Core GPU datasheet

    NVIDIA Corporation . NVIDIA H100 Tensor Core GPU datasheet. https://www.nvidia.com/en-us/data-center/h100/, 2023. Accessed: 2024-06-15

    2023

  84. [114]

    Timeloop: A systematic approach to dnn accelerator evaluation

    Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, Joel Emer, et al. Timeloop: A systematic approach to dnn accelerator evaluation. In 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 304--315. IEEE, 2019. doi:...

    2019 doi

  85. [115]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, \'I \ n igo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 1...

    2024 doi

  86. [116]

    Carbon emissions and large neural network training

    David Patterson, Joseph Gonzalez, Quoc Le, et al. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021

    2021 arXiv

  87. [117]

    Patterson and John L

    David A. Patterson and John L. Hennessy. Computer Organization and Design: The Hardware/Software Interface. Morgan Kaufmann, 5th edition, 2014. ISBN 978-0124077263

    2014

  88. [118]

    Efficiently scaling transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, et al. Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems (MLSys), volume 5. mlsys.org, 2023. URL https://proceedings.mlsys.org/paper_files/paper/2023/hash/c4be71ab8d24cdfb45e3d06dbfca2780...

    2023

  89. [119]

    Sparks, and Ameet Talwalkar

    Hang Qi, Evan R. Sparks, and Ameet Talwalkar. PALEO : A performance model for deep neural networks. In Proceedings of the 5th International Conference on Learning Representations (ICLR). OpenReview.net, 2017

    2017

  90. [120]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE, 2020. doi:10.1109/sc41...

    2020 doi arXiv

  91. [121]

    Machine Learning Systems: Principles and Practices of Engineering Artificially Intelligent Systems

    Vijay Janapa Reddi et al. Machine Learning Systems: Principles and Practices of Engineering Artificially Intelligent Systems. Harvard University, 2025 a . URL https://mlsysbook.ai

    2025

  92. [122]

    TinyTorch : A progressive educational framework for machine learning systems

    Vijay Janapa Reddi et al. TinyTorch : A progressive educational framework for machine learning systems. Harvard University, 2025 b . URL https://mlsysbook.ai/tinytorch

    2025

  93. [123]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, et al. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Proceedings of the 5th International Conference on Learning Representations (ICLR). OpenReview.net, 2017

    2017

  94. [124]

    Megatron-LM : Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM : Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  95. [125]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. In Proceedings of the 13th International Conference on Learning Representations (ICLR). OpenReview.net, 2025

    2025

  96. [126]

    Stephenson, Lia S

    Arthur G. Stephenson, Lia S. LaPiana, Daniel R. Mulville, Peter J. Rutledge, Frank H. Bauer, David Folta, Greg A. Dukeman, Robert Sackheim, and Peter Norvig. Mars Climate Orbiter Mishap Investigation Board Phase I Report . Technical report, National Aeronautics and Space Admin...

    1999

  97. [127]

    The bitter lesson

    Rich Sutton. The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019. Accessed: 2024-06-15

    2019

  98. [128]

    Tanenbaum and Albert S

    Andrew S. Tanenbaum and Albert S. Woodhull. Operating Systems: Design and Implementation. Prentice Hall, 3rd edition, 2006

    2006

  99. [129]

    SimAI : Unifying architecture design and performance tuning for large-scale large language model training with scalability and precision

    Xizheng Wang et al. SimAI : Unifying architecture design and performance tuning for large-scale large language model training with scalability and precision. In Proceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI). USENIX Association, 2025

    2025

  100. [130]

    Roofline

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline. Communications of the ACM, 52 0 (4): 0 65--76, 2009. ISSN 0001-0782, 1557-7317. doi:10.1145/1498765.1498785. URL https://doi.org/10.1145/1498765.1498785

    2009 doi

  101. [131]

    Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale

    William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale. In 2023 IEEE International Symposium on Performance Analysis of Sys...

    2023 doi

  102. [132]

    Machine learning fleet efficiency: Analyzing and optimizing large-scale Google TPU systems with ML productivity goodput, 2025

    Arissa Wongpanich, Tayo Oguntebi, Jose Baiocchi Paredes, Yu Emma Wang, Phitchaya Mangpo Phothilimthana, Ritwika Mitra, Zongwei Zhou, Naveen Kumar, and Vijay Janapa Reddi. Machine learning fleet efficiency: Analyzing and optimizing large-scale Google TPU systems with ML product...

    2025 arXiv

  103. [133]

    Emer, and Vivienne Sze

    Yannan Nellie Wu, Joel S. Emer, and Vivienne Sze. Accelergy: An architecture-level energy estimation methodology for accelerator designs. In 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pages 1--8. IEEE, 2019. doi:10.1109/iccad45719.2019.8942149. UR...

    2019 doi

  104. [134]

    John W. Young. A first order approximation to the optimum checkpoint interval. Communications of the ACM, 17 0 (9): 0 530--531, 1974. ISSN 0001-0782, 1557-7317. doi:10.1145/361147.361115. URL https://doi.org/10.1145/361147.361115

    1974 doi

  105. [135]

    Yu, Yubo Gao, Pavel Golikov, and Gennady Pekhimenko

    Geoffrey X. Yu, Yubo Gao, Pavel Golikov, and Gennady Pekhimenko. Habitat: A runtime-based computational performance predictor for deep neural network training. In Proceedings of the 2021 USENIX Annual Technical Conference (ATC). USENIX Association, 2021

    2021

  106. [136]

    LLM inference unveiled: Survey and roofline model insights

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, et al. LLM inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363, 2024

    2024 arXiv

  107. [137]

    Llmcompass: Enabling efficient hardware design for large language model inference

    Hengrui Zhang, August Ning, Rohan Baskar Prabhakar, and David Wentzlaff. Llmcompass: Enabling efficient hardware design for large language model inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 1080--1096. IEEE, 2024. doi:1...

    2024 doi

  108. [138]

    Sglang: Efficient execution of structured language model programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, et al. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems 37, pages 62557--62583. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024. doi:10.52...

    2024 doi

  109. [139]

    DistServe : Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe : Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24...

    2024

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.