REVIEW 10 cited by
Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
read the original abstract
Large language models (LLMs) have shown remarkable performance across a wide range of applications, often outperforming human experts. However, deploying these gigantic models efficiently for diverse inference use cases requires carefully designed hardware platforms with ample computing, memory, and network resources. With constant innovation in LLM serving optimizations and model architecture evolving at breakneck speed, the hardware requirements to meet Service Level Objectives (SLOs) remain an open research question. To answer the question, we present an analytical tool, GenZ, to efficiently navigate the relationship between diverse LLM model architectures(Dense, GQA, MoE, Mamba), LLM serving optimizations(Chunking, Speculative decoding, quanitization), and AI platform design parameters. Our tool estimates LLM inference performance metrics for the given scenario. We have validated against real hardware platforms running various different LLM models, achieving a max geomean error of 5.82.We use GenZ to identify compute, memory capacity, memory bandwidth, network latency, and network bandwidth requirements across diverse LLM inference use cases. We also study diverse architectural choices in use today (inspired by LLM serving platforms from several vendors) to help inform computer architects designing next-generation AI hardware accelerators and platforms. The trends and insights derived from GenZ can guide AI engineers deploying LLMs as well as computer architects designing next-generation hardware accelerators and platforms. Ultimately, this work sheds light on the platform design considerations for unlocking the full potential of large language models across a spectrum of applications. The source code is available at https://github.com/abhibambhaniya/GenZ-LLM-Analyzer . Users can also be tried it on at https://genz-llm-analyzer.streamlit.app/ without any setup on your web browser.
Forward citations
Cited by 10 Pith papers
-
TileSight: A First-Principles Tile-Centric Analytical GPU Performance Model from Cores to Clusters
A tile-centric analytical model predicts GPU kernel latency and cache behavior purely from microbenchmark-calibrated hardware rates, reaching about 12% GEMM error and 13% end-to-end LLM serving error across five GPU lines.
-
MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference
MIST is a new simulator for heterogeneous multi-stage LLM inference that combines hardware traces with analytical models to explore configuration trade-offs in hybrid CPU-accelerator systems.
-
MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systems
A dimensionally strict analytical framework codifies 22 ML systems walls into 28 composable resolvers for sub-second full-stack design-space exploration and hardware synthesis.
-
SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving
Throughput saturation in LLM serving is caused by DRAM-bandwidth saturation in decode-phase attention kernels, and SLIM predicts this knee from sparse profiling to avoid wasteful batching.
-
LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving
Scaling on-chip M3D caches to 1–4 GB reduces simulated LLM serving energy by up to 44% in prefill and 30% in decode, but benefits saturate and are workload-dependent.
-
Think Before You Grid-Search: Floor-First Triage for LLM Serving
A five-dimensional resource-vector floor model computes latency bounds and capacity walls for LLM serving, predicting when TP16 or EP16+DP attention layouts dominate based on operating point.
-
Think Before You Grid-Search: Floor-First Triage for LLM Serving
LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.
-
SOLAR: AI-Powered Speed-of-Light Performance Analysis
SOLAR automates derivation of validated speed-of-light performance bounds for deep learning models from source code via LLM-to-IR translation and analytical computation.
-
Sim-FA: A GPGPU Simulator Framework for Fine-Grained Asynchronous Pipeline Analysis
A trace-driven simulator models Hopper TMA/WGMMA pipelines at WarpGroup granularity, matching H800 FlashAttention-3 latency to 5.7% MAPE, and its analytical model explains GenZ's long-sequence DRAM underestimation.
-
Sim-FA: A GPGPU Simulator Framework for Fine-Grained Asynchronous Pipeline Analysis
Sim-FA is a new simulator that instruments FlashAttention-3 for cycle-accurate GPGPU analysis, achieving 5.7% average error on H800 while explaining inaccuracies in existing DRAM traffic models.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.