Pith. sign in

REVIEW 46 cited by

LLM Inference Unveiled: Survey and Roofline Model Insights

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.16363 v6 pith:IUKHAW67 submitted 2024-02-26 cs.CL cs.AI

Zhihang Yuan , Yuzhang Shang , Yang Zhou , Zhen Dong , Zhe Zhou , Chenhao Xue , Bingzhe Wu , Zhikai Li
show 6 more authors
Qingyi Gu Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer
This is my paper · ORCID
classification cs.CLcs.AI
keywords inferencemodelefficientfieldframeworkhardwarerooflinesurvey
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The field of efficient Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Although the field has expanded and is vibrant, there hasn't been a concise framework that analyzes the various methods of LLM Inference to provide a clear understanding of this domain. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference techniques. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems, such as why LLMs are memory-bound, how much memory and computation they need, and how to choose the right hardware. We systematically collate the latest advancements in efficient LLM inference, covering crucial areas such as model compression (e.g., Knowledge Distillation and Quantization), algorithm improvements (e.g., Early Exit and Mixture-of-Expert), and both hardware and system-level enhancements. Our survey stands out by analyzing these methods with roofline model, helping us understand their impact on memory access and computation. This distinctive approach not only showcases the current research landscape but also delivers valuable insights for practical implementation, positioning our work as an indispensable resource for researchers new to the field as well as for those seeking to deepen their understanding of efficient LLM deployment. The analyze tool, LLM-Viewer, is open-sourced.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 46 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spec Sheets Are Not Kernels: An ISA- and Source-Level Audit of INT8 Availability on NVIDIA Blackwell Ultra

    cs.AR 2026-08 accept novelty 7.0 of 10

    On NVIDIA Blackwell Ultra, INT8 W8A8 is undeployable by default because the PTX ISA, CUTLASS, vLLM, and SGLang all lack a fifth-generation INT8 tensor-core path, despite the datasheet listing INT8 support.

  2. TileSight: A First-Principles Tile-Centric Analytical GPU Performance Model from Cores to Clusters

    cs.DC 2026-07 conditional novelty 7.0 of 10

    A tile-centric analytical model predicts GPU kernel latency and cache behavior purely from microbenchmark-calibrated hardware rates, reaching about 12% GEMM error and 13% end-to-end LLM serving error across five GPU lines.

  3. XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization

    cs.LG 2025-08 conditional novelty 7.0 of 10

    XQuant caches quantized layer inputs and rematerializes Keys and Values on the fly, achieving up to 12.5x KV cache memory savings with about 0.1 perplexity degradation.

  4. DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model

    cs.AR 2025-05 conditional novelty 7.0 of 10

    A near-core decompression accelerator plus a 3D roofline model speeds up compressed LLM matrix multiplication by up to 4x in simulation.

  5. FlatAttention: Dataflow and Fabric Collectives Co-Optimization for Efficient Multi-Head Attention on Tile-Based Many-PE Accelerators

    cs.AR 2025-05 conditional novelty 7.0 of 10

    FlatAttention groups tiles and uses NoC collectives to reduce HBM traffic by up to 16x and reach 89.3% utilization for multi-head attention on many-PE accelerators.

  6. Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format

    cs.AR 2024-11 conditional novelty 7.0 of 10

    Anda stores LLM activations in a group-shared-exponent format with per-module mantissa widths and uses bit-serial hardware that skips unused bits, reporting roughly 2.4x speedup, 4.0x area efficiency, and 3.1x energy ...

  7. MLSYSIM: First-Principles Infrastructure Modeling for Machine Learning Systems

    cs.DC 2026-06 accept novelty 6.5 of 10

    A dimensionally strict analytical framework codifies 22 ML systems walls into 28 composable resolvers for sub-second full-stack design-space exploration and hardware synthesis.

  8. SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Throughput saturation in LLM serving is caused by DRAM-bandwidth saturation in decode-phase attention kernels, and SLIM predicts this knee from sparse profiling to avoid wasteful batching.

  9. NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement

    cs.AR 2026-07 conditional novelty 6.0 of 10

    A heterogeneous GPU-PNM serving system that routes short prompts to GPUs and long prompts to near-memory sparse-attention devices, with runtime migration for growing contexts, reports large throughput and tail-latency...

  10. HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding

    cs.AR 2026-07 conditional novelty 6.0 of 10

    A two-stage token-then-element KV cache pruning scheme with a reconfigurable sorting accelerator is reported to reduce decoding memory traffic by ~7x and energy by up to 90% while keeping accuracy within ~1%.

  11. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    cs.PF 2026-07 conditional novelty 6.0 of 10

    LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.

  12. KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Sliding-window bidirectional-attention KV compression with Token2Chunk and periodic PagedAttention eviction keeps near-full reasoning accuracy at ~20% cache while lifting memory-constrained throughput ~12.75%.

  13. Sim-FA: A GPGPU Simulator Framework for Fine-Grained Asynchronous Pipeline Analysis

    cs.AR 2026-05 unverdicted novelty 6.0 of 10

    Sim-FA is a new simulator that instruments FlashAttention-3 for cycle-accurate GPGPU analysis, achieving 5.7% average error on H800 while explaining inaccuracies in existing DRAM traffic models.

  14. Teaching LLMs to Speak Spectroscopy

    astro-ph.IM 2025-08 conditional novelty 6.0 of 10

    A LLaMA-3.1-8B model fine-tuned with LoRA on digit-serialized SDSS spectra predicts redshifts with MAE 0.043 and retains 85% of its astronomy QA performance.

  15. Kinetics: Rethinking Test-Time Scaling Laws

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.

  16. LatentLLM: Attention-Aware Joint Tensor Compression

    cs.LG 2025-05 conditional novelty 6.0 of 10

    LatentLLM compresses pretrained LLMs and multimodal models with attention-aware joint low-rank tensor decomposition, outperforming SVD-based baselines on OPT perplexity and LLaVA ScienceQA.

  17. Scaling Law for Quantization-Aware Training

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.

  18. Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression

    cs.AR 2025-05 conditional novelty 6.0 of 10

    Ecco compresses LLM weights and KV cache with shared k-means patterns and Huffman coding, and adds a parallel hardware decoder, to roughly quadruple memory capacity and speed up decode by up to ~3x.

  19. Radio: Rate-Distortion Optimization for Large Language Model Compression

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Radio assigns bit depths to LLM weight groups by minimizing a rate-distortion objective via dual ascent, improving perplexity at 2-4 bits compared with GPTQ, AWQ, and OWQ.

  20. CarbonCall: Sustainability-Aware Function Calling for Large Language Models on Edge Devices

    cs.PF 2025-04 conditional novelty 6.0 of 10

    A sustainability-aware function-calling framework that prunes tools, throttles power to match grid carbon intensity, and switches between quantized LLM variants to cut carbon emissions on edge devices.

  21. SplitReason: Learning To Offload Reasoning

    cs.CL 2025-04 conditional novelty 6.0 of 10

    A 1.5-billion-parameter reasoning model is trained to emit special tags that offload the hardest parts of its chain of thought to a 32-billion-parameter model, improving AIME24 accuracy by 28.3 percentage points while...

  22. Memory Offloading for Large Language Model Inference with Latency SLO Guarantees

    cs.DC 2025-02 conditional novelty 6.0 of 10

    Select-N meets LLM latency SLOs by choosing and continuously adjusting an offloading interval that controls how much model state lives in CPU memory, increasing host memory usage and throughput versus prior offloading...

  23. EcoServe: Designing Carbon-Aware AI Inference Systems

    cs.DC 2025-02 conditional novelty 6.0 of 10

    EcoServe combines four strategies (reuse, rightsize, reduce, recycle) in an ILP optimizer to cut modeled carbon emissions for LLM serving by up to 47% while keeping SLOs.

  24. Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A training-free method that selects a subset of prompt tokens with a small speculator model to accelerate LLM prefill, yielding up to 7.66x TTFT speedup on Llama-3.1-405B.

  25. Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correction

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A DQN-based candidate selection and a competitive correction block improve MT ensembling quality while reducing inference cost on English-Hindi and Hindi-English tasks.

  26. BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference

    cs.CL 2025-01 conditional novelty 6.0 of 10

    BlockDialect assigns one of 16 FP4 'dialect' formats to each 32-64 element block of weights and activations, achieving near-full-precision accuracy with roughly 4-bit storage and integer-friendly MACs.

  27. DFModel: Design Space Optimization of Large-Scale Systems Exploiting Dataflow Mappings

    cs.AR 2024-12 conditional novelty 6.0 of 10

    DFModel formulates inter-chip and intra-chip dataflow mapping as a mixed-integer program and claims near-optimal mappings for LLM, DLRM, HPL, and FFT workloads on a wide range of accelerator systems.

  28. E-CAR: Efficient Continuous Autoregressive Image Generation via Multistage Modeling

    cs.CV 2024-12 reject novelty 6.0 of 10

    A stage-wise continuous autoregressive model with multistage flow matching gets large speedups on 256x256 ImageNet generation, but with a clear FID cost versus DiT and MAR.

  29. SpeechPrune: Context-aware Token Pruning for Speech Information Retrieval

    eess.AS 2024-12 conditional novelty 6.0 of 10

    A query-guided token pruning method improves speech-LLM accuracy on a new long-form audio benchmark while cutting computation.

  30. EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importance

    cs.CL 2024-12 conditional novelty 6.0 of 10

    EMS improves KV cache compression with a Global-Local importance score and a head-wise Evict-then-Merge strategy, outperforming prior methods on LongBench and Needle-in-a-Haystack.

  31. LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

    cs.AI 2026-08 conditional novelty 5.0 of 10

    LLMVisor fits a piecewise-linear formula over token counts, context length, squared tokens, and batch size to predict and attribute LLM batch latency in real time.

  32. A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computing

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Tunable energy landscapes whose thermal averages equal sigmoid, softmax, and matrix-vector products can, in principle, form the basis of a low-energy analog computer, with a superconducting double-well device as a fir...

  33. HPIM: Heterogeneous Processing-In-Memory-based Accelerator for Large Language Models Inference

    cs.AR 2025-09 conditional novelty 5.0 of 10

    A heterogeneous processing-in-memory accelerator that runs attention in SRAM and weight-heavy matrix-vector products in HBM reportedly cuts single-batch LLM decoding latency by up to 34x in simulation.

  34. APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A bit-sliced Tensor Core scheme with a bipolar integer encoding and adaptive kernel autotuning reports 1.65-3.99x speedups for ultra-low-bit GPTQ-quantized LLM inference on three NVIDIA GPUs.

  35. MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization

    cs.CV 2025-02 conditional novelty 5.0 of 10

    MQuant quantizes multimodal LLMs to W4A8 using per-modality static scales, token reordering, and Hadamard-outlier suppression, claiming near-floating-point accuracy and up to 30% latency reduction on five MLLMs.

  36. Genetic AI: Evolutionary Games for ab initio dynamic Multi-Objective Optimization

    cs.NE 2025-01 reject novelty 5.0 of 10

    Genetic AI uses evolutionary game dynamics over data rows and columns to rank discrete solutions without training data or predefined weights.

  37. Multi-matrix Factorization Attention

    cs.LG 2024-12 conditional novelty 5.0 of 10

    MFA and MFA-KR factorize the attention QK circuit with shared low-rank key and value projections, matching or beating MHA accuracy at a small fraction of the KV cache.

  38. Less is More: Optimizing Function Calling for LLM Execution on Edge Devices

    cs.PF 2024-11 conditional novelty 5.0 of 10

    Selectively reducing the number of tools presented to an LLM, using embedding similarity over individual tools or clusters, improves function-calling success and efficiency on edge devices.

  39. RAILS: Retrieval-Augmented Intelligence for Learning Software Development

    cs.SE 2025-06 reject novelty 4.0 of 10

    RAILS, a RAG pipeline with FAISS retrieval and a compile-error feedback loop, is reported to outperform baseline GPT-3.5-Turbo prompting on 78 Java import error cases.

  40. Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques

    cs.LG 2025-06 unverdicted novelty 4.0 of 10

    A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.

  41. $\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.

  42. How to keep pushing ML accelerator performance? Know your rooflines!

    cs.AR 2025-05 conditional novelty 4.0 of 10

    A JSSC survey organizing ML accelerator techniques around two roofline curves, one for throughput and one for energy efficiency, with the insight that their bottleneck knees often sit at different arithmetic intensities.

  43. Token Sequence Compression for Efficient Multimodal Computing

    cs.CV 2025-04 conditional novelty 4.0 of 10

    Simple cluster-based averaging of visual tokens is competitive with, and sometimes better than, attention-based token selection on LLaVA and VILA benchmarks, though the advantage over prior methods is mixed.

  44. Taming the Titans: A Survey of Efficient LLM Inference Serving

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.

  45. A Survey on Inference Optimization Techniques for Mixture of Experts Models

    cs.LG 2024-12 conditional novelty 3.0 of 10

    A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.

  46. freePruner: A Training-free Approach for Large Multimodal Model Acceleration

    cs.CV 2024-11 conditional novelty 3.0 of 10

    freePruner selects 50 percent of visual tokens using attention-based importance and keeps accuracy close to the original model, enabling a training-free about 2x acceleration for LMMs.

Pith tools