Pith. sign in

REVIEW 15 cited by

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.06865 v2 pith:MUXVZ2YZ submitted 2023-03-13 cs.LG cs.AIcs.PF

Ying Sheng , Lianmin Zheng , Binhang Yuan , Zhuohan Li , Max Ryabinin , Daniel Y. Fu , Zhiqiang Xie , Beidi Chen
show 6 more authors
Clark Barrett Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang
This is my paper · ORCID
classification cs.LGcs.AIcs.PF
keywords flexgenhigh-throughputinferencememorysinglethroughputbatchbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tensors. FlexGen further compresses the weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at https://github.com/FMInference/FlexGen

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 47 citations worldwide. Full citation record

  1. ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression

    cs.LG 2024-12 conditional novelty 7.0 of 10

    ClusterKV recalls KV cache tokens by semantic clusters instead of fixed positions, preserving accuracy at small cache budgets while speeding up long-context LLM inference.

  2. Transition-Aware Backend Dispatch for Edge LLM Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Transition-aware backend dispatch—conditioning each operator's backend choice on the previous backend—reduces replayed latency, energy, and EDP by up to 17.4%, 14.4%, and 28.5% versus static assignment in seven edge-L...

  3. PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PagedWeight pages quantized MoE expert weights on and off the GPU at runtime, releasing memory to the KV cache while using sensitivity, routing, and prompt signals to choose which experts to shrink.

  4. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  5. High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A learned top-K distribution-restoration layer recovers near-FP16 long-context accuracy under 1-bit KV-cache quantization.

  6. IAM: Efficient Inference through Attention Mapping between Different-scale LLMs

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Attention matrices of same-series small and large LLMs are similar enough that replacing up to 50% of a large model's attention layers with the small model's matrices preserves most performance while reducing KV cache...

  7. Hardware-Efficient Attention for Fast Decoding

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.

  8. BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching

    cs.LG 2024-11 conditional novelty 6.0 of 10

    BlendServe combines resource-aware batching with prefix sharing using a resource-aware prefix tree and dual scanner, achieving up to 1.44x throughput vs vLLM/SGLang in offline LLM inference.

  9. Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Ternary language models trained on 1.2 trillion tokens continue to improve, and a new GPU kernel speeds up their inference up to 5x end-to-end.

  10. TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering

    cs.CL 2025-06 conditional novelty 5.0 of 10

    TaDA mean-centers key and value activations along the head dimension, quantizes the deviations with per-layer precision search, and matches 16-bit baseline accuracy while using about 27 to 35 percent of the cache memory.

  11. Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

    cs.LG 2025-01 conditional novelty 5.0 of 10

    A post-training quantization scheme that uses per-group leading-one detection to keep four salient bits from an 8/16-bit integer base, achieving 4-bit weights, activations, and KV cache without fine-tuning or rotation.

  12. Glinthawk: A Two-Tiered Architecture for Offline LLM Inference

    cs.LG 2025-01 conditional novelty 5.0 of 10

    A two-tier inference system that offloads attention and KV cache to cheap CPU nodes raises offline LLM throughput about 6x and lowers hardware cost about 2.8x in a T4-based prototype.

  13. More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Storing 4x as many KV cache tokens at 4-bit precision outperforms storing 1x tokens at 16-bit precision on long-context benchmarks at equal data-memory budgets.

  14. Taming the Titans: A Survey of Efficient LLM Inference Serving

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.

  15. Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud

    cs.DC 2024-11 unverdicted novelty 1.0 of 10

    A review of cold start mitigation for serverless LLM inference summarizes ServerlessLLM's multi-tier checkpoint loading and live migration, reporting 6-8x faster startup relative to PyTorch and SafeTensors without add...

Pith tools