Pith. sign in

REVIEW 7 cited by

FlashDecoding++: Faster Large Language Model Inference on GPUs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.01282 v4 pith:7WGZ73QC submitted 2023-11-02 cs.LG cs.CL

Ke Hong , Guohao Dai , Jiaming Xu , Qiuli Mao , Xiuhong Li , Jun Liu , Kangdi Chen , Yuhan Dong
show 1 more author
Yu Wang
This is my paper
classification cs.LGcs.CL
keywords flashdecodinginferencesoftmaxdataflowdifferentflathardwareperformance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

As the Large Language Model (LLM) becomes increasingly important in various domains. However, the following challenges still remain unsolved in accelerating LLM inference: (1) Synchronized partial softmax update. The softmax operation requires a synchronized update operation among each partial softmax result, leading to ~20% overheads for the attention computation in LLMs. (2) Under-utilized computation of flat GEMM. The shape of matrices performing GEMM in LLM inference is flat, leading to under-utilized computation and >50% performance loss after padding zeros in previous designs. (3) Performance loss due to static dataflow. Kernel performance in LLM depends on varied input data features, hardware configurations, etc. A single and static dataflow may lead to a 50.25% performance loss for GEMMs of different shapes in LLM inference. We present FlashDecoding++, a fast LLM inference engine supporting mainstream LLMs and hardware back-ends. To tackle the above challenges, FlashDecoding++ creatively proposes: (1) Asynchronized softmax with unified max value. FlashDecoding++ introduces a unified max value technique for different partial softmax computations to avoid synchronization. (2) Flat GEMM optimization with double buffering. FlashDecoding++ points out that flat GEMMs with different shapes face varied bottlenecks. Then, techniques like double buffering are introduced. (3) Heuristic dataflow with hardware resource adaptation. FlashDecoding++ heuristically optimizes dataflow using different hardware resource considering input dynamics. Due to the versatility of optimizations in FlashDecoding++, FlashDecoding++ can achieve up to 4.86x and 2.18x speedup on both NVIDIA and AMD GPUs compared to Hugging Face implementations. FlashDecoding++ also achieves an average speedup of 1.37x compared to state-of-the-art LLM inference engines on mainstream LLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models

    cs.CV 2025-09 conditional novelty 7.0 of 10

    FG-Attn accelerates video DiT inference by computing attention over only the relevant Mx1 key slices per query group, using an asynchronous gather-load kernel, and reports up to 1.65x end-to-end speedup with negligibl...

  2. Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling

    cs.DC 2026-08 conditional novelty 6.0 of 10

    AFlex combines operator-level disaggregation with per-operator DVFS to cut LLM serving energy per token by up to 49% without violating P90 TTFT/TPOT SLOs.

  3. QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A two-stage prompting workflow with a TL language lets LLMs generate FlashAttention-class GPU kernels that match or beat hand-optimized libraries on multiple GPU generations.

  4. SD-Acc: Accelerating Stable Diffusion through Phase-aware Sampling and Hardware Co-Optimizations

    cs.AR 2025-07 conditional novelty 5.0 of 10

    Phase-aware sampling cuts Stable Diffusion's compute by roughly 2.4x to 5.7x with only small CLIP-score changes, and the accompanying FPGA accelerator turns this into 2.7x to 6.0x energy savings over an Nvidia V100 GPU.

  5. TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering

    cs.CL 2025-06 conditional novelty 5.0 of 10

    TaDA mean-centers key and value activations along the head dimension, quantizes the deviations with per-layer precision search, and matches 16-bit baseline accuracy while using about 27 to 35 percent of the cache memory.

  6. Past-Future Scheduler for LLM Serving under SLA Guarantees

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A past-future scheduler predicts output lengths from historical requests and only admits new requests when the predicted future memory peak of the running batch fits in GPU memory.

  7. Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure

    cs.DC 2025-07 reject novelty 4.0 of 10

    A CXL-based disaggregated memory architecture with hybrid XLink interconnects is proposed and prototyped, claiming large speedups for memory-bound AI and HPC workloads.

Pith tools