Pith. sign in

REVIEW 3 major objections 3 minor 3 cited by

ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Cluster-level collective primitives fuse LLM decoding stages on-chip, achieving a 1.61x average speedup over leading inference frameworks.

desk verdict A genuine and well-designed systems contribution on fusing decoding operators over Hopper DSMEM, but one headline speedup pair is arithmetically impossible and the evaluation reporting needs cleanup before the 1.61x claim is fully credible. read the letter →

arxiv 2508.18850 v1 pith:LYESDLBR submitted 2025-08-26 cs.DC cs.AI

Xinhao Luo , Zihan Liu , Yangjie Zhou , Shihan Fang , Ziyu Huang , Yu Feng , Chen Zhang , Shixuan Sun
show 3 more authors
Zhenzhe Zheng Jingwen Leng Minyi Guo
This is my paper
classification cs.DCcs.AI
keywords LLMinferenceoperatorfusionthreadblockclustersdistributedsharedmemorycollectivecommunicationdecodinglatencyHopperGPUDSMEM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM token generation is dominated by decoding, where each token must pass through projection, attention, and output-projection kernels that today write intermediates to global memory and synchronize at kernel boundaries. ClusterFusion argues that Hopper GPUs' thread-block clusters and distributed shared memory can eliminate those round-trips: it introduces ClusterReduce and ClusterGather, two collective primitives that exchange and reduce data among blocks inside a cluster over the on-chip interconnect. With these primitives, a cluster of thread blocks computes a full attention head end-to-end in one fused kernel, keeping Q, K, V, softmax statistics, and attention output on chip. On H100, the resulting framework reports a 1.61x average end-to-end latency speedup over leading inference frameworks, with the largest gains on the core projection-and-attention modules and smaller speedups (roughly 1.1–1.2x) at batch size 16.

What carries the argument

ClusterReduce and ClusterGather are binary-tree collectives executed over log2 N rounds across N = 2^k thread blocks; each round doubles the communication stride and exchanges data through the SM-to-SM network (distributed shared memory, DSMEM) instead of global memory. ClusterReduce performs element-wise reduction with a constant message size, while ClusterGather accumulates remote data with a doubling message size. The framework maps each attention head to one cluster of blocks, keeping data-dependent dimensions inside the cluster and distributing data-independent dimensions across clusters, which allows a single fused kernel to carry QKV projection, partial-attention computation, online-s

What would settle it

Run the fused kernel at cluster size 4 while sweeping the number of concurrently active attention heads (e.g., 8, 32, 128) on an H100 and measure per-token latency against the same model run with unfused kernels. If end-to-end TPOT at high head counts, or with all SMs issuing DSMEM traffic simultaneously, does not stay below the best global-memory baseline, the claimed bandwidth trade-off and 1.61x average speedup would be falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the block-isolated execution model — where thread blocks resolve dependencies by writing to and reading from global memory — is not the only practical dataflow for decoding. By treating a thread-block cluster as a cooperative unit and exposing cluster-scoped collective communication as reusable primitives, the QKV projection, attention, and output projection of a Transformer block can be composed into one fused kernel. The cluster keeps the full Q/K/V vectors, softmax statistics, and attention output in on-chip memory: ClusterGather assembles per-block Q/K/V fragments, and ClusterReduce combines partial attention results and softmax statistics. On an H10

Load-bearing premise

The load-bearing premise is that the profiled SM-to-SM latency and bandwidth of distributed shared memory (about 190 cycles at cluster size 2, with bandwidth staying competitive only at small cluster sizes) remain true under real kernel execution with many concurrent clusters and synchronization waits; if DSMEM throughput degrades under contention or the sync/wait overhead dominates, the on-chip fused kernel would not beat global-memory baselines.

Editorial extensions

If this is right

  • If the claimed speedups hold, single-token decoding latency drops by about 38% on average, directly improving per-token latency for interactive LLM services.
  • The fused kernel cuts kernel launch overhead by nearly an order of magnitude even against CUDA-Graph-optimized baselines, so fusion scope, not launch cost, becomes the binding constraint.
  • Global memory traffic for the three core modules drops because Q/K/V, attention output, and softmax statistics never leave the cluster, reducing the memory system's role in decoding.
  • The optimal cluster size is workload-dependent (2–4 for typical head counts; larger clusters hurt due to interconnect latency and reduced active SMs), so clusterFusion implies cluster size should be a tunable parameter.
  • The same primitives generalize to latent-attention (MLA) style models, suggesting cluster-centric fusion applies beyond standard multi-head attention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence the paper does not develop: the same cluster collectives could fuse the FFN and normalization into the same kernel at small batch sizes, where the fused scope would then be bounded by shared-memory capacity rather than cluster size.
  • A testable extension: applying ClusterReduce and ClusterGather to prefilling or to MoE expert-parallel decoding, where partial results are reduced across experts, would reveal whether the 1.61x benefit transfers beyond the three fused modules.
  • The reported 1.1–1.2x speedup at batch size 16 suggests that on-chip fusion wins when decoding is memory-bound; as batch grows and compute intensity rises, the bottleneck shifts and the fusion benefit may shrink further or invert.
  • Because the microbenchmark latency (about 190 cycles at cluster size 2) is measured in isolation, a stress test with all SMs issuing concurrent DSMEM traffic would tell whether the claimed bandwidth trade-offs hold under real occupancy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The manuscript proposes ClusterFusion, an execution framework for LLM decoding on NVIDIA Hopper GPUs. It introduces two cluster-level collective primitives, ClusterReduce and ClusterGather, built on distributed shared memory (DSMEM), and uses them to fuse QKV projection, attention, and output projection into a single kernel for both standard MHA and DeepSeek MLA. The paper also develops a DSMEM traffic model to choose among dataflow variants (SplitToken vs. SplitHead), reports microbenchmarks, ablations, and end-to-end TPOT/core-module speedups on an H100 for Llama2-7B and DeepSeek-V2-Lite, and claims a 1.61x average end-to-end speedup over state-of-the-art inference frameworks. Source code is provided.

Significance. The central idea is timely and practically useful: Hopper DSMEM is currently exposed through low-level per-thread-block data movement, and a reusable cluster-collective abstraction that enables genuine operator fusion for decoding is a credible systems contribution. The paper's design is plausible, the traffic-model-based comparison of dataflow variants is a reasonable methodology, and the multi-model/multi-framework evaluation targets a real deployment bottleneck. The public source code and the explicit validation of the traffic model in Fig. 20 are strengths. However, the evaluation as printed contains internal inconsistencies that currently prevent the headline performance claim from being trusted without correction.

major comments (3)
  1. [§4.1, DeepSeek-V2-Lite/TensorRT-LLM results] The reported end-to-end speedup over TensorRT-LLM for DeepSeek-V2-Lite (1.51x) and the reported core fused-module speedup over TensorRT-LLM (1.35x) are mutually incompatible. Let C be the baseline core-module latency and O the latency of the remaining unaccelerated components (FFN, RMSNorm). The fused core latency is C/1.35, so the end-to-end speedup is (C+O)/(C/1.35 + O), which is at most 1.35. Since the paper states that FFN and RMSNorm adopt existing optimized implementations rather than new acceleration, the 1.51x value cannot be produced by the described system. Because this entry contributes to the headline 1.61x average, the abstract's central number is affected. The authors must identify which of the two numbers is misreported, rerun the comparison, or state explicitly if the two measurements cover different execution scopes/timers.
  2. [§4.1, §4.2, Appendix C.1] The main text cites Fig. 17 and Fig. 18 for the single-batch end-to-end and core-module results, but Figs. 17 and 18 in Appendix C.1 are the batch-size-16 results, with different speedup values (e.g., Llama2-7B TPOT speedups 1.11x/1.09x/1.12x/1.32x, not the cited 1.41x/1.39x/1.43x/2.03x). The single-batch data are in Figs. 8 and 9. Similarly, §4.2 refers to Fig. 19 for the single-batch memory-transfer/launch-overhead analysis, but Fig. 19 is the multi-batch panel; the single-batch panel is Fig. 12. The text also refers to 'Alg. 5' for the main fused dataflow, whereas Alg. 5 is the SplitHead variant in Appendix B.2; the main dataflow is Alg. 3. These cross-reference errors are not merely cosmetic: as printed, the quantitative claims in §4.1 cannot be verified against the figures to which they are attached.
  3. [§3.2, traffic model] The traffic formula for the fused dataflow is inconsistent with Algorithm 3. Algorithm 3 performs ClusterGather on the QKV tensors whose per-block size is 3h and ClusterReduce on the attention output whose size is H (plus negligible softmax statistics). The total DSMEM traffic should therefore be TrafficGather(3h, N) + TrafficReduce(H, N). The printed formula is TrafficReduce(3h, N) + TrafficGather(H, N), and Appendix B.2 later uses the opposite assignment for SplitToken. Since this analytical model is used to justify the choice of SplitToken over SplitHead, the formula should be corrected and the resulting comparison re-verified; the experimental validation in Fig. 20 is reassuring but does not remove the need for the formal expression to be correct.
minor comments (3)
  1. [Evaluation methodology] No run-to-run variation, number of repetitions, or error bars are reported for any benchmark figure (e.g., Figs. 8, 9, 11, 13, Table 1). A sentence describing the measurement methodology would improve reproducibility and make the reported differences more credible.
  2. [Figures and notation] Several smaller quality issues should be fixed: the caption/notation in Fig. 5 should state clearly whether the reported global bandwidth is per-SM, per-cluster, or aggregate; 'Obatin' appears in Alg. 3; 'SplieHead' appears in Alg. 5; 'Anslysis' appears in §4.3; 'NVIDA' appears in reference [27].
  3. [Appendix C.1] The multi-batch results show much smaller speedups (e.g., 1.07x-1.32x on Llama2-7B). The abstract's broad claim of '1.61x on average across different models and configurations' should be qualified to clarify that it refers to the batch-size-1 setting; otherwise readers may overgeneralize the headline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ClusterFusion's primitives, traffic-based dataflow choice, and benchmark claims are independent; the only self-citation is a non-load-bearing related-work mention.

full rationale

ClusterFusion's derivation chain is self-contained. The two primitives are defined by Algorithms 1 and 2, and the DSMEM traffic formulas in Section 3.2 and Appendix B are direct counts of the messages exchanged in those algorithms (size × log2 N × N for ClusterReduce, and the analogous gather count), so choosing SplitToken over SplitHead by comparing these counts is an analytical design choice, not a fitted prediction. The performance claims in Section 4.1 are benchmark measurements against SGLang, vLLM, TensorRT-LLM, and MLC-LLM on H100; none of these comparisons is an output of the traffic model. Cluster size is a tuned system configuration (Section 4.1), explicitly varied by workload, not a parameter fitted to produce the reported speedups. The only self-citation is Ref. [24] (VQ-LLM) in Appendix A's related-work list ("Quantization [22, 24]"); it is not a load-bearing premise, so it does not constitute circularity. The skeptical observation that the DeepSeek/TensorRT-LLM end-to-end speedup (1.51×) exceeds the corresponding core-module speedup (1.35×) is an internal-consistency or correctness concern, not a circularity concern: it does not show that any claimed result is equivalent to its inputs by construction. Consequently, no circular step is present.

Assumptions & free parameters 2 free parameters · 7 assumptions · 2 invented entities

The paper introduces two software primitives and a cluster-centric dataflow. The main free parameter is the cluster size, tuned per workload. The axioms are standard reduction/softmax math plus hardware behavior assumptions from the paper's own DSMEM profiling. No new physical entities are introduced; ClusterReduce and ClusterGather are implemented software abstractions with measured behavior.

free parameters (2)
  • Cluster size N (thread blocks per cluster) = 4 for main results; 2 for 128-head workloads; 8 and 16 tested but worse
    Tuned per configuration to maximize throughput (Fig. 11); the paper states the optimal cluster size varies across workloads (Sec. 4.1) and is therefore a chosen parameter.
  • Dataflow variant (SplitToken vs SplitHead) = SplitToken selected
    Selected via the analytic DSMEM traffic model (Sec. B.2) and confirmed by measurement (Fig. 20). It is a hand-chosen design option, not a continuous fitted constant.
assumptions (7)
  • standard math Reduction operators (sum, max) are associative and can be applied in any order across thread blocks.
    Used in ClusterReduce (Alg. 1) and in the online-softmax rescaling steps of Algs. 3 and 4.
  • standard math Online softmax rescaling using per-block local max/sum recovered via a cluster reduction yields numerically correct attention output.
    This is the standard FlashAttention/FlashDecoding math, invoked in Alg. 3 steps 5-6 and Alg. 4 steps 8-9.
  • domain assumption Hopper DSMEM provides the profiled latency (about 190 cycles at N=2) and bandwidth characteristics without hidden global-memory fallback.
    The entire fusion strategy depends on these measurements (Fig. 5); if real kernel execution with contention behaves differently, the speedup could vanish.
  • domain assumption A thread block cluster can be treated as a fully connected logical network for collective communication using exponential-stride exchange.
    Underlies Algs. 1 and 2; assumes a binary-tree pattern over DSMEM is correct and efficient for arbitrary cluster sizes that are powers of two.
  • domain assumption Mapping one attention head per cluster and keeping data-dependent dimensions within the cluster preserves correctness for both MHA and MLA.
    Used to structure the fused dataflow (Sec. 3.2 and Appendix B.1). It relies on attention heads being independent and the KV cache being readable on-chip.
  • ad hoc to paper The DSMEM traffic model (TrafficReduce/TrafficGather) is the dominant cost that determines the best dataflow variant.
    Used to select SplitToken over SplitHead (Sec. B.2). It is validated empirically but is a paper-specific modeling assumption, not a standard theorem.
  • domain assumption Global memory atomicAdd correctly accumulates partial output-projection results from different head clusters.
    Alg. 3 step 8 and Alg. 5 step 6 use atomicAdd to combine per-head partial contributions; correctness assumes no precision-sensitive ordering effects.
invented entities (2)
  • ClusterReduce independent evidence
    purpose: On-chip all-reduce of a tensor across thread blocks in a cluster via DSMEM.
    A software abstraction over PTX DSMEM operations; its latency is benchmarked on H100 in Table 1, providing external evidence of its behavior.
  • ClusterGather independent evidence
    purpose: On-chip all-gather of a tensor across thread blocks in a cluster.
    A software abstraction over PTX DSMEM operations; benchmarked on H100 in Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive." pith.science (2026). https://pith.science/paper/LYESDLBR

@misc{pith2026250818850,
  author       = {Pith},
  title        = {Pith review of: ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYESDLBR}},
  note         = {Machine review of arXiv:2508.18850}
}
read the original abstract

Large language model (LLM) decoding suffers from high latency due to fragmented execution across operators and heavy reliance on off-chip memory for data exchange and reduction. This execution model limits opportunities for fusion and incurs significant memory traffic and kernel launch overhead. While modern architectures such as NVIDIA Hopper provide distributed shared memory and low-latency intra-cluster interconnects, they expose only low-level data movement instructions, lacking structured abstractions for collective on-chip communication. To bridge this software-hardware gap, we introduce two cluster-level communication primitives, ClusterReduce and ClusterGather, which abstract common communication patterns and enable structured, high-speed data exchange and reduction between thread blocks within a cluster, allowing intermediate results to be on-chip without involving off-chip memory. Building on these abstractions, we design ClusterFusion, an execution framework that schedules communication and computation jointly to expand operator fusion scope by composing decoding stages such as QKV Projection, Attention, and Output Projection into a single fused kernels. Evaluations on H100 GPUs show that ClusterFusion outperforms state-of-the-art inference frameworks by 1.61x on average in end-to-end latency across different models and configurations. The source code is available at https://github.com/xinhao-luo/ClusterFusion.

Figures

Figures reproduced from arXiv: 2508.18850 by the authors.

Figure 1
Figure 1. Typical Transformer Block as the fundamental [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Existing QKV Projection, Attention, and Output Projection dataflow graph. Projection: Z = Concat  Softmax  Q1K⊤ √ 1 dk  V1, . . . , Softmax  QhK⊤ √ h dk  Vh  WO (1) Here, dk is the dimension of each head’s query (Q) and key (K). The FFN module applies three linear layers with a non-linear activation in between, formulated as: FFN(Z) = W3 (σ(W1Z) ⊙ W2Z) (2) where σ is a non-linear activation (e.g., GELU), and ⊙… view at source ↗
Figure 5
Figure 5. SM-to-SM access latency (left), bandwidth (middle), and [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (12 more)
Figure 6
Figure 6. Figure 6: Illustration of cluster-level collective communication primitives: [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Cluster-centric fused QKV Projection, Attention and Output Projection dataflow graph. 3.2 Cluster-Centric Dataflow Design with Primitives Building upon the proposed cluster-level primitives, we now illustrate how they are employed to construct a cluster-centric dataflo…
Figure 8
Figure 8. Figure 8: TPOT of Llama2-7B (left) and DeepSeek-V2-Lite (right) on H100. 505 955 SGLang MLC-LLM vLLM ClusterFusion TRT-LLM 565 566 577 581 618 Latency (us) 0 50 100 150 200 250 300 Sequence Length 1k 2k 4k 8k 16k Sequence Length 1k 2k 4k 8k 16k [PITH_FULL_IMAGE:figures/full_fig…
Figure 10
Figure 10. Figure 10: Sequence length distribution in ShareGPT [1] and Splitwise [35, 2] datasets. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Latency of core module in ClusterFusion with varying settings. SGLang vLLM ClusterFusion Llama2-7B DeepSeek-V2-Lite Memory Transfer Size (MB) 400 500 600 700 200 250 300 Sequence Length 1k 2k 4k 8k 16k Sequence Length 1k 2k 4k 8k 16k SGLang vLLM ClusterFusion Kernel L…
Figure 19
Figure 19. Figure 19: The performance gain stems from the fact that [PITH_FULL_IMAGE:figures/full_fig_p009_19.png]
Figure 13
Figure 13. Figure 13: TPOT of ClusterFusion on Llama2-7B with and without DSMEM. To demonstrate the importance of the on-chip interconnect leveraged in our dataflow design, we conduct a microbenchmark to evaluate the cluster-level collective communication primitives intro￾duced in Sec. 3.1…
Figure 14
Figure 14. Figure 14: Overview of MLA computation: original (left) and with weight absorption optimization [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: This dataflow performs the entire MLA computation on-chip, eliminating any intermediate [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Cluster-centric fused QKV Projection, Attention and Output Projection dataflow graph. on-chip register memory. As illustrated in Alg. 5, this dataflow is parallel in the number of heads. Each head corresponds to a cluster of N = 2k thread blocks (k ≤ 4), where each bl…
Figure 17
Figure 17. Figure 17: TPOT of Llama2-7B (left) and DeepSeek-V2-Lite (right) on H100. 2314 SGLang MLC-LLM vLLM ClusterFusion TRT-LLM 932 936 1016 1108 1436 Latency (us) 0 500 1000 1500 2000 0 200 400 600 800 Sequence Length 1k 2k 4k 8k 16k Sequence Length 1k 2k 4k 8k 16k [PITH_FULL_IMAGE:f…
Figure 19
Figure 19. Figure 19: Comparison of global memory data transfer size [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GPU-Tile-Sim: A Tile-Centric GPU Simulation Framework for LLM Hardware-Software Co-Design

    cs.DC 2026-07 conditional novelty 6.5 of 10

    A tile-graph GPU simulator predicts optimized LLM kernel performance on A100/H100 with MAPE about 1–9% by modeling dependencies and overlap rather than full instruction pipelines.

  2. CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernels

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    CuBridge adapts expert CUDA attention kernels via LLM-driven lift-transfer-lower to produce correct, high-performance implementations for new variants across GPUs.

  3. ClusterFusion++: Expanding Cluster-Level Fusion to Full Transformer-Block Decoding

    cs.DC 2026-04 unverdicted novelty 5.0 of 10

    ClusterFusion++ fuses the entire Transformer block (LayerNorm to residual) via CUDA extensions and achieves 1.34x throughput on Pythia-2.8B with near-identical output fidelity.

Reference graph

Works this paper leans on

54 extracted references · 37 canonical work pages · cited by 3 Pith papers

  1. [1]

    https://sharegpt.com/

    Sharegpt. https://sharegpt.com/

    work page

  2. [2]

    https://github.com/Azure/AzurePublicDataset/blob/master/ AzureLLMInferenceDataset2023.md

    Splitwise. https://github.com/Azure/AzurePublicDataset/blob/master/ AzureLLMInferenceDataset2023.md

    work page

  3. [3]

    https://patents.google.com/patent/US20230289189A1/en

    work page

  4. [4]

    Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale. In Felix Wolf, Sameer Shende, Candace Culhane, Sadaf R. Alam, and Heike Jagode, editors, SC22:...

    work page 2022

  5. [5]

    Jason Ansel, Edward Z. Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael V oznesen- sky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Mic...

    work page 2024

  6. [6]

    Cerebras wse-3 chip

    Cerebras. Cerebras wse-3 chip. https://www.cerebras.ai/chip

    work page

  7. [7]

    Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishna- murthy

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Q. Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishna- murthy. TVM: an automated end-to-end optimizing compiler for deep learning. In13th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2018, Carlsbad, CA, USA, October ...

    work page 2018

  8. [8]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024

    work page 2024

Show all 54 references
  1. [9]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS...

    2022

  2. [10]

    Flash-decoding for long-context inference

    Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context inference. https://crfm.stanford.edu/2023/10/12/flashdecoding.html, 2023

    2023

  3. [11]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Hao Yang, Haowei...

    2024 arXiv

  4. [12]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2024 arXiv

  5. [13]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y . Wu, Y . K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence. CoRR, abs/2401.14196, 2024

    2024 arXiv

  6. [14]

    Waferllm: A wafer-scale LLM inference system

    Congjie He, Yeqi Huang, Pei Mu, Ziming Miao, Jilong Xue, Lingxiao Ma, Fan Yang, and Luo Mai. Waferllm: A wafer-scale LLM inference system. CoRR, abs/2502.04563, 2025

    2025 arXiv

  7. [15]

    Flashdecoding++: Faster large language model inference with asynchronization, flat GEMM optimization, and heuristics

    Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference with asynchronization, flat GEMM optimization, and heuristics. In Proceedings of the Seventh Annual Conference on Machi...

    2024

  8. [16]

    Dissecting the graphcore IPU architecture via microbenchmarking

    Zhe Jia, Blake Tillman, Marco Maggioni, and Daniele Paolo Scarpazza. Dissecting the graphcore IPU architecture via microbenchmarking. CoRR, abs/1912.03413, 2019

    1912 arXiv

  9. [17]

    Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention, 2024

    2024

  10. [18]

    Flashmla: Efficient mla decoding kernels

    Shengyu Liu Jiashi Li. Flashmla: Efficient mla decoding kernels. https://github.com/ deepseek-ai/FlashMLA, 2025

    2025

  11. [19]

    Aamodt, and John Kim

    Zhixian Jin, Christopher Rocca, Jiho Kim, Hans Kasan, Minsoo Rhu, Ali Bakhoda, Tor M. Aamodt, and John Kim. Uncovering real gpu noc characteristics: Implications on intercon- nect architecture. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 8...

    2024

  12. [20]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, S...

    2023

  13. [21]

    Chatgpt: A meta-analysis after 2.5 months

    Christoph Leiter, Ran Zhang, Yanran Chen, Jonas Belouadi, Daniil Larionov, Vivian Fresen, and Steffen Eger. Chatgpt: A meta-analysis after 2.5 months. CoRR, abs/2302.13795, 2023. 11

    2023 arXiv

  14. [22]

    AWQ: activation-aware weight quantization for on-device LLM compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of the Seventh Annual Conference on Mac...

    2024

  15. [23]

    Scaling deep learning computation over the inter-core connected intelligence processor with T10

    Yiqi Liu, Yuqi Xue, Yu Cheng, Lingxiao Ma, Ziming Miao, Jilong Xue, and Jian Huang. Scaling deep learning computation over the inter-core connected intelligence processor with T10. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, SOSP 2024, Aust...

    2024

  16. [24]

    VQ- LLM: high-performance code generation for vector quantization augmented LLM inference

    Zihan Liu, Xinhao Luo, Junxian Guo, Wentao Ni, Yangjie Zhou, Yue Guan, Cong Guo, Weihao Cui, Yu Feng, Minyi Guo, Yuhao Zhu, Minjia Zhang, Chen Jin, and Jingwen Leng. VQ- LLM: high-performance code generation for vector quantization augmented LLM inference. In IEEE Internationa...

    2025

  17. [25]

    Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y . Zhang, Zhilin Yang, Xinyu Zhou, ...

    2025 arXiv

  18. [26]

    MLC-LLM, 2023-2025

    MLC team. MLC-LLM, 2023-2025

    2023

  19. [27]

    Nvidia hopper architecture

    NVIDA. Nvidia hopper architecture. https://www.nvidia.com/en-us/data-center/ technologies/hopper-architecture/

  20. [28]

    Cuda c++ programming guide

    NVIDIA. Cuda c++ programming guide. https://docs.nvidia.com/cuda/ cuda-c-programming-guide/index.html

  21. [29]

    Cudagraph

    NVIDIA. Cudagraph. https://developer.nvidia.com/blog/cuda-graphs/

  22. [30]

    NVIDIA. Cutlass. https://github.com/NVIDIA/cutlass

  23. [31]

    Nvidia nsight compute

    NVIDIA. Nvidia nsight compute. https://developer.nvidia.com/nsight-compute

  24. [32]

    Nvidia nsight system

    NVIDIA. Nvidia nsight system. https://developer.nvidia.com/nsight-systems

  25. [33]

    Tensorrt-llm

    NVIDIA. Tensorrt-llm. https://github.com/NVIDIA/TensorRT-LLM

  26. [34]

    Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, ...

    2019

  27. [35]

    Splitwise: Efficient generative LLM inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In 51st ACM/IEEE Annual International Symposium on Computer Architecture, ISCA 2024, Buenos Aires, ...

    2024

  28. [36]

    Code llama: Open foundation models for code

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défoss...

    2023 arXiv

  29. [37]

    Sglang deepseek model optimizations

    SGLang. Sglang deepseek model optimizations. https://github.com/sgl-project/ sgl-learning-materials/blob/main/slides/sglang_deepseek_model_ optimizations.pdf. 12

  30. [38]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Syst...

    2024

  31. [39]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024

    2024 arXiv

  32. [40]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019

    1911 arXiv

  33. [41]

    Welder: Scheduling deep learning memory access via tile-graph

    Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, and Lidong Zhou. Welder: Scheduling deep learning memory access via tile-graph. In 17th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2023, Boston, MA, USA, Ju...

    2023

  34. [42]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019

    1909 arXiv

  35. [43]

    A contrastive framework for neural text generation

    Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orlean...

    2022

  36. [44]

    Philippe Tillet, Hsiang-Tsung Kung, and David D. Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@PLDI 2019, Phoenix, AZ, U...

    2019

  37. [45]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  38. [46]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, De...

    2017

  39. [47]

    Optimizing deep learning inference via global analysis and tensor expres- sions

    Chunwei Xia, Jiacheng Zhao, Qianqi Sun, Zheng Wang, Yuan Wen, Teng Yu, Xiaobing Feng, and Huimin Cui. Optimizing deep learning inference via global analysis and tensor expres- sions. In Proceedings of the 29th ACM International Conference on Architectural Support for Programmi...

    2024

  40. [48]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via se...

    2024 arXiv

  41. [49]

    Flashinfer: Efficient and customizable attention engine for LLM inference serving

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for LLM inference serving. CoRR, abs/2501.01005, 2025

    2025 arXiv

  42. [50]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y . X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attentio...

    2025 arXiv

  43. [51]

    Bytetransformer: A high-performance transformer boosted for variable- length inputs

    Yujia Zhai, Chengquan Jiang, Leyuan Wang, Xiaoying Jia, Shang Zhang, Zizhong Chen, Xin Liu, and Yibo Zhu. Bytetransformer: A high-performance transformer boosted for variable- length inputs. In IEEE International Parallel and Distributed Processing Symposium, IPDPS 2023, St. P...

    2023

  44. [52]

    Gonzalez, Clark W

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Advances in Neural Informati...

    2024

  45. [53]

    Tileflow: A framework for modeling fusion dataflow via tree-based analysis

    Size Zheng, Siyuan Chen, Siyuan Gao, Liancheng Jia, Guangyu Sun, Runsheng Wang, and Yun Liang. Tileflow: A framework for modeling fusion dataflow via tree-based analysis. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2023, Toron...

    2023

  46. [54]

    Mononn: Enabling a new monolithic optimization space for neural network inference tasks on modern gpu-centric architectures

    Donglin Zhuang, Zhen Zheng, Haojun Xia, Xiafei Qiu, Junjie Bai, Wei Lin, and Shuaiwen Leon Song. Mononn: Enabling a new monolithic optimization space for neural network inference tasks on modern gpu-centric architectures. In 18th USENIX Symposium on Operating Systems Design an...

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.