Pith. sign in

REVIEW 2 major objections 4 minor 54 references

CTA-pipelining runs dependent GPU kernels concurrently across devices at CTA granularity, cutting single-batch multi-layer GEMM latency by up to 31.8% versus micro-batching and 29.6% versus tensor parallelism.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 16:29 UTC pith:2ROHNTB3

load-bearing objection Solid systems paper with real multi-GPU CTA-level spatial pipelining numbers on H200/B200; gains are real for pure GEMM/MLP but transfer to full transformers is still open. the 2 major comments →

arxiv 2607.07862 v1 pith:2ROHNTB3 submitted 2026-07-08 cs.DC cs.LG

CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems

Tingkai Liu , Muralidhar Andoorveedu , Sanjoy Das , Sanjay Patel , Volodymyr Kindratenko This is my paper
classification cs.DC cs.LG
keywords CTA-pipeliningmulti-GPU systemslatency-oriented scalingtensor parallelismmicro-batchingshared-memory multi-GPULLM inferenceGEMM
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern multi-GPU systems already present a unified shared-memory space over NVLink, yet software still treats the fabric as a network and still relies on Tensor Parallelism or coarse micro-batching for latency. The paper claims that a finer spatial technique is possible: launch data-dependent kernels simultaneously on different GPUs and let each Cooperative Thread Array (CTA) of the consumer start as soon as its producer tiles become ready. The only kernel changes needed are short prologue and epilogue snippets that consult a cross-device work-queue and scoreboard; the original compute body is left untouched. On 2-layer GEMMs that stand in for MLP blocks, the method reduces end-to-end latency by as much as 31.8 percent versus optimally tuned micro-batching and 29.6 percent versus Tensor Parallelism, and can be stacked with Tensor Parallelism as an orthogonal scaling dimension. The result matters for any latency-critical single-batch inference path that still has sequential operator chains long enough to pipeline across devices.

Core claim

CTA-pipelining is a practical spatial scaling method that lets dependent kernels run concurrently across GPUs by exposing only CTA-level data dependencies through a lightweight shared-memory protocol (dependency array, atomic scoreboard, inter-device work-queue). When integrated into both classical and warp-specialized multi-stage persistent GEMM kernels, the protocol overhead is either negligible or completely hidden inside existing pipeline stalls, yielding up to 31.8 percent lower latency than micro-batching and 29.6 percent lower latency than Tensor Parallelism on representative MLP-style 2-layer GEMMs, while remaining combinable with Tensor Parallelism.

What carries the argument

The CTA-pipelining protocol: producer CTAs write tiles across NVLink, then atomically decrement a scoreboard; when a consumer CTA becomes ready its ID is pushed into a consumer-side work-queue that the consumer prologue polls, remapping its own CTA ID before executing the original kernel body.

Load-bearing premise

The measured gains on pure multi-layer GEMMs with fixed square weights and no intermediate non-linearities or residual paths will still appear once residual connections, layer-norm and attention kernels interrupt clean CTA-level producer-consumer chains.

What would settle it

Replace the pure 2-layer GEMM chain with a full transformer MLP block that includes residual add, layer-norm and GeLU (or SwiGLU) between the two matrix multiplies, re-measure single-batch latency against the same micro-batching and Tensor-Parallelism baselines on the identical 8-GPU B200 system, and check whether the reported 29–32 percent reductions survive.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes CTA-pipelining, a software protocol that injects lightweight prologue/epilogue snippets into existing GPU kernels so that data-dependent kernels can execute concurrently across GPUs at CTA granularity, using a dependency array, scoreboard of atomic counters, and an inter-device workqueue placed in consumer memory. Integration is shown for both classical SM90 and warp-specialized multi-stage persistent SM100 CUTLASS GEMM kernels; protocol overhead is quantified via injected timestamps (Fig. 2). On multi-layer GEMMs that stand in for MLP layers (fixed 8192 imes8192 BF16 weights), the method is compared against cuBLAS micro-batching (chunk-size sweep) and NCCL Tensor Parallelism on 8-GPU H200/B200 systems, reporting up to 31.8 % and 29.6 % latency reductions respectively, and is further shown to compose with TP by reducing All-Reduce world size while preserving larger per-GPU tiles.

Significance. If the measured gains hold under the stated scope, the work supplies a practical, minimally invasive spatial-scaling dimension that sits between inter-layer pipeline parallelism and operator-level tensor parallelism, and that can be combined with the latter. The concrete CUTLASS integration (both classical and warp-specialized styles), the explicit overhead traces that demonstrate hiding inside multi-stage pipelines, and the head-to-head numbers against strong cuBLAS/NCCL baselines on current H200/B200 hardware constitute reproducible engineering evidence rather than a purely conceptual proposal. The discussion of NVLink topology limits and possible hardware assists for Lamport-style consistency further sketches a co-design path for future shared-memory multi-GPU systems.

major comments (2)
  1. Section IV and the abstract carefully scope the strongest claim to “2-layer GEMM, representing the MLP operation.” The dependency-array construction in §III-A assumes a pure tile-to-tile producer-consumer chain. Residual adds, LayerNorm, and attention kernels that appear in a full transformer block break that chain; the manuscript never demonstrates that the same prologue/epilogue protocol still yields net latency reduction once those operators are present. A single end-to-end transformer-layer micro-benchmark (or an explicit statement that the claim is limited to pure GEMM pipelines) is needed before the abstract’s LLM-serving framing can be taken at face value.
  2. Figure 7 and the accompanying text claim that combining CTA-pipelining with TP reduces both communication volume and, in some regimes, pure compute time. The compute-time benefit is attributed to “preserving larger matrix dimensions,” yet the same section notes an offsetting one-wave ramp-up cost. No ablation isolates the two effects (e.g., pure TP at the reduced world size versus the hybrid schedule). Without that separation it is unclear how much of the reported gain is truly orthogonal scaling versus simply a different TP degree.
minor comments (4)
  1. Figure 2 caption states “time is not at the correct scale”; absolute micro-second values are given in the text, but a consistent scale bar or dual-axis presentation would make the hiding argument easier to verify.
  2. The optional cuStreamWaitValue32 micro-optimization is mentioned in §III-A5 but never quantified; a one-line latency comparison would clarify whether it is recommended in practice.
  3. Typographical inconsistencies appear (“V olodymyr”, “TP .”, “R-MB” table header); a final proof-reading pass is warranted.
  4. Related-work discussion of Kitsune [46] and mega-kernels is present but brief; a short paragraph contrasting software-only CTA-pipelining with those hardware-centric or recompilation-heavy approaches would strengthen positioning.

Circularity Check

0 steps flagged

No significant circularity; latency claims are direct wall-clock measurements against independent cuBLAS/NCCL baselines on real hardware, not reductions of fitted parameters or self-defined quantities.

full rationale

The paper's central claims (up to 31.8% vs micro-batching and 29.6% vs TP on 2-layer GEMM) are empirical end-to-end latencies measured with Nsight Systems on H200/B200 systems using unmodified CUTLASS kernels plus injected prologue/epilogue, compared to strong independent baselines (cuBLAS + NCCL AllReduce under Megatron-style sharding). Protocol overhead is profiled via injected timestamps (Fig. 2) rather than derived from a fitted model. Dependency arrays and scoreboards are constructed from static tile dataflow of the GEMM itself, not from a circular definition of the reported speedup. Citations are to external libraries (CUTLASS, cuBLAS, NCCL) and standard parallelism literature (Megatron-LM, GPipe, etc.); none is a load-bearing uniqueness theorem or ansatz imported from overlapping authors that forces the result. No equation equates a predicted quantity to a fitted input by construction. The work is therefore self-contained against its stated benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

The central empirical claim rests on standard CUDA/NVLink memory semantics, the correctness of the injected prologue/epilogue protocol, and the representativeness of multi-layer GEMM for MLP latency. No free parameters are fitted to produce the speedups; the invented entities are the protocol data structures themselves, which are engineering constructs rather than physical postulates.

axioms (3)
  • domain assumption CUDA asynchronous writes plus system-wide threadfence provide correct cross-device visibility ordering for producer-consumer hand-off over NVLink.
    Invoked throughout Section III-A; without it the scoreboard/workqueue protocol can observe stale data.
  • domain assumption Multi-layer GEMM with fixed 8192×8192 weights and no intermediate activations is a faithful proxy for MLP latency behavior under the evaluated sequence lengths.
    Stated in Section IV; the strongest claim is framed around this proxy.
  • domain assumption Warp-specialized multi-stage persistent kernels leave sufficient idle warp time to hide prologue/epilogue and fence latency.
    Used in Section III-C and Figure 2b to claim near-zero visible overhead.
invented entities (1)
  • CTA-pipelining dependency structure (dependency array + scoreboard + inter-device workqueue) no independent evidence
    purpose: Enable correct, low-overhead CTA-level producer-consumer signaling across GPUs without rewriting core kernels.
    Core engineering construct introduced in Section III-A; no independent physical existence outside the protocol.

pith-pipeline@v1.1.0-grok45 · 23243 in / 2614 out tokens · 23560 ms · 2026-07-10T16:29:17.080248+00:00 · methodology

0 comments
read the original abstract

The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated shared-memory structures. However, current software still mostly treats these coherent interconnects simply as high-speed networks. Simultaneously, the demand for serving Large Language Models under latency constraints has shifted GPU workload optimization from being throughput-driven to latency-bound, necessitating latency-oriented scaling methods beyond Tensor Parallelism (TP). Thus, we introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems. As a latency-oriented spatial scaling technique, CTA-pipelining leverages dependencies at the Cooperative Thread Array level, enabling concurrent execution of dependent kernels across GPUs. We demonstrate its capability using CUTLASS, cuBLAS, and NCCL libraries on 8-GPU H200 and B200 systems. Results show on 2-layer GEMM, representing the MLP operation, CTA-pipelining reduces latency by up to 31.8% compared to micro-batching, and 29.6% compared to TP. It can also be combined with TP as an orthogonal scaling dimension to further push the latency boundary.

Figures

Figures reproduced from arXiv: 2607.07862 by Muralidhar Andoorveedu, Sanjay Patel, Sanjoy Das, Tingkai Liu, Volodymyr Kindratenko.

Figure 1
Figure 1. Figure 1: Setups of enabling CTA-pipelining across 2 GPUs. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: First, the producer CTA writes its output directly [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The detail execution trace of adding CTA-pipelining [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Multi-layer GEMM (16384×8192×8192) latency re￾duction, comparing CTA-pipelining against micro-batching implemented with cuBLAS. The number of GEMM layers increases with the number of GPUs. but also from cross-device write latency, as the final wave of TMA writes must post before the kernel can terminate. In con￾trast, our execution trace analysis in Figure 2b demonstrates that, while CTA-pipelining require… view at source ↗
Figure 5
Figure 5. Figure 5: CTA-Pipelining latency vs. micro-batching and Tensor [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The experiment setup for comparing executing 2 Layers [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The execution latency breakdown of 2-layer GEMM with different input sequence lengths, comparing pure TP and the [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 54 canonical work pages · 8 internal anchors

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” inProceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY , USA: Curran Associates Inc., 2017, p. 6000–6010. [Online]. Available: https://papers.nips.cc/paper files...

    work page 2017

  2. [2]

    Scaling Laws for Neural Language Models

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” 2020. [Online]. Available: https://arxiv.org/abs/2001.08361

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  3. [3]

    Taming the Titans: A Survey of Efficient LLM Inference Serving

    R. Zhen, J. Li, Y . Ji, Z. Yang, T. Liu, Q. Xia, X. Duan, Z. Wang, B. Huai, and M. Zhang, “Taming the titans: A survey of efficient llm inference serving,” 2025. [Online]. Available: https://arxiv.org/abs/2504.19720

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in llm inference with sarathi-serve,” inProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’24. USA: USENIX Association, 2024. [Online]. Available: https://dl.acm.org/doi/10.55...

    work page doi:10.5555/3691938.3691945 2024

  5. [5]

    Nvidia gb200 nvl72: Rack-scale system built for the age of ai reasoning

    NVIDIA, “Nvidia gb200 nvl72: Rack-scale system built for the age of ai reasoning.” [Online]. Available: https://www.nvidia.com/en-us/ data-center/gb200-nvl72/

    work page

  6. [6]

    Nvidia nvlink and nvlink switch

    ——, “Nvidia nvlink and nvlink switch.” [Online]. Available: https://www.nvidia.com/en-us/data-center/gb200-nvl72/

    work page

  7. [7]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 611–626. [Online]. Availabl...

    work page doi:10.1145/3600006.3613165 2023

  8. [8]

    Sglang: efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y . Sheng, “Sglang: efficient execution of structured language model programs,” inProceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS ’24. Red Hook, NY , USA: Curran Associates Inc., 2024. ...

    work page 2024

  9. [9]

    llm-d: a kubernetes-native high-performance distributed llm inference framework

    llm d, “llm-d: a kubernetes-native high-performance distributed llm inference framework.” [Online]. Available: https://llm-d.ai

    work page

  10. [10]

    Aibrix: Towards scalable, cost-effective large language model inference infrastructure,

    T. A. Team, J. Shan, V . Gupta, L. Xu, H. Shi, J. Zhang, N. Wang, L. Xu, R. Kang, T. Liu, Y . Zhang, Y . Zhu, S. Jin, G. Lim, B. Chen, Z. Chen, X. Liu, X. Chen, K. Yin, C.-P. Chung, C. Jiang, Y . Lu, J. Chen, C. Lin, W. Xiang, R. Shi, and L. Xie, “Aibrix: Towards scalable, cost-effective large language model inference infrastructure,”

    work page

  11. [11]

    AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure

    [Online]. Available: https://arxiv.org/abs/2504.03648

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Nvidia dynamo: Scale and serve ai inference—fast

    NVIDIA, “Nvidia dynamo: Scale and serve ai inference—fast.” [Online]. Available: https://www.nvidia.com/en-us/ai/dynamo/

    work page

  13. [13]

    Parallelkittens: Systematic and practical simplification of multi-gpu ai kernels,

    S. H. Sul, S. Arora, B. F. Spector, and C. R ´e, “Parallelkittens: Systematic and practical simplification of multi-gpu ai kernels,” 2025. [Online]. Available: https://arxiv.org/abs/2511.13940

    work page arXiv 2025

  14. [14]

    Lightning: Scaling the gpu programming model beyond a single gpu,

    S. Heldens, P. Hijma, B. Van Werkhoven, J. Maassen, and R. V . van Nieuwpoort, “Lightning: Scaling the gpu programming model beyond a single gpu,” in2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2022, pp. 492–503. [Online]. Available: https://doi.org/10.1109/IPDPS53621.2022.00054

    work page doi:10.1109/ipdps53621.2022.00054 2022

  15. [15]

    Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

    S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V . Korthikanti, E. Zhang, R. Child, R. Y . Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y . He, M. Houston, S. Tiwary, and B. Catanzaro, “Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model,” 202...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  16. [16]

    Alpa: Automating inter- and Intra-Operator parallelism for distributed deep learning,

    L. Zheng, Z. Li, H. Zhang, Y . Zhuang, Z. Chen, Y . Huang, Y . Wang, Y . Xu, D. Zhuo, E. P. Xing, J. E. Gonzalez, and I. Stoica, “Alpa: Automating inter- and Intra-Operator parallelism for distributed deep learning,” in16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp. 559–578. ...

    work page 2022

  17. [17]

    Efficient large-scale language model training on gpu clusters using megatron-lm,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V . Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Ana...

    work page doi:10.1145/3458817.3476209 2021

  18. [18]

    Synergistic tensor and pipeline parallelism,

    M. Qi, J. Peng, J. Zhang, J. Zhu, Y . Li, and H. Ma, “Synergistic tensor and pipeline parallelism,” 2025. [Online]. Available: https: //arxiv.org/abs/2510.27257

    work page arXiv 2025

  19. [19]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism,

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V . Le, Y . Wu, and z. Chen, “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” inAdvances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/201...

    work page 2019

  20. [20]

    Pipedream: generalized pipeline parallelism for dnn training,

    D. Narayanan, A. Harlap, A. Phanishayee, V . Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” inProceedings of the 27th ACM Symposium on Operating Systems Principles, ser. SOSP ’19. New York, NY , USA: Association for Computing Machinery, 2019, pp. 1–15. [Online]. Availab...

    work page doi:10.1145/3341301.3359646 2019

  21. [21]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020. [Online]. Available: https://arxiv.org/abs/1909.08053

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  22. [22]

    Megablocks: Efficient sparse training with mixture-of-experts,

    T. Gale, D. Narayanan, C. Young, and M. Zaharia, “Megablocks: Efficient sparse training with mixture-of-experts,” inProceedings of Machine Learning and Systems, D. Song, M. Carbin, and T. Chen, Eds., vol. 5. Curan, 2023, pp. 288–304. [Online]. Available: https://proceedings.mlsys.org/paper files/paper/ 2023/file/5a54f79333768effe7e8927bcccffe40-Paper-mlsy...

    work page 2023

  23. [23]

    Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving,” inProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’24. USA: USENIX Association, 2024. [Online]. Available: https://dl.acm.org/doi/1...

    work page doi:10.5555/3691938.3691949 2024

  24. [24]

    Flashattention: fast and memory-efficient exact attention with io- awareness,

    T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: fast and memory-efficient exact attention with io- awareness,” inProceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY , USA: Curran Associates Inc., 2022. [Online]. Available: https://proceedings.neurips.cc/paper files/pap...

    work page 2022

  25. [25]

    Neutrino Production via $e^-e^+$ Collision at $Z$-boson Peak

    R. Y . Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y . He, “Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale,” inSC22: International Conference for High Performance Computing, Networking, Storage and Analysis, 2022, pp. 1–15. [Online]. Available...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1109/sc41404.2022.00051 2022

  26. [26]

    Mirage: A multi-level superoptimizer for tensor programs,

    M. Wu, X. Cheng, S. Liu, C. Shi, J. Ji, K. Ao, P. Velliengiri, X. Miao, O. Padon, and Z. Jia, “Mirage: A multi-level superoptimizer for tensor programs,” in19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). Boston, MA: USENIX Association, Jul. 2025. [Online]. Available: https://dl.acm.org/doi/10.5555/3767901. 3767914

    work page doi:10.5555/3767901 2025

  27. [27]

    Mirage persistent kernel: A compiler and runtime for mega-kernelizing tensor programs,

    X. Cheng, Z. Zhang, Y . Zhou, J. Ji, J. Jiang, Z. Zhao, Z. Xiao, Z. Ye, Y . Huang, R. Lai, H. Jin, B. Hou, M. Wu, Y . Dong, A. Yip, Z. Ye, S. Wang, W. Yang, X. Miao, T. Chen, and Z. Jia, “Mirage persistent kernel: A compiler and runtime for mega-kernelizing tensor programs,”

    work page

  28. [28]

    Available: https://arxiv.org/abs/2512.22219

    [Online]. Available: https://arxiv.org/abs/2512.22219

    work page arXiv

  29. [29]

    SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

    A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee, “Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills,” 2023. [Online]. Available: https: //arxiv.org/abs/2308.16369

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    Cutlass library

    NVIDIA, “Cutlass library.” [Online]. Available: https://docs.nvidia.com/ cutlass/latest/

    work page

  31. [31]

    cublas: Basic linear algebra on nvidia gpus

    ——, “cublas: Basic linear algebra on nvidia gpus.” [Online]. Available: https://developer.nvidia.com/cublas

    work page

  32. [32]

    Nvidia collective communications library (nccl)

    ——, “Nvidia collective communications library (nccl).” [Online]. Available: https://developer.nvidia.com/nccl

    work page

  33. [33]

    Cuda programming guide

    ——, “Cuda programming guide.” [Online]. Available: https://docs. nvidia.com/cuda/cuda-programming-guide/index.html

    work page

  34. [34]

    Nvidia tesla: A unified graphics and computing architecture,

    E. Lindholm, J. Nickolls, S. Oberman, and J. Montrym, “Nvidia tesla: A unified graphics and computing architecture,”IEEE Micro, vol. 28, no. 2, pp. 39–55, 2008. [Online]. Available: https: //doi.org/10.1109/MM.2008.31

    work page doi:10.1109/mm.2008.31 2008

  35. [35]

    Nvidia h100 tensor core gpu architecture,

    NVIDIA, “Nvidia h100 tensor core gpu architecture,” 2022. [Online]. Available: https://resources.nvidia.com/en-us-tensor-core/ nvidia-hopper-architecture-whitepaper

    work page 2022

  36. [36]

    Nvidia groq 3 lpx: The inference accelerator for nvidia vera rubin

    ——, “Nvidia groq 3 lpx: The inference accelerator for nvidia vera rubin.” [Online]. Available: https://www.nvidia.com/en-us/data-center/ lpx/

    work page

  37. [37]

    Nimble: lightweight and parallel gpu task scheduling for deep learning,

    W. Kwon, G.-I. Yu, E. Jeong, and B.-G. Chun, “Nimble: lightweight and parallel gpu task scheduling for deep learning,” inProceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Red Hook, NY , USA: Curran Associates Inc.,

    work page

  38. [38]

    Available: https://proceedings.neurips.cc/paper files/ paper/2020/hash/5f0ad4db43d8723d18169b2e4817a160-Abstract.html

    [Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2020/hash/5f0ad4db43d8723d18169b2e4817a160-Abstract.html

    work page 2020

  39. [39]

    Welder: Scheduling deep learning memory access via tile-graph,

    Y . Shi, Z. Yang, J. Xue, L. Ma, Y . Xia, Z. Miao, Y . Guo, F. Yang, and L. Zhou, “Welder: Scheduling deep learning memory access via tile-graph,” in17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). Boston, MA: USENIX Association, Jul. 2023, pp. 701–718. [Online]. Available: https://www.usenix.org/conference/osdi23/presentation/shi

    work page 2023

  40. [40]

    Tvm: an automated end-to-end optimizing compiler for deep learning,

    T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, M. Cowan, H. Shen, L. Wang, Y . Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy, “Tvm: an automated end-to-end optimizing compiler for deep learning,” inProceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’18. USA: USENIX Association, 2018, p. 579–594. [Online]. ...

    work page doi:10.5555/3291168.3291211 2018

  41. [41]

    Automatic horizontal fusion for gpu kernels,

    A. Li, B. Zheng, G. Pekhimenko, and F. Long, “Automatic horizontal fusion for gpu kernels,” inProceedings of the 20th IEEE/ACM International Symposium on Code Generation and Optimization, ser. CGO ’22. IEEE Press, 2022, p. 14–27. [Online]. Available: https://doi.org/10.1109/CGO53902.2022.9741270

    work page doi:10.1109/cgo53902.2022.9741270 2022

  42. [42]

    Rammer: Enabling holistic deep learning compiler optimizations with rTasks,

    L. Ma, Z. Xie, Z. Yang, J. Xue, Y . Miao, W. Cui, W. Hu, F. Yang, L. Zhang, and L. Zhou, “Rammer: Enabling holistic deep learning compiler optimizations with rTasks,” in14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, Nov. 2020, pp. 881–897. [Online]. Available: https://dl.acm.org/doi/abs/10.5555/3488766.3488816

    work page doi:10.5555/3488766.3488816 2020

  43. [43]

    Versapipe: a versatile programming framework for pipelined computing on gpu,

    Z. Zheng, C. Oh, J. Zhai, X. Shen, Y . Yi, and W. Chen, “Versapipe: a versatile programming framework for pipelined computing on gpu,” in Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO-50 ’17. New York, NY , USA: Association for Computing Machinery, 2017, p. 587–599. [Online]. Available: https://doi.org/10...

    work page doi:10.1145/3123939.3123978 2017

  44. [44]

    A framework for fine-grained synchronization of dependent gpu kernels,

    A. Jangda, S. Maleki, M. M. Dehnavi, M. Musuvathi, and O. Saarikivi, “A framework for fine-grained synchronization of dependent gpu kernels,” inProceedings of the 2024 IEEE/ACM International Symposium on Code Generation and Optimization, ser. CGO ’24. IEEE Press, 2024, p. 93–105. [Online]. Available: https://doi.org/10.1109/CGO57630.2024.10444873

    work page doi:10.1109/cgo57630.2024.10444873 2024

  45. [45]

    Hytis: Hybrid tile scheduling for gpu gemm with enhanced wave utilization and cache locality,

    Z. Zhang, H. Wang, H. Xu, D. Yang, X. Zhou, and D. Cheng, “Hytis: Hybrid tile scheduling for gpu gemm with enhanced wave utilization and cache locality,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’25. New York, NY , USA: Association for Computing Machinery, 2025, p. 1604–1618. [...

    work page doi:10.1145/3712285.3759771 2025

  46. [46]

    Nanoflow: towards optimal large language model serving throughput,

    K. Zhu, Y . Gao, Y . Zhao, L. Zhao, G. Zuo, Y . Gu, D. Xie, T. Tang, Q. Xu, Z. Ye, K. Kamahori, C.-Y . Lin, Z. Wang, S. Wang, A. Krishnamurthy, and B. Kasikci, “Nanoflow: towards optimal large language model serving throughput,” inProceedings of the 19th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI ’25. USA: USENIX Associati...

    work page doi:10.5555/3767901.3767942 2025

  47. [47]

    FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion

    L.-W. Chang, W. Bao, Q. Hou, C. Jiang, N. Zheng, Y . Zhong, X. Zhang, Z. Song, C. Yao, Z. Jiang, H. Lin, X. Jin, and X. Liu, “Flux: Fast software-based communication overlap on gpus through kernel fusion,” 2024. [Online]. Available: https://arxiv.org/abs/2406.06858

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [48]

    Overlap communication with dependent computation via decomposition in large deep learning models,

    S. Wang, J. Wei, A. Sabne, A. Davis, B. Ilbeyi, B. Hechtman, D. Chen, K. S. Murthy, M. Maggioni, Q. Zhang, S. Kumar, T. Guo, Y . Xu, and Z. Zhou, “Overlap communication with dependent computation via decomposition in large deep learning models,” inProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and O...

    work page doi:10.1145/3567955.3567959 2023

  49. [49]

    Kitsune: Enabling dataflow execution on gpus with spatial pipelines,

    M. Davies, N. Crago, K. Sankaralingam, and S. Keckler, “Kitsune: Enabling dataflow execution on gpus with spatial pipelines,”ACM Trans. Archit. Code Optim., vol. 22, no. 4, Dec. 2025. [Online]. Available: https://doi.org/10.1145/3777466

    work page doi:10.1145/3777466 2025

  50. [50]

    Nvidia nsight systems

    NVIDIA, “Nvidia nsight systems.” [Online]. Available: https://developer. nvidia.com/nsight-systems

    work page

  51. [51]

    Triton: an intermediate language and compiler for tiled neural network computations,

    P. Tillet, H. T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” inProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, ser. MAPL 2019. New York, NY , USA: Association for Computing Machinery, 2019, p. 10–19. [Online]. Available: https://doi.org/10...

    work page doi:10.1145/3315508.3329973 2019

  52. [52]

    cutile python

    NVIDIA, “cutile python.” [Online]. Available: https://docs.nvidia.com/ cuda/cutile-python/

    work page

  53. [53]

    Nvidia hgx platform

    ——, “Nvidia hgx platform.” [Online]. Available: https://www.nvidia. com/en-us/data-center/hgx/

    work page

  54. [54]

    Performance analysis of k-ary n-cube interconnection networks,

    W. Dally, “Performance analysis of k-ary n-cube interconnection networks,”IEEE Transactions on Computers, vol. 39, no. 6, pp. 775–785, 1990. [Online]. Available: https://doi.org/10.1109/12.53599

    work page doi:10.1109/12.53599 1990