Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Tilus is a tile-level GPGPU language that turns any 1-to-8-bit low-precision tensor type into kernels that beat existing compilers and hand-tuned code.

desk verdict Tilus is a solid, well-engineered compiler paper for sub-byte quantization kernels with real artifacts; the main soft spot is an unproven no-cost register reinterpretation guarantee, not a showstopper. read the letter →

arxiv 2504.12984 v3 pith:XMZACH7K submitted 2025-04-17 cs.LG cs.AIcs.PL

Yaoyao Ding , Bohan Hou , Xiao Zhang , Allan Lin , Tianqi Chen , Cody Yu Hao , Yida Wang , Gennady Pekhimenko This is my paper · ORCID
classification cs.LGcs.AIcs.PL
keywords GPUprogramminglanguagelow-precisionquantizationarbitrarybitwidthalgebraiclayoutsystemregistertensorstensorcoresLLMservingkernelcompilation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Tilus, a tile-level GPGPU language, and argues that arbitrary low-precision data types—integers and floats from 1 to 8 bits—can be made both easy to program and fast on GPUs. Its central claim is that a language exposing thread-block-level instructions, hierarchical memory, and an algebraic layout system can eliminate the shared-memory layout conversions and missing software pipelining that make odd-width quantization (3, 5, 6, 7 bits) slow today. If true, the practical accuracy-efficiency trade-off for quantized LLM serving widens: 4-bit quantization is no longer the only fast option, and hand-written kernels for each new width become unnecessary. The paper reports speedups of 1.75x over a leading tile-oriented compiler, 2.61x over a schedule-oriented low-precision compiler, and 1.29x and 1.03x over hand-tuned 5/6-bit and 4-bit kernels, all from one parameterized kernel template.

What carries the argument

The central object is the register tensor layout, defined as a function f(t,i) that maps thread index t and local element index i to the logical index of the tensor element. Tilus builds this function from primitive local(n1,n2) and spatial(n1,n2) layouts combined by a Kronecker product, giving a unified representation in terms of mode_shape, spatial_modes, and local_modes. The load-bearing operation is the View instruction: it reinterprets a register tensor's data type and layout whenever the number of threads and the number of bits per thread match, making sub-byte reinterpretations a zero-cost register-level operation. This is what lets Tilus load arbitrary low-precision tiles as hardware-friendly uint8 and defer type conversion to vectorized casts.

What would settle it

Compile a Tilus program whose body is only LoadGlobal(u8), then View as an i6 layout, then StoreGlobal, and run it on a fixed GPU with a fixed tile; compile the same program with the View replaced by an explicit copy through shared memory. If the View version is not measurably faster or equal, and its generated machine code contains register moves, shuffles, or extra memory instructions for any legal layout, the zero-cost reinterpretation claim fails for that layout.

Watch

Extended reading notes

Core claim

Tilus's discovery is that the layout of a register tensor—the mapping from a thread index and a local element index to the logical tensor index—can be composed algebraically out of two primitives, local and spatial, using a Kronecker product. Because these layouts are closed under the product, any layout can be written in a unified form, and two register tensors with the same number of threads and the same number of bits per thread can be reinterpreted from one data type and layout to another with a View instruction that costs nothing. This turns low-precision weight loading into a two-step pipeline—load the packed bytes as uint8, then View them as int6, int5, or float6—replacing the shared-memory layout conversion that tile-oriented compilers need and the type-level packing that limits schedule-oriented compilers to power-of-two widths. Casting from the low-precision type to float16 is then done entirely in registers with vectorized bit-manipulation instructions. On this basis Tilus reports up to 2.61x speedup over the schedule-oriented baseline and beats hand-tuned kernels for the widths those kernels support.

Load-bearing premise

The argument assumes that two register tensors with the same number of threads and the same number of bits per thread can always be reinterpreted from one data type and layout into another at zero runtime cost; if register allocation or bit ordering makes that reinterpretation cost real for some layouts, the performance claims for those widths would shrink even though the language would still work.

Editorial extensions

If this is right

  • A single Tilus program template covers uint1-uint8, int2-int8, and float3-float8, so new quantization widths need no new hand-written kernels.
  • Odd widths like 3, 5, 6, and 7 bits become viable in LLM serving, allowing accuracy-preserving quantization where 4-bit is too aggressive and 8-bit wasteful.
  • Weight loading for low-precision matmul can be software-pipelined via asynchronous copies and register-level reinterpretation, removing the shared-memory layout-conversion bottleneck.
  • End-to-end decode latency at small batch sizes improves across 9B, 32B, and 70B models on A100, L40S, and H100 GPUs.
  • The same virtual-machine instruction set allows target-specific instruction selection, so new GPU features can be added without rewriting kernels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's GPU matmul demonstrations, the bit-per-thread reinterpretation argument is width-invariant, so the same View strategy should extend to 9-to-15-bit types packable into 16-bit lanes and to non-GPU vector hardware.
  • Because layouts are closed under the Kronecker product and View is a bijection on the (thread, bit) space, the layout algebra could double as a verification target: one could mechanically prove that a View produces the logical tensor content requested, catching register-allocation bugs the compiler misses.
  • The smallest reported margin is 1.03x against a hand-tuned 4-bit kernel, which suggests the remaining gap to expert kernels is thin; sustaining it on new architectures will depend on the View remaining free in generated machine code.
  • The load-reinterpret path is not matmul-specific, so a natural test is whether Tilus extends to activation quantization with mixed per-tensor widths or to attention kernels, which the paper does not evaluate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Tilus, a tile-level GPGPU programming language with an algebraic layout system for register tensors, a thread-block-level virtual machine instruction set, and native support for low-precision data types with arbitrary bit widths from 1 to 8. The central performance claim is that a View instruction can reinterpret register tensors between different data types and layouts at zero cost when the number of threads and the total number of bits per thread match, thereby eliminating the shared-memory layout conversions that dominate Triton-style low-precision kernels. The paper reports speedups over Triton, Ladder, QuantLLM, and Marlin, includes end-to-end LLM serving results across three models and three GPU architectures, and provides open-source artifacts with a Docker workflow.

Significance. If the zero-cost View claim is valid, Tilus is a significant contribution: it demonstrates that a single parameterized program template can generate efficient kernels for the full spectrum of 1-8 bit quantization formats, a capability that existing compilers and hand-written kernels do not provide at this level of generality or performance. The algebraic layout system is a clean formalization with no fitted constants (the only free parameters are tile sizes), and the artifacts are open-sourced with a reproducible benchmark protocol, including median-of-50 operator runs, median-of-10 model runs, L2 cache clearing, and auto-tuned baselines. The reported speedup over Marlin, the strongest hand-tuned baseline, is modest (1.03x), but the breadth of supported data types is the main value proposition. The main risk to significance is that the zero-cost View instruction is the mechanism behind the efficiency gains, and its semantics are not sufficiently specified to establish that the cost is truly zero.

major comments (3)
  1. [§3.2, Fig. 2(c), Table 1] The zero-cost View claim is load-bearing for the reported speedups but is not established. The compatibility rule in Fig. 2(c) — same number of threads and same total bits per thread — is insufficient unless the physical bit-level register layout is specified. The layout algebra in Section 4 maps a logical index to a (thread, local-index) pair, but it never defines the bit offset, ordering, or padding of sub-byte elements within a thread's registers. Consequently, reinterpreting 3×u8 per thread as 4×i6 per thread may require bit extraction and reassembly instructions (shifts, LOP3, PRMT) unless the bit patterns happen to align. Table 1 defines View as "at no cost" by fiat, but no proof or codegen-level analysis is provided. Please specify the register representation and either prove bit-compatibility for the layouts used in Fig. 2 and Section 7.2, or present the emitted PTX/SASS to demonstrate that View compiles to zero instructions for these cases.
  2. [§7.2] The general method for efficient low-precision loading is only defined for layouts where each thread holds an integer number of bytes. The formula n1 = gcd(n, 16), n2 = n / gcd(n1, 16) requires an integer n (bytes per thread). The paper claims support for arbitrary 1-8 bit widths and benchmarks u1 and u2 in Figure 10, but for per-thread bit counts that are not multiples of 8 (e.g., 4 bits per thread for u1 in a 32-thread warp), this u8-reinterpretation recipe does not apply. The text states that the method "generalizes to loading any low-precision tensor with arbitrary layout," but it does not explain how non-byte-aligned per-thread sizes are handled. Please extend the construction to non-integer bytes per thread, or clarify the alternative mechanism used for the u1/u2 benchmarks, and show the corresponding View layouts for those cases.
  3. [Abstract, §9.1-§9.2, §A.3.4] The headline aggregate speedups (1.75x, 2.61x, 1.29x, 1.03x) are not defined: the paper does not specify whether these are means, medians, or maxima over the workloads in Figure 10, and no variance information is reported anywhere. Since 1.03x against Marlin is close to plausible run-to-run noise, the comparison would be much more convincing with per-configuration distributions or confidence intervals. In addition, the end-to-end experiments use dummy weights (as stated in A.3.4), so the paper contains no accuracy validation of the quantized models; this should be stated prominently in the main text, since the motivation in Section 2.1 concerns the accuracy-efficiency trade-off of 5-7 bit quantization.
minor comments (5)
  1. [Section 4.2] The Kronecker product definition uses the symbol ⊙ for element-wise product without defining it explicitly for index vectors; a reader familiar with matrix operations may confuse it with the Hadamard product. Please add one sentence clarifying that ⊙ operates component-wise on the logical index.
  2. [References] References [56] and [57] are both "Attention is All you Need" by Vaswani et al.; the duplicate should be removed or replaced with the intended citing work.
  3. [Figure 10] The speedup labels on the bars are difficult to read, especially for small values; consider reporting the exact numbers in a table or using a different labeling scheme.
  4. [Section 5] The description of the unified layout representation would benefit from a worked example showing how mode_shape, spatial_modes, and local_modes are derived from a given Kronecker-product expression; the current example in Figure 6 is not explicitly connected to the primitives of Section 4.
  5. [Section 1] The statement that existing approaches for generating low-precision kernels are limited to power-of-two bit widths is too strong, given that the related work section itself cites APNN-TC, Any-Precision LLM, and ABQ-LLM, which support arbitrary bit widths; please add a qualifier such as "efficiently" or "in a compiler setting."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Tilus's layout system is a formalization, its speedups are external benchmarks, and no fitted parameter or self-citation is relabeled as a prediction.

full rationale

The paper's central claims are a new tile-level language, an algebraic layout system, and empirical kernel speedups. The layout algebra (Section 4) is defined from primitive local/spatial layouts and Kronecker products; it does not presuppose the performance results it later reports. The View operation's 'no cost' property (Table 1, Section 3.2) is an implementation and code-generation claim, not a consequence of the compatibility conditions of equal thread count and equal bits per thread; those conditions define semantic reinterpretability, while the claimed efficiency is verified by the benchmarks and by the described lowering strategy in Section 8. Auto-tuning of tile sizes is applied to Tilus and to the baseline compilers, so it does not constitute fitting a parameter to a target result and then calling it a prediction. References to prior work, including Hidet as the backend, are implementation dependencies rather than load-bearing evidence for the paper's uniqueness or performance claims. No fitted constants, no imported uniqueness theorem, and no derivation that reduces to its own inputs were found.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central efficiency claim rests on the compact bit-packing convention and the View reinterpretation guarantee; both are implemented and validated by experiments but not proven for the full space of layouts. Tile sizes are tuned per operator, which is standard for this kind of evaluation.

free parameters (1)
  • Tile sizes (BM, BN, BK) = auto-tuned; about 200 configurations per operator
    Reported speedups in Section 9 are at the auto-tuned point; these are engineering search parameters, not physical constants.
assumptions (5)
  • domain assumption A thread-block-level Tilus program executes correctly on the GPU execution and memory hierarchy (registers, shared memory, global memory).
    The SIMB programming model in Section 6 assumes this faithful mapping; this is standard for GPU compilers.
  • domain assumption Low-precision values are packed compactly with no bit gaps, and bitwise load/store (Figure 8) preserves arithmetic semantics after casting.
    Section 7.1 defines this storage and the bit operations; the cast-to-f16 correctness depends on it.
  • domain assumption Register tensors with the same thread count and same bits per thread are freely reinterpretable between compatible layouts at no cost.
    Stated in Section 3.2 (Figure 2(c)) and used in Section 7.2; asserted for the general case without a proof of all layout mappings.
  • domain assumption Tensor core mma instruction layouts can be expressed in the Kronecker-product layout algebra (e.g., mma.m16n8k16 as local(2,1).spatial(8,4).local(1,2)).
    Section 4 states this and instruction selection in Section 8.1 relies on matching these layouts; a hardware fact assumed by the design.
  • standard math The Kronecker product of layouts is associative.
    Section 4.2 claims this without giving the proof; associativity underlies layout composition and division.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation." pith.science (2026). https://pith.science/paper/XMZACH7K

@misc{pith2026250412984,
  author       = {Pith},
  title        = {Pith review of: Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XMZACH7K}},
  note         = {Machine review of arXiv:2504.12984}
}
abstract

Serving Large Language Models (LLMs) is critical for AI-powered applications, yet it demands substantial computational resources, particularly in memory bandwidth and computational throughput. Low-precision computation has emerged as a key technique to improve efficiency while reducing resource consumption. Existing approaches for generating low-precision kernels are limited to weight bit widths that are powers of two and suffer from suboptimal performance because of high-level GPU programming abstractions. These abstractions restrict critical optimizations, such as fine-grained register management and optimized memory access patterns, that are essential for efficient low-precision computations. In this paper, we introduce Tilus, a domain-specific language designed for General-Purpose GPU (GPGPU) computing that supports low-precision data types with arbitrary bit widths from 1 to 8 while maintaining GPU programmability. Tilus features a thread-block-level programming model, a hierarchical memory space, a novel algebraic layout system, and extensive support for diverse low-precision data types. Tilus programs are compiled into highly efficient GPU programs through automatic vectorization and instruction selection. Extensive experiments demonstrate that Tilus efficiently supports a full spectrum of low-precision data types, and outperforms state-of-the-art low-precision kernels. Compared to existing compilers such as Triton and Ladder, as well as hand-optimized kernels such as QuantLLM and Marlin, Tilus achieves performance improvements of: $1.75\times$, $2.61\times$, $1.29\times$ and $1.03\times$, respectively. We open-source Tilus at https://github.com/NVIDIA/tilus.

Figures

Figures reproduced from arXiv: 2504.12984 by the authors.

Figure 1
Figure 1. The weight loading pipeline of Triton, Ladder, and our approach. The tensors could be in global memory (GMEM), shared memory (SMEM), or registers (REGS). 3 System Overview 3.1 Key Ideas Our work introduces a domain-specific language, Tilus, that provides fine-grained control over shared memory and reg￾isters, making it possible to program efficient low-precision deep learning kernels. Tilus supports low-precision da… view at source ↗
Figure 2
Figure 2. This figure provides a concrete example of how the Tilus is used to implement low-precision matrix multiplication (FP16 × INT6). Figure (a) illustrates the virtual machine program, highlighting key features such as the algebraic layout system (Section 4), thread-block-level instructions (Section 6), and efficient low-precision data support. Figure (b) illustrates the kernel’s data flow, emphasizing tensor movement a… view at source ↗
Figure 3
Figure 3. Layout of operand A in a Tensor Core instruction. The operand, with 16 × 8 elements, is distributed across 32 threads, with each thread storing four elements. The logical index of each element is determined by a layout function given the thread index t and the local element index i. refer to this layout as a distributed layout [53]. Such a layout can be defined as a function 𝑓 that maps a thread index 𝑡 and a local … view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Examples of Kronecker products over layouts. In the figure, layout (c) is the product of layouts (a) and (b), while layout (e) is the product of layouts (d) and (a). The layouts used in modern deep learning workloads, as well as those defined by hardware instructions, …
Figure 4
Figure 4. Figure 4: Two types of primitive layouts: local and spatial. A local layout stores all tile elements within a single thread, whereas a spatial layout distributes them across multiple threads, with each thread holding only a single element. With the formal definition of layout, w…
Figure 6
Figure 6. Figure 6: Example of the unified layout representation. We use a unified representation for all layouts of reg￾ister tensors in Tilus. This representation gives each lay￾out four attributes: shape, mode_shape, spatial_modes, and local_modes. The shape is a sequence of integers t…
Figure 7
Figure 7. Figure 7: A Tilus program contains parameters and a body. The body is a list of control-flow statements or block-level instructions. The majority of functionality, such as tensor allocation and computation, is provided by instructions. 6.3 Thread-Block-Level Instruction Set Each…
Figure 9
Figure 9. Figure 9: Program to rearrange tensor B with data type int6, used in the "Change Layout" step of [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Speedup of low-precision kernels in Triton, QuantLLM, Ladder, and Tilus (Ours) compared against the standard half-precision kernel from cuBLAS. Benchmarked data types include uint8 (u8), f6e3m2 (f6), int4 (i4), uint4 (u4), uint2 (u2), and uint1 (u1). Each workload (BS…
Figure 11
Figure 11. Figure 11: Speedup of quantized matrix multiplication com￾pared against the cuBLAS FP16 kernel. A full spectrum of quantized data types is evaluated. Tilus supports low-precision matrix multiplications of the form matmul(A, B), where operand A can have data types with 32, 16, or…
Figure 13
Figure 13. Figure 13: End-to-end performance of the QWen2.5-30B model across NVIDIA A100, L40S, and H100 GPUs. The weight data types for vLLM, Ladder, and Tilus are float16, uint4, and uint4, respectively. OOM indicates out-of￾memory error, and ERR indicates a runtime error. 9.5.1 Speedup …
Figure 12
Figure 12. Figure 12: ). Analysis of Ladder’s generated kernels revealed suboptimal use of CUDA Cores for 1–15 tokens and Tensor Cores for 16 or more tokens, as key optimizations like soft￾ware pipelining [26] and k-dimension parallelization [44] were not implemented, leading to poor perfo…
Figure 14
Figure 14. Figure 14: Speedup of quantized matmuls across different batch sizes from both prefill and decode stages [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Dato: A Task-Based Programming Model for Dataflow Accelerators

    cs.PL 2025-09 conditional novelty 7.0 of 10

    Dato introduces first-class stream and layout types plus an automatic virtual-to-physical mapping search, reporting up to 2.81x speedups on AMD NPUs and 98% of theoretical peak on an FPGA.

Reference graph

Works this paper leans on

78 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    Tor M Aamodt, Wilson Wai Lun Fung, and Timothy G Rogers. 2018. The SIMT Core: Instruction and Register Data Flow. InGeneral-Purpose Graphics Processor Architectures. Springer, 21–66

    work page 2018

  2. [2]

    Andrew Adams, Karima Ma, Luke Anderson, Riyadh Baghdadi, Tzu- Mao Li, Michaël Gharbi, Benoit Steiner, Steven Johnson, Kayvon Fata- halian, Frédo Durand, and Jonathan Ragan-Kelley. 2019. Learning to Optimize Halide with Tree Search and Random Programs. ACM Trans. Graph. 38, 4, Article 121 (jul 2019), 12 pages. doi: 10.1145/3306346. 3322967

    doi:10.1145/3306346 2019

  3. [3]

    Aditya Agrawal, Matthew Hedlund, and Blake Hechtman. 2024. eXmY: A Data Type and Technique for Arbitrary Bit Precision Quantization. arXiv:2405.13938 [cs.LG] https://arxiv.org/abs/2405.13938

    arXiv 2024

  4. [4]

    Byung Hoon Ahn, Prannoy Pilligundla, Amir Yazdanbakhsh, and Hadi Esmaeilzadeh. 2020. Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation. In International Confer- ence on Learning Representations . https://openreview.net/forum?id= rygG4AVFvH

    work page 2020

  5. [5]

    Rink, Michael Schaarschmidt, Timur Sitdikov, Agnieszka Swietlik, Dimitrios Vytiniotis, and Joel Wee

    Sami Alabed, Daniel Belov, Bart Chrzaszcz, Juliana Franco, Dominik Grewe, Dougal Maclaurin, James Molloy, Tom Natan, Tamara Norman, Xiaoyue Pan, Adam Paszke, Norman A. Rink, Michael Schaarschmidt, Timur Sitdikov, Agnieszka Swietlik, Dimitrios Vytiniotis, and Joel Wee

  6. [6]

    AMD Corporation. 2024. CDNA 3 Architecture for Accelerated Comput- ing. Available at https://www.amd.com/en/technologies/cdna.html

    work page 2024

  7. [7]

    AMD Corporation. 2024. HIP: Heterogeneous-Compute Interface for Portability. Available at https://rocm.docs.amd.com/projects/HIP/en/ latest/

    work page 2024

  8. [8]

    Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=dfqsW38v1X

    work page 2024

Show all 78 references
  1. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey W...

    2020

  2. [10]

    Wen-Pu Cai, Ming-Yang Li, and Wu-Jun Li. 2024. Lcq: Low-rank codebook based quantization for large language models.arXiv preprint arXiv:2405.20973 (2024)

    2024 arXiv

  3. [11]

    Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa

  4. [12]

    Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Q. Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In OSDI

    2018

  5. [13]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer

  6. [14]

    Yaoyao Ding, Cody Hao Yu, Bojian Zheng, Yizhi Liu, Yida Wang, and Gennady Pekhimenko. 2023. Hidet: Task-Mapping Programming Par- adigm for Deep Learning Tensor Programs. In Proceedings of the 28th ACM International Conference on Architectural Support for Program- ming Language...

    2023

  7. [15]

    Yaoyao Ding, Ligeng Zhu, Zhihao Jia, Gennady Pekhimenko, and Song Han. 2021. Ios: Inter-operator scheduler for cnn acceleration. Proceedings of Machine Learning and Systems 3 (2021), 167–180

    2021

  8. [16]

    Pratik Fegade, Tianqi Chen, Phillip Gibbons, and Todd Mowry. 2021. Cortex: A compiler for recursive deep learning models. Proceedings of Machine Learning and Systems 3 (2021), 38–54

    2021

  9. [17]

    Pratik Fegade, Tianqi Chen, Phillip Gibbons, and Todd Mowry

  10. [18]

    Boyuan Feng, Yuke Wang, Tong Geng, Ang Li, and Yufei Ding. 2021. APNN-TC: accelerating arbitrary precision neural networks on ampere GPU tensor cores. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (St. Louis, Mi...

    2021

  11. [19]

    Siyuan Feng, Bohan Hou, Hongyi Jin, Wuwei Lin, Junru Shao, Ruihang Lai, Zihao Ye, Lianmin Zheng, Cody Hao Yu, Yong Yu, and Tianqi Chen

  12. [20]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)

    2022 arXiv

  13. [21]

    Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh

    Elias Frantar, Roberto L. Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. 2024. MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models. arXiv:2408.11743 [cs.LG] https: //arxiv.org/abs/2408.11743

    2024 arXiv

  14. [22]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  15. [23]

    Bastian Hagedorn, Bin Fan, Hanfeng Chen, Cris Cecka, Michael Gar- land, and Vinod Grover. 2023. Graphene: An IR for Optimized Tensor Computations on GPUs. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating S...

    2023

  16. [24]

    In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2(Vancouver, BC, Canada)(ASPLOS 2023)

    TensorIR: An Abstraction for Automatic Tensorized Program Optimization. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2(Vancouver, BC, Canada)(ASPLOS 2023). Association for Computing Mac...

    2023

  17. [25]

    Adnan Hoque, Less Wright, Chih-Chieh Yang, Mudhakar Srivatsa, and Raghu Ganti. 2024. Accelerating a Triton Fused Kernel for W4A16 Quantized Inference with SplitK work decomposition. arXiv preprint arXiv:2402.00025 (2024)

    2024 arXiv

  18. [26]

    Guyue Huang, Yang Bai, Liu Liu, Yuke Wang, Bei Yu, Yufei Ding, and Yuan Xie. 2023. Alcop: Automatic load-compute pipelining in deep learning compiler for ai-gpus. Proceedings of Machine Learning and Systems 5 (2023), 680–694

    2023

  19. [27]

    IEEE Standards Association. 2019. IEEE Standard for Floating-Point Arithmetic (IEEE 754-2019). IEEE, New York, NY. doi:10.1109/IEEESTD. 2019.8766229 Available at https://standards.ieee.org/standard/754- 2019.html

    2019

  20. [28]

    Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Za- haria, and Alex Aiken. 2019. TASO: optimizing deep learning computa- tion with automatic generation of graph substitutions. In Proceedings of the 27th ACM Symposium on Operating Systems Principles . 47–62

    2019

  21. [29]

    Fletcher

    Kartik Hegde, Hadi Asghari-Moghaddam, Michael Pellauer, Neal Crago, Aamer Jaleel, Edgar Solomonik, Joel Emer, and Christopher W. Fletcher. 2019. ExTensor: An Accelerator for Sparse Tensor Alge- bra. In Proceedings of the 52nd Annual IEEE/ACM International Sym- posium on Microa...

    2019

  22. [30]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei- Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. In MLSys

    2024

  23. [31]

    Yang Liu and Mirella Lapata. 2019. Text Summarization with Pre- trained Encoders. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , xxx, xxx, xx...

    2019 doi

  24. [32]

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2025. SpinQuant: LLM Quantiza- tion with Learned Rotations. In The Thirteenth International Confer- ence on Learning Repres...

    2025

  25. [33]

    Weile Luo, Ruibo Fan, Zeyu Li, Dayou Du, Hongyuan Liu, Qiang Wang, and Xiaowen Chu. 2025. Dissecting the NVIDIA Hopper Architec- ture through Microbenchmarking and Multiple Level Analysis. arXiv preprint arXiv:2501.12084 (2025)

    2025 arXiv

  26. [34]

    Gonzalez, Hao Zhang, and Ion Sto- ica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Sto- ica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating...

    2023

  27. [35]

    Shaobo Ma, Chao Fang, Haikuo Shao, and Zhongfeng Wang. 2025. Efficient Arbitrary Precision Acceleration for Large Language Mod- els on GPU Tensor Cores. In Proceedings of the 30th Asia and South Pacific Design Automation Conference (Tokyo, Japan) (ASPDAC ’25). Association for ...

    2025

  28. [36]

    Odemuyiwa, Michael Pel- lauer, Joel S

    Nandeeka Nayak, Xinrui Wu, Toluwanimi O. Odemuyiwa, Michael Pel- lauer, Joel S. Emer, and Christopher W. Fletcher. 2024. FuseMax: Lever- aging Extended Einsums to Optimize Attention Accelerator Design. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO)....

    2024

  29. [37]

    John Nickolls and William J. Dally. 2010. The GPU Computing Era. IEEE Micro 30, 2 (2010), 56–69. doi:10.1109/MM.2010.41

    2010 doi

  30. [38]

    NVIDIA Corporation. 2021. CUTLASS: CUDA Templates for Linear Algebra Subroutines and Solvers. https://github.com/NVIDIA/cutlass

    2021

  31. [39]

    Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. 2020. RAMMER: Enabling Holistic Deep Learning Compiler Optimizations with Rtasks. USENIX Association, USA, 17

    2020

  32. [40]

    NVIDIA Corporation. 2024. CUDA C++ Programming Guide. Version 12.0. Available at https://docs.nvidia.com/cuda/cuda-c-programming- guide/

    2024

  33. [41]

    NVIDIA Corporation. 2024. Parallel Thread Execution ISA Version 12.0 . Available at https://docs.nvidia.com/cuda/parallel-thread-execution/ index.html

    2024

  34. [42]

    NVIDIA Corporation. 2024. SASS: Streaming Assembler for NVIDIA GPUs. Available athttps://docs.nvidia.com/cuda/cuda-binary-utilities/ index.html

    2024

  35. [43]

    OpenAI. 2024. ChatGPT. https://chat.openai.com/. Accessed: 2024- 11-12; Generative AI language model

    2024

  36. [44]

    NVIDIA Corporation. 2023. NVIDIA cuBLAS Library. https://developer. nvidia.com/cublas Version 12.2

    2023

  37. [45]

    Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, and Jae W. Lee. 2024. Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs. InProceedings of the 41st International Conference on Machine Learning

    2024

  38. [46]

    Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. In Acm Sigplan Notices, Vol. 48. ACM, 519–530

    2013

  39. [47]

    Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. 2023. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537 (2023)

    2023 arXiv

  40. [48]

    Amit Sabne. 2020. XLA : Compiling Machine Learning for Peak Per- formance

    2020

  41. [49]

    Muhammad Osama, Duane Merrill, Cris Cecka, Michael Garland, and John D. Owens. 2023. Stream-K: Work-Centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU. In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel ...

    2023

  42. [50]

    Haichen Shen, Jared Roesch, Zhi Chen, Wei Chen, Yong Wu, Mu Li, Vin Sharma, Zachary Tatlock, and Yida Wang. 2021. Nimble: Effi- ciently Compiling Dynamic Neural Networks for Model Inference. In Proceedings of Machine Learning and Systems , A. Smola, A. Dimakis, and I. Stoica (...

    2021

  43. [51]

    Shizhi Tang, Jidong Zhai, Haojie Wang, Lin Jiang, Liyan Zheng, Zhen- hao Yuan, and Chen Zhang. 2022. FreeTensor: A Free-Form DSL with Holistic Optimizations for Irregular Tensor Programs. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language D...

    2022

  44. [52]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mes- nard, Bobak Shahriari, Alexandre Ramé, et al . 2024. Gemma 2: Im- proving open language models at a practical size. arXiv preprint arXiv:2408.00118 (2024)

    2024 arXiv

  45. [53]

    Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: an interme- diate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (Phoenix, AZ, USA) (MAPL 2019). A...

    2019

  46. [54]

    Junru Shao, Xiyou Zhou, Siyuan Feng, Bohan Hou, Ruihang Lai, Hongyi Jin, Wuwei Lin, Masahiro Masuda, Cody Hao Yu, and Tianqi Chen. 2024. Tensor program optimization with probabilistic programs. In Proceedings of the 36th International Conference on Neural Informa- tion Process...

    2024

  47. [55]

    Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen

    Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zach DeVito, William S. Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen. 2018. Tensor Comprehensions: Framework- Agnostic High-Performance Machine Learning Abstractions. ArXiv abs/1802.04730 (2018)

    2018 arXiv

  48. [57]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Pro- cessing Systems , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S...

    2017

  49. [58]

    Lei Wang, Lingxiao Ma, Shijie Cao, Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan Yang, Ting Cao, Yuqing Yang, and Mao Yang. 2024. Ladder: Enabling Efficient Low-Precision Deep Learning Computing through Hardware-aware Tensor Transforma- tion. In 18th US...

    2024

  50. [59]

    Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. QuIP#: even better LLM quantization with hadamard incoherence and lattice codebooks. In Proceedings of the 41st International Conference on Machine Learning (Vienna, Austria) (ICML’24). JML...

    2024

  51. [60]

    Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, and Shuaiwen Leon Song. 2024. Quant-LLM: Accelerating the Serving of Large Lan- guage Models via FP6-Centric ...

    2024

  52. [61]

    Jiarong Xing, Leyuan Wang, Shang Zhang, Jack Chen, Ang Chen, and Yibo Zhu. 2022. Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance. In Proceedings of Machine Learning and Systems, Vol. 4

    2022

  53. [62]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al

  54. [63]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Asso- ciation, Carlsbad, C...

    2022

  55. [64]

    BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexan- dra Sasha Luccioni, François Yvon, et al . 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.0...

    2022 arXiv

  56. [65]

    Jie Zhao, Xiong Gao, Ruijie Xia, Zhaochuang Zhang, Deshi Chen, Lei Chen, Renwei Zhang, Zhen Geng, Bin Cheng, and Xue- feng Jin. 2022. Apollo: Automatic Partition-based Operator Fu- sion through Layer by Layer Optimization. In Proceedings of Ma- chine Learning and Systems , D. ...

    2022

  57. [66]

    Bojian Zheng, Ziheng Jiang, Cody Hao Yu, Haichen Shen, Joshua Fromm, Yizhi Liu, Yida Wang, Luis Ceze, Tianqi Chen, and Gennady Pekhimenko. 2022. DietCode: Automatic Optimiza- tion for Dynamic Tensor Programs. In Proceedings of Machine Learning and Systems , D. Marculescu, Y. C...

    2022

  58. [67]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, and Ion Stoica. 2020. Ansor: Generating High- Performance Tensor Programs for Deep Learning. In 14th USENIX Symposium on Operating ...

    2020

  59. [68]

    arXiv preprint arXiv:2407.10671 (2024)

    Qwen2 technical report. arXiv preprint arXiv:2407.10671 (2024)

    2024 arXiv

  60. [69]

    Size Zheng, Yun Liang, Shuo Wang, Renze Chen, and Kaiwen Sheng

  61. [70]

    Chao Zeng, Songwei Liu, Yusheng Xie, Hong Liu, Xiaojian Wang, Miao Wei, Shu Yang, Fangmin Chen, and Xing Mei. 2025. ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence 39, 21 (Apr. 202...

    2025 doi

  62. [71]

    Hongyu Zhu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Wei Cui, Fan Yang, Mao Yang, Lidong Zhou, Asaf Cidon, and Gennady Pekhimenko. 2022. ROLLER: Fast and Efficient Tensor Compilation for Deep Learning. In 16th USENIX Symposi...

    2022

  63. [72]

    Zhu, W.Y

    K. Zhu, W.Y. Zhao, Z. Zheng, T.Y. Guo, P.Z. Zhao, J.J. Bai, J. Yang, X.Y. Liu, L.S. Diao, and W. Lin. 2021. DISC: A Dynamic Shape Compiler for Machine Learning Workloads. In Proceedings of the 1st Workshop on Machine Learning and Systems (Online, United Kingdom) (EuroMLSys ’21...

    2021

  64. [74]

    Size Zheng, Renze Chen, Anjiang Wei, Yicheng Jin, Qin Han, Liqiang Lu, Bingyang Wu, Xiuhong Li, Shengen Yan, and Yun Liang. 2022. AMOS: enabling <u>a</u>utomatic <u>m</u>apping for tensor com- putations <u>o</u>n <u>s</u>patial accelerators with hardware abstraction. In Procee...

    2022

  65. [77]

    Zhen Zheng, Xuanda Yang, Pengzhan Zhao, Guoping Long, Kai Zhu, Feiwen Zhu, Wenyi Zhao, Xiaoyong Liu, Jun Yang, Jidong Zhai, Shuai- wen Leon Song, and Wei Lin. 2022. AStitch: Enabling a New Multi- Dimensional Optimization Space for Memory-Intensive ML Training and Inference on ...

    2022

  66. [2020]

    Proceedings of the Twenty-Fifth International Conference on Ar- chitectural Support for Programming Languages and Operating Systems (2020)

    FlexTensor: An Automatic Schedule Exploration and Opti- mization Framework for Tensor Computation on Heterogeneous Sys- tem. Proceedings of the Twenty-Fifth International Conference on Ar- chitectural Support for Programming Languages and Operating Systems (2020)

    2020

  67. [2022]

    In Proceedings of Machine Learn- ing and Systems , D

    The CoRa Tensor Compiler: Compilation for Ragged Ten- sors with Minimal Padding. In Proceedings of Machine Learn- ing and Systems , D. Marculescu, Y. Chi, and C. Wu (Eds.), Vol. 4. 721–747. https://proceedings.mlsys.org/paper/2022/file/ d3d9446802a44259755d38e6d163e820-Paper.pdf

    2022

  68. [2023]

    In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23)

    QuIP: 2-bit quantization of large language models with guar- antees. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 196, 34 pages

  69. [2024]

    In Proceedings of the 36th International Conference on Neural Informa- tion Processing Systems (New Orleans, LA, USA) (NIPS ’22)

    LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Conference on Neural Informa- tion Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 2198, 15 pages

  70. [2025]

    In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam, Netherlands) (ASPLOS ’25)

    PartIR: Composing SPMD Partitioning Strategies for Machine Learning. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1 (Rotterdam, Netherlands) (ASPLOS ’25). Association Tilus: A Tile-Leve...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.