Pith. sign in

REVIEW 3 major objections 4 minor 69 references

APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By decomposing quantized LLM matrices into 1-bit bipolar pieces, APT-LLM runs arbitrary-precision inference on GPU Tensor Cores and reports up to 3.99x speedup over FP16 on RTX 3090.

desk verdict Useful kernel-engineering paper on arbitrary-precision LLM MatMul, but the bit-reconstruction math as written doesn't close; fixable, but central. read the letter →

arxiv 2508.19087 v1 pith:EPJ4X3IR submitted 2025-08-26 cs.LG cs.AIcs.AR

Shaobo Ma , Chao Fang , Haikuo Shao , Zhongfeng Wang This is my paper
classification cs.LGcs.AIcs.AR
keywords LLMinferenceaccelerationTensorCoresarbitrary-precisionquantizationbipolar-INTbit-wisematrixmultiplicationGPUsharedmemoryschedulingadaptivekernelmappinglow-bit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

APT-LLM tries to close the gap between ultra-low-bit LLM quantization and what GPU Tensor Cores actually support. Its central device is a new data format, bipolar-INT, in which every bit represents -1 or +1 and conversion from signed INT is a lossless sign-bit flip, plus a bit-wise MatMul that decomposes operands into 1-bit matrices, runs them through Tensor Core 1-bit GEMMs, and reconstructs the exact product by shift-and-add. The paper claims this enables arbitrary precision with better Tensor Core utilization, reporting up to 3.99x speedup over FP16 and 2.16x over NVIDIA CUTLASS INT4 on RTX 3090. It also introduces shared-memory-centric data recovery and adaptive kernel mapping, which the ablation credits for a further 3.22x speedup over a naive implementation.

What carries the argument

Bipolar-INT is a numeric format where bit i contributes +2^i or -2^i instead of 0 or 2^i; it is obtained from signed INT by flipping the sign bit, so hat-x-prime = 2*hat-x + 1. Because linear quantization W = s*hat-W + z becomes W = (s/2)*hat-W-prime + (z - s/2), the conversion is algebraically lossless after adjusting scale and zero. The carrying object is bit-wise MatMul reconstitution: decompose weight and activation into bit matrices W(i) and X(j), let the Tensor Core execute pairwise 1-bit GEMMs to produce intermediate matrices Y(i,j), then recover Y = sum over i,j of Y(i,j) * 2^(i+j).

What would settle it

Run the proposed kernel on a small known 2-bit case, such as two 2x2 matrices containing both same-sign and opposite-sign bipolar entries, and compare the shift-and-add reconstructed output against the exact signed integer product. Any mismatch in the low bits, or an error that grows with K, would show that the raw 1-bit MMA output is not algebraic +1/-1 multiplication and that the written reconstruction is incomplete.

Watch

Extended reading notes

Core claim

The paper's central claim is that any fixed-point INT quantized LLM can be converted, without retraining or meaningful accuracy loss, to bipolar-INT, where each bit contributes +2^i or -2^i. Because an n-bit value is then a linear combination of bit matrices, a MatMul of two such values is exactly the shift-and-add sum of pairwise 1-bit Tensor Core products. This makes INT2, INT3, and mixed precisions like W3A4 first-class citizens on GPU Tensor Cores that natively support only 1-bit and 4-bit integer GEMM. The authors further claim that performing the shift-and-add recovery inside shared memory or register fragments rather than global memory, and adaptively selecting block and warp tile siz

Load-bearing premise

The reconstruction assumes that the 1-bit Tensor Core MMA returns the algebraic product of +1/-1 bipolar bits, so the final result is obtained by shift-and-add alone; if the hardware operation is actually bitwise AND, XOR, or equality over 0/1 bits, extra correction terms are needed and are not stated in the paper.

Editorial extensions

If this is right

  • Ultra-low-bit W1A2, W2A2, and W3A4 quantized LLMs can run on existing Ampere, Ada, and Hopper Tensor Cores without first converting to supported INT4 or INT8 formats, yielding the reported up to 3.99x speedup over FP16 on RTX 3090.
  • Any standard linear-quantized model can switch to bipolar-INT by updating scale s to s/2 and zero z to z - s/2, with no retraining required.
  • Performing data recovery in shared memory or fragments substantially reduces latency; the ablation credits memory scheduling with 2.15x and kernel mapping with an additional 1.50x.
  • On newer GPUs the absolute speedups persist but shrink, reaching up to 2.44x over FP16 and 1.65x over CUTLASS integer baselines, because baseline throughput improved while the non-Tensor-Core recovery phase did not scale as well.
  • The adaptive kernel mapping shows that no single hyperparameter configuration is optimal for all LLM layers or phases, which motivates the lookup-table search used before inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mathematical correctness of every kernel result depends on an undocumented hardware detail: the 1-bit Tensor Core MMA returns the algebraic product of bipolar +1/-1 bits. If it instead returns bitwise AND or XOR on 0/1 values, the shift-and-add reconstruction needs row and column sum correction terms that the paper never states.
  • The same bit-decomposition strategy could be combined with native INT2 or INT4 MMA where available, reducing the number of intermediate bit-pair products and lowering recovery cost; the paper does not explore this hybrid path.
  • The adaptive kernel mapping uses a lookup table for common LLM matrix shapes. A closed-form cost model would generalize it to arbitrary shapes and would also make prediction possible for when the speedup disappears on GPUs with very fast native INT4 paths.
  • The perplexity tables show small drift after bipolar-INT conversion, which the paper attributes to storing the adjusted scale and zero in FP16/FP32; a numerical analysis of that rounding sensitivity would be a natural stress test of the lossless claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes APT-LLM, a GPU acceleration scheme for arbitrary-precision quantized LLM inference on NVIDIA Tensor Cores. The scheme has three components: (i) a bipolar-INT data format intended to replace signed INT losslessly and to make bit-level operations uniform; (ii) a bit-wise MatMul reconstitution method that decomposes operands into bit planes, performs 1-bit Tensor-Core MatMuls, and reconstructs the output by shift-and-add; and (iii) a GPU memory scheduling strategy plus an adaptive kernel-mapping framework that selects kernel hyperparameters from a precomputed lookup table. The paper reports speedups over FP16 and CUTLASS INT4/INT8 baselines on RTX 3090, RTX 4090, and H800, and a perplexity comparison at W4A4 showing no degradation for several quantization methods.

Significance. If the central dataflow is correct, the paper addresses a genuine gap: existing Tensor Cores do not natively support the INT2/INT3 formats used by recent ultra-low-bit LLM quantization methods, and the proposed bit-sliced approach could make arbitrary precisions practical on current GPUs. The paper has concrete strengths: extensive kernel and end-to-end LLM benchmarks across three GPU generations; an ablation study separating memory scheduling from kernel mapping; and a W4A4 perplexity sanity check across multiple quantization methods. However, the mathematical description of the bit-wise reconstitution is incomplete, and the correctness of every reported speedup depends on it. The manuscript is therefore not yet self-contained enough to support its central claim as written.

major comments (3)
  1. [Sec. III-B / Fig. 5] The reconstitution formula in Fig. 5, Y = Σ_{i,j} 2^{i+j} Y(i,j), is only valid if the 1-bit Tensor-Core MatMul returns the exact bipolar dot product S = Σ_k (2A_{i,k}-1)(2C_{j,k}-1). The text states that TCs use 'AND or XOR logic gates' for 1-bit MatMul. If Y(i,j) is the AND popcount P = Σ A·C, then S = 4P - 2ΣA - 2ΣC + K; if it is the XOR popcount Q, then S = K - 2Q. Neither is equal to P or Q, and the required row-sum, column-sum, and K-dependent correction terms are not derived anywhere in the paper. Since every kernel result and downstream speedup depends on this reconstruction, the central correctness argument is incomplete. Please state exactly what Y(i,j) is, add the correction terms, and/or provide a microbenchmark identifying the primitive actually returned by the Ampere/Ada/Hopper Tensor Cores.
  2. [Sec. III-A] The bipolar-INT conversion relation is stated as 'ˆx′ = 2ˆx + 1' in the text, but the definition of bipolar-INT given above is (x)_D = Σ_{i=0}^{n-1} (2x(i)−1)·2^i. For an individual bit b, the bipolar coefficient is 2b−1, not 2b+1. The 'sign bit flip' description in Fig. 4 is also not obviously consistent with the algebraic definition for two's-complement values. Because the claim 'seamlessly replace the INT format without any loss of accuracy' rests on this conversion, the paper needs to define the mapping unambiguously and correct the equation.
  3. [Sec. VI-C / Table V] The accuracy evaluation is limited to W4A4. The speedup claims are made for W1A2, W2A2, and W3A4, but no perplexity or downstream-accuracy numbers are reported for these precisions. The paper argues the conversion is mathematically lossless, yet the actual quantized-model behavior at these precisions is not demonstrated. Additionally, latency numbers in Tables II–IV and the perplexity numbers in Table V are reported without error bars or run-to-run variance, which makes it difficult to judge whether the reported differences are significant. Please add accuracy results for the precisions used in the main speedup claims and report variance or at least multiple seeds.
minor comments (4)
  1. [Sec. VI-B1] The text says W1A2 approaches CUTLASS INT1 in the 64/4k/4k task, 'achieving a 92.5% speedup.' From Table II the intended statement is that APT W1A2 achieves 92.5% of CUTLASS INT1's speedup (6.40× vs. 6.92×), not a 92.5% speedup.
  2. [Sec. VI-B2] The decode-phase paragraph refers to 'the LLAMA3-7B model,' while the rest of the paper uses LLaMA3-8B. This is likely a typo and should be corrected.
  3. [Sec. IV-B] The term 'Fragment' is capitalised and used as if it were a defined technical term, but it is not defined in the text. Define it explicitly (e.g., fragment as the register-resident tile used by a Tensor Core warp instruction) or use a different word.
  4. [Fig. 5 / Sec. III-B] The displayed reconstitution equation in Fig. 5 is not numbered. Numbering this equation and explicitly defining the dimensions and data types of Y(i,j) would help the reader verify the dataflow.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity: speedups are measured against external baselines; the only self-citation is non-load-bearing. The bit-wise reconstitution has a correctness gap (unverified TC 1-bit semantics) but that is not a circular step.

full rationale

The paper's central speedup claims are benchmarked against external FP16, CUTLASS INT4/INT8, and APNN-TC baselines; no target quantity is fitted or defined in terms of the claimed result. The bipolar-INT format conversion is an algebraic reparameterization: substituting x' = 2x + 1 into W = s * x + z gives W = (s/2) * x' + (z - s/2), so the 'lossless' claim follows from algebra, not from circular definition. The adaptive kernel search is standard autotuning (Sec. V-B) and does not disguise a fitted parameter as a prediction. The only self-citation, ANDA [52], appears in a related-work enumeration and is not load-bearing. A genuine correctness risk exists in Sec. III-B: the paper defines (x)_D = sum_i (2x_i - 1) * 2^i and reconstructs Y = sum_{i,j} Y(i,j) * 2^(i+j), but this is valid only if the 1-bit Tensor Core MMA output Y(i,j) is the dot product of bipolar bit values. The paper asserts this ('The pairwise multiplication of W(i) and X(j) yields a 32-bit intermediate result matrix Y(i,j)') while also stating 'NVIDIA GPUs facilitate the utilization of either AND or XOR logic gates for executing 1-bit MatMul operations'. If the TC returns an AND/XOR popcount, the reconstruction requires correction terms (row sums, column sums, K-dependent constants) that are not derived. This is a serious correctness incompleteness, but it is not circularity: the conclusion is not assumed by the premise; rather, an unverified hardware semantic is used as a bridge. Accordingly, the circularity score is 2: one minor non-load-bearing self-citation, no reduction of the central claim to its inputs.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central speedup claim depends on tuned kernel tile sizes and a configuration lookup table (free parameters), on the unverified assumption that 1-bit TC MMA outputs are directly algebraic products, and on the hardware behavior of bitwise tensor core operations. Bipolar-INT is a re-encoding, not an independently evidenced entity.

free parameters (2)
  • Kernel tile sizes (BM, BN, BK, WM, WN, WK, TR, TC, WB) = Varies by matrix shape; examples in Fig 16 (e.g., Down W1A2 prefill: B_w=64, B_x=128, T_R=2, T_C=4, W_B=16)
    Chosen by exhaustive search over the tiling constraint to maximize measured speedup on the benchmarked MatMul shapes; the reported speedups are conditional on these tuned values.
  • Kernel configuration lookup table = Not fully enumerated in the paper; only LLaMA3-8B configurations shown
    Built from the same workloads used for evaluation; approximate matching is used for unseen shapes, so optimality is relative to the search space and the training workloads.
assumptions (5)
  • domain assumption NVIDIA Tensor Cores can execute 1-bit MMA efficiently with AND or XOR accumulation at the throughput assumed by the design.
    Invoked in Sec III-B to justify the 1-bit MatMul stage; cites [21] but does not include a microbenchmark in this paper.
  • domain assumption The 32-bit intermediate matrices Y(i,j) from the 1-bit MMA equal the algebraic products of the bipolar bit values, so the final result is a pure shift-and-add of the Y(i,j).
    Sec III-B and Fig 5; this is the missing reconstruction step, likely false if the MMA returns popcounts of AND/XOR without post-processing.
  • standard math The tiling relation (Bw*Bx)/(WM*WN) = WB*TR*TC describes all feasible kernel configurations.
    Sec V-B; a bookkeeping identity that counts tiles, not a guarantee of optimality.
  • domain assumption Double-buffered shared memory fully overlaps global-memory loads with compute on the target GPUs.
    Sec IV-B; standard CUDA pipelining, assumed to deliver the claimed latency hiding.
  • domain assumption GPTQ-quantized W1A2/W2A2/W3A4 models retain task accuracy sufficient for the inference speedup comparisons.
    Sec VI-C uses these precisions for speedup claims but only reports perplexity for W4A4 (Table V).
invented entities (1)
  • bipolar-INT data format
    purpose: Eliminates sign-bit asymmetry in bit-sliced TC MatMul; achieved by the affine map x' = 2x+1 on quantized integers, giving a symmetric range of odd integers.
    It is a re-encoding of two's complement, not a new observable; its utility is demonstrated internally via perplexity and speedup experiments, with no external handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration." pith.science (2026). https://pith.science/paper/EPJ4X3IR

@misc{pith2026250819087,
  author       = {Pith},
  title        = {Pith review of: APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPJ4X3IR}},
  note         = {Machine review of arXiv:2508.19087}
}
abstract

Large language models (LLMs) have revolutionized AI applications, yet their enormous computational demands severely limit deployment and real-time performance. Quantization methods can help reduce computational costs, however, attaining the extreme efficiency associated with ultra-low-bit quantized LLMs at arbitrary precision presents challenges on GPUs. This is primarily due to the limited support for GPU Tensor Cores, inefficient memory management, and inflexible kernel optimizations. To tackle these challenges, we propose a comprehensive acceleration scheme for arbitrary precision LLMs, namely APT-LLM. Firstly, we introduce a novel data format, bipolar-INT, which allows for efficient and lossless conversion with signed INT, while also being more conducive to parallel computation. We also develop a matrix multiplication (MatMul) method allowing for arbitrary precision by dismantling and reassembling matrices at the bit level. This method provides flexible precision and optimizes the utilization of GPU Tensor Cores. In addition, we propose a memory management system focused on data recovery, which strategically employs fast shared memory to substantially increase kernel execution speed and reduce memory access latency. Finally, we develop a kernel mapping method that dynamically selects the optimal configurable hyperparameters of kernels for varying matrix sizes, enabling optimal performance across different LLM architectures and precision settings. In LLM inference, APT-LLM achieves up to a 3.99$\times$ speedup compared to FP16 baselines and a 2.16$\times$ speedup over NVIDIA CUTLASS INT4 acceleration on RTX 3090. On RTX 4090 and H800, APT-LLM achieves up to 2.44$\times$ speedup over FP16 and 1.65$\times$ speedup over CUTLASS integer baselines.

Figures

Figures reproduced from arXiv: 2508.19087 by the authors.

Figure 2
Figure 2. The proposed APT-LLM scheme provides comprehensive optimiza￾tions across format, memory, and kernel levels, significantly improving the inference performance of arbitrary-precision quantized LLMs on GPUs. lish a framework for determining optimal hyperparameter configurations. This optimization process considers both the matrix dimensions and precision requirements of each specific MatMul operation, resulting in sign… view at source ↗
Figure 3
Figure 3. Comparison between the prefill and decode phases of LLM inference. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Comparison between bipolar-INT and signed INT. Bipolar-INT is well-suited for TCs’ parallel computing due to its unified operations and can be easily derived from INT. where i denotes the bit position of x. From the perspective of parallel computation, in the context of signed INT quantization, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (10 more)
Figure 6
Figure 6. Figure 6: Procedure of matrix decomposition and reassembly to [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Recovery-oriented memory scheduling strategy for arbitrary precision [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Detailed overview of the computational flow of APT. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Comparison of throughput between APT and other methods in the [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Comparison of throughput between APT and other methods in LLM [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Comparison of throughput between APT and other methods on different GPU platforms. model. It can be observed that in GEMV tasks, the perfor￾mance gap between BTC and APT is significantly reduced. This is because BTC also utilizes 1-bit TCs as its primary computation r…
Figure 13
Figure 13. Figure 13: Comparison of APT and other methods in accelerating LLM inference on RTX 3090. LLaMA3-8B Qwen2.5-7B Qwen2.5-14B OPT-6.7B Bloom-7B 0.0 0.5 1.0 1.5 2.0 Speedup FP16 CUTLASS INT4 APT W3A4 APT W2A2 APT W1A2 (a) Prefill on RTX 4090 LLaMA3-8B Qwen2.5-7B Qwen2.5-14B OPT-6.7B…
Figure 14
Figure 14. Figure 14: Comparison of APT and other methods in accelerating LLM inference on different GPU platforms. 2) Decode phase [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: Ablation study demonstrating the impact of memory scheduling and [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]
Figure 16
Figure 16. Figure 16: Kernel mapping hyperparameter configurations for different layers [PITH_FULL_IMAGE:figures/full_fig_p012_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 50 canonical work pages

  1. [1]

    Sparks of artificial general intelligence: Early exper- iments with gpt-4,

    S. Bubeck et al., “Sparks of artificial general intelligence: Early exper- iments with gpt-4,” arXiv preprint arXiv:2303.12712 , 2023

    arXiv 2023

  2. [2]

    The llama 3 herd of models,

    A. Dubey et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  3. [3]

    Deepseek-v3 technical report,

    A. Liu et al. , “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  4. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    arXiv 2025

  5. [5]

    Jumping nlp curves: A review of natural language processing research,

    E. Cambria et al. , “Jumping nlp curves: A review of natural language processing research,” IEEE Computational Intelligence Magazine (CIM), vol. 9, no. 2, pp. 48–57, 2014

    work page 2014

  6. [6]

    Scaling laws for neural language models,

    J. Kaplan et al. , “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020

    arXiv 2001

  7. [7]

    Chateda: A large language model powered autonomous agent for eda,

    H. Wu et al., “Chateda: A large language model powered autonomous agent for eda,” IEEE Transactions on Computer-Aided Design of Inte- grated Circuits and Systems (TCAD) , vol. 43, no. 10, pp. 3184–3197, 2024

    work page 2024

  8. [8]

    Dtatrans: Leveraging dynamic token-based quantization with accuracy compensation mechanism for efficient transformer archi- tecture,

    T. Yang et al., “Dtatrans: Leveraging dynamic token-based quantization with accuracy compensation mechanism for efficient transformer archi- tecture,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD) , vol. 42, no. 2, pp. 509–520, 2022

    work page 2022

Show all 69 references
  1. [9]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers et al. , “Qlora: Efficient finetuning of quantized llms,” Advances in neural information processing systems (NeurIPS) , vol. 36, pp. 10 088–10 115, 2023

    2023

  2. [10]

    Optq: Accurate quantization for generative pre-trained transformers,

    E. Frantar et al., “Optq: Accurate quantization for generative pre-trained transformers,” in The Eleventh International Conference on Learning Representations (ICLR), 2023

    2023

  3. [11]

    A precision-scalable risc-v dnn processor with on- device learning capability at the extreme edge,

    L. Huang et al. , “A precision-scalable risc-v dnn processor with on- device learning capability at the extreme edge,” in 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), 2024, pp. 927–932

    2024

  4. [12]

    Token-scaled logit distillation for ternary weight gen- erative language models,

    M. Kim et al. , “Token-scaled logit distillation for ternary weight gen- erative language models,” Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2024

    2024

  5. [13]

    Onebit: Towards extremely low-bit large language models,

    Y . Xu et al., “Onebit: Towards extremely low-bit large language models,” arXiv preprint arXiv:2402.11295 , 2024

    2024 arXiv

  6. [14]

    Smoothquant: Accurate and efficient post-training quantization for large language models,

    G. Xiao et al. , “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning (ICML) . PMLR, 2023, pp. 38 087–38 099

    2023

  7. [15]

    Omniquant: Omnidirectionally calibrated quantization for large language models,

    W. Shao et al. , “Omniquant: Omnidirectionally calibrated quantization for large language models,” in The Twelfth International Conference on Learning Representations (ICLR) , 2024

    2024

  8. [16]

    Holes: Boosting large language models efficiency with hardware-friendly lossless encoding,

    F. Liu et al. , “Holes: Boosting large language models efficiency with hardware-friendly lossless encoding,” in 2024 IEEE 42nd International Conference on Computer Design (ICCD) , 2024, pp. 207–214

    2024

  9. [17]

    Quantization via distillation and contrastive learning,

    Z. Pei et al. , “Quantization via distillation and contrastive learning,” IEEE Transactions on Neural Networks and Learning Systems (TNNLS), vol. 35, no. 12, pp. 17 164–17 176, 2024

    2024

  10. [18]

    Atom: Low-bit quantization for efficient and accurate llm serving,

    Y . Zhao et al. , “Atom: Low-bit quantization for efficient and accurate llm serving,” Proceedings of Machine Learning and Systems (MLSys) , vol. 6, pp. 196–209, 2024

    2024

  11. [19]

    Nvidia a100 tensor core gpu: Performance and innovation,

    J. Choquette et al. , “Nvidia a100 tensor core gpu: Performance and innovation,” IEEE Micro, vol. 41, no. 2, pp. 29–35, 2021

    2021

  12. [20]

    Rtx on—the nvidia turing gpu,

    J. Burgess, “Rtx on—the nvidia turing gpu,” IEEE Micro, vol. 40, no. 2, pp. 36–44, 2020

    2020

  13. [21]

    Dissecting tensor cores via microbenchmarks: Latency, throughput and numeric behaviors,

    W. Sun et al., “Dissecting tensor cores via microbenchmarks: Latency, throughput and numeric behaviors,” IEEE Transactions on Parallel and Distributed Systems (TPDS) , vol. 34, no. 1, pp. 246–261, 2022

    2022

  14. [22]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,

    Y . Lin et al., “Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,” arXiv preprint arXiv:2405.04532 , 2024

    2024 arXiv

  15. [23]

    G-blastn: accelerating nucleotide alignment by graphics processors,

    K. Zhao et al., “G-blastn: accelerating nucleotide alignment by graphics processors,” Bioinformatics, vol. 30, no. 10, pp. 1384–1391, 2014

    2014

  16. [24]

    Accelerating performance of gpu-based workloads using cxl,

    M. Arif et al. , “Accelerating performance of gpu-based workloads using cxl,” in Proceedings of the 13th Workshop on AI and Scientific Computing at Scale using Flexible Computing , 2023, pp. 27–31

    2023

  17. [25]

    Superneurons: Dynamic gpu memory management for training deep neural networks,

    L. Wang et al., “Superneurons: Dynamic gpu memory management for training deep neural networks,” in Proceedings of the 23rd ACM SIG- PLAN symposium on principles and practice of parallel programming (PPoPP), 2018, pp. 41–53

    2018

  18. [26]

    Gpt3. int8 (): 8-bit matrix multiplication for trans- formers at scale,

    T. Dettmers et al., “Gpt3. int8 (): 8-bit matrix multiplication for trans- formers at scale,” Advances in neural information processing systems (NeurIPS), vol. 35, pp. 30 318–30 332, 2022

    2022

  19. [27]

    Quant-llm: Accelerating the serving of large language models via fp6-centric algorithm-system co-design on modern gpus,

    H. Xia et al. , “Quant-llm: Accelerating the serving of large language models via fp6-centric algorithm-system co-design on modern gpus,” in 2024 USENIX Annual Technical Conference (ATC) , 2024, pp. 699–713

    2024

  20. [28]

    Tsm2x: High-performance tall-and-skinny matrix– matrix multiplication on gpus,

    C. Rivera et al. , “Tsm2x: High-performance tall-and-skinny matrix– matrix multiplication on gpus,” Journal of Parallel and Distributed Computing (JPDC), vol. 151, pp. 70–85, 2021

    2021

  21. [29]

    Stream-k: Work-centric parallel decomposition for dense matrix-matrix multiplication on the gpu,

    M. Osama et al. , “Stream-k: Work-centric parallel decomposition for dense matrix-matrix multiplication on the gpu,” in Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (PPoPP), 2023, pp. 429–431

    2023

  22. [30]

    Warp-aware adaptive energy efficiency calibration for multi-gpu systems,

    Z. Wang et al., “Warp-aware adaptive energy efficiency calibration for multi-gpu systems,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), vol. 42, no. 5, pp. 1676–1690, 2023

    2023

  23. [31]

    Dg-replace: A dataflow-driven gpu-accelerated an- alytical global placement framework for machine learning accelerators,

    A. B. Kahng et al., “Dg-replace: A dataflow-driven gpu-accelerated an- alytical global placement framework for machine learning accelerators,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD) , vol. 44, no. 2, pp. 696–708, 2025

    2025

  24. [32]

    Enabling efficient sparse multiplications on gpus with heuristic adaptability,

    J. Xu et al. , “Enabling efficient sparse multiplications on gpus with heuristic adaptability,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD) , pp. 1–1, 2024

    2024

  25. [33]

    Bstc: A novel binarized-soft-tensor-core design for acceler- ating bit-based approximated neural nets,

    A. Li et al., “Bstc: A novel binarized-soft-tensor-core design for acceler- ating bit-based approximated neural nets,” International Conference for High Performance Computing, Networking, Storage and Analysis (SC) , pp. 1–30, 2019

    2019

  26. [34]

    Accelerating binarized neural networks via bit-tensor-cores in turing gpus,

    A. Li et al., “Accelerating binarized neural networks via bit-tensor-cores in turing gpus,” IEEE Transactions on Parallel and Distributed Systems (TPDS), vol. 32, pp. 1878–1891, 2020

    2020

  27. [35]

    Demystifying the nvidia ampere architecture through microbenchmarking and instruction-level analysis,

    H. Abdelkhalik et al. , “Demystifying the nvidia ampere architecture through microbenchmarking and instruction-level analysis,” in 2022 IEEE High Performance Extreme Computing Conference (HPEC), 2022, pp. 1–8

    2022

  28. [36]

    Dissecting the nvidia turing t4 gpu via microbenchmark- ing,

    Z. Jia et al., “Dissecting the nvidia turing t4 gpu via microbenchmark- ing,” arXiv preprint arXiv:1903.07486 , 2019

    1903 arXiv

  29. [38]

    Gtco: Graph and tensor co-design for transformer-based image recognition on tensor cores,

    Y . Bai et al., “Gtco: Graph and tensor co-design for transformer-based image recognition on tensor cores,” IEEE Transactions on Computer- Aided Design of Integrated Circuits and Systems (TCAD), vol. 43, no. 2, pp. 586–599, 2024

    2024

  30. [39]

    Reducing shared memory footprint to leverage high throughput on tensor cores and its flexible api extension library,

    H. Ootomo et al. , “Reducing shared memory footprint to leverage high throughput on tensor cores and its flexible api extension library,” in Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region (HPCAsia) , 2023, pp. 1–8

    2023

  31. [40]

    Tc-gnn: Bridging sparse gnn computation and dense tensor cores on gpus,

    Y . Wang et al. , “Tc-gnn: Bridging sparse gnn computation and dense tensor cores on gpus,” in 2023 USENIX Annual Technical Conference (ATC), 2023, pp. 149–164

    2023

  32. [41]

    A survey on efficient inference for large language models,

    Z. Zhou et al. , “A survey on efficient inference for large language models,” arXiv preprint arXiv:2404.14294 , 2024. 14

    2024 arXiv

  33. [42]

    Transformer tricks: Precomputing the first layer,

    N. Graef, “Transformer tricks: Precomputing the first layer,” arXiv preprint arXiv:2402.13388, 2024

    2024 arXiv

  34. [43]

    Model tells you what to discard: Adaptive kv cache compression for llms,

    S. Ge et al. , “Model tells you what to discard: Adaptive kv cache compression for llms,” arXiv preprint arXiv:2310.01801 , 2023

    2023 arXiv

  35. [44]

    Efficiently scaling transformer inference,

    R. Pope et al., “Efficiently scaling transformer inference,” Proceedings of Machine Learning and Systems (MLSys) , vol. 5, pp. 606–624, 2023

    2023

  36. [45]

    Llm inference unveiled: Survey and roofline model insights,

    Z. Yuan et al. , “Llm inference unveiled: Survey and roofline model insights,” arXiv preprint arXiv:2402.16363 , 2024

    2024 arXiv

  37. [46]

    Squeezellm: Dense-and-sparse quantization,

    S. Kim et al. , “Squeezellm: Dense-and-sparse quantization,” in Inter- national Conference on Machine Learning (ICML) . PMLR, 2024, pp. 23 901–23 923

    2024

  38. [47]

    Quantsr: accurate low-bit quantization for efficient image super-resolution,

    H. Qin et al., “Quantsr: accurate low-bit quantization for efficient image super-resolution,” Advances in Neural Information Processing Systems (NeurIPS), vol. 36, pp. 56 838–56 848, 2023

    2023

  39. [48]

    Accurate lora-finetuning quantization of llms via infor- mation retention,

    H. Qin et al. , “Accurate lora-finetuning quantization of llms via infor- mation retention,” in Proceedings of the 41st International Conference on Machine Learning (ICML) , 2024, pp. 41 498–41 516

    2024

  40. [49]

    Bimatting: Efficient video matting via binarization,

    H. Qin et al. , “Bimatting: Efficient video matting via binarization,” Advances in Neural Information Processing Systems (NeurIPS) , vol. 36, pp. 43 307–43 321, 2023

    2023

  41. [50]

    Bibert: Accurate fully binarized bert,

    H. Qin et al. , “Bibert: Accurate fully binarized bert,” in International Conference on Learning Representations (ICLR) , 2022

    2022

  42. [51]

    Bebert: Efficient and robust binary ensemble bert,

    J. Tian et al. , “Bebert: Efficient and robust binary ensemble bert,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2023, pp. 1–5

    2023

  43. [52]

    Anda: Unlocking efficient llm inference with a variable- length grouped activation data format,

    C. Fang et al., “Anda: Unlocking efficient llm inference with a variable- length grouped activation data format,” in 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), 2025, pp. 1467–1481

    2025

  44. [53]

    Binaryconnect: Training deep neural networks with binary weights during propagations,

    M. Courbariaux et al. , “Binaryconnect: Training deep neural networks with binary weights during propagations,” Advances in neural informa- tion processing systems (NeurIPS) , vol. 28, 2015

    2015

  45. [54]

    Apnn-tc: Accelerating arbitrary precision neural net- works on ampere gpu tensor cores,

    B. Feng et al. , “Apnn-tc: Accelerating arbitrary precision neural net- works on ampere gpu tensor cores,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2021, pp. 1–13

    2021

  46. [55]

    O3bnn: An out-of-order architecture for high- performance binarized neural network inference with fine-grained prun- ing,

    T. Geng et al. , “O3bnn: An out-of-order architecture for high- performance binarized neural network inference with fine-grained prun- ing,” in Proceedings of the ACM International Conference on Super- computing (ICS), 2019, pp. 461–472

    2019

  47. [56]

    O3bnn-r: An out-of-order architecture for high- performance and regularized bnn inference,

    T. Geng et al. , “O3bnn-r: An out-of-order architecture for high- performance and regularized bnn inference,” IEEE Transactions on parallel and distributed systems (TPDS) , vol. 32, no. 1, pp. 199–213, 2020

    2020

  48. [57]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,

    S. Han et al. , “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149, 2015

    2015 arXiv

  49. [58]

    Energy-efficient neural network accelerator based on outlier-aware low-precision computation,

    E. Park et al. , “Energy-efficient neural network accelerator based on outlier-aware low-precision computation,” in ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA) , 2018, pp. 688–698

    2018

  50. [59]

    Haq: Hardware-aware automated quantization with mixed precision,

    K. Wang et al. , “Haq: Hardware-aware automated quantization with mixed precision,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , 2019, pp. 8612–8620

    2019

  51. [60]

    Lq-nets: Learned quantization for highly accurate and compact deep neural networks,

    D. Zhang et al. , “Lq-nets: Learned quantization for highly accurate and compact deep neural networks,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 365–382

    2018

  52. [61]

    Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,

    S. Zhou et al., “Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,”arXiv preprint arXiv:1606.06160, 2016

    2016 arXiv

  53. [62]

    Bitnet: Scaling 1-bit transformers for large language models,

    H. Wang et al. , “Bitnet: Scaling 1-bit transformers for large language models,” ArXiv, vol. abs/2310.11453, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:264172438

    2023 arXiv

  54. [63]

    CUTLASS,

    V . Thakkar et al. , “CUTLASS,” Jan. 2023. [Online]. Available: https://github.com/NVIDIA/cutlass

    2023

  55. [64]

    Understanding gemm performance and energy on nvidia ada lovelace: A machine learning-based analytical approach,

    P. Halim et al., “Understanding gemm performance and energy on nvidia ada lovelace: A machine learning-based analytical approach,” arXiv preprint arXiv:2411.16954, 2024

    2024 arXiv

  56. [65]

    Benchmarking and dissecting the nvidia hopper gpu architecture,

    W. Luo et al. , “Benchmarking and dissecting the nvidia hopper gpu architecture,” in 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS) , 2024, pp. 656–667

    2024

  57. [66]

    Qwen2. 5 technical report,

    A. Yang et al. , “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  58. [67]

    Opt: Open pre-trained transformer language models,

    S. Zhang et al., “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068 , 2022

    2022 arXiv

  59. [68]

    Bloom: A 176b-parameter open-access multilingual language model,

    B. Workshop et al., “Bloom: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100 , 2022

    2022 arXiv

  60. [69]

    Quarot: Outlier-free 4-bit inference in rotated llms,

    S. Ashkboos et al., “Quarot: Outlier-free 4-bit inference in rotated llms,” Advances in Neural Information Processing Systems (NeurIPS) , vol. 37, pp. 100 213–100 240, 2024

    2024

  61. [70]

    Pointer sentinel mixture models,

    S. Merity et al. , “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.