Pith. sign in

REVIEW 4 major objections 6 minor 51 references

UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that metadata traffic, not arithmetic, throttles low-bit sparse LLM decoding on edge GPUs, and that a shared-bitmap encoding with a register-level decoding kernel removes enough of it to beat sparse and dense baselines by…

desk verdict Real kernel contribution in IE-BME and LSPD, but the end-to-end results rest on an unstated row-reordering permutation that likely changes the model's outputs; kernel-level work deserves review, framework-level claims need fixes. read the letter →

arxiv 2608.09291 v1 pith:6YZ27DYH submitted 2026-08-10 cs.DC

Tianhao Jiang , Hang Gu , Teng Wang , Qianyu Cheng , ZhenDong Zheng , Cheng Tang , Qiyue Su , Wenqi Lou
show 4 more authors
Lei Gong Chao Wang Xi Li Xuehai Zhou
This is my paper · ORCID
classification cs.DC
keywords EdgeLLMinferenceunstructuredpruninglow-bitquantizationSpMMTensorCoreunionbitmapencodingpayload-to-metadataratioefficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that once LLM weights are pruned and quantized to 4-bit weights and activations (W4A4), the dominant cost in small-batch decoding is no longer arithmetic or even the weight bytes themselves, but the metadata, the indices, offsets, and bitmaps that tell the kernel where the surviving nonzeros live. The authors define the Payload-to-Metadata Ratio (PMR) and show analytically that lower bit widths shrink PMR, dragging the effective compute intensity of a sparse kernel down toward the bandwidth ceiling. To reverse that, they propose UnionSparse, whose Index-Efficient Bitmap Encoding (IE-BME) lets one shared union bitmap describe four aligned 4-bit payload streams, and whose Low-Bit Shared-Memory Parallel Decoding (LSPD) turns those bitmaps directly into Tensor Core register fragments. Reported results under W4A4 quantization put UnionSparse 2.30x and 1.43x ahead of FlashLLM and SpInfer, and 1.56x and 3.46x ahead of CUTLASS and cuBLAS Tensor Core. If the claim holds, index efficiency is a first-order design axis for edge LLM inference, not an implementation detail.

What carries the argument

The central object is the Union Bitmap, a single logical activity bit shared by $Q=R_A C_A$ area-aligned low-bit payload positions and stored as packed 64-bit words; it replaces $Q$ per-area bitmaps or per-element indices, which is what raises PMR and lowers metadata traffic. The decoding machinery is Low-Bit Shared-Memory Parallel Decoding (LSPD), which assigns each packed union-bitmap segment to a thread, computes base offsets with popcount instructions and prefix sums, enumerates active bits, and uses each compact rank to gather one value from all $Q$ aligned compressed streams, packing them directly into the register fragment layout expected by Tensor Core matrix-multiply-accumulate (MMA) instructions. A bitmap-aware row-reordering step, solved offline by grouped subset dynamic programming, permutes rows inside each local tile to reduce the number of active union-mask entries and shorten the compressed streams. The Decoupled Warp-Specialized Producer-Consumer (DW-PC) pipeline then assigns one warp to prefetch the next weight and activation tiles into a double buffer while the other warps decode and compute the current tile. The PMR identity and the effective compute-intensity formula are the analytical core, explaining why higher-PMR formats shift rightward on the roofline and move away from the memory-bound region.

What would settle it

Run the paper's OPT-13B W4A4 end-to-end configuration with the inverse of the bitmap-aware row permutation applied after each affected layer, and compare latency, output equality, and generation quality against the unpermuted model; if the inverse-corrected model either loses the reported speedup or produces different tokens on identical prompts, the end-to-end claim is not established.

Watch

Extended reading notes

Core claim

The core discovery is that payload extraction, not payload size, becomes the binding constraint when quantized sparsity and small decode batches meet. With 4-bit weights a nonzero occupies a single byte, so the index or bitmap needed to locate it can rival or exceed the value itself; the paper quantifies this with the Payload-to-Metadata Ratio $\mathrm{PMR} = B_w M K(1-s)\,/\,S_{\mathrm{meta}}$ and an effective compute intensity that shows metadata inflating weight-side traffic by the factor $(1+1/\mathrm{PMR})$. IE-BME raises PMR by grouping $Q=R_A C_A$ area-aligned value positions under one logical union bit, so one active bit selects $Q$ low-bit payloads aligned in $Q$ streams. LSPD then reconstructs MMA-ready weight fragments by popcounting packed 64-bit union-bitmap words, generating compact global indices, and gathering all $Q$ streams in one traversal, eliminating per-area sparse indices and the bank conflicts they cause. Under W4A4 on an edge GPU with 30–70% sparsity, the paper reports 1.56x over CUTLASS, 3.46x over cuBLAS Tensor Core, 2.30x over FlashLLM, and 1.43x over SpInfer at kernel level, with the largest gains at decode widths 1–8 and end-to-end OPT-13B speedups up to 2.63x over FasterTransformer.

Load-bearing premise

The end-to-end results assume that the row-permutation step applied to each weight tile during format conversion is either undone later or harmless to the model's outputs, but the paper never says how the permutation is reversed or propagated across layers, and it reports no accuracy or perplexity for the reordered model.

Editorial extensions

If this is right

  • Under W4A4 quantization, pruning alone cannot deliver its expected speedup: the kernel remains bandwidth-bound unless the sparse format also shrinks metadata, so index efficiency must be part of any low-bit sparse inference stack.
  • Across decode widths 1–8, UnionSparse reports 3.50x–4.73x speedups over cuBLAS Tensor Core, so the benefit concentrates in the small-batch regime where edge LLM decoding operates.
  • At larger decode width, N=32, the advantage narrows, implying that dense, regular Tensor Core execution reclaims ground once the workload is compute-rich rather than metadata-bound.
  • End-to-end on OPT-13B with 50% pruning and W4A4 quantization, decode-phase SpMM accounts for 69–75% of per-token latency, and UnionSparse reports up to 2.63x throughput over FasterTransformer at short output lengths.
  • The PMR analysis extends beyond the tested 4-bit case: the paper's W2A4 results show the same qualitative trend, suggesting the metadata bottleneck worsens at lower bit widths and index-efficient formats become even more valuable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the PMR/CI formulation could serve as an offline screening tool, predicting from a format's metadata bytes, sparsity, and batch width whether a kernel will sit on the memory-bound roofline before implementation, and thus guiding format selection for new models.
  • Editorial inference: the grouped subset dynamic program for row reordering is exponential in tile height and the paper reports about four hours for OPT-13B; a cheaper greedy or local-search heuristic would scale the offline stage to larger models, and measuring how much of the 99.52 versus 99.90 microsecond ablation gain it preserves would isolate the value of exact optimization.
  • Editorial inference: the same union-bitmap amortization should transfer to any collection of aligned low-bit arrays that share an activity pattern, such as multi-head attention projections or grouped KV-cache entries, making the core idea a general encoding principle rather than only an SpMM kernel.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes UnionSparse, a sparsity framework for low-bit (W4A4) sparse LLM inference on edge GPUs. The core ideas are a new sparse format, IE-BME, which amortizes one logical bitmap across multiple low-bit payload areas, and a SpMM kernel combining a decoupled warp-specialized producer-consumer pipeline (DW-PC) with low-bit shared-memory parallel decoding (LSPD). The authors introduce a Payload-to-Metadata Ratio (PMR) and argue that raising PMR improves effective compute intensity, thereby moving low-bit sparse kernels away from the memory-bound regime. They report kernel-level speedups over SpInfer, FlashLLM, CUTLASS, and cuBLAS, and end-to-end OPT-13B throughput and energy results on Jetson AGX Orin, with additional experiments on Jetson Thor.

Significance. The problem addressed is timely and important: low-bit quantization shrinks weight payloads without proportionally shrinking sparse metadata, and this imbalance is particularly acute on bandwidth-limited edge GPUs. The paper contains useful engineering contributions: the IE-BME representation is clearly described, the LSPD decoding scheme is well motivated, the ablation study isolates the contribution of the main kernel optimizations, and source code is provided. The kernel-level comparison against SpInfer under INT4 is meaningful, and the profiling analysis gives concrete evidence on why the proposed kernel reduces bank conflicts and improves ALU utilization. However, several load-bearing issues in the theoretical analysis and in the end-to-end evaluation currently prevent the paper's central claims from being accepted as stated.

major comments (4)
  1. [§V-B and §VII-E] Bitmap-aware row reordering permutes rows within each local tile, which permutes the output features of the corresponding layer. The paper never states how this permutation is undone before the next layer or folded into the columns of the subsequent weight matrix, and it reports no perplexity or accuracy for the reordered OPT-13B model. Because reordering is performed independently per local tile per matrix, different layers generally have different permutations, so the permutation cannot be silently absorbed by the next linear layer, especially with residual connections, LayerNorm, and nonlinearities in between. The reported end-to-end throughput and energy numbers in §VII-E and Table IV may therefore correspond to a model computing different outputs. Kernel-level comparisons are unaffected, but the framework-level claims require either a description of how the permutation is neutralized or an accuracy/PP L evaluation of the reordered model.
  2. [§III-C, Eq. (2)] Equation (2) is dimensionally inconsistent: the numerator 2 M N (1−s) has dimensions M×N, while the denominator has terms B_w M (1−s)(1 + 1/PMR) and B_w N, missing the reduction dimension K in both the FLOP count and the traffic terms. A correct effective compute intensity for an M×K weight times K×N activation product should be roughly 2 M N K (1−s) / [B_w M K (1−s)(1 + 1/PMR) + B_a K N], where B_a is the activation byte width. Because Figure 5 uses Eq. (2) to compute the x-axis effective compute intensity, the roofline comparison and the associated claim that higher PMR shifts operating points rightward rest on an invalid formula. This needs to be corrected and the resulting roofline recomputed.
  3. [§III-B, §V-A] The PMR definition in Eq. (1) uses the ideal number of nonzero elements M K (1−s) as the payload size, but IE-BME stores Q payload values for every active union-mask entry, including area-specific zeros. As stated in §V-A, an active group emits a value from every area even if some area values are zero. For random 50% sparsity with Q=4, roughly 93.75% of groups are active, so the stored compressed payload is about 1.875× the number of true nonzeros, whereas CSR and other formats store only nonzeros. The PMR and the traffic model in Eq. (2) therefore undercount IE-BME's actual weight traffic and overstate its effective compute intensity. Since the paper's central narrative is that IE-BME improves payload efficiency, the analysis should use the actually stored payload size (e.g., Q × number of active union bits) when comparing formats and when computing effective CI.
  4. [§VII-A, Table II] The headline 2.30× speedup over FlashLLM is not a W4A4 comparison. Section VII-A states that FlashLLM is evaluated 'with its original load-sparse, compute-dense FP16 path using inputs pre-converted to the required format,' meaning FlashLLM reads and computes on FP16 weights rather than INT4 weights. A W4A4 kernel is compared against a kernel that moves 4× the weight bytes per element and computes with higher-precision arithmetic, so the reported speedup over FlashLLM is inflated by the baseline's precision mismatch. Either an FP16-equivalent variant of UnionSparse should be compared against FlashLLM, or FlashLLM should be given a comparable INT4 path; otherwise the abstract's 'under W4A4 ... outperforms FlashLLM ... by 2.30×' claim is not supported by the experiments as described.
minor comments (6)
  1. [§VII-B] The text says 'Jetson Thor GPU shown in Figure II' but the reference should be to Table II, which contains the Thor results; the figure reference is incorrect.
  2. [Acknowledgment] There is a typo in 'Studentsnnovation and Entrepreneurship Foundation'; it should be 'Student Innovation and Entrepreneurship Foundation'.
  3. [Table III] The incremental gain of Reorder over LSPD+DB is 0.38 µs (99.90 µs to 99.52 µs), less than 0.4%; the text calls this a 'small but consistent' improvement, but no variance or repeated-run statistics are reported to support the consistency claim.
  4. [Figure 13] The x-axis labels such as '1*64*64' are not defined; the caption mentions B, I, O but the notation should be spelled out in the caption or legend.
  5. [§III-C] Equation (2) uses B_w for both weight and activation traffic; the activation term should be written with an explicit activation byte width (e.g., B_a) and the dimensions of all quantities should be stated.
  6. [§VII-E] The end-to-end experiment uses 'an average input length of 10 tokens,' but no distribution or range is given; short-prompt edge workloads can be sensitive to input-length variance, so the setting should be described more precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PMR and effective compute intensity are definitional accounting identities, and the reported speedups are measured against external baselines rather than derived from fitted inputs.

full rationale

The paper's derivation chain is self-contained. PMR (Eq. 1) is defined as payload bytes over metadata bytes; Eq. 2 rewrites the standard FLOP/byte ratio with weight-side traffic decomposed via PMR, so the claim that higher PMR raises effective compute intensity is a transparent accounting identity rather than a fitted parameter renamed as a prediction. The roofline plot uses an analytically computed x-axis against measured kernel performance, and all headline speedups (2.30x vs FlashLLM, 1.43x vs SpInfer, etc.) are arithmetic means of measured per-case latencies against external baselines, so the central result is not forced by construction. Self-citations ([19], [28], [29]) appear in background and related-work contexts and are not load-bearing. The ablation study measures the incremental latency of LSPD, double buffering, and reorder empirically rather than reading the gains off the objective function. The reviewer-identified concerns are correctness/validity risks, not circularity: the Section V-B row reordering is never shown to be semantically inverted end-to-end on OPT-13B, and the Thor evaluation is explicitly qualified by the statement 'we don’t use optimization method based on hardware characteristic of Jetson thor and choose RA=2 and CA=2'. Under the hard rules, neither of these raises the circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central design choices are hand-selected hyperparameters rather than quantities fit to match target numbers. The main unstated premises are semantic correctness of row reordering and transferability of the Wanda plus Atom pipeline; these are assumptions the paper inherits from prior work or leaves implicit.

free parameters (5)
  • Area grid (R_A, C_A) = 2 x 2
    Hand-chosen partition of each local tile into R_A by C_A areas. Figure 7 uses 2x2 for illustration, and the Thor experiment explicitly uses R_A=2 and C_A=2.
  • Local tile shape (H_L, K_L) = 16 x 64
    Chosen for Jetson AGX Orin. Figure 7 shows an 8x32 schematic of the 16x64 local tile.
  • Bitmap segment width G = 32 bits per segment (two 16-bit rows per 64-bit word)
    Implementation choice for thread-to-segment mapping in LSPD, affecting parallel decoding granularity.
  • Warp count and producer-consumer roles = 4 warps, Warp 0 as producer
    Design choice in Algorithm 2, controlling overlap and SM occupancy.
  • SplitK scheduling heuristic = INITSCHEDULE(BatchID, SplitK), values not specified
    Kernel schedule heuristic used in Algorithm 2 line 3; the exact scheduling parameters are not reported.
assumptions (4)
  • domain assumption Weight sparsity is static and unstructured; the union mask over grouped positions exactly represents nonzeros after reordering.
    Section V-A partitions the pruned tile into areas and defines B_t as a logical OR. This models the workload but excludes dynamic or structured sparsity.
  • domain assumption Reordering rows of W within each local tile preserves end-to-end semantics because the induced output permutation is neutralized in adjacent layers or at the final output.
    Section V-B minimizes union-mask activity by permuting rows but never states how the permutation is undone. No accuracy numbers are reported in Section VII-E.
  • domain assumption Wanda pruning to 50% sparsity and Atom W4A4 quantization keep the model usable; accuracy is assumed from prior work.
    Section VII-E applies these methods without reporting perplexity or task accuracy for the resulting OPT-13B.
  • domain assumption The roofline CI model in Equation (2) captures first-order performance, with memory traffic equal to payload plus metadata plus activations and cache and instruction effects omitted.
    Section III-C uses this model to explain the measured trend in Figure 5; it is a deliberate simplification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge." pith.science (2026). https://pith.science/paper/6YZ27DYH

@misc{pith2026260809291,
  author       = {Pith},
  title        = {Pith review of: UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6YZ27DYH}},
  note         = {Machine review of arXiv:2608.09291}
}
read the original abstract

Edge LLM inference combines sparsity and low-bit quantization to meet device memory, latency, and power limits. Yet quantization shrinks weight payloads without proportionally reducing sparse metadata, so index traffic and nonzero extraction become critical SpMM bottlenecks. We introduce the Payload-to-Metadata Ratio (PMR) and show that improving PMR raises effective compute intensity in decoding. We present UnionSparse, an index-efficient framework that combines Index-Efficient Bitmap Encoding (IE-BME) with a SpMM kernel using Low-Bit Shared-Memory Parallel Decoding (LSPD). IE-BME amortizes metadata and aligns sparse traversal with fragment assembly, while LSPD improves small-batch execution. Under W4A4 quantization and 30%--70% sparsity, UnionSparse outperforms FlashLLM and SpInfer by 2.30x and 1.43x, and CUTLASS and cuBLAS Tensor Core by 1.56x and 3.46x, respectively. These results establish payload-extraction efficiency as a first-order concern for low-bit sparse inference on edge GPUs. Source code is available at: https://github.com/Victor-Alen/UnionSparse.

Figures

Figures reproduced from arXiv: 2608.09291 by the authors.

Figure 1
Figure 1. Representative LLM deployment scenarios and service workflow. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Breakdown of OPT-13B inference on Jetson AGX Orin 64GB. [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. PMR comparison across precision levels and sparse formats. The left [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (9 more)
Figure 6
Figure 6. Figure 6: Overall framework of UnionSparse with the representation and execution optimization. The representation part reduces logical union-mask activity [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Hierarchical IE-BME representation. A sparse weight matrix is partitioned into global tiles (GTs), each containing multiple local tiles (LTs). For an LT () [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Decoupled Warp-Specialized Producer-Consumer (DW-PC) pipeline. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Low-Bit Shared-Memory Parallel Decoding (LSPD). (a) Thread-to-segment mapping on the LT-level union bitmap. (b) Bitmap-guided offset generation [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Kernel-Level Speedup across Decode Widths [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Kernel-Level Speedup across Decode Widths [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 13
Figure 13. Figure 13: End-to-end latency breakdown of UnionSparse during prefill and [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 12
Figure 12. Figure 12: Kernel Utilization Breakdown under W4A4 Quantization for Two [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 14
Figure 14. Figure 14: End-to-End Inference Performance of UnionSparse on OPT-13B with W4A4 Quantization. [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 35 canonical work pages

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems, vol. 30, 2017

    2017

  2. [3]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [4]

    Edgellm: Fast on-device llm inference with speculative decoding,

    D. e. a. Xu, “Edgellm: Fast on-device llm inference with speculative decoding,”IEEE Transactions on Mobile Computing, 2025

    work page 2025

  4. [5]

    Tenet: An efficient sparsity-aware lut-centric architec- ture for ternary llm inference on edge,

    Z. e. a. Huang, “Tenet: An efficient sparsity-aware lut-centric architec- ture for ternary llm inference on edge,”arXiv preprint arXiv:2509.13765, 2025. IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS 13

    arXiv 2025

  5. [6]

    Efficient inference for edge large language models,

    G. e. a. Cai, “Efficient inference for edge large language models,” Tsinghua Science and Technology, 2026

    work page 2026

  6. [7]

    Hqp: Sensitivity-aware hybrid quantization and pruning for ultra-low-latency edge ai inference,

    D. Gopalan and R. Ali, “Hqp: Sensitivity-aware hybrid quantization and pruning for ultra-low-latency edge ai inference,”arXiv preprint arXiv:2602.06069, 2026

    work page arXiv 2026

  7. [8]

    Sustainable llm inference for edge ai,

    E. J. e. a. Husom, “Sustainable llm inference for edge ai,”arXiv preprint arXiv:2504.03360, 2025

    arXiv 2025

  8. [9]

    A unified and resource-aware framework for adaptive inference optimization on edge devices,

    Y . e. a. Wang, “A unified and resource-aware framework for adaptive inference optimization on edge devices,”Electronics, vol. 14, no. 11, p. 2188, 2025

    work page 2025

Show all 51 references
  1. [10]

    Vec-lut: Vector table lookup for parallel ultra-low-bit llm inference on edge devices,

    X. e. a. Li, “Vec-lut: Vector table lookup for parallel ultra-low-bit llm inference on edge devices,”arXiv preprint arXiv:2512.06443, 2025

    2025 arXiv

  2. [11]

    H2eal: Hybrid-bonding architecture with hybrid sparse attention for efficient long-context llm inference,

    Z. e. a. Fu, “H2eal: Hybrid-bonding architecture with hybrid sparse attention for efficient long-context llm inference,”arXiv preprint arXiv:2508.16653, 2025

    2025

  3. [12]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” in The Eleventh International Conference on Learning Representations (ICLR), 2023

    2023

  4. [13]

    Awq: Activation-aware weight quanti- zation for on-device LLM compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for on-device LLM compression and acceleration,”Proceedings of Machine Learning and Systems, vol. 6, 2024

    2024

  5. [14]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,

    Y . Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han, “Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,”Proceedings of Machine Learning and Systems, vol. 7, 2025

    2025

  6. [15]

    Flatquant: Flatness matters for llm quantization,

    Y . Sun, R. Liu, H. Bai, H. Bao, K. Zhao, Y . Li, J. Hu, X. Yu, L. Hou, C. Yuanet al., “Flatquant: Flatness matters for llm quantization,”arXiv preprint arXiv:2410.09426, 2024

    2024 arXiv

  7. [16]

    LLM.int8(): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” arXiv preprint arXiv:2208.07339, 2022

    2022 arXiv

  8. [17]

    SmoothQuant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” inProceedings of the 40th International Con- ference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. ...

    2023

  9. [18]

    Sparsegpt: Massive language models can be accurately pruned in one-shot,

    E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202, 2023, pp. 10 323–10 337

    2023

  10. [19]

    Unleashing network/accelerator co-exploration potential on fpgas: A deeper joint search,

    W. Lou, L. Gong, C. Wang, J. Qian, X. Wang, C. Li, and X. Zhou, “Unleashing network/accelerator co-exploration potential on fpgas: A deeper joint search,”IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2024

    2024

  11. [20]

    Palm-e: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yuet al., “Palm-e: An embodied multimodal language model,”arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  12. [21]

    Foundation models in autonomous driving: A survey on scenario generation and scenario analysis,

    Y . Gao, M. Piccinini, Y . Zhang, D. Wang, K. Moller, R. Brusnicki, B. Zarrouki, A. Gambi, J. F. Totz, K. Stormset al., “Foundation models in autonomous driving: A survey on scenario generation and scenario analysis,”IEEE Open Journal of Intelligent Transportation Systems, 2026

    2026

  13. [22]

    Intelligent assistant language understanding on device,

    C. Aas, H. Abdelsalam, I. Belousova, S. Bhargava, J. Cheng, R. Daland, J. Driesen, F. Flego, T. Guigue, A. Johannsenet al., “Intelligent assistant language understanding on device,”arXiv preprint arXiv:2308.03905, 2023

    2023 arXiv

  14. [23]

    Private llm inference on consumer black- well gpus: A practical guide for cost-effective local deployment in smes,

    J. Knoop and H. Holtmann, “Private llm inference on consumer black- well gpus: A practical guide for cost-effective local deployment in smes,” arXiv preprint arXiv:2601.09527, 2026

    2026

  15. [24]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY ...

    2023

  16. [25]

    Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks,

    T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste, “Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks,”Journal of Machine Learning Research, vol. 22, no. 1, pp. 10 882–11 105, 2021

    2021

  17. [26]

    Wrp: Weight recover prune for structured sparsity,

    Z. Tan, X. Zhang, and Z. Wei, “Wrp: Weight recover prune for structured sparsity,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 6433– 6443

    2024

  18. [27]

    Structured pruning for large language models using coupled components elimination and minor fine-tuning,

    H. Zhang, X. XiaolongShi, J. Sun, and G. Sun, “Structured pruning for large language models using coupled components elimination and minor fine-tuning,” inFindings of the Association for Computational Linguistics: NAACL 2024, 2024, pp. 1–12

    2024

  19. [28]

    Unisparta: A unified sparse tensor program tuning framework,

    Z. Wang, L. Gong, X. Qu, C. Tang, W. Lou, T. Wang, Q. Cheng, X. Chen, C. Wang, and X. Zhou, “Unisparta: A unified sparse tensor program tuning framework,”IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2025

    2025

  20. [29]

    Closertome: A unified framework for accurate and transferable latency prediction across heterogeneous devices,

    C. Tang, G. Sui, W. Lou, Z. Wang, J. Tuo, W. Xie, Y . Gao, Y . Zhu, L. Gong, C. Wanget al., “Closertome: A unified framework for accurate and transferable latency prediction across heterogeneous devices,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 4...

    2026

  21. [30]

    Sparse gpu kernels for deep learning,

    T. Gale, M. Zaharia, C. Young, and E. Elsen, “Sparse gpu kernels for deep learning,” inSC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2020, pp. 1–14

    2020

  22. [31]

    Dtc-spmm: Bridging the gap in accelerating general sparse matrix multiplication with tensor cores,

    R. Fan, W. Wang, and X. Chu, “Dtc-spmm: Bridging the gap in accelerating general sparse matrix multiplication with tensor cores,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, 2024

    2024

  23. [32]

    High performance unstructured spmm computation using tensor cores,

    P. Okanovic, G. Kwasniewski, P. Sylos Labini, M. Besta, F. Vella, and T. Hoefler, “High performance unstructured spmm computation using tensor cores,” arXiv preprint arXiv:2408.11551, 2024, accepted at SC ’24

    2024 arXiv

  24. [33]

    Acc-spmm: Accelerating general-purpose sparse matrix-matrix multiplication with gpu tensor cores,

    H. Zhao, S. Li, J. Wang, C. Zhou, J. Wang, Z. Xin, S. Li, Z. Liang, Z. Pan, F. Liuet al., “Acc-spmm: Accelerating general-purpose sparse matrix-matrix multiplication with gpu tensor cores,” inProceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Pa...

    2025

  25. [34]

    V oltrix: Sparse {Matrix-Matrix}multiplication on tensor cores with asynchronous and balanced kernel optimization,

    Y . Xia, W. Wang, D. Yang, X. Zhou, and D. Cheng, “V oltrix: Sparse {Matrix-Matrix}multiplication on tensor cores with asynchronous and balanced kernel optimization,” in2025 USENIX Annual Technical Con- ference (USENIX ATC 25), 2025, pp. 699–714

    2025

  26. [35]

    {SparTA}:{Deep-Learning}model sparsity via{Tensor- with-Sparsity-Attribute},

    N. Zheng, B. Lin, Q. Zhang, L. Ma, Y . Yang, F. Yang, Y . Wang, M. Yang, and L. Zhou, “{SparTA}:{Deep-Learning}model sparsity via{Tensor- with-Sparsity-Attribute},” in16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022, pp. 213–232

    2022

  27. [36]

    cusparse library user guide,

    “cusparse library user guide,” NVIDIA CUDA Documentation, 2024, https://docs.nvidia.com/cuda/cusparse/

    2024

  28. [37]

    cuSPARSELt,

    NVIDIA, “cuSPARSELt,” https://developer.nvidia.com/cusparselt, 2024

    2024

  29. [38]

    Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity,

    H. Xia, Z. Zheng, Y . Li, D. Zhuang, Z. Zhou, X. Qiu, Y . Li, W. Lin, and S. L. Song, “Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity,”arXiv preprint arXiv:2309.10285, 2023

    2023 arXiv

  30. [39]

    Spinfer: Leveraging low-level sparsity for efficient large language model inference on gpus,

    R. Fan, X. Yu, P. Dong, Z. Li, G. Gong, Q. Wang, W. Wang, and X. Chu, “Spinfer: Leveraging low-level sparsity for efficient large language model inference on gpus,” inProceedings of the Twentieth European Conference on Computer Systems (EuroSys ’25), 2025, pp. 243–260

    2025

  31. [40]

    Sputnik: Sparse matrix multiplication on gpus,

    “Sputnik: Sparse matrix multiplication on gpus,” Open-Source Project Repository, 2024, https://github.com/google-research/sputnik

    2024

  32. [41]

    Nvidia jetson agx orin developer kit,

    “Nvidia jetson agx orin developer kit,” NVIDIA Developer Documentation, 2024, https://developer.nvidia.com/embedded/ jetson-agx-orin-developer-kit

    2024

  33. [42]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvronet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  34. [43]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. Singh Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,” arXiv prepri...

    2022 arXiv

  35. [44]

    Qwen technical report,

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huanget al., “Qwen technical report,” arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  36. [45]

    Llama 3: Open foundation and instruction-tuned models,

    Meta, “Llama 3: Open foundation and instruction-tuned models,” 2024

    2024

  37. [46]

    Cutlass: Cuda templates for linear algebra subroutines and solvers,

    “Cutlass: Cuda templates for linear algebra subroutines and solvers,” NVIDIA Developer Documentation and Open-Source Repository, 2024, https://github.com/NVIDIA/cutlass

    2024

  38. [47]

    cublas library user guide,

    “cublas library user guide,” NVIDIA CUDA Documentation, 2024, https://docs.nvidia.com/cuda/cublas/

    2024

  39. [48]

    Fastertransformer,

    “Fastertransformer,” NVIDIA Open-Source Repository, 2023, https:// github.com/NVIDIA/FasterTransformer

    2023

  40. [49]

    A simple and effective pruning approach for large language models,

    M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A simple and effective pruning approach for large language models,” inThe Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  41. [50]

    Atom: Low-bit quantization for efficient and accurate llm serving,

    Y . Zhao, C.-Y . Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,”Proceedings of Machine Learning and Systems, vol. 6, pp. 196–209, 2024. IEEE TRANSACTIONS ON COMPUTER...

    2024

  42. [2019]

    He examines various aspects of embedded systems, with a focus on performance, availability, flexibility, and energy efficiency

    He is currently a research scientist with the School of Computer Science, University of Science and Technology of China. He examines various aspects of embedded systems, with a focus on performance, availability, flexibility, and energy efficiency. His research interests inclu...

    2003

  43. [2023]

    He is currently pursuing the Eng.D

    In the same year, he began his master’s studies with the School of Computer Science and Technology, University of Science and Technology of China (USTC), Hefei, China. He is currently pursuing the Eng.D. degree with the School of Software Engineering, USTC. His research focuse...

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.