Pith. sign in

REVIEW 18 cited by

LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2206.09557 v4 pith:QYCTOENE submitted 2022-06-20 cs.DC cs.CL

Gunho Park , Baeseong Park , Minsub Kim , Sungjae Lee , Jeonghoon Kim , Beomseok Kwon , Se Jung Kwon , Byeongwook Kim
show 2 more authors
Youngjoo Lee Dongsoo Lee
This is my paper
classification cs.DCcs.CL
keywords lut-gemmquantizationdequantizationefficientprocessachievingcomparedcompression
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Recent advances in self-supervised learning and the Transformer architecture have significantly improved natural language processing (NLP), achieving remarkably low perplexity. However, the growing size of NLP models introduces a memory wall problem during the generation phase. To mitigate this issue, recent efforts have focused on quantizing model weights to sub-4-bit precision while preserving full precision for activations, resulting in practical speed-ups during inference on a single GPU. However, these improvements primarily stem from reduced memory movement, which necessitates a resource-intensive dequantization process rather than actual computational reduction. In this paper, we introduce LUT-GEMM, an efficient kernel for quantized matrix multiplication, which not only eliminates the resource-intensive dequantization process but also reduces computational costs compared to previous kernels for weight-only quantization. Furthermore, we proposed group-wise quantization to offer a flexible trade-off between compression ratio and accuracy. The impact of LUT-GEMM is facilitated by implementing high compression ratios through low-bit quantization and efficient LUT-based operations. We show experimentally that when applied to the OPT-175B model with 3-bit quantization, LUT-GEMM substantially accelerates token generation latency, achieving a remarkable 2.1$\times$ improvement on a single GPU when compared to OPTQ, which relies on the costly dequantization process.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices

    cs.DC 2025-12 conditional novelty 7.0 of 10

    Vec-LUT delivers up to 4.2x speedup over prior LUT methods for parallel ultra-low-bit LLM inference on edge devices by unifying lookups across tokens and adding cache-aware tensor layouts.

  2. OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

    cs.LG 2025-07 conditional novelty 7.0 of 10

    OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.

  3. QLoRA: Efficient Finetuning of Quantized LLMs

    cs.LG 2023-05 conditional novelty 7.0 of 10

    QLoRA finetunes 4-bit quantized LLMs via LoRA adapters to match full-precision performance while using far less memory, enabling 65B-scale training on single GPUs and producing Guanaco models near ChatGPT level.

  4. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    cs.LG 2022-10 unverdicted novelty 7.0 of 10

    GPTQ quantizes 175B-parameter GPT models to 3-4 bits per weight in one shot using approximate second-order information, achieving negligible accuracy degradation and 3-4x inference speedups.

  5. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

    cs.LG 2022-08 conditional novelty 7.0 of 10

    LLM.int8() performs 8-bit inference for transformers up to 175B parameters with no accuracy loss by combining vector-wise quantization for most features with 16-bit mixed-precision handling of systematic outlier dimensions.

  6. MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.

  7. StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Near-memory DeQuantization Blocks in the HBM base die dequantize weight-only quantized LLM weights on the load path, cutting GPU dequant overhead and yielding up to 7.08× mpGEMM speedup and 2.20× decode throughput in ...

  8. MxGLUT: A Reconfigurable LUT-Centric Broadcast Dataflow Accelerator for Mixed-Precision GEMM

    cs.AR 2026-07 unverdicted novelty 6.0 of 10

    MxGLUT introduces a reconfigurable LUT-centric broadcast dataflow accelerator with mixed-precision LUT-based PEs that unifies FP8-INT4 and FP8-FP8 GEMM without separate FP datapaths, reporting up to 2.16x prefill spee...

  9. When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMs

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    Experiments across code LLMs show no-review collapses fastest, human-gated filters slow collapse, and AI self-gates lose effect over time, degenerating to ungated self-training under self-confirming acceptance as prov...

  10. Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inference

    cs.AR 2026-04 unverdicted novelty 6.0 of 10

    A formalized design-space framework with generator and TSMC 16nm-validated cost model shows that LUT reuse gains depend on activation type and that larger cores improve density, yielding 2.2x area reduction over multi...

  11. MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learning

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    MP-ISMoE uses Gaussian noise perturbed iterative quantization and interactive side mixture-of-experts to deliver higher accuracy than prior memory-efficient transfer learning methods while keeping similar parameter an...

  12. Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

    cs.CL 2026-03 unverdicted novelty 6.0 of 10

    ZipCal curates calibration data for LLM pruning and quantization by maximizing lexical diversity via Zipfian power laws, outperforming random sampling and matching perplexity-based methods at 240x speed.

  13. Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective

    cs.PF 2025-08 conditional novelty 6.0 of 10

    No single LLM quantization method dominates performance, energy, and quality; the best choice depends on task, request length, load, parallelism, and GPU type.

  14. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

    cs.CL 2023-06 conditional novelty 6.0 of 10

    AWQ quantizes LLM weights to low bits by scaling salient channels based on activation statistics, outperforming prior methods on language, coding, math, and multi-modal benchmarks.

  15. IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    IO-SVD performs SVD-based LLM compression by constructing a KL-aware double-sided whitening space and using first-order loss estimates for heterogeneous rank allocation.

  16. Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization

    cs.CL 2026-03 conditional novelty 5.0 of 10

    ZipCal curates calibration sets for LLM pruning and quantization by maximizing Zipfian lexical diversity, matching model-perplexity methods at ~240× lower cost.

  17. BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models

    cs.LG 2026-02 unverdicted novelty 5.0 of 10

    BPDQ creates variable quantization grids from bit-planes and scalar coefficients, refined iteratively with second-order data to minimize output error, enabling 2-bit serving of Qwen2.5-72B on one RTX 3090 at 83.85% GS...

  18. Inference performance evaluation for LLMs on edge devices with a novel benchmarking framework and metric

    cs.PF 2025-08 reject novelty 2.0 of 10

    A benchmarking study of LLM inference on three edge devices with a proposed MBU metric that reduces to a standard throughput normalization.

Pith tools