Pith. sign in

REVIEW 10 cited by

QQQ: Quality Quattuor-Bit Quantization for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.09904 v3 pith:YNOCU5S3 submitted 2024-06-14 cs.LG

Ying Zhang , Peng Zhang , Mincong Huang , Jingyang Xiang , Yujie Wang , Chao Wang , Yineng Zhang , Lei Yu
show 2 more authors
Chuan Liu Wei Lin
This is my paper
classification cs.LG
keywords timesgemmperformancequantizationspeedw4a8inferencemodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Quantization is a proven effective method for compressing large language models. Although popular techniques like W8A8 and W4A16 effectively maintain model performance, they often fail to concurrently speed up the prefill and decoding stages of inference. W4A8 is a promising strategy to accelerate both of them while usually leads to a significant performance degradation. To address these issues, we present QQQ, a Quality Quattuor-bit Quantization method with 4-bit weights and 8-bit activations. QQQ employs adaptive smoothing and Hessian-based compensation, significantly enhancing the performance of quantized models without extensive training. Furthermore, we meticulously engineer W4A8 GEMM kernels to increase inference speed. Our specialized per-channel W4A8 GEMM and per-group W4A8 GEMM achieve impressive speed increases of 3.67$\times$ and 3.29 $\times$ over FP16 GEMM. Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 $\times$, 2.10$\times$, and 1.25$\times$ compared to FP16, W8A8, and W4A16, respectively.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

    cs.DC 2026-06 conditional novelty 7.0 of 10

    APEX4 co-designs pure INT4 GEMM kernels with ρ-aware granularity adaptation to deliver up to 2.09× end-to-end speedup on GPUs with low ρ while keeping LLaMA-2-70B perplexity within 0.63 of FP16.

  2. Studying quantization trade-offs for efficient inference deployment in machine translation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Combining 200-400 token document chunking with W4A8/W8A8 quantization improves MT serving efficiency, while long-context translation quality collapses for quantized EuroLLM but not for quantized Hy-MT2.

  3. Studying quantization trade-offs for efficient inference deployment in machine translation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Quantized Hy-MT2 models stay accurate at long context, but quantized EuroLLM 9B/22B models collapse (up to ~60% chrF++ drop) while W4A8/W8A8 plus 200–400-token chunking improves serving throughput.

  4. MxGLUT: A Reconfigurable LUT-Centric Broadcast Dataflow Accelerator for Mixed-Precision GEMM

    cs.AR 2026-07 unverdicted novelty 6.0 of 10

    MxGLUT introduces a reconfigurable LUT-centric broadcast dataflow accelerator with mixed-precision LUT-based PEs that unifies FP8-INT4 and FP8-FP8 GEMM without separate FP datapaths, reporting up to 2.16x prefill spee...

  5. APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

    cs.DC 2026-06 unverdicted novelty 6.0 of 10

    APEX4 co-designs pure INT4 GEMM kernels with ρ-aware granularity adaptation to deliver up to 2.09× end-to-end speedup on GPUs with low ρ while keeping perplexity within 0.63 of FP16 on LLaMA-2-70B.

  6. Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    SplitQ improves low-bit PTQ for VLMs by isolating modality-specific outlier channels via MOCD and applying dual-branch adaptive calibration via ACC, outperforming prior methods on six datasets across W4A8 to W3A2 settings.

  7. LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving

    cs.DC 2025-09 conditional novelty 6.0 of 10

    A W4A8 GEMM kernel using shift-based overflow-safe dequantization and a hardware-scheduled pipeline reports up to 2.9x speedup over prior W4A8 kernels.

  8. Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving

    cs.DC 2026-06 unverdicted novelty 5.0 of 10

    Organizes the heterogeneous LLM prefill-decode design space along four axes and extracts three boundary decisions with guidance on precision, KV representation, and ownership.

  9. Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving

    cs.DC 2026-06 unverdicted novelty 5.0 of 10

    The paper organizes heterogeneous prefill-decode LLM serving into a four-axis design space and identifies three recurring boundary decisions that require joint choices.

  10. Baichuan-M2: Scaling Medical Capability with Large Verifier System

    cs.LG 2025-09 conditional novelty 5.0 of 10

    Baichuan-M2, a 32B medical LLM trained with a patient simulator and a clinical rubric generator as RL verifiers, reports state-of-the-art HealthBench scores (60.1 overall, 34.7 hard), ahead of all open-source models.

Pith tools