Pith. sign in

REVIEW 9 cited by

LLM-FP4: 4-Bit Floating-Point Quantized Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.16836 v1 pith:LCMYUK6X submitted 2023-10-25 cs.CL cs.AIcs.ARcs.CV

LLM-FP4: 4-Bit Floating-Point Quantized Transformers

Shih-yang Liu , Zechun Liu , Xijie Huang , Pingcheng Dong , Kwang-Ting Cheng This is my paper
classification cs.CL cs.AIcs.ARcs.CV
keywords quantizationactivationfloating-pointllm-fp4modelsweightsactivationsbits
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We propose LLM-FP4 for quantizing both weights and activations in large language models (LLMs) down to 4-bit floating-point values, in a post-training manner. Existing post-training quantization (PTQ) solutions are primarily integer-based and struggle with bit widths below 8 bits. Compared to integer quantization, floating-point (FP) quantization is more flexible and can better handle long-tail or bell-shaped distributions, and it has emerged as a default choice in many hardware platforms. One characteristic of FP quantization is that its performance largely depends on the choice of exponent bits and clipping range. In this regard, we construct a strong FP-PTQ baseline by searching for the optimal quantization parameters. Furthermore, we observe a high inter-channel variance and low intra-channel variance pattern in activation distributions, which adds activation quantization difficulty. We recognize this pattern to be consistent across a spectrum of transformer models designed for diverse tasks, such as LLMs, BERT, and Vision Transformer models. To tackle this, we propose per-channel activation quantization and show that these additional scaling factors can be reparameterized as exponential biases of weights, incurring a negligible cost. Our method, for the first time, can quantize both weights and activations in the LLaMA-13B to only 4-bit and achieves an average score of 63.1 on the common sense zero-shot reasoning tasks, which is only 5.8 lower than the full-precision model, significantly outperforming the previous state-of-the-art by 12.7 points. Code is available at: https://github.com/nbasyl/LLM-FP4.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FlipGuard: Defending Large Language Models Against Quantization-Conditioned Backdoor Attacks

    cs.CR 2026-06 unverdicted novelty 7.0

    FlipGuard perturbs LLM weights prior to quantization to neutralize quantization-conditioned backdoor attacks, evaluated via the Defense Effectiveness Ratio on multiple models and quantization schemes.

  2. Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling

    cs.CL 2025-12 conditional novelty 7.0

    Four Over Six adaptively scales blocks in NVFP4 quantization to smaller FP4 values, making representable value distributions more uniform and reducing quantization error especially for near-maximal values.

  3. Breaking the Rounding Trap: Securing LLMs against Quantization-Conditioned Backdoors

    cs.CR 2026-06 conditional novelty 6.0

    QuantGuard uses differentiable rounding control to break quantization-boundary backdoors in LLMs, reducing post-quantization attack success to clean-model levels across six models and INT8/FP4/NF4.

  4. Breaking the Rounding Trap: Securing LLMs against Quantization-Conditioned Backdoors

    cs.CR 2026-06 unverdicted novelty 6.0

    QuantGuard is a pre-quantization method using differentiable rounding controls, error-guided reversal constraints, output consistency, and weight regularization on a small calibration set to suppress quantization-cond...

  5. LiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection

    cs.LG 2026-06 unverdicted novelty 6.0

    LiftQuant uses dimensional lifting of weights to a higher-dimensional 1-bit lattice followed by projection to achieve tunable continuous bit-widths in LLM quantization while remaining hardware-friendly.

  6. LiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection

    cs.LG 2026-06 unverdicted novelty 6.0

    LiftQuant enables continuous bit-width LLM quantization via dimensional lifting and projection from a 1-bit lattice, allowing 2.4-bit compression of 70B models that outperforms fixed 2-bit baselines on identical hardware.

  7. LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation

    cs.LG 2026-04 unverdicted novelty 6.0

    LBLLM achieves better accuracy than prior binarization methods for LLMs by decoupling weight and activation quantization through initialization, layer-wise distillation, and learnable activation scaling.

  8. Balancing FP8 Computation Accuracy and Efficiency on Digital CIM via Shift-Aware On-the-fly Aligned-Mantissa Bitwidth Prediction

    cs.AR 2026-02 unverdicted novelty 5.0

    A 28nm digital CIM accelerator for FP8 uses on-the-fly shift-aware bitwidth prediction, FIFO alignment, and scalable MACs to reach 20.4 TFLOPS/W and 2.8x better efficiency than prior work while supporting variable man...

  9. StatQAT: Statistical Quantizer Optimization for Deep Networks

    stat.ML 2026-05 unverdicted novelty 4.0

    A statistical error analysis framework yields iterative and analytic quantizers that improve accuracy and stability when incorporated into quantization-aware training for integer and floating-point formats.