Pith. sign in

REVIEW 16 cited by

ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.08302 v3 pith:THRPCIKS submitted 2023-03-15 cs.LG cs.AIcs.CL

Zhewei Yao , Xiaoxia Wu , Cheng Li , Stephen Youn , Yuxiong He This is my paper
classification cs.LGcs.AIcs.CL
keywords quantizationmodelmethodsmodelsactivationanalysiscompensationcomprehensive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Post-training quantization (PTQ) has emerged as a promising technique for mitigating memory consumption and computational costs in large language models (LLMs). However, a systematic examination of various quantization schemes, model families, and quantization bit precision has been absent from the literature. In this paper, we conduct a comprehensive analysis of these factors by investigating the effects of PTQ on weight-only, activation-only, and weight-and-activation quantization using diverse methods such as round-to-nearest (RTN), GPTQ, ZeroQuant, and their variants. We apply these methods to two distinct model families with parameters ranging from 125M to 176B. Our contributions include: (1) a sensitivity analysis revealing that activation quantization is generally more susceptible to weight quantization, with smaller models often outperforming larger models in terms of activation quantization; (2) an evaluation and comparison of existing PTQ methods to optimize model size reduction while minimizing the impact on accuracy, revealing that none of the current methods can achieve the original model quality for quantization with either INT4-weight or INT4-weight-and-INT8-activation; (3) based on these insights, we propose an optimized method called Low-Rank Compensation (LoRC), which employs low-rank matrices to enhance model quality recovery with a minimal increase in model size.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation

    cs.LG 2026-05 unverdicted novelty 8.0 of 10

    GPTQ-intrinsic LoRA augments GPTQ with intrinsic low-rank compensation via Hessian modification to achieve layer-wise reconstruction bounds that match information-theoretic lower bounds under structural assumptions.

  2. TwinQuant: Learnable Subspace Decomposition for 4-Bit LLM Quantization

    cs.DC 2026-06 unverdicted novelty 7.0 of 10

    TwinQuant learns quantization-friendly subspaces for 4-bit LLM weights via manifold optimization and a fused kernel, preserving near-FP16 accuracy with up to 1.8x speedup on LLaMA3 and Qwen3 models.

  3. GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Rotation and fine-grained group quantization can work together if rotation spans several quantization groups and outlier channels are permuted onto harmonic Hadamard rows, enabling 4-bit LLM inference with integer-onl...

  4. ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    ProjQ constrains post-training quantization noise to a low-rank manifold through orthogonal subspace projection, enabling better compensation by LoRA adapters and preserving greater model plasticity than standard PTQ.

  5. Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    SplitQ improves low-bit PTQ for VLMs by isolating modality-specific outlier channels via MOCD and applying dual-branch adaptive calibration via ACC, outperforming prior methods on six datasets across W4A8 to W3A2 settings.

  6. DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

    cs.LG 2026-05 conditional novelty 6.0 of 10

    DECO matches dense model performance at 20% expert activation via ReLU-based routing with learnable scaling and the NormSiLU activation, plus a 3x real-hardware speedup.

  7. DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    DECO sparse MoE matches dense Transformer performance at 20% expert activation with a 3x hardware inference speedup.

  8. DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    DECO is a sparse MoE architecture with ReLU-based routing, learnable expert scaling, and NormSiLU activation that matches dense Transformer performance at 20% expert activation and delivers 2.93x speedup on Jetson AGX Orin.

  9. MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    MCAP uses load-time Monte Carlo profiling to estimate layer importance, enabling dynamic quantization (W4A8 vs W4A16) and memory tiering (GPU/RAM/SSD) that delivers 1.5-1.8x higher decode throughput than llama-cpp Q4_...

  10. You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

    cs.CL 2025-11 conditional novelty 6.0 of 10

    TAQ estimates per-layer importance from hidden representations and output sensitivity on task calibration data to allocate mixed precision in a training-free PTQ setting, outperforming task-agnostic baselines on accur...

  11. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

    cs.LG 2023-05 accept novelty 6.0 of 10

    FrugalGPT learns query-specific cascades across heterogeneous LLM APIs to match or exceed top-model accuracy at far lower cost.

  12. You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations

    cs.CL 2025-11 reject novelty 5.0 of 10

    A per-layer, per-task bit-allocation rule based on hidden-activation entropy and variance reportedly matches full-precision TriviaQA accuracy within about one point — but the paper's tables and abstract are internally...

  13. Enhancing Model Privacy in Federated Learning with Random Masking and Quantization

    cs.LG 2025-08 reject novelty 5.0 of 10

    FedQSN hides part of the server model with random masks and quantizes the remainder to give clients a degraded proxy, reporting a large global-vs-proxy performance gap with modest loss in the final global model.

  14. Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A decision framework mapping five production constraints to established model-optimization techniques, synthesized from 25+ cited papers, with four worked industry pipelines.

  15. Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks

    cs.CL 2025-07 reject novelty 3.0 of 10

    Across Arabic, English, and Kannada benchmarks, 4-bit and 8-bit quantization preserves most accuracy while aggressive pruning degrades larger multilingual models more than smaller ones.

  16. A Survey on Efficient Inference for Large Language Models

    cs.CL 2024-04 accept novelty 3.0 of 10

    The paper surveys techniques to speed up and reduce the resource needs of LLM inference, organized by data-level, model-level, and system-level changes, with comparative experiments on representative methods.

Pith tools