Pith. sign in

REVIEW 49 cited by

LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.17888 v1 pith:IGKD77JM submitted 2023-05-29 cs.CL

Zechun Liu , Barlas Oguz , Changsheng Zhao , Ernie Chang , Pierre Stock , Yashar Mehdad , Yangyang Shi , Raghuraman Krishnamoorthi
show 1 more author
Vikas Chandra
This is my paper · ORCID
classification cs.CL
keywords quantizationmethodsdownlargemodelmodelstrainingaware
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and support long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 49 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HadamRNN: Binary and Sparse Ternary Orthogonal RNNs

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Sign-flipped Sylvester Hadamard matrices make orthogonal RNN recurrent weights binary or sparse ternary while preserving long-memory performance.

  2. Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format

    cs.AR 2024-11 conditional novelty 7.0 of 10

    Anda stores LLM activations in a group-shared-exponent format with per-module mantissa widths and uses bit-serial hardware that skips unused bits, reporting roughly 2.4x speedup, 4.0x area efficiency, and 3.1x energy ...

  3. BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration

    cs.LG 2024-11 conditional novelty 7.0 of 10

    BitMoD quantizes LLM weights to 3-4 bits using per-group custom floating-point formats with a repurposed zero value, and accelerates them with a bit-serial PE and dequantization unit.

  4. Hidden Language Consistency Phenomena in Reasoning LLMs

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Reasoning models often stop using the requested language as problems get harder, and this language breakdown can make accuracy look better than it is.

  5. Reliability Scaling Laws for Quantized Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.

  6. High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

    cs.LG 2026-06 conditional novelty 6.0 of 10

    A learned top-K distribution-restoration layer recovers near-FP16 long-context accuracy under 1-bit KV-cache quantization.

  7. MoPEQ: Mixture of Mixed Precision Quantized Experts

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Assigning 2, 3, or 4 bits to MoE experts by Hessian trace sensitivity keeps VLM accuracy close to uniform 4-bit while reducing model size.

  8. LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving

    cs.DC 2025-09 conditional novelty 6.0 of 10

    A W4A8 GEMM kernel using shift-based overflow-safe dequantization and a hardware-scheduled pipeline reports up to 2.9x speedup over prior W4A8 kernels.

  9. SiLQ: Simple Large Language Model Quantization-Aware Training

    cs.LG 2025-07 conditional novelty 6.0 of 10

    SiLQ fine-tunes 8B-parameter LLMs with quantized weights, activations, and cache for a small fraction of extra training tokens, matching or beating leading post-training quantization methods.

  10. GeLaCo: An Evolutionary Approach to Layer Compression

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Evolutionary search over layer-merging configurations, scored by module-wise activation similarity, yields competitive LLM compression and the first size-quality Pareto fronts.

  11. Is Quantization a Deal-breaker? Empirical Insights from Large Code Models

    cs.SE 2025-07 conditional novelty 6.0 of 10

    4-bit AWQ quantization of CodeLlama and DeepSeek-Coder largely preserves static-analysis measures of code quality on Java and Python tasks, with small readability and maintainability shifts.

  12. CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CCQ compresses LLMs to 2.0-2.75 bits per weight using convolutional codes and bit-shift decoding, shrinking 671B-parameter models to under 200GB.

  13. ROSAQ: Rotation-based Saliency-Aware Weight Quantization for Efficiently Compressing Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ROSAQ uses PCA projection to find salient transformer weight channels, keeps those in FP16, quantizes the rest to INT3/4, and reports modest gains over previous quantization methods.

  14. Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    UPQ, a progressive FP16-to-INT4-to-INT2 pipeline with teacher-student distillation, is the first to quantize open-source instruction-tuned LLMs to 2-bit without proprietary post-training data.

  15. Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.

  16. RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RoSTE couples quantization-aware supervised fine-tuning with per-layer Hadamard rotation selection, reducing quantization outliers and improving 4-bit quantized LLM accuracy over SFT-then-PTQ baselines.

  17. SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Sharing one MLP layer's weights across several layers plus low-rank adapters recovers most of a pretrained LLM's quality with a fraction of the storage and faster phone inference.

  18. FBQuant: FeedBack Quantization for Large Language Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    FBQuant redefines sub-branch compensation as Q(W - Sigma) + Sigma, bounding per-weight reconstruction error by half the quantizer step and improving 3-bit LLM accuracy.

  19. Irrational Complex Rotations Empower Low-bit Optimizers

    cs.LG 2025-01 reject novelty 6.0 of 10

    π-Quant's core representation theorem fails: the curve e^{iθ}+e^{iπθ} is dense in the disk but does not cover it, and Lemma 3.2's angle formulas are internally inconsistent.

  20. 1.58-bit FLUX

    cs.CV 2024-12 reject novelty 6.0 of 10

    A post-training method reduces 99.5% of FLUX.1-dev's transformer weights to ternary values and reports roughly comparable text-to-image quality with large storage and memory savings.

  21. Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Noise Perturbation Fine-tuning (NPFT) reduces the quantization sensitivity of outlier weights via LoRA fine-tuning with random perturbations, letting standard RTN quantization match GPTQ on LLaMA2-7B-4bit without mixe...

  22. MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache

    cs.CL 2024-11 conditional novelty 6.0 of 10

    MiniKV combines 2-bit KV quantization with heavy-hitter and recent-window token selection plus pyramid layer budgets, achieving 86% KV cache compression with about 98.5% of full-model LongBench accuracy.

  23. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  24. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A 2-bit base plus three 2-bit residual stages gives one checkpoint that runs at 2, 4, 6, or 8 bits, matching a prior multi-precision baseline at 6-8 bits in most tested models.

  25. Quantize with Confidence? An Empirical Study of Quantization for Code Generation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.

  26. LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

    cs.CV 2025-09 conditional novelty 5.0 of 10

    LUQ assigns ultra-low bit widths to transformer layers whose output activations have low entropy, cutting multimodal LLM memory by up to 40 percent with modest benchmark loss.

  27. Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs

    cs.AI 2025-08 conditional novelty 5.0 of 10

    The study introduces TruthfulnessEval and reports that 4-bit quantization preserves simple true/false accuracy, but explicit 'lie' prompts make quantized and full-precision LLMs output falsehoods even when internal pr...

  28. P-CoT: A Pedagogically-motivated Participatory Chain-of-Thought Prompting for Phonological Reasoning in LLMs

    cs.CL 2025-07 reject novelty 5.0 of 10

    P-CoT prompting improves many LLM results on PhonologyBench tasks, but it does not consistently beat baselines across all models and tasks as the paper claims.

  29. LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillation

    cs.LG 2025-05 reject novelty 5.0 of 10

    LCD clusters LLM weights into tiny codebooks under a Hessian-guided objective and uses lookup-table inference to reach 2-3 bits, with reported speedups up to 6.2x.

  30. FedHQ: Hybrid Runtime Quantization for Federated Learning

    cs.LG 2025-05 reject novelty 5.0 of 10

    A federated learning method assigns each client either PTQ or QAT using hardware and data-distribution scores, reporting speedups and accuracy gains on three small image datasets.

  31. Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

    cs.LG 2025-01 conditional novelty 5.0 of 10

    A post-training quantization scheme that uses per-group leading-one detection to keep four salient bits from an 8/16-bit integer base, achieving 4-bit weights, activations, and KV cache without fine-tuning or rotation.

  32. Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference

    cs.LG 2024-12 conditional novelty 5.0 of 10

    A greedy, ablation-based pruning method extracts a standalone task-specific subnetwork from GPT-2 Small, reducing parameters by up to 82.77% while keeping accuracy on three synthetic single-token tasks.

  33. Toward LLM-Agent-Based Modeling of Transportation Systems: A Conceptual Framework

    cs.AI 2024-12 conditional novelty 5.0 of 10

    LLM-driven agents with profiles, memory, and feedback loops can generate plausible daily travel activities and learn to adjust commute timing in a small proof-of-concept, pointing toward a new direction for agent-base...

  34. SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization

    cs.LG 2024-12 conditional novelty 5.0 of 10

    SKIM combines channel-wise mixed-precision bit allocation with a trainable column-scaling vector for K-means-based post-training quantization, improving perplexity at 3-bit and supporting arbitrary average bit widths.

  35. Llama Guard 3-1B-INT4: Compact and Efficient Safeguard for Human-AI Conversations

    cs.DC 2024-11 conditional novelty 5.0 of 10

    A 1B Llama Guard safety model, compressed to 4-bit weights and a 20-token output vocabulary, runs on a phone at 30+ tokens per second with English F1 slightly better than the full-precision 1B model.

  36. Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A decision framework mapping five production constraints to established model-optimization techniques, synthesized from 25+ cited papers, with four worked industry pipelines.

  37. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

  38. RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Models

    cs.RO 2025-06 conditional novelty 4.0 of 10

    RLRC combines 90% structured pruning, supervised fine-tuning, PPO reinforcement learning, and optional 4-bit quantization to recover OpenVLA's success rate on ManiSkill while cutting memory up to 8x and boosting throu...

  39. Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks

    cs.LG 2025-05 conditional novelty 4.0 of 10

    The paper makes the case that tensorized neural networks offer valuable compression, scaling, and interpretability advantages that the deep learning community has not yet fully exploited.

  40. NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

    cs.LG 2025-05 conditional novelty 4.0 of 10

    NQKV is a block-wise 4-bit KV cache quantizer that uses a normal-distribution lookup table to let OPT models run with larger batches or longer contexts at negligible measured accuracy cost.

  41. Quantitative Analysis of Performance Drop in DeepSeek Model Quantization

    cs.LG 2025-05 conditional novelty 4.0 of 10

    A hand-tuned 3-bit quantization scheme, DQ3_K_M, matches 4-bit accuracy on DeepSeek-R1 and V3 while shrinking the models enough for single-machine deployment.

  42. SWSC: Shared Weight for Similar Channel in LLM

    cs.LG 2025-01 conditional novelty 4.0 of 10

    SWSC combines channel K-means clustering with an SVD low-rank error correction to compress LLM weights, and reports lower perplexity than RTN quantization on Llama-2-7B Q and K projections at 2 to 3 average bits.

  43. Deploying Foundation Model Powered Agent Services: A Survey

    cs.DC 2024-12 accept novelty 4.0 of 10

    This survey proposes a layered framework (execution, resource, model, agent, application) for deploying foundation-model-powered agent services across edge-cloud environments, and reviews optimization techniques at ea...

  44. QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models

    cs.LG 2024-12 reject novelty 4.0 of 10

    A pruning-plus-quantization framework that uses Bayesian optimization to choose per-layer bit widths reports roughly 30 percent memory savings on 7B-13B LLMs at approximately unchanged zero-shot accuracy.

  45. FASTNav: Fine-tuned Adaptive Small-language-models Trained for Multi-point Robot Navigation

    cs.RO 2024-11 conditional novelty 4.0 of 10

    Fine-tuned small language models, coached by a GPT-4 teacher through iterative prompting, can perform multi-point robot navigation on edge devices with success rates approaching larger models.

  46. Large Language models for Time Series Analysis: Techniques, Applications, and Challenges

    cs.LG 2025-05 reject novelty 3.0 of 10

    A review of LLM-based time series analysis that proposes several taxonomies, but is undermined by citation errors and a lack of systematic methodology.

  47. The Rise of Small Language Models in Healthcare: A Comprehensive Survey

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A comprehensive survey of small language models in healthcare, with a taxonomy of building, adapting, and compressing them for clinical NLP tasks.

  48. Rethinking Post-Training Quantization: Introducing a Statistical Pre-Calibration Approach

    cs.LG 2025-01 reject novelty 3.0 of 10

    A soft-thresholding rule equivalent to selecting the top alpha percent of weights by magnitude is presented as a statistical pre-calibration for LLM quantization, with results close to but not clearly better than cali...

  49. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools