Pith. sign in

REVIEW 10 cited by

Evaluating Quantized Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.18158 v2 pith:FXUUOTXP submitted 2024-02-28 cs.CL cs.AI

Shiyao Li , Xuefei Ning , Luning Wang , Tengxuan Liu , Xiangsheng Shi , Shengen Yan , Guohao Dai , Huazhong Yang
show 1 more author
Yu Wang
This is my paper
classification cs.CLcs.AI
keywords quantizationevaluationllmseffectevaluatinglanguagelargemethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Post-training quantization (PTQ) has emerged as a promising technique to reduce the cost of large language models (LLMs). Specifically, PTQ can effectively mitigate memory consumption and reduce computational overhead in LLMs. To meet the requirements of both high efficiency and performance across diverse scenarios, a comprehensive evaluation of quantized LLMs is essential to guide the selection of quantization methods. This paper presents a thorough evaluation of these factors by evaluating the effect of PTQ on Weight, Activation, and KV Cache on 11 model families, including OPT, LLaMA2, Falcon, Bloomz, Mistral, ChatGLM, Vicuna, LongChat, StableLM, Gemma, and Mamba, with parameters ranging from 125M to 180B. The evaluation encompasses five types of tasks: basic NLP, emergent ability, trustworthiness, dialogue, and long-context tasks. Moreover, we also evaluate the state-of-the-art (SOTA) quantization methods to demonstrate their applicability. Based on the extensive experiments, we systematically summarize the effect of quantization, provide recommendations to apply quantization techniques, and point out future directions. The code can be found in https://github.com/thu-nics/qllm-eval.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Studying quantization trade-offs for efficient inference deployment in machine translation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Quantized Hy-MT2 models stay accurate at long context, but quantized EuroLLM 9B/22B models collapse (up to ~60% chrF++ drop) while W4A8/W8A8 plus 200–400-token chunking improves serving throughput.

  2. Where Facts Go Missing: A Layerwise Taxonomy and Per-Layer Attribution of Information Omission in Air-Gapped LLMAgent Pipelines

    cs.MA 2026-07 conditional novelty 6.0 of 10

    In a controlled 75,476-trial stress test, about 73% of omitted-fact failures in LLM agent pipelines are traced to deterministic middleware (redaction, pagination, truncation) rather than model behavior.

  3. HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Tasks

    cs.LG 2026-07 conditional novelty 6.0 of 10

    An integrated black-box audit protocol profiles parametric hindsight in LLMs and shows the date-trigger reflex tracks training generation, not scale, while effective knowledge cutoffs span 22 months.

  4. Reliability Scaling Laws for Quantized Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.

  5. Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts

    cs.CL 2026-07 reject novelty 6.0 of 10

    Post-training NF4 quantization preserves LLM accuracy but size- and benchmark-dependently shifts chain-of-thought failure modes, including Hollow Convergence, in ways surface detectors cannot catch.

  6. Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models

    cs.SD 2025-07 conditional novelty 6.0 of 10

    A benchmark of eight post-training quantization methods on Whisper and Moonshine edge speech models across seven datasets, finding 8-bit is safe and 3-bit weights are viable for larger models with advanced methods like SpQR.

  7. Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMs

    cs.LG 2025-06 conditional novelty 6.0 of 10

    UPQ, a progressive FP16-to-INT4-to-INT2 pipeline with teacher-student distillation, is the first to quantize open-source instruction-tuned LLMs to 2-bit without proprietary post-training data.

  8. Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Quantizing LLMs to 4 or 8 bits cuts GPU memory by up to 75% with generally small performance changes across eight biomedical NLP benchmarks.

  9. Pruning General Large Language Models into Customized Expert Models

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Cus-Prun identifies and removes neurons that are irrelevant to a user's target language, domain, and task, producing specialized expert models without post-training.

  10. Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

    cs.CL 2025-05 reject novelty 4.0 of 10

    The paper claims that most massive activations in LLMs are residual-propagated 'fake' outliers that can be removed without accuracy loss.

Pith tools