Pith. sign in

REVIEW 46 cited by

SqueezeLLM: Dense-and-Sparse Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.07629 v4 pith:QEL4HKLM submitted 2023-06-13 cs.CL cs.LG

Sehoon Kim , Coleman Hooper , Amir Gholami , Zhen Dong , Xiuyu Li , Sheng Shen , Michael W. Mahoney , Kurt Keutzer This is my paper · ORCID
classification cs.CLcs.LG
keywords quantizationmodelsinferencememorysqueezellmbaselinecompareddense-and-sparse
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Generative Large Language Models (LLMs) have demonstrated remarkable results for a wide range of tasks. However, deploying these models for inference has been a significant challenge due to their unprecedented resource requirements. This has forced existing deployment frameworks to use multi-GPU inference pipelines, which are often complex and costly, or to use smaller and less performant models. In this work, we demonstrate that the main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, specifically for single batch inference. While quantization has emerged as a promising solution by representing weights with reduced precision, previous efforts have often resulted in notable performance degradation. To address this, we introduce SqueezeLLM, a post-training quantization framework that not only enables lossless compression to ultra-low precisions of up to 3-bit, but also achieves higher quantization performance under the same memory constraint. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format. When applied to the LLaMA models, our 3-bit quantization significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x as compared to the state-of-the-art methods with the same memory requirement. Furthermore, when deployed on an A6000 GPU, our quantized models achieve up to 2.3x speedup compared to the baseline. Our code is available at https://github.com/SqueezeAILab/SqueezeLLM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 46 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

    cs.CL 2026-08 conditional novelty 7.0 of 10

    AdaMTP uses entropy-based segmentation to adaptively mask multi-token prediction losses, improving quality and speed over fixed-horizon multi-token prediction.

  2. PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

    cs.LG 2025-11 conditional novelty 7.0 of 10

    A training-free method that merges pairs of MoE experts at the individual-weight level and packs the required masks into unused exponent bits, cutting expert memory by 50% with minimal accuracy loss.

  3. OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

    cs.LG 2025-07 conditional novelty 7.0 of 10

    OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.

  4. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  5. ICQuant: Index Coding enables Low-bit LLM Quantization

    cs.LG 2025-05 conditional novelty 7.0 of 10

    ICQuant stores outlier positions in LLM weight matrices with about 0.3 bits per weight by exploiting their near-uniform spacing, so 2-bit scalar quantization matches fine-tuned vector quantization without training.

  6. Pushing the Limits of Large Language Model Quantization via the Linearity Theorem

    cs.LG 2024-11 conditional novelty 7.0 of 10

    A new theorem and method (HIGGS) make per-layer quantization error a reliable predictor of final model perplexity, enabling state-of-the-art data-free and dynamic bit-width LLM compression.

  7. CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A parametric cubic codebook with two shape parameters per group reduces quantization reconstruction error versus uniform integer and finite floating-point baselines for 1-8-bit LLM weights, and can be executed directl...

  8. NANQ: Noise-Floor-Aware Mixed-Precision Non-Uniform Quantization for Analog Compute-in-Memory

    cs.LG 2026-08 conditional novelty 6.0 of 10

    NANQ constructs non-uniform quantization levels from the measured magnitude-dependent noise of an eFlash CIM array and uses precision-saturation thresholds to assign layer bit-widths, improving low-bit on-chip inferen...

  9. MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.

  10. StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Near-memory DeQuantization Blocks in the HBM base die dequantize weight-only quantized LLM weights on the load path, cutting GPU dequant overhead and yielding up to 7.08× mpGEMM speedup and 2.20× decode throughput in ...

  11. CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CCQ compresses LLMs to 2.0-2.75 bits per weight using convolutional codes and bit-shift decoding, shrinking 671B-parameter models to under 200GB.

  12. Information-Bottleneck Driven Binary Neural Network for Change Detection

    cs.CV 2025-07 conditional novelty 6.0 of 10

    BiCD is a 1-bit change detection network whose auxiliary IB-style losses improve F1 by about 1 to 3 points over other binary networks, with no extra inference cost.

  13. Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.

  14. Efficient Serving of LLM Applications with Probabilistic Demand Modeling

    cs.DC 2025-06 conditional novelty 6.0 of 10

    Hermes models LLM applications as probabilistic demand graphs and uses the Gittins scheduling policy plus backend prewarming to cut average completion time by over 70% in testbed runs.

  15. NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

    cs.LG 2025-05 conditional novelty 6.0 of 10

    NSNQuant applies a Normalize-Shift-Normalize transform plus a Hadamard rotation to make KV cache channels match a standard normal distribution, so one codebook trained on random noise can quantize them without calibration.

  16. Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.

  17. Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression

    cs.AR 2025-05 conditional novelty 6.0 of 10

    Ecco compresses LLM weights and KV cache with shared k-means patterns and Huffman coding, and adds a parallel hardware decoder, to roughly quadruple memory capacity and speed up decode by up to ~3x.

  18. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MxMoE assigns mixed bitwidths at the linear-block level in MoE models using an ILP that balances measured quantization sensitivity and expert activation frequency, then auto-generates fused Group-GEMM kernels, improvi...

  19. MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MoEQuant combines expert-balanced self-sampled calibration data with affinity-weighted quantization error to improve low-bit quantization of MoE LLMs.

  20. R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference

    cs.LG 2025-04 conditional novelty 6.0 of 10

    A training-free inference scheme combines input-activation sparsity with weight low-rank decomposition to reach 50% model-level sparsity on modern LLMs while keeping accuracy close to the dense model.

  21. TeleSparse: Practical Privacy-Preserving Verification of Deep Neural Networks

    cs.LG 2025-04 conditional novelty 6.0 of 10

    Pruning weights and teleporting activations before proof generation cuts ZK-SNARK prover memory by up to 67% and proof time by up to 54% on vision models at about 1% accuracy cost.

  22. FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference

    cs.AR 2025-04 conditional novelty 6.0 of 10

    A Fisher-weighted, block-level mixed-precision quantization method plus a custom mixed-precision datapath runs 70% of Llama-2-7B weight and activation blocks in FP4 with under 1% perplexity loss versus all-FP8.

  23. SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Sharing one MLP layer's weights across several layers plus low-rank adapters recovers most of a pretrained LLM's quality with a fraction of the storage and faster phone inference.

  24. Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Hierarchy Drafting accelerates LLM inference without retraining by retrieving draft tokens from three databases in order of temporal locality, beating prior database drafting methods on Spec-Bench.

  25. QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache

    cs.LG 2025-02 conditional novelty 6.0 of 10

    QuantSpec accelerates long-context LLM decoding by giving the self-draft model a 4-bit hierarchical KV cache shared with an INT8-quality target cache, achieving up to ~2.5x speedup and mostly >90% acceptance.

  26. Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Dobi-SVD compresses LLMs via differentiable SVD rank selection, IPCA-based weight reconstruction, and quantized storage remapping, reporting competitive perplexity at 40% parameters.

  27. Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A unified analysis of subspace perturbations in zero-order optimization identifies subspace alignment as the key driver of convergence, and leads to MeZO-BCD, a block-coordinate method with up to 2.77x wall-clock spee...

  28. FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices

    cs.AI 2025-01 conditional novelty 6.0 of 10

    FlexQuant generates a family of shared-parameter quantized LLMs by gradually replacing modules with lower-bit versions, cutting storage and improving memory granularity.

  29. BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference

    cs.CL 2025-01 conditional novelty 6.0 of 10

    BlockDialect assigns one of 16 FP4 'dialect' formats to each 32-64 element block of weights and activations, achieving near-full-precision accuracy with roughly 4-bit storage and integer-friendly MACs.

  30. 1.58-bit FLUX

    cs.CV 2024-12 reject novelty 6.0 of 10

    A post-training method reduces 99.5% of FLUX.1-dev's transformer weights to ternary values and reports roughly comparable text-to-image quality with large storage and memory savings.

  31. ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals

    cs.LG 2024-12 conditional novelty 6.0 of 10

    ResQ quantizes LLMs to 4-bit with a low-rank 8-bit residual subspace chosen by PCA, beating prior 4-bit methods on perplexity and accuracy.

  32. Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Noise Perturbation Fine-tuning (NPFT) reduces the quantization sensitivity of outlier weights via LoRA fine-tuning with random perturbations, letting standard RTN quantization match GPTQ on LLaMA2-7B-4bit without mixe...

  33. Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A 2-bit base plus three 2-bit residual stages gives one checkpoint that runs at 2, 4, 6, or 8 bits, matching a prior multi-precision baseline at 6-8 bits in most tested models.

  34. Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation

    cs.CL 2026-01 conditional novelty 5.0 of 10

    LLMs tend to assign high probability to target keywords in the first two generation steps; a plugin using those keywords with parallel constrained generation gives comparable short-text quality and large speedups on s...

  35. Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs

    cs.AI 2025-08 conditional novelty 5.0 of 10

    The study introduces TruthfulnessEval and reports that 4-bit quantization preserves simple true/false accuracy, but explicit 'lie' prompts make quantized and full-precision LLMs output falsehoods even when internal pr...

  36. FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs

    cs.LG 2025-04 conditional novelty 5.0 of 10

    FineQ quantizes LLM weights to 2.33 bits by protecting outliers inside 3-weight clusters and adds a temporal-coding accelerator that replaces multipliers with adders.

  37. Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

    cs.LG 2025-01 conditional novelty 5.0 of 10

    A post-training quantization scheme that uses per-group leading-one detection to keep four salient bits from an 8/16-bit integer base, achieving 4-bit weights, activations, and KV cache without fine-tuning or rotation.

  38. PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networks

    cs.DC 2025-01 conditional novelty 5.0 of 10

    PICE uses progressive inference, cloud sketches plus parallel edge expansion, to raise LLM serving throughput by 1.5-2x and cut latency by up to 43 percent in testbed experiments.

  39. SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization

    cs.LG 2024-12 conditional novelty 5.0 of 10

    SKIM combines channel-wise mixed-precision bit allocation with a trainable column-scaling vector for K-means-based post-training quantization, improving perplexity at 3-bit and supporting arbitrary average bit widths.

  40. Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry

    cs.DS 2026-08 reject novelty 4.0 of 10

    A proposed 'Riemann GeoResolver' framework for inverse-distance attention in hyperbolic and spherical geometry, whose central PL inequality rests on an invalid derivative computation.

  41. BAQ: Efficient Bit Allocation Quantization for Large Language Models

    cs.LG 2025-06 conditional novelty 4.0 of 10

    BAQ assigns per-column bitwidths to LLM weights by solving a convex quantization-loss problem, and reports large 2-bit perplexity gains over GPTQ.

  42. Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression

    cs.LG 2025-05 conditional novelty 4.0 of 10

    ACBench tests compressed LLMs on agentic tasks and finds 4-bit quantization keeps tool use and workflow generation strong while hurting real-world application performance.

  43. NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

    cs.LG 2025-05 conditional novelty 4.0 of 10

    NQKV is a block-wise 4-bit KV cache quantizer that uses a normal-distribution lookup table to let OPT models run with larger batches or longer contexts at negligible measured accuracy cost.

  44. On multi-token prediction for efficient LLM inference

    cs.CL 2025-02 conditional novelty 4.0 of 10

    NTP-trained LLMs can predict a second token by marginalizing over first-token probabilities, but adapting them with extra heads only partially closes the gap to that baseline.

  45. Edge Intelligence in Civil Aviation: Paradigms, Techniques, and Applications

    cs.AI 2026-07 conditional novelty 3.0 of 10

    A survey of edge-AI techniques and four cloud–edge–device deployment paradigms, mapped to civil aviation applications and future trends.

  46. CPTQuant - A Novel Mixed Precision Post-Training Quantization Techniques for Large Language Models

    cs.CL 2024-12 reject novelty 3.0 of 10

    CPTQuant proposes three sensitivity-based mixed precision quantization methods and claims up to 4x compression on BERT and OPT models, but the comparisons and method details are incomplete.

Pith tools