Pith. sign in

REVIEW 1 major objections 6 minor 2 cited by

MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization

T0 review · 1 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MQuant shows that fully static W4A8 quantization can keep multimodal LLMs within one percentage point of floating-point accuracy while cutting latency.

desk verdict A practical MLLM quantization framework with a genuinely useful RMS insight, but the near-lossless claim is contradicted by its own Table 3 and the baselines are handicapped. read the letter →

arxiv 2502.00425 v2 pith:7JFTDSNO submitted 2025-02-01 cs.CV cs.AI

JiangYong Yu , Sifan Zhou , Dawei Yang , Shuo Wang , Shuoyu Li , Xing Hu , Chen Xu , Zukang Xu
show 2 more authors
Changyong Shu Zhihang Yuan
This is my paper · ORCID
classification cs.CVcs.AI
keywords post-trainingquantizationmultimodallargelanguagemodelsstaticW4A8Hadamardrotationvision-languageinferenceaccelerationtokenreordering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that multimodal large language models are hard to quantize for three specific reasons: visual tokens dominate and outnumber textual tokens, their activation scales are very different, and Hadamard rotations used to remove outliers create new weight outliers. MQuant is a post-training quantization framework that targets each of those causes. It assigns separate static scales to visual and textual tokens, reorders the mixed token sequence before inference while preserving the causal attention structure, and splits out outlier-prone channels created by online Hadamard rotations. On five MLLMs, the paper reports that W4A8 quantization stays within one point of floating-point accuracy on OCR and VQA benchmarks, while prefill becomes about 23% faster and decode about 100% faster relative to per-token dynamic quantization. If correct, this would make fast, memory-efficient MLLM inference possible without the runtime cost of dynamic scale computation.

What carries the argument

The load-bearing object is the unified causal mask of AIFS together with the per-modality static scales of MSQ. The unified mask turns an interleaved text-image-text sequence into a visual-block-first sequence while preserving the original autoregressive reach, and because the position embeddings are shifted along with the tokens, the attention matrix is claimed to be identical to the native order. The second mechanism is the channel-mean outlier identity for online Hadamard rotations: for a Hadamard matrix whose first row is all ones, the first row of the rotated weight is $\sqrt{n}$ times the mean of each column, and when this exceeds the original maximum the layer becomes hard to quantize. RMS splits exactly those channels into a separate GEMV and zeros the offending row in the main GEMM, which is what lets the rotated W4A8 model recover near-floating-point accuracy.

What would settle it

Run the unquantized model on the same multi-image or video prompt twice, once in the native interleaved order and once after the AIFS reordering, and compare the attention outputs elementwise in full precision; any difference beyond floating-point roundoff in the first transformer block would show the invariance is approximate rather than exact.

Watch

Extended reading notes

Core claim

The central claim is that MLLM quantization fails for identifiable, fixable reasons rather than being inherently lossy. Modality-Specific Static Quantization (MSQ) calibrates one static scale for all visual tokens and a different one for all textual tokens, so wide visual activations do not force coarse quantization onto narrow textual activations. Attention-Invariant Flexible Switching (AIFS) reorders the interleaved sequence into visual-then-textual form and adjusts the causal mask and position embeddings so that every attention score is numerically unchanged, making the two modality scales usable as contiguous static-quantized blocks. Rotation Magnitude Suppression (RMS) addresses the finding that the first row of an online fast Hadamard rotation equals $\sqrt{n}$ times the column mean of the weight matrix, which can create a new outlier channel; RMS detects those channels and routes them through a separate GEMV path. With these three components, the paper reports W4A8 results within one point of the floating-point model on five MLLMs, with static per-tensor scaling instead of per-token dynamic scaling.

Load-bearing premise

The load-bearing premise is that AIFS's token reordering leaves every attention score exactly unchanged for arbitrary interleaved multimodal sequences, because the paper's latency gains come from doing that reorder once before inference and trusting the shifted causal mask and position embeddings to reproduce the original model behavior.

Editorial extensions

If this is right

  • W4A8 static quantization can replace per-token dynamic quantization for MLLM inference without the accuracy loss a single global static scale would cause.
  • Prefill latency drops by roughly 23% and decode latency by roughly 100% in the paper's measurements, making high-resolution image inputs and long generations cheaper.
  • Memory use drops by more than half relative to BF16 (reported up to 152.9% improvement at 840x840), which matters for edge deployment.
  • AIFS is compatible with FlashAttention, so the reordering can be folded into existing attention kernels with negligible added latency.
  • The scheme is claimed to be general across model families and scales, from about 7B to 72B parameters, under the same W4A8 setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if AIFS's invariance holds for arbitrary interleaved inputs, the same modality-decoupled reordering should extend to video frames and multi-image dialogues, where the visual token count grows even faster.
  • Editorial inference: the channel-mean outlier mechanism is stated for any weight matrix fed through an online Hadamard rotation, so RMS-style row splitting may transfer to non-multimodal rotated transformers whenever the equivalent of Eq. 9 is met.
  • Editorial inference: the reported sub-1% degradation is measured on OCR and VQA benchmarks; extending the evaluation to open-ended instruction-following or long-form generation would test whether the near-lossless claim survives tasks where small distribution shifts change whole responses.
  • Editorial inference: with left-padded batches the padding tokens are assigned the visual scale, so multi-batch deployments may need to confirm that padded positions do not distort visual-scale calibration when batch compositions vary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper proposes MQuant, a post-training quantization framework for multimodal large language models (MLLMs) under fully static W4A8/W8A8 settings. MQuant has three components: Modality-Specific Static Quantization (MSQ) assigns separate static activation scales to visual and textual tokens; Attention-Invariant Flexible Switching (AIFS) reorders tokens so that each modality forms a contiguous block while adjusting the causal mask and position embeddings; Rotation Magnitude Suppression (RMS) identifies and splits channels that, after a Hadamard transform, develop large first-row outliers. Experiments are reported on five MLLMs (InternVL2-8B, Qwen-VL-Chat-9.6B, MiniCPM-V-2.6-8B, GLM-4V-9B, Qwen2-VL-7B/72B) over TextVQA, DocVQA, OCRBench, and MME, with comparisons to RTN, SmoothQuant, QuaRot, and several MLLM-specific quantizers, plus latency and memory measurements. The paper claims that MQuant achieves near-floating-point accuracy with less than 1% degradation under W4A8 while speeding up prefill by 23% and decode by 100%.

Significance. If the technical claims hold, MQuant would be a practical step toward fully static low-bit inference for MLLMs, avoiding the per-token dynamic quantization overhead that is common in this domain. The paper's strengths include a clearly decomposed set of mechanisms, a theoretical characterization of Hadamard-induced weight outliers in Eq. 9 with empirical verification in Table 2, ablation studies isolating each component (Tables 7 and 10), and a code release. The per-modality static scaling idea is simple and plausible, and the reported gains over static-per-tensor baselines are substantial. However, the significance is tempered by the fact that several of the accuracy claims in the abstract and Section 1 are contradicted by the paper's own Table 3, and by the absence of a rigorous proof that the AIFS reordering is numerically equivalent for arbitrary multimodal inputs.

major comments (1)
  1. [Section 3.1 (Eq. 5) and Appendix A.2] The claim that MQuant under W4A8 achieves 'less than 1% accuracy loss on all MLLMs' (abstract and Section 1) and that W8A8 is 'near-lossless' is contradicted by Table 3. Specific examples: InternVL2-8B drops 2.55 points on DocVQA (90.97 to 88.42) and 69 points on OCRBench (794 to 725, an 8.7% relative drop) under W4A8; Qwen-VL-Chat-9.6B drops 121 points on MME (1834 to 1713, a 6.6% relative drop) under W4A8 and 143 points on MME even under W8A8; GLM-4V-9B drops 1.49 points on DocVQA and 33 points on MME under W4A8. Several of these exceed 1% whether measured absolutely or relatively. Since the near-lossless claim is the paper's headline, the manuscript should either correct the claim (e.g., 'within 1% on several models and within 2-3% on several benchmarks') or transparently report per-model degradation in the abstract and conclusion.
minor comments (6)
  1. [Throughout] 'casual attention' and 'casual mask' are used repeatedly (e.g., Section 3.1, Figure 3 caption) where 'causal' is meant.
  2. [Section 4.3] In the paragraph after Table 9, 'MBQ+AIFS with Flash Attention' and 'MBQ and AIFS designs' appear to be copy-paste errors; these should read 'MSQ+AIFS' or 'MQuant'.
  3. [Section 4.4] The text contains typos 'loessless accuracy' (should be 'lossless') and 'online per-toekn dynamic quantization' (should be 'per-token').
  4. [Table 3] The table's formatting is confusing: the 'Visual LLM' column is empty, and the grouping of W8A8 versus W4A8 rows is ambiguous because the row labels concatenate 'RTN W8A8 W4A8' and 'RTN W4A8 W4A8'. The table should be split into explicit W8A8 and W4A8 blocks.
  5. [References] References [1] and [2] both cite the GPT-4 technical report with different formatting; one should be removed or merged.
  6. [Section 4.4] The sentence 'AIFS yields speedups of 20%–80% as resolution increases' (in the discussion of Figure 7) does not specify the comparison baseline; it should state whether the comparison is against BF16 or per-token dynamic quantization, and for which input configuration.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MQuant's components are evaluated against external baselines, and its equivalence and outlier claims are mathematical constructions or externally tested, not fitted inputs.

full rationale

MQuant's central claims are not circular. AIFS is a construction: the reordered sequence E_u and the unified causal mask M^u (Eqs. 3-5 and Appendix A.2, Eq. 11) are defined from the original token indices, so the asserted numerical equivalence with the original masked attention is an identity that can be checked directly, not a quantity fitted to the method's own outputs. RMS derives the channel-mean outlier condition from the Hadamard matrix's first row (Eqs. 7-9), then verifies prevalence empirically (Table 2) and tests the fix via ablations (Table 7); the condition is mathematical and the gain is measured externally. All accuracy results are measured against BF16 and external baselines (RTN, SmoothQuant, Quarot, Q-VLM, MBQ, QSLAW) on held-out benchmarks, and no benchmark number is constructed from MQuant's own scale factors or masks. Several citations are to the authors' own prior work (e.g., RPTQ, I-LLM, OstQuant, MambaQuant, Q-PETR), but they appear as related-work context or as additional LLM-PTQ examples; the method itself builds on external results (Quarot, QuIP/Quip#, SliceGPT, SmoothQuant, GPTQ, AWQ), so no load-bearing step reduces to a self-citation. The abstract's '<1% degradation on all MLLMs' is contradicted by several entries in Table 3, and the multi-image AIFS equivalence is asserted rather than proved for arbitrary interleavings; these are correctness and completeness concerns, not circularity. Score 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The ledger contains calibration-fitted quantization scales and several equivalence assumptions about reordering, normalization conversion, and Hadamard outliers. The paper introduces no new physical entities; its claims rest on these assumptions plus external benchmark evaluations.

free parameters (2)
  • Per-modality static activation scales s_v and s_t for each linear layer = calibrated from 256 samples of each benchmark's training set
    These two scales per layer replace dynamic per-token scales and directly determine activation quantization error; the central near-lossless accuracy claim depends on how well they fit calibration data.
  • Weight quantization scales and zero-points (per-channel, group size 128) = GPTQ/AWQ calibrated on the same 256-sample set
    W4 weight quantization uses GPTQ/AWQ, whose scales and zero-points are fitted to calibration data; MQuant's reported W4 accuracy includes these fitted values.
assumptions (4)
  • domain assumption LayerNorm in vision encoders/projectors can be replaced by RMSNorm with recentering adjustments without changing the model's outputs, enabling Hadamard rotations.
    Invoked in Section 3.2 and Appendices A.13-A.14 to make QuaRot-style rotations applicable to MLLMs; the algebraic equivalence holds in exact arithmetic, and its effect under low-bit quantization is assumed benign.
  • domain assumption AIFS token reordering with the permuted causal mask and shifted position indices is exactly output-equivalent to the original sequence for all interleaved multimodal inputs, including multi-image and video.
    Stated in Section 3.1 and Eq. 5 as 'numerically the same'; the mask formula handles a single contiguous visual block and no proof is given for multiple visual segments or for M-RoPE-style position embeddings.
  • standard math The first row of a Walsh-Hadamard matrix contains identical entries up to 1/sqrt(n), so (H W)_0j equals sqrt(n) times the j-th input channel mean.
    This is a standard Hadamard matrix property used in the RMS derivation (Section 3.2, Eq. 8).
  • domain assumption Suppressing only the first-row outliers flagged by Eq. 9 is sufficient to restore quantization accuracy; other rows of the transformed weight do not create comparable outliers.
    The paper's ablation (Table 7) supports this empirically on Qwen2-VL-7B, but the theoretical analysis does not characterize quantization error contributions beyond the first row.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization." pith.science (2026). https://pith.science/paper/7JFTDSNO

@misc{pith2026250200425,
  author       = {Pith},
  title        = {Pith review of: MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JFTDSNO}},
  note         = {Machine review of arXiv:2502.00425}
}
read the original abstract

Multimodal large language models (MLLMs) have garnered widespread attention due to their ability to understand multimodal input. However, their large parameter sizes and substantial computational demands severely hinder their practical deployment and application.While quantization is an effective way to reduce model size and inference latency, its application to MLLMs remains underexplored. In this paper, we propose MQuant, a post-training quantization (PTQ) framework designed to tackle the unique challenges of multimodal large language models (MLLMs). Conventional quantization often struggles with MLLMs because of (a) high inference latency from large visual token counts, (b) distributional disparities between visual and textual tokens, and (c) extreme outliers introduced by Hadamard-based transformations. To address these issues, MQuant introduces: Modality-Specific Static Quantization (MSQ), assigning distinct static scales for visual vs. textual tokens; Attention-Invariant Flexible Switching (AIFS), reordering tokens to preserve casual attention while eliminating expensive token-wise scale computations; Rotation Magnitude Suppression (RMS), mitigating weight outliers arising from online Hadamard rotations. On five mainstream MLLMs (including Qwen-VL, MiniCPM-V, CogVLM2), MQuant under W4A8 achieves near-floating-point accuracy (<1% degradation) while reducing inference latency by up to 30%, significantly outperforming existing PTQ baselines. Our MQuant effectively bridges the gap for efficient and accurate MLLMs inference in resource-constrained devices. Code has been released in https://github.com/StiphyJay/MQuant.

Figures

Figures reproduced from arXiv: 2502.00425 by the authors.

Figure 1
Figure 1. (a) Prefill visual tokens counts across different MLLMs as the image splits/resolution increases. (b) The activation values of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. MLLM’s architecture. As shown in the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of Modality-Specific Static Quantization [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: (a) The pipeline of Quarot, showing offline and par [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: An overview of our proposed RMS. We separate [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Illustration the weight distributions for the down-proj [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The Speedup of AIFS+MSQ on Qwen2-VL-7B. the same speedup as naïve per-tensor static quantization but also maintains near-floating-point accuracy across linear layers [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The illustration of causal mask when batch size > 1. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The number of prefill visual tokens across different MLLMs as the image splits or resolution increases. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 11
Figure 11. Figure 11: The illustration of transformation from Pre-LN to RMSNorm. Therefore, as shown in [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 10
Figure 10. Figure 10: The proposed Post-LN + Rotate Scheme. operation is to recenter the output of main branches. Notably, since 𝑿𝑘+1 = 𝑿𝑘 +ℓ2 (𝑔(ℓ1 (𝐿𝑁 (𝑿𝑘 )))), after applying ❶ and ❷, the input and the output of main branch are re-centered with zero-mean, while the input of residual bra…
Figure 12
Figure 12. Figure 12: Illustration the weight distributions for the down-proj layer in different MLLMs’s visual encoder under three conditions: [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    EffiVLM-Bench is a benchmark study showing token compression is task- and model-dependent, KV cache methods are more loyal, and parameter compression preserves accuracy better at typical ratios.

  2. Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques

    cs.LG 2025-06 unverdicted novelty 4.0 of 10

    A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.

Reference graph

Works this paper leans on

77 extracted references · 31 canonical work pages · cited by 2 Pith papers

  1. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  3. [4]

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by MM ’25, October 27–31, 2025, Dublin, Ireland JiangY ong Y u & Sifan Zhou et al. Table 13: Multi-Batch speedup comparison of MSQ + AIFS on W4A8 setting. Each row shows the cumulative total of text tokens, ...

    work page arXiv 2024

  4. [5]

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier- free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456 (2024)

    arXiv 2024

  5. [6]

    JL Ba. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016)

    arXiv 2016

  6. [7]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, et al. 2023. Qwen Technical Report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  7. [8]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Jun- yang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)

    arXiv 2023

  8. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems (2020)

    work page 2020

Show all 77 references
  1. [10]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, et al . 2024. InternLM2 Technical Report. arXiv:2403.17297 [cs.CL]

    2024 arXiv

  2. [11]

    Jerry Chee, Yaohui Cai, V olodymyr Kuleshov, and Christopher M De Sa. 2024. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems 36 (2024)

    2024

  3. [12]

    Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. 2024. Prefixquant: Static quantization beats dynamic through prefixed outliers in llms. arXiv preprint arXiv:2410.05265 (2024)

    2024 arXiv

  4. [13]

    Zhixuan Chen, Xing Hu, Dawei Yang, Zukang Xu, Zhihang Yuan, Sifan Zhou, et al. [n. d.]. MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance. In Forty- second International Conference on Machine Learning

  5. [14]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al . 2024. How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites. arXiv preprint arXiv:2404.16821 (2024)

    2024 arXiv

  6. [15]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scal- ing up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Com...

    2024

  7. [16]

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al . 2024. MobileVLM V2: Faster and Stronger Baseline for Vision Language Model. arXiv preprint arXiv:2402.03766 (2024)

    2024 arXiv

  8. [17]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359

    2022

  9. [18]

    Alexey Dosovitskiy. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR (2021)

    2021

  10. [19]

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. 2024. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Mult...

    2024

  11. [20]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  12. [21]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv preprint arXiv:2210.17323 (2022)

    2022 arXiv

  13. [22]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji

  14. [23]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv preprint arXiv:2406.12793 (2024)

    2024 arXiv

  15. [24]

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al . 2024. CogVLM2: Vi- sual Language Models for Image and Video Understanding. arXiv preprint arXiv:2408.16500 (2024)

    2024 arXiv

  16. [25]

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024. MiniCPM: Un- veiling the Potential of Small Language Models with Scalable Training Strategies. arXiv preprint arXiv:2404.06395 (2024)

    2024 arXiv

  17. [26]

    Xing Hu, Yuan Chen, Dawei Yang, Sifan Zhou, Zhihang Yuan, Jiangyong Yu, and Chen Xu. 2024. I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models. arXiv preprint arXiv:2405.17849 (2024)

    2024 arXiv

  18. [27]

    Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. 2025. OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting. arXiv preprint arXiv:2501.13987 (2025)

    2025 arXiv

  19. [28]

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al . 2024. Language is not all you need: Aligning perception with language models.NeurIPS 36 (2024)

    2024

  20. [29]

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021. OpenCLIP. https://doi.org/10.5281/zenodo.5143773 If you use ...

    2021 doi

  21. [30]

    Xiaoyan Jiang, Hang Yang, Kaiying Zhu, Xihe Qiu, Shibo Zhao, and Sifan Zhou

  22. [31]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrap- ping language-image pre-training with frozen image encoders and large language models. ICML (2023), 19730–19742

    2023

  23. [32]

    Liang Li, Qingyuan Li, Bo Zhang, and Xiangxiang Chu. 2024. Norm tweaking: High-performance low-bit quantization of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 38. 18536–18544

    2024

  24. [33]

    Qingyuan Li, Yifan Zhang, Liang Li, Peng Yao, Bo Zhang, Xiangxiang Chu, Yerui Sun, Li Du, and Yuchen Xie. 2023. Fptq: Fine-grained post-training quantization for large language models. arXiv preprint arXiv:2308.15987 (2023)

    2023 arXiv

  25. [34]

    Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang

  26. [35]

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Rui- hang Chu, Shaoteng Liu, and Jiaya Jia. 2024. Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models. arXiv preprint arXiv:2403.18814 MQuant: Unleashing the Inference Potential of Mult...

    2024 arXiv

  27. [36]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han

  28. [37]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual Instruction Tuning. NeurIPS 36 (2024)

    2024

  29. [38]

    arXiv:2412.19509 [cs.CV] https://arxiv.org/abs/2412.19509

    MBQ: Modality-Balanced Quantization for Large Vision-Language Models. arXiv:2412.19509 [cs.CV] https://arxiv.org/abs/2412.19509

    arXiv

  30. [39]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Ad- vances in Neural Information Processing Systems 35 ...

    2022

  31. [40]

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. DocVQA: A dataset for VQA on document images. In WACV. 2200–2209

    2021

  32. [41]

    arXiv preprint arXiv:2306.00978 (2023)

    AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv preprint arXiv:2306.00978 (2023)

    2023 arXiv

  33. [42]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  34. [43]

    Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. 2023. On the hidden mystery of OCR in large multimodal models. arXiv preprint arXiv:2305.07895 (2023)

    2023 arXiv

  35. [44]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models that can read. In CVPR. 8317–8326

    2019

  36. [45]

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. RoFormer: En- hanced Transformer with Rotary Position Embedding. arXiv:2104.09864 [cs.CL]

    2021 arXiv

  37. [46]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML. PMLR, 8748–8763

    2021

  38. [47]

    Fuwen Tan, Royson Lee, Łukasz Dudziak, Shell Xu Hu, Sourav Bhattacharya, Timothy Hospedales, Georgios Tzimiropoulos, and Brais Martinezs. 2024. Mo- bileQuant: Mobile-friendly Quantization for On-device Language Models. In The 2024 Conference on Empirical Methods in Natural Lan...

    2024

  39. [48]

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2023. OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models. CoRR abs/2308.13137 (2023)

    2023 arXiv

  40. [49]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  41. [50]

    Albert Tseng, Jerry Chee, Qingyao Sun, V olodymyr Kuleshov, and Christopher De Sa. 2024. Quip#: Even better LLM quantization with hadamard incoherence and lattice codebooks. Forty-first International Conference on Machine Learning (2024)

    2024

  42. [51]

    Fuwen Tan, Royson Lee, Łukasz Dudziak, Shell Xu Hu, Sourav Bhattacharya, Timothy Hospedales, Georgios Tzimiropoulos, and Brais Martinez. 2024. Mo- bilequant: Mobile-friendly quantization for on-device language models. arXiv preprint arXiv:2408.13933 (2024)

    2024 arXiv

  43. [52]

    Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu

  44. [53]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  45. [54]

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023. CogVLM: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 (2023)

    2023 arXiv

  46. [55]

    Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. 2022. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems (2022)

    2022

  47. [56]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems (2017)

    2017

  48. [57]

    Jingjing Xie, Yuxin Zhang, Mingbao Lin, Liujuan Cao, and Rongrong Ji. 2024. Advancing multimodal large language models with quantization-aware scale learning for efficient adaptation. In Proceedings of the 32nd ACM International Conference on Multimedia. 10582–10591

    2024

  49. [58]

    In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    Q-VLM: Post-training Quantization for Large Vision-Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

  50. [59]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing Vision-Language Mode...

    2024 arXiv

  51. [60]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 (2024)

    2024 arXiv

  52. [61]

    Jiangyong Yu, Changyong Shu, Dawei Yang, Sifan Zhou, Zichen Yu, Xing Hu, and Yan Chen. 2025. Q-PETR: Quant-aware Position Embedding Transformation for Multi-View 3D Object Detection. arXiv preprint arXiv:2502.15488 (2025)

    2025

  53. [62]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438 (2022)

    2022 arXiv

  54. [63]

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun

  55. [64]

    Chen Xu, Yuxuan Yue, Zukang Xu, Xing Hu, Zhixuan Chen, Sifan Zhou, Zhihang Yuan, Dawei Yang, et al. [n. d.]. RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization. In Forty-second International Conference on Machine Learning

  56. [65]

    Zukang Xu, Yuxuan Yue, Xing Hu, Dawei Yang, Zhihang Yuan, Zixu Jiang, Zhixuan Chen, Sifan Zhou, et al . 2025. MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods. The Thirteenth International Conference on Learning Representations (2025)

    2025

  57. [66]

    Biao Zhang et al . 2019. Root mean square layer normalization. Advances in Neural Information Processing Systems 32 (2019)

    2019

  58. [67]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. In EMNLP (Demos)

    2023

  59. [68]

    Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. 2023. RPTQ: Reorder- based Post-training Quantization for Large Language Models. arXiv preprint arXiv:2304.01089 (2023)

    2023 arXiv

  60. [69]

    Sifan Zhou, Liang Li, Xinyu Zhang, Bo Zhang, Shipeng Bai, Miao Sun, Ziyu Zhao, Xiaobo Lu, and Xiangxiang Chu. 2024. LiDAR-PTQ: Post-Training Quantization for Point Cloud 3D Object Detection. International Conference on Learning Representations (2024)

    2024

  61. [70]

    arXiv preprint arXiv:2312.05821 (2023)

    ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models. arXiv preprint arXiv:2312.05821 (2023)

    2023 arXiv

  62. [71]

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, et al. 2024. Llm inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363 (2024)

    2024 arXiv

  63. [72]

    Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. 2024. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402.12065

    2024 arXiv

  64. [75]

    Ying Zhang, Peng Zhang, Mincong Huang, Jingyang Xiang, Yujie Wang, Chao Wang, Yineng Zhang, Lei Yu, Chuan Liu, and Wei Lin. 2024. QQQ: Qual- ity Quattuor-Bit Quantization for Large Language Models. arXiv preprint arXiv:2406.09904 (2024)

    2024 arXiv

  65. [77]

    Sifan Zhou, Shuo Wang, Zhihang Yuan, Mingjia Shi, Yuzhang Shang, and Dawei Yang. 2025. GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning. In Findings of the Association for Computational Linguistics: ACL 2025. Association for...

    2025

  66. [78]

    text-image-text

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304.10592 (2023). A Appendix A.1 MQuant Algorithm Here, we present overall MQuant algorithm ...

    2023 arXiv

  67. [2022]

    NeurIPS 35 (2022), 23716–23736

    Flamingo: A visual language model for few-shot learning. NeurIPS 35 (2022), 23716–23736

    2022

  68. [2023]

    arXiv preprint arXiv:2306.13394 (2023)

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 (2023)

    2023 arXiv

  69. [2024]

    arXiv preprint arXiv:2409.17020 (2024)

    Ptq4ris: Post-training quantization for referring image segmentation. arXiv preprint arXiv:2409.17020 (2024)

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.