Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

MBQ: Modality-Balanced Quantization for Large Vision-Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that quantizing large vision-language models loses accuracy because calibration treats vision and language tokens equally, and that weighting each modality by its measured sensitivity (average absolute gradient of the SFT…

desk verdict Modality-balanced calibration is a real idea with one clean ablation, but the main accuracy claims are confounded by calibration-set mismatch; fixable with a same-calibration sweep. read the letter →

arxiv 2412.19509 v2 pith:WAGZU4O6 submitted 2024-12-27 cs.CV cs.AI

Shiyao Li , Yingchun Hu , Xuefei Ning , Xihui Liu , Ke Hong , Xiaotao Jia , Xiuhong Li , Yaqi Yan
show 5 more authors
Pei Ran Guohao Dai Shengen Yan Huazhong Yang Yu Wang
This is my paper · ORCID
classification cs.CVcs.AI
keywords post-trainingquantizationvision-languagemodelsmodalitybalancingchannel-wiseequalizationgradientsensitivityweight-onlyweight-activation3-bit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper identifies a first-order cause of accuracy loss when post-training quantizing large vision-language models: vision tokens and language tokens are not equally sensitive to quantization error, yet standard calibration treats them the same. Using a first-order Taylor bound, it shows the loss increase from a quantized layer is controlled by the product of quantization error and the average absolute gradient of the SFT loss per token type. It then proposes MBQ, which weights the vision and language portions of the block-wise reconstruction loss by these average gradients when searching for channel-wise equalization factors. On seven- to seventy-billion-parameter models, MBQ improves task accuracy over strong baselines by up to 4.4% under W3A16 and 11.6% under W4A8 quantization, and it ships a fused 3-bit dequantization-GEMV kernel for a 1.4x decoding speedup. A reader should care because it suggests that modality imbalance, not merely activation outliers, is a primary obstacle to low-bit VLM deployment.

What carries the argument

The load-bearing object is the modality-balanced reconstruction loss: instead of minimizing $\|Q(WE)(E^{-1}X)-WX\|^2$ over all tokens equally, MBQ minimizes $\bar{|g_v|}\,\|WX_v-Q(WE)(E^{-1}X_v)\| + \bar{|g_l|}\,\|WX_l-Q(WE)(E^{-1}X_l)\|$ (with absolute error, not squared), where $\bar{|g_v|}$ and $\bar{|g_l|}$ are average absolute gradients of the SFT loss w.r.t. the layer's vision and language output tokens. This is derived from a first-order Taylor bound on the loss change, Eqs. (8)-(13), and the weights are computed once per layer from a small calibration set (128 image-caption pairs) and reused as constant weights during the equalization-factor search. The MAE form is chosen because it derives directly from the bound and outperforms an MSE-weighted variant in ablations.

What would settle it

Compute the per-layer gradient ratios on a different calibration set (for example, VQA or chart-understanding pairs), re-estimate the MBQ weights, and compare accuracy against the original COCO-derived weights: if the weights shift substantially yet produce worse accuracy than a fixed ratio, or if the improvement vanishes when the calibration set changes, the sensitivity measure is not the load-bearing mechanism. Conversely, an experiment that randomizes the per-modality weights (such as swapping vision and language weights) and shows the accuracy gain disappears would confirm that the gradient-based balancing itself is what drives the result.

Watch

Extended reading notes

Core claim

The central claim is that the sensitivity of a VLM's output to feature perturbation differs by an order of magnitude between vision and language tokens, and that calibration objectives which ignore this imbalance systematically over-budget quantization error. For a linear layer, MBQ bounds the change in SFT loss by $\bar{|g_v|} \cdot \|Y_v - \hat Y_v\| + \bar{|g_l|} \cdot \|Y_l - \hat Y_l\|$, where $\bar{|g|}$ are average absolute gradients of the loss w.r.t. the output tokens of each modality. It therefore minimizes a reconstruction loss in which each modality's error is weighted by its measured sensitivity, for both weight-only and weight-activation settings. With these weights, the method consistently outperforms equal-treatment baselines such as AWQ and SmoothQuant on MMMU, SEED, OCRBench, VizWiz, ScienceQA, and TextVQA across LLaVA-onevision, InternVL2, and Qwen2-VL families.

Load-bearing premise

The method assumes that the average absolute gradient of the SFT loss, computed over just 128 image-caption pairs from a single dataset, gives a stable and transferable per-layer measure of how much each modality's tokens matter to the final answer, across all layers and model families.

Editorial extensions

If this is right

  • If MBQ is right, applying unmodified LLM PTQ methods to VLMs will keep leaving accuracy on the table, because the equal-treatment objective is a systematic bias, not just noise.
  • Quantizing the ViT encoder alongside the VLM is feasible at W4A8 with little or no drop, since vision tokens are the more redundant modality.
  • MBQ is a plug-in: it can be combined with rotation-based quantizers (MBQ (Rot)) to bring W4A8 large VLMs within about one point of FP16 accuracy.
  • Any future equalization-based PTQ method applied to multimodal models should incorporate per-modality weighting to avoid over-emphasizing the insensitive modality.
  • A fused W3 weight-only kernel can accelerate VLM decoding by 1.4x on an RTX 4090, making low-bit multimodal serving practical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gradient-ratio story is causal, the best calibration dataset for VLM quantization may be the one that matches the deployment distribution of vision versus language token counts, not merely a high-quality caption set; that is a testable hypothesis the paper does not run.
  • The same per-modality weighting principle could extend to other compression steps, such as pruning or KV-cache quantization, where sensitivity also differs by token type.
  • The 128-sample COCO caption calibration may underestimate vision-token sensitivity on tasks where the image is the primary answer source (for example, fine-grained OCR or diagram reading); an experiment re-estimating the per-layer weights on task-specific data would show whether the fixed weights transfer.
  • Since the method averages gradients over tokens, layers where vision gradients are near zero may make the balanced objective degenerate; the paper notes this for token-wise weighting, and the same caution applies to the per-modality average.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Modality-Balanced Quantization (MBQ) for post-training quantization of large vision-language models. Observing in Fig. 1 that the average absolute gradient of the SFT loss with respect to language-token features is an order of magnitude larger than for vision tokens, the authors weight the block-wise reconstruction loss by per-modality average absolute gradients during calibration, yielding a per-layer balanced objective for weight-only (W3A16, W4A16) and weight-activation (W4A8, W8A8) quantization. The main results report accuracy improvements over AWQ, GPTQ, and SmoothQuant of up to 4.4% and 11.6% on W3A16 and W4A8, respectively, across 7B-72B LLaVA, InternVL2, and Qwen2-VL models, plus a fused W3 GEMV kernel that achieves up to 1.4x end-to-end decoding speedup.

Significance. If the claimed improvements are robust, the paper makes a useful contribution by identifying modality imbalance as a distinct failure cause in VLM quantization and by proposing a simple, generalizable weighting scheme that plugs into CWE-based PTQ methods. The same-calibration ablation in Table 4, where adding MAE modality-balance to a COCO-calibrated CWE baseline improves MMMU from 38.7 to 42.0 (W3A16) and from 29.2 to 42.6 (W4A8), is the strongest evidence that the modality-weighting mechanism, rather than calibration choice, drives the gain on LLaVA-onevision-7B. The paper also ships code and a practical W3 kernel, which are clear strengths. However, as detailed below, the headline comparisons with Pile-calibrated baselines are not controlled for calibration data, and the theoretical derivation in Sec. 3.2 contains an invalid inequality. These issues are fixable with additional experiments and a reframing of the theoretical claim.

major comments (3)
  1. [Sec. 4.1.1 / Sec. 4.3.1, Tables 2-4] The main results in Tables 2 and 3 compare MBQ, calibrated on 128 ShareGPT4V COCO image-caption pairs, with AWQ, GPTQ, and SmoothQuant calibrated on the Pile (language-only) set. This confounds the method with the calibration set. The only same-calibration evidence is Table 4, for LLaVA-onevision-7B on MMMU and SEED. In that table, COCO-calibrated CWE without balance collapses on W4A8/SEED (10.2 vs. SmoothQuant's 41.6 with Pile), and MBQ restores it; this supports the modality-balance mechanism for one model, but it does not establish the advertised 'up to 4.4%/11.6%' margins over SOTA baselines across six models and six benchmarks. The paper's own Sec. 4.3.1 reports that switching SmoothQuant to COCO calibration degrades SEED by 31.4%, so the main-table comparisons conflate calibration choice with modality balancing. A full same-calibration comparison sweep across all models and benchmarks is needed to support the headline claim.
  2. [Sec. 3.2, Eqs. (8)-(13)] The derivation of the balanced loss is not mathematically valid as written: the inequality ||g_v^T * Δ_v|| <= average_abs(g_v) * ||Δ_v|| in Eq. (12) does not follow when average_abs(g_v) is the average absolute gradient over tokens rather than a matrix norm or a per-element maximum. Consequently, Eqs. (14)-(17) are not actually derived from the first-order Taylor expansion in a rigorous sense. The empirical results can stand on their own, but the current text overstates the theoretical grounding; please either correct the bound or present the objective as a heuristic importance-weighting scheme.
  3. [Sec. 4.1.1 and Sec. 3.2] The entire method rests on the assumption that the per-modality average absolute gradient of the SFT loss, computed on a fixed 128-pair COCO set, is a reliable and transferable per-layer importance weight. The paper shows only one illustrative layer (Fig. 1) and does not report the layer-wise variation or model-to-model variation of the vision/language gradient ratio, nor the fraction of zero-gradient vision tokens (which Sec. 4.3.2 indicates is large). Without such statistics, it is unclear whether the weighting is stable across layers and models, and whether the average is a sufficient statistic. Please add a sensitivity analysis of the gradient ratios.
minor comments (6)
  1. [Abstract] The abstract says 'under W3 and W4A8 quantization'; please use 'W3A16 and W4A8' for precision.
  2. [Sec. 2.1, Eq. (2)] The expression for Sasym is ambiguous; use parentheses, e.g., Sasym = (max(W_FP16) - Z) / (2^N - 1).
  3. [Abstract / Sec. 4.2] The 'up to 4.4% and 11.6%' figures in the abstract should be checked against Sec. 4.2, which reports an 18.5% average improvement on LLaVA-onevision-72B W3A16; please make the reported 'up to' figures consistent across the paper.
  4. [Table 2] The SEED score for RTN W3A16 on LLaVA-onevision-7B (5.9) is dramatically lower than all other cells in that row; please verify this entry or explain the collapse.
  5. [Sec. 4.3.2] The two additional reweighting strategies (random token grouping and token-wise gradients) are described without reporting the OCRBench numbers; please include those values in the text or in a table.
  6. [Sec. 3.2, Eq. (12)] Define the notation explicitly: let average_abs(g_v) = (1/N_v) * sum_{i=1}^{N_v} |g_{v,i}|, where N_v is the number of vision tokens.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: sensitivity-weighted reconstruction is a first-order proxy, and the reported benchmark gains are external to the calibration objective.

full rationale

The derivation chain is self-contained and not circular. MBQ estimates per-modality sensitivity as the average absolute gradient of the SFT loss with respect to the output features of each linear layer (Fig. 1 and Sec. 3.1), then uses Eqs. (8)-(13), a first-order Taylor bound, to justify minimizing the weighted reconstruction error in Eqs. (14)-(17). The weighted loss is an upper-bound proxy for the change in the calibration SFT loss, so improvements on that proxy are expected by construction; however, the paper's central claims are benchmark accuracies on MMMU, SEED, OCRBench, etc., which are not used to set the gradient weights or any final hyperparameter. The 0.1 factor appears only in the oracle ablation of Sec. 3.1, not in the final method. The W3/W4 speedup claims are measured on hardware and are also external. Self-citations ([23], [24], [36], [54]) are background or prior evaluation methodology and are not load-bearing for the modality-balancing mechanism. The calibration-set difference between MBQ (COCO captions) and baselines (Pile text) is a real experimental confound that weakens the controlled comparison, but it is a benchmark-fairness concern, not a circular reduction: no equation or fitted parameter in MBQ is defined in terms of the reported evaluation metrics. Therefore no step reduces by construction to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. It uses one hand-chosen factor in the motivating oracle experiment, plus a calibration set size. The main assumptions are that gradient sensitivity is a valid and transferable importance measure, and that per-modality averaging is sufficient.

free parameters (2)
  • Oracle modality-balancing factor (vision weight) = 0.1
    Used only in the Sec. 3.1 oracle ablation (Eqs. 6-7) to demonstrate the effect; the final MBQ method derives per-layer weights from gradients and does not use this fixed factor.
  • Calibration set size = 128 image-caption pairs
    Chosen by hand in Sec. 4.1.1; the central claim depends on having enough calibration data to estimate stable per-modality gradient averages.
assumptions (4)
  • domain assumption SFT-loss gradients with respect to output token features measure the impact of quantization perturbation on final task loss.
    Sec. 3.1 and Eqs. (8)-(13) use this as the core sensitivity indicator, and assume first-order Taylor approximation is adequate for 3-4 bit quantization error magnitudes.
  • ad hoc to paper Average absolute gradient per modality is a sufficient statistic for token sensitivity.
    Eq. (13) replaces per-token gradient vectors with scalar per-modality means; the paper does not justify this reduction beyond computational convenience.
  • domain assumption Calibration distribution (COCO image captioning) transfers to evaluation benchmarks (MMMU, TextVQA, SEED, etc.).
    Sec. 4.1.1 uses 128 ShareGPT4V COCO caption pairs; the method assumes sensitivity weights learned on captions generalize to other VQA and reasoning tasks.
  • domain assumption Vision tokens are redundant and less sensitive than language tokens across VLM families.
    Sec. 1 and Sec. 3.1 cite data redundancy and language bias from Zhang et al. [50]; this motivates but is not required for the gradient-based method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MBQ: Modality-Balanced Quantization for Large Vision-Language Models." pith.science (2026). https://pith.science/paper/WAGZU4O6

@misc{pith2026241219509,
  author       = {Pith},
  title        = {Pith review of: MBQ: Modality-Balanced Quantization for Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WAGZU4O6}},
  note         = {Machine review of arXiv:2412.19509}
}
read the original abstract

Vision-Language Models (VLMs) have enabled a variety of real-world applications. The large parameter size of VLMs brings large memory and computation overhead which poses significant challenges for deployment. Post-Training Quantization (PTQ) is an effective technique to reduce the memory and computation overhead. Existing PTQ methods mainly focus on large language models (LLMs), without considering the differences across other modalities. In this paper, we discover that there is a significant difference in sensitivity between language and vision tokens in large VLMs. Therefore, treating tokens from different modalities equally, as in existing PTQ methods, may over-emphasize the insensitive modalities, leading to significant accuracy loss. To deal with the above issue, we propose a simple yet effective method, Modality-Balanced Quantization (MBQ), for large VLMs. Specifically, MBQ incorporates the different sensitivities across modalities during the calibration process to minimize the reconstruction loss for better quantization parameters. Extensive experiments show that MBQ can significantly improve task accuracy by up to 4.4% and 11.6% under W3 and W4A8 quantization for 7B to 70B VLMs, compared to SOTA baselines. Additionally, we implement a W3 GPU kernel that fuses the dequantization and GEMV operators, achieving a 1.4x speedup on LLaVA-onevision-7B on the RTX 4090. The code is available at https://github.com/thu-nics/MBQ.

Figures

Figures reproduced from arXiv: 2412.19509 by the authors.

Figure 1
Figure 1. The gradients of loss function w.r.t. the token features [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The inference process of Large VLMs. The blue patches [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization

    cs.CV 2025-02 conditional novelty 5.0 of 10

    MQuant quantizes multimodal LLMs to W4A8 using per-modality static scales, token reordering, and Hadamard-outlier suppression, claiming near-floating-point accuracy and up to 30% latency reduction on five MLLMs.

Reference graph

Works this paper leans on

63 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 1

    2015

  3. [3]

    Quarot: Outlier-free 4-bit inference in rotated llms

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Al- istarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456, 2024. 1

    arXiv 2024

  4. [4]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1, 4

    arXiv 2023

  5. [5]

    Paligemma: A versatile 3b vlm for trans- fer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer. arXiv preprint arXiv:2407.07726, 2024. 2

    arXiv 2024

  6. [6]

    Madtp: Multi- modal alignment-guided dynamic token pruning for accel- erating vision-language transformer

    Jianjian Cao, Peng Ye, Shengze Li, Chong Yu, Yan- song Tang, Jiwen Lu, and Tao Chen. Madtp: Multi- modal alignment-guided dynamic token pruning for accel- erating vision-language transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15710–15719, 2024. 2

    work page 2024

  7. [7]

    Quip: 2-bit quantization of large language models with guarantees

    Jerry Chee, Yaohui Cai, V olodymyr Kuleshov, and Christo- pher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024. 1

    work page 2024

  8. [8]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 1, 4

    arXiv 2023

Show all 63 references
  1. [9]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer, 2025. 2

    2025

  2. [10]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 1, 2, 3

    2015 arXiv

  3. [11]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  4. [12]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 2

    2023 arXiv

  5. [13]

    Flashattention-2: Faster attention with bet- ter parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023. 8

    2023 arXiv

  6. [14]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression

    Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, De- nis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078, 2023. 1

    2023 arXiv

  7. [15]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. 1, 2

    2022 arXiv

  8. [16]

    Moa: Mixture of sparse attention for automatic large language model compression.arXiv preprint arXiv:2406.14909, 2024

    Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression.arXiv preprint arXiv:2406.14909, 2024. 1

    2024

  9. [17]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, An- ish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. 6

    2020 arXiv

  10. [18]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  11. [19]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 4

    2023 arXiv

  12. [20]

    Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024. 4, 2

    2024

  13. [21]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 4, 2

    2024 arXiv

  14. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  15. [23]

    Llm-mq: Mixed-precision quantiza- tion for efficient llm deployment

    Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang, and Yu Wang. Llm-mq: Mixed-precision quantiza- tion for efficient llm deployment. In The Efficient Natural Language and Speech Processing Workshop with NeurIPS ,

  16. [24]

    Evaluating quantized large language models

    Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xi- angsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. In Proceedings of the 41st International Conference on Ma- chine Learning, pages 28480–28524, 2024. 6

    2024

  17. [25]

    Eagle: Speculative sampling requires rethinking feature un- certainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature un- certainty. arXiv preprint arXiv:2401.15077, 2024. 1

    2024 arXiv

  18. [26]

    Moe-llava: Mixture of experts for large vision- language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision- language models. arXiv preprint arXiv:2401.15947 , 2024. 2

    2024 arXiv

  19. [27]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems , 6:87–100,

  20. [28]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 2

    2024

  21. [29]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532, 2024. 2, 4, 8

    2024 arXiv

  22. [30]

    Boosting multimodal large language models with visual tokens withdrawal for rapid inference

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. arXiv preprint arXiv:2405.05803, 2024. 2

    2024 arXiv

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2023. 1, 2, 4

    2023

  24. [32]

    Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024. 4

    2024

  25. [33]

    Spinquant–llm quantization with learned rotations

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant–llm quantization with learned rotations. arXiv preprint arXiv:2405.16406, 2024. 1, 2

    2024 arXiv

  26. [34]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  27. [35]

    Mm1: Methods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 2

    2024 arXiv

  28. [36]

    Skeleton-of-thought: Prompting llms for efficient parallel generation

    Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton-of-thought: Prompting llms for efficient parallel generation. In The Twelfth International Conference on Learning Representations, 2024. 1

    2024

  29. [37]

    Vl-mamba: Ex- ploring state space models for multimodal learning

    Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. Vl-mamba: Ex- ploring state space models for multimodal learning. arXiv preprint arXiv:2403.13600, 2024. 2

    2024 arXiv

  30. [38]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388,

  31. [39]

    Omniquant: Omnidirectionally calibrated quantization for large language models

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137, 2023. 2, 1

    2023 arXiv

  32. [40]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019. 4

    2019

  33. [41]

    Flatquant: Flatness matters for llm quantization

    Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, et al. Flatquant: Flatness matters for llm quantization. arXiv preprint arXiv:2410.09426, 2024. 1, 7

    2024 arXiv

  34. [42]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 1

    2017

  35. [43]

    Smoothquant: Accurate and effi- cient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and effi- cient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087–38099. PMLR, 2023. 1, 2, 4

    2023

  36. [44]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. 1

    2023 arXiv

  37. [45]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703, 2024. 2

    2024 arXiv

  38. [46]

    Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation

    Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yux- iong He. Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation. arXiv preprint arXiv:2303.08302, 2023. 6

    2023 arXiv

  39. [47]

    Mmmu: A massive multi-discipline mul- timodal understanding and reasoning benchmark for expert agi

    Xiang Yue et al. Mmmu: A massive multi-discipline mul- timodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024. 4

    2024

  40. [48]

    Sigmoid loss for language image pre-training,

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training,

  41. [49]

    Lmms- eval: Reality check on the evaluation of large multimodal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms- eval: Reality check on the evaluation of large multimodal models, 2024. 4

    2024

  42. [50]

    Debi- asing multimodal large language models

    Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Debi- asing multimodal large language models. CoRR, 2024. 3

    2024

  43. [51]

    H2o: Heavy-hitter ora- cle for efficient generative inference of large language mod- els

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R´e, Clark Barrett, et al. H2o: Heavy-hitter ora- cle for efficient generative inference of large language mod- els. Advances in Neural Information Processing ...

    2023

  44. [52]

    Cobra: Extending mamba to multi-modal large language model for efficient inference

    Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint arXiv:2403.14520, 2024. 2

    2024 arXiv

  45. [53]

    Atom: Low-bit quantization for efficient and accurate llm serving

    Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems, 6:196–209, 2024. 1

    2024

  46. [54]

    A survey on efficient inference for large language models

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294, 2024. 1

    2024 arXiv

  47. [55]

    Llava-phi: Efficient multi-modal as- sistant with small language model, 2024

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal as- sistant with small language model, 2024. 2 MBQ: Modality-Balanced Quantization for Large Vision-Language Models Supplementary Material LargeVision-LanguageModel Descr...

    2024

  48. [56]

    The Inference Process of VLMs The inference process of VLMs is shown in Fig

    Additional Preliminaries 6.1. The Inference Process of VLMs The inference process of VLMs is shown in Fig. 2. The whole inference system consists of three key components: • Language Tokenizer: Transform natural language sen- tences into a series of language tokens. • ViT Encod...

  49. [57]

    LLM Quantization Post-Training Quantization (PTQ) techniques are widely used in LLMs to accelerate the inference process

    Related Work 7.1. LLM Quantization Post-Training Quantization (PTQ) techniques are widely used in LLMs to accelerate the inference process. They employ the low-precision data format and computation to reduce the memory and computation overhead. To accelerate the memory-bound d...

  50. [58]

    W4A16 and W8A8 Results on Large VLMs As shown in Tab

    Additional Experiments 8.1. W4A16 and W8A8 Results on Large VLMs As shown in Tab. 11, we present the evaluation results for W4A16 and W8A8 quantized VLMs from the LLaV A- onevision, InternVL2, and Qwen2-VL families. In most cases, the proposed MBQ achieves accuracy comparable ...

  51. [59]

    No output: The quantized VLM generates no or only a few valid tokens, as shown in Example 2

  52. [60]

    Randomness: The quantized VLM randomly generates meaningless symbols, as shown in Example 4

  53. [61]

    Repetition: The quantized VLM keeps repeating some certain tokens, as shown in Example 1 and Example 6

  54. [62]

    Condition Missing: The quantized VLM misses the key points in the questions as shown in Example 3

  55. [63]

    No Output

    Semantic Error: The quantized VLM can understand the questions but still generates wrong answers with meaningful and fluent language, as shown in Example 5. We summarize the number of samples corresponding to the above five error types for each quantized VLM, the re- sults are...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.