{"id":"1b829f78-e92c-47c0-846f-dd4cfe6f1289","arxiv_id":"2412.19509","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A modality-weighted quantization method improves accuracy of 3-bit and 4-bit vision-language models by protecting sensitive language tokens during calibration.","lead":"Researchers found that text tokens are much more sensitive than image tokens when compressing vision-language models, and used that difference to build MBQ, a quantization method that cuts accuracy loss at 3- and 4-bit precision. It could make 7B to 70B multimodal AI models cheaper to run on GPUs without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main-table gains may be inflated by calibration-set mismatch: MBQ uses 128 COCO caption pairs while AWQ/GPTQ/SmoothQuant baselines use Pile text calibration; same-calibration evidence is limited to one 7B model and two benchmarks.","rationale":"I read the paper in good faith. The modality-balance idea is plausible, and Table 4's comparison of COCO-calibrated CWE with and without modality weighting is the right kind of isolation experiment. The concern I find most load-bearing is not that the authors concealed the calibration effect—they explicitly discuss it in Section 4.3.1—but that the headline numerical claims in Tables 2 and 3 are computed against baselines calibrated on a different, text-only distribution. The controlled evidence in Table 4 covers one 7B model and two benchmarks, and one of those cells (W4A8/SEED) shows that switching to COCO calibration alone can cause a catastrophic drop in the baseline, so MBQ's large improvement there may be specific to repairing that collapse. This is a fixable experimental-design gap rather than a fatal flaw, so the reader's CONDITIONAL verdict remains appropriate. I partially agree with the reader: their rationale flags the calibration mismatch, but their stated weakest assumption is about the reliability and transferability of the gradient-based sensitivity weights, which is a related but distinct concern. The proposed test—rerunning all baselines with MBQ's calibration set—would settle whether the central numerical claim is robust.","tokens_in":21668,"tokens_out":9873,"duration_ms":101155,"concrete_test":"Reproduce the main results in Tables 2 and 3 using the exact MBQ calibration set (128 COCO caption pairs) for AWQ, GPTQ, and SmoothQuant, keeping their original optimization objectives and, wherever feasible, the same CWE search code as MBQ minus the modality weights. Compute per-model/per-bitwidth average deltas over the six benchmarks. If most deltas drop below about 1%, the advertised 4.4%/11.6% margins are largely a calibration-set artifact; if the deltas remain comparable to the Table 4 pattern (large on LLaVA-onevision, small on Qwen2-VL), the modality-balance claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1.1 specifies MBQ calibration on 128 ShareGPT4V COCO image-caption pairs; Section 4.3.1 states AWQ and SmoothQuant baselines use the Pile (language-only) calibration set. Tables 2 and 3 therefore compare MBQ against baselines that differ both in calibration content and calibration modality, so the headline improvements ('up to 4.4%' under W3A16 and 'up to 11.6%' under W4A8) are not controlled for the calibration set. Table 4 is the only same-calibration ablation: for LLaVA-onevision-7B it compares COCO-calibrated CWE without modality weighting ('-') against MBQ on MMMU and SEED. That ablation supports the modality-balance mechanism for that one model, but it does not establish the advertised margins over SOTA baselines across six models and six benchmarks. The W4A8/SEED cell is especially fragile: COCO-calibrated CWE without balance collapses to 10.2, versus SmoothQuant's 41.6 with Pile calibration, so MBQ's large gain there may mostly reflect repairing a calibration-induced collapse rather than a general modality-balancing benefit. Without a full same-calibration comparison sweep, the central empirical claim remains underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Modality-Balanced Quantization (MBQ) for post-training quantization of large vision-language models. Observing in Fig. 1 that the average absolute gradient of the SFT loss with respect to language-token features is an order of magnitude larger than for vision tokens, the authors weight the block-wise reconstruction loss by per-modality average absolute gradients during calibration, yielding a per-layer balanced objective for weight-only (W3A16, W4A16) and weight-activation (W4A8, W8A8) quantization. The main results report accuracy improvements over AWQ, GPTQ, and SmoothQuant of up to 4.4% and 11.6% on W3A16 and W4A8, respectively, across 7B-72B LLaVA, InternVL2, and Qwen2-VL models, plus a fused W3 GEMV kernel that achieves up to 1.4x end-to-end decoding speedup.","tokens_in":22005,"tokens_out":7084,"duration_ms":63625,"significance":"If the claimed improvements are robust, the paper makes a useful contribution by identifying modality imbalance as a distinct failure cause in VLM quantization and by proposing a simple, generalizable weighting scheme that plugs into CWE-based PTQ methods. The same-calibration ablation in Table 4, where adding MAE modality-balance to a COCO-calibrated CWE baseline improves MMMU from 38.7 to 42.0 (W3A16) and from 29.2 to 42.6 (W4A8), is the strongest evidence that the modality-weighting mechanism, rather than calibration choice, drives the gain on LLaVA-onevision-7B. The paper also ships code and a practical W3 kernel, which are clear strengths. However, as detailed below, the headline comparisons with Pile-calibrated baselines are not controlled for calibration data, and the theoretical derivation in Sec. 3.2 contains an invalid inequality. These issues are fixable with additional experiments and a reframing of the theoretical claim.","major_comments":[{"comment":"The main results in Tables 2 and 3 compare MBQ, calibrated on 128 ShareGPT4V COCO image-caption pairs, with AWQ, GPTQ, and SmoothQuant calibrated on the Pile (language-only) set. This confounds the method with the calibration set. The only same-calibration evidence is Table 4, for LLaVA-onevision-7B on MMMU and SEED. In that table, COCO-calibrated CWE without balance collapses on W4A8/SEED (10.2 vs. SmoothQuant's 41.6 with Pile), and MBQ restores it; this supports the modality-balance mechanism for one model, but it does not establish the advertised 'up to 4.4%/11.6%' margins over SOTA baselines across six models and six benchmarks. The paper's own Sec. 4.3.1 reports that switching SmoothQuant to COCO calibration degrades SEED by 31.4%, so the main-table comparisons conflate calibration choice with modality balancing. A full same-calibration comparison sweep across all models and benchmarks is needed to support the headline claim.","section":"Sec. 4.1.1 / Sec. 4.3.1, Tables 2-4"},{"comment":"The derivation of the balanced loss is not mathematically valid as written: the inequality ||g_v^T * Δ_v|| <= average_abs(g_v) * ||Δ_v|| in Eq. (12) does not follow when average_abs(g_v) is the average absolute gradient over tokens rather than a matrix norm or a per-element maximum. Consequently, Eqs. (14)-(17) are not actually derived from the first-order Taylor expansion in a rigorous sense. The empirical results can stand on their own, but the current text overstates the theoretical grounding; please either correct the bound or present the objective as a heuristic importance-weighting scheme.","section":"Sec. 3.2, Eqs. (8)-(13)"},{"comment":"The entire method rests on the assumption that the per-modality average absolute gradient of the SFT loss, computed on a fixed 128-pair COCO set, is a reliable and transferable per-layer importance weight. The paper shows only one illustrative layer (Fig. 1) and does not report the layer-wise variation or model-to-model variation of the vision/language gradient ratio, nor the fraction of zero-gradient vision tokens (which Sec. 4.3.2 indicates is large). Without such statistics, it is unclear whether the weighting is stable across layers and models, and whether the average is a sufficient statistic. Please add a sensitivity analysis of the gradient ratios.","section":"Sec. 4.1.1 and Sec. 3.2"}],"minor_comments":[{"comment":"The abstract says 'under W3 and W4A8 quantization'; please use 'W3A16 and W4A8' for precision.","section":"Abstract"},{"comment":"The expression for Sasym is ambiguous; use parentheses, e.g., Sasym = (max(W_FP16) - Z) / (2^N - 1).","section":"Sec. 2.1, Eq. (2)"},{"comment":"The 'up to 4.4% and 11.6%' figures in the abstract should be checked against Sec. 4.2, which reports an 18.5% average improvement on LLaVA-onevision-72B W3A16; please make the reported 'up to' figures consistent across the paper.","section":"Abstract / Sec. 4.2"},{"comment":"The SEED score for RTN W3A16 on LLaVA-onevision-7B (5.9) is dramatically lower than all other cells in that row; please verify this entry or explain the collapse.","section":"Table 2"},{"comment":"The two additional reweighting strategies (random token grouping and token-wise gradients) are described without reporting the OCRBench numbers; please include those values in the text or in a table.","section":"Sec. 4.3.2"},{"comment":"Define the notation explicitly: let average_abs(g_v) = (1/N_v) * sum_{i=1}^{N_v} |g_{v,i}|, where N_v is the number of vision tokens.","section":"Sec. 3.2, Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The calibration-set mismatch is the primary correctness risk for the paper's headline claim. I would require a full same-calibration comparison sweep across all models and benchmarks before publication, and I would ask the authors to either correct the inequality in Sec. 3.2 or reframe the derivation as a heuristic. The paper is otherwise promising: the mechanism is clear, Table 4 provides a strong same-calibration ablation for one model, the kernel work is practical, and the code is released."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the core observation is the right one: vision and language tokens in VLMs have very different sensitivity to feature perturbation, and a block-wise calibration that ignores this will over-invest in the robust modality. The gradient visualization in Fig. 1 is convincing, and the oracle experiment with factor 0.1 shows the effect is real. The method itself, weighting the reconstruction loss by per-modality average absolute gradients, is a clean extension of CWE and easy to implement.\n\nThe strongest evidence is Table 4: on LLaVA-onevision-7B, switching from COCO-calibrated CWE without balancing to COCO-calibrated MBQ lifts MMMU from 38.7 to 42.0 (W3A16) and from 29.2 to 42.6 (W4A8). That isolates the modality-balancing mechanism from the calibration set. But the table covers one model and two benchmarks. The main results in Tables 2 and 3 compare MBQ, calibrated on 128 COCO caption pairs, against AWQ and SmoothQuant calibrated on Pile text. That is a double confound: different content and different modality. The stress-test note is right to call this out. In particular, the W4A8/SEED collapse of COCO-calibrated CWE to 10.2 suggests that without MBQ, COCO calibration can hurt badly; the headline gains over Pile-calibrated SmoothQuant are partly repairing that collapse. The AWQ disaster on LLaVA-72B W3A16 (33.4 MMMU vs RTN 53.9) is probably the same story.\n\nThe derivation in Sec. 3.2 is not rigorous. The step from ||g_v^T Δ_v + g_l^T Δ_l|| to \\bar{|g_v|} ||Δ_v|| + \\bar{|g_l|} ||Δ_l|| is not an inequality in the direction claimed; averaging absolute gradients and multiplying by a norm does not bound an inner product. The MAE weighting is therefore a reasonable heuristic, not a derived optimum. That is fine, but the authors should say so.\n\nThe GPU kernel and end-to-end speedup are straightforward systems work, solid but secondary.\n\nBottom line: this paper deserves a serious referee. The idea is timely, the one controlled ablation is encouraging, and the systems piece is real. But the main empirical claims need a same-calibration sweep across all models and benchmarks, and the derivation should be corrected or reframed as heuristic. If those are addressed, it could be a useful contribution to efficient VLM deployment. I'd take it to a reading group to discuss the proper way to control calibration in PTQ comparisons.","headline":"Modality-balanced calibration is a real idea with one clean ablation, but the main accuracy claims are confounded by calibration-set mismatch; fixable with a same-calibration sweep.","tokens_in":22548,"tokens_out":3868,"would_cite":true,"duration_ms":36839,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that quantizing large vision-language models loses accuracy because calibration treats vision and language tokens equally, and that weighting each modality by its measured sensitivity (average absolute gradient of the SFT…","keywords":["post-training quantization","vision-language models","modality balancing","channel-wise equalization","gradient sensitivity","weight-only quantization","weight-activation quantization","3-bit quantization"],"falsifier":"Compute the per-layer gradient ratios on a different calibration set (for example, VQA or chart-understanding pairs), re-estimate the MBQ weights, and compare accuracy against the original COCO-derived weights: if the weights shift substantially yet produce worse accuracy than a fixed ratio, or if the improvement vanishes when the calibration set changes, the sensitivity measure is not the load-bearing mechanism. Conversely, an experiment that randomizes the per-modality weights (such as swapping vision and language weights) and shows the accuracy gain disappears would confirm that the gradient-based balancing itself is what drives the result.","tokens_in":1815,"feed_emoji":"⚖️","tokens_out":1870,"duration_ms":65803,"temperature":0.7,"pith_summary":"The paper identifies a first-order cause of accuracy loss when post-training quantizing large vision-language models: vision tokens and language tokens are not equally sensitive to quantization error, yet standard calibration treats them the same. Using a first-order Taylor bound, it shows the loss increase from a quantized layer is controlled by the product of quantization error and the average absolute gradient of the SFT loss per token type. It then proposes MBQ, which weights the vision and language portions of the block-wise reconstruction loss by these average gradients when searching for channel-wise equalization factors. On seven- to seventy-billion-parameter models, MBQ improves task accuracy over strong baselines by up to 4.4% under W3A16 and 11.6% under W4A8 quantization, and it ships a fused 3-bit dequantization-GEMV kernel for a 1.4x decoding speedup. A reader should care because it suggests that modality imbalance, not merely activation outliers, is a primary obstacle to low-bit VLM deployment.","feed_headline":"Per-modality calibration lifts quantized VLM accuracy by 11.6%","feed_subtitle":"Weighting vision vs language tokens by gradient sensitivity recovers accuracy on 7B-70B VLMs at 3-4 bit.","key_machinery":"The load-bearing object is the modality-balanced reconstruction loss: instead of minimizing $\\|Q(WE)(E^{-1}X)-WX\\|^2$ over all tokens equally, MBQ minimizes $\\bar{|g_v|}\\,\\|WX_v-Q(WE)(E^{-1}X_v)\\| + \\bar{|g_l|}\\,\\|WX_l-Q(WE)(E^{-1}X_l)\\|$ (with absolute error, not squared), where $\\bar{|g_v|}$ and $\\bar{|g_l|}$ are average absolute gradients of the SFT loss w.r.t. the layer's vision and language output tokens. This is derived from a first-order Taylor bound on the loss change, Eqs. (8)-(13), and the weights are computed once per layer from a small calibration set (128 image-caption pairs) and reused as constant weights during the equalization-factor search. The MAE form is chosen because it derives directly from the bound and outperforms an MSE-weighted variant in ablations.","core_discovery":"The central claim is that the sensitivity of a VLM's output to feature perturbation differs by an order of magnitude between vision and language tokens, and that calibration objectives which ignore this imbalance systematically over-budget quantization error. For a linear layer, MBQ bounds the change in SFT loss by $\\bar{|g_v|} \\cdot \\|Y_v - \\hat Y_v\\| + \\bar{|g_l|} \\cdot \\|Y_l - \\hat Y_l\\|$, where $\\bar{|g|}$ are average absolute gradients of the loss w.r.t. the output tokens of each modality. It therefore minimizes a reconstruction loss in which each modality's error is weighted by its measured sensitivity, for both weight-only and weight-activation settings. With these weights, the method consistently outperforms equal-treatment baselines such as AWQ and SmoothQuant on MMMU, SEED, OCRBench, VizWiz, ScienceQA, and TextVQA across LLaVA-onevision, InternVL2, and Qwen2-VL families.","pith_inferences":["If the gradient-ratio story is causal, the best calibration dataset for VLM quantization may be the one that matches the deployment distribution of vision versus language token counts, not merely a high-quality caption set; that is a testable hypothesis the paper does not run.","The same per-modality weighting principle could extend to other compression steps, such as pruning or KV-cache quantization, where sensitivity also differs by token type.","The 128-sample COCO caption calibration may underestimate vision-token sensitivity on tasks where the image is the primary answer source (for example, fine-grained OCR or diagram reading); an experiment re-estimating the per-layer weights on task-specific data would show whether the fixed weights transfer.","Since the method averages gradients over tokens, layers where vision gradients are near zero may make the balanced objective degenerate; the paper notes this for token-wise weighting, and the same caution applies to the per-modality average."],"forward_implications":["If MBQ is right, applying unmodified LLM PTQ methods to VLMs will keep leaving accuracy on the table, because the equal-treatment objective is a systematic bias, not just noise.","Quantizing the ViT encoder alongside the VLM is feasible at W4A8 with little or no drop, since vision tokens are the more redundant modality.","MBQ is a plug-in: it can be combined with rotation-based quantizers (MBQ (Rot)) to bring W4A8 large VLMs within about one point of FP16 accuracy.","Any future equalization-based PTQ method applied to multimodal models should incorporate per-modality weighting to avoid over-emphasizing the insensitive modality.","A fused W3 weight-only kernel can accelerate VLM decoding by 1.4x on an RTX 4090, making low-bit multimodal serving practical."],"supporting_citations":[{"why":"Supplies the channel-wise equalization objective and the weight-only baseline (AWQ) that MBQ modifies.","marker":"[27]"},{"why":"Supplies the weight-activation quantization baseline (SmoothQuant) and the equalization recipe for activations.","marker":"[43]"},{"why":"Provides the COCO caption dataset used for calibration and for the gradient-sensitivity measurement.","marker":"[10]"},{"why":"Provides the improved COCO captions (ShareGPT4V) that form the instructional calibration set.","marker":"[8]"},{"why":"One of the three model families evaluated (LLaVA-onevision), used for both small and large scale tests.","marker":"[21]"},{"why":"Defines the SFT loss whose gradients serve as the sensitivity indicators (visual instruction tuning framework).","marker":"[31]"},{"why":"The rotation-based quantization method combined with MBQ to form MBQ (Rot) for large-model experiments.","marker":"[41]"},{"why":"Provides the open-source W4A8 kernel used in the end-to-end speedup measurements.","marker":"[29]"}],"fun_headline_variants":["Modality-balanced quantization lifts VLM accuracy up to 11.6%","Sensitivity-weighted calibration boosts quantized VLM accuracy by 11.6%","Gradient-weighted token calibration improves low-bit VLMs","Per-modality gradient weighting yields 11.6% gain in quantized VLMs","Vision-language token sensitivities guide quantization for better accuracy"],"cache_read_input_tokens":24576,"weakest_assumption_plain":"The method assumes that the average absolute gradient of the SFT loss, computed over just 128 image-caption pairs from a single dataset, gives a stable and transferable per-layer measure of how much each modality's tokens matter to the final answer, across all layers and model families.","fun_headline_variants_meta":{"raw":{"variants":["Modality-balanced quantization lifts VLM accuracy up to 11.6%","Sensitivity-weighted calibration boosts quantized VLM accuracy by 11.6%","Gradient-weighted token calibration improves low-bit VLMs","Per-modality gradient weighting yields 11.6% gain in quantized VLMs","Vision-language token sensitivities guide quantization for better accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000985,"raw_usage":{"total_tokens":4228,"prompt_tokens":1041,"completion_tokens":3187,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":3094}},"tokens_in":657,"tokens_out":3187,"duration_ms":23200,"temperature":1.0,"reasoning_tokens":3094,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:16:26.564856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the per-layer gradient ratios on a different calibration set (for example, VQA or chart-understanding pairs), re-estimate the MBQ weights, and compare accuracy against the original COCO-derived weights: if the weights shift substantially yet produce worse accuracy than a fixed ratio, or if the improvement vanishes when the calibration set changes, the sensitivity measure is not the load-bearing mechanism. Conversely, an experiment that randomizes the per-modality weights (such as swapping vision and language weights) and shows the accuracy gain disappears would confirm that the gradient-based balancing itself is what drives the result.","supporting_citations":[{"cited_title":"Awq: Activation-aware weight quantization for on-device llm compression and acceleration","cited_arxiv_id":null,"evidence_quote":"Supplies the channel-wise equalization objective and the weight-only baseline (AWQ) that MBQ modifies."},{"cited_title":"Smoothquant: Accurate and effi- cient post-training quantization for large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the weight-activation quantization baseline (SmoothQuant) and the equalization recipe for activations."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Defines the SFT loss whose gradients serve as the sensitivity indicators (visual instruction tuning framework)."}],"review_version":1}