Pith. sign in

REVIEW 3 major objections 7 minor 13 references

For a fixed GPU budget, a larger 4-bit vision-language model beats a smaller full-precision one on both accuracy and the confidence signal you need to defer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 14:52 UTC pith:DAYS4B5E

load-bearing objection Clean three-arm bake-off with a usable deployment takeaway; the AUROC gaps are directionally solid but rest on thin error counts, so treat the exact magnitudes as provisional. the 3 major comments →

arxiv 2607.24440 v1 pith:DAYS4B5E submitted 2026-07-27 cs.CV cs.CLcs.LG

Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation

M M Asif Ferdous This is my paper
classification cs.CV cs.CLcs.LG
keywords vision-language modelsuncertaintyquantizationmodel scaleselective predictionerror-detection AUROCimage degradationverbalized confidence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When a vision-language model runs on a phone or a cheap GPU, images arrive blurred, dark, or compressed, and the system must know when to answer and when to hand off. A practitioner with fixed memory can pick a small full-precision model, the same model quantized, or a larger model squeezed into the same footprint. This paper measures those three choices on the same degraded photos and shows that scale sharply improves the model’s internal token-probability signal for detecting its own errors, while the confidence number it writes in plain language stays weak and often near chance. Four-bit quantization barely hurts accuracy but damages both the internal signal and the model’s willingness to emit a parseable confidence. The practical upshot is clear: spend the memory budget on parameters, not precision, and defer on mean token probability rather than stated confidence—except under severe low light, where no configuration is reliable alone.

Core claim

Across 5,700 predictions on six photographic degradations, scaling Qwen2-VL from 2B to 7B at fixed 4-bit precision lifts mean internal error-detection AUROC from 0.80 to 0.98 while verbalized confidence only moves from 0.61 to 0.69, so the gap between what the model knows and what it says widens with size. Quantizing the 2B model from fp16 to 4-bit costs only 1.6 accuracy points but drops internal AUROC from 0.95 to 0.80 and collapses the verbalized parse rate from 99% to 64%. Of the three configurations that fit a 16 GB GPU, 7B-4bit dominates on accuracy and on the internal uncertainty signal.

What carries the argument

Error-detection AUROC on two confidence signals—verbalized integer confidence elicited by prompt, and internal mean token probability over the answer span—evaluated under controlled scale-only and quantization-only contrasts, then turned into selective-prediction risk–coverage curves and clean-calibrated threshold transfer under degradation.

Load-bearing premise

The scale and quantization effects measured here transfer beyond one model family, one food-photo multiple-choice task, and roughly one hundred images per condition.

What would settle it

Repeat the same three-arm design on another open VLM family and an open-ended VQA benchmark with larger n; if a larger quantized model no longer beats a smaller full-precision one on internal error-detection AUROC and threshold-transfer safety, the deployment claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Under a fixed memory budget, prefer a larger 4-bit VLM over a smaller full-precision one for both accuracy and deferral.
  • Use mean token probability over the answer span as the deferral signal; do not rely on stated confidence regardless of model size.
  • A deferral threshold calibrated on clean images with the larger quantized model’s internal signal can hold under degradation by cutting coverage rather than admitting errors.
  • Error-detection AUROC, not calibration error, is the metric that reveals whether a confidence signal can actually rank correct vs incorrect answers.
  • Severe underexposure still collapses accuracy and weakens the internal signal, so an upstream image-quality check is required in addition to confidence thresholding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If verbalized confidence is a fluent surface behavior rather than a truthful self-report, training or post-hoc methods that only improve parse rate or calibration of stated scores may not close the internal–verbalized gap.
  • The same budget logic may apply to other edge multimodal stacks (audio-language, document VLM) where practitioners currently default to the smallest full-precision checkpoint.
  • Quantization-aware training or calibration that specifically preserves token-level ranking, not just accuracy, could recover the internal AUROC lost at 4-bit without giving up the memory win.
  • Forced multiple-choice may inflate absolute accuracy and mask refusal behavior; open-ended settings could widen the practical value of a strong internal deferral signal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper studies how two practical levers — model scale (2B vs. 7B) and 4-bit NF4 quantization — affect two confidence signals in Qwen2-VL (verbalized numeric confidence and mean answer-span token probability) under six photographic degradations at three severities, on 100 Food-101 images posed as four-way multiple choice. Three arms (2B-fp16, 2B-4bit, 7B-4bit) run on identical images/prompts under a ~16 GB memory budget, yielding orthogonal scale and quantization contrasts. Headline findings: scale raises mean internal error-detection AUROC from 0.80 to 0.98 while verbalized AUROC stays weak (0.61→0.69); quantization costs little accuracy (−1.6 pts) but degrades the internal signal (0.95→0.80) and collapses the verbalized parse rate (0.99→0.64); hence the deployment recommendation to prefer 7B-4bit and to defer on internal token probability. A clean-calibrated threshold-transfer experiment and a risk–coverage analysis operationalize the recommendation, and the paper argues error-detection AUROC, not ECE, is the diagnostic metric for near-constant confidence signals.

Significance. If the quantitative gaps hold up under paired analysis, the paper makes a useful and non-obvious contribution: the first precision-controlled measurement of quantization's effect on VLM confidence signals, evidence that the internal/verbalized divergence is not repaired by scale (contradicting a plausible small-model-artifact reading), and a concrete, falsifiable deployment recommendation for the common 16 GB budget scenario. The methodological point that ECE certifies near-constant, non-discriminative confidence while AUROC exposes it is well argued with the arm-A example. The work ships full reproducibility (code, per-prediction CSVs, item manifest, greedy decoding, fixed seeds), and its design choices — answer-span-restricted internal confidence to avoid circularity, bare-label scoring to avoid conflating capability with format compliance — are careful. Significance is bounded by the single model family, single dataset, and four-way forced-choice format, all acknowledged in §6.

major comments (3)
  1. [§4.2–4.4, Table 2, §3.7] The headline numbers are differences in mean error-detection AUROC between arms (quantization: 0.95→0.80, §4.3; scale: 0.80→0.98, §4.2; dominance in Table 2 / RQ3), but the evidence for these differences is weaker than the per-condition CIs suggest. With n≈100 items per condition (§3.3) and per-condition accuracy of 0.89–0.94 (Table 2), each per-condition AUROC is estimated against roughly 6–11 incorrect examples, and near-ceiling conditions are dropped as n/a (§3.7), preferentially removing the cells where arms are most likely to agree. On ~10 negatives, re-ranking one or two shared items moves a condition's AUROC by ~0.1, so the claim-defining 0.15–0.18 mean gaps could be driven by a small number of items. The paper reports percentile bootstrap CIs per estimate (§3.7) and concedes in §6 that some intervals are wide, but never reports a paired test of the arm differences. Because all th
  2. [§4.3, Table 2, §3.5] The verbalized AUROC for arm B (0.61, Table 2) is computed only on the 64% of replies that parsed (parse rate collapse, §4.3). If parse failure correlates with degraded or otherwise hard inputs — which is plausible, since the parse rate is itself reported as condition-dependent — then 0.61 is estimated on a non-representative subset, and the direction of the resulting bias in the internal–verbalized gap (and in the 'gap widens with scale' claim of §4.2, which contrasts 0.19 at 2B with 0.29 at 7B where parsing is 100%) is unknown. A standard fix is to score non-parsed replies as minimal/zero confidence (no stated confidence = defer) and recompute the verbalized AUROC on the full prediction set, or at minimum to report the parse rate per condition and show whether failures concentrate in the high-severity conditions. As written, the cross-arm verbalized comparison mixes signal quality with
  3. [§4.4, Table 3] The threshold-transfer experiment fits a single clean-image operating point (answer 90%) and reports three conditions per arm in Table 3, while the text (§4.4) says the threshold was applied to 'every condition'. The zero-admitted-errors result for 7B-4bit rests on very small denominators (coverage 0.39–0.44 of n≈100, i.e. ~40 answered items) and on one operating point chosen ad hoc. Please (i) report all 19 conditions or state the selection rule for the rows shown, (ii) give a confidence statement on the 0% admitted-error rates (e.g., an upper binomial bound), and (iii) show sensitivity to at least one other operating point (e.g., 70% or 80% clean coverage), since the RQ3 deployment claim — that the 7B threshold 'never admits errors' — is the paper's most actionable and most fragile statement.
minor comments (7)
  1. [Abstract, §3.4, §4.2] Rendering artifacts in the abstract and §4.2: 'modelstatesin', 'model'sinternaluncertainty', 'model'sverbalizedconfidence', '5,700across'. Presumably a PDF-extraction issue, but worth checking the source.
  2. [§3.5 vs. Appendix A] Appendix A states that for bare-label replies 'the internal confidence ... is the mean token probability over the full generated span,' whereas §3.5 (Eq. 1) defines c_int over the answer span only. For bare replies the full span presumably equals the answer, but this should be stated explicitly, since the answer-span restriction is presented as essential to avoid circularity.
  3. [§3.2] The 32-token generation cap (§3.2) may interact with the bare-label/format-failure behaviour of the 2B arms (truncated two-line replies would parse as failures). A sentence confirming the cap was never (or rarely) binding, e.g. from the released CSVs, would close this loophole.
  4. [Figures 1, 3] Figure 1: the colour legend (dark blue = 7B internal, red = verbalized) is described only in the caption; the two arms' verbalized curves are hard to distinguish. Consider direct labelling. Figure 3's pooled-vs-mean AURC discrepancy is honestly flagged in the caption but would be cleaner as a footnoted number.
  5. [Table 2] Table 2 would benefit from the number of conditions actually contributing to each AUROC mean (after n/a exclusion, §3.7), since the counts differ across arms and signals.
  6. [Appendix A] The claim that the two elicitation formats 'agree on 100% accuracy' (App. A) is reassuring but n=20 clean items is small; consider softening to 'agree on all 20 pilot items'.
  7. [§2] Missing related work: recent studies of token-probability vs. verbalized confidence divergence in LLMs (e.g., post-2023 elicitation-robustness literature) would strengthen the §2 framing beyond Xiong et al. [13] and Tian et al. [11].

Circularity Check

0 steps flagged

Empirical bake-off with external metrics; no circular derivation chain.

full rationale

The paper is a controlled three-arm empirical comparison (2B-fp16, 2B-4bit, 7B-4bit) of accuracy and two confidence signals under image degradation. Load-bearing quantities—accuracy, error-detection AUROC, AURC, ECE, Brier, parse rate, and selective-prediction operating points—are standard external metrics computed on labeled Food101 items with fixed gold answers. Internal confidence is the mean token probability over the generated answer span (Eq. 1); verbalized confidence is a parsed integer from model text. Neither is defined in terms of the other, and no free parameter is fitted to a subset of outcomes and then reported as an independent prediction of a closely related quantity. Contrasts isolate scale (B vs C at fixed precision) and quantization (A vs B at fixed size) on identical inputs. Citations are to external prior work (Xiong, Tian, Guo, Hendrycks, etc.); there is no self-citation chain, uniqueness theorem, or ansatz imported from the same author that forces the result. Ceiling conditions are dropped as n/a rather than redefined. The deployment recommendation is a ranking of measured operating points, not a first-principles derivation. Small-N AUROC instability and limited generality are statistical/scope concerns, not circularity. Finding: no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

Load-bearing content is empirical protocol, not new physical entities or fitted laws. The claim rests on standard ML evaluation axioms, domain assumptions about what counts as a usable deferral signal and a fair memory-budget comparison, and a few study-specific operational definitions (answer-span mean token probability, NF4 4-bit, Food101 MCQ, chosen elicitation template).

free parameters (4)
  • Elicitation template choice (few-shot two-line format) = few-shot template (highest parse rate)
    Selected by pilot parse rate on 20 clean items per size; verbalized results depend on this prompt choice even though alternatives are reported in Appendix A.
  • Deferral operating point for threshold transfer = 90% clean coverage
    Clean-image threshold set to answer 90% of clean inputs; transfer conclusions depend on this coverage target.
  • Internal confidence aggregation = arithmetic mean over answer-span tokens
    Mean token probability over the answer span (Eq. 1); alternative aggregations (min, geometric mean, full sequence) are not swept.
  • Degradation severity parameter grid = three severities per six families as in Table 1
    Hand-chosen JPEG qualities, blur radii, brightness/noise factors, etc. (Table 1) define the shift distribution on which AUROC gaps are measured.
axioms (6)
  • domain assumption Error-detection AUROC of a confidence score is the right primary criterion for whether that score is usable for deferral/selective prediction.
    Stated in §1, §3.7, and §4.5; used to elevate AUROC over ECE when confidence variance is near zero.
  • domain assumption Mean probability of generated answer tokens is a valid internal confidence signal comparable across precision settings.
    Definition Eq. (1) in §3.5; quantization is interpreted as perturbing this distribution.
  • ad hoc to paper A 2B-fp16 vs 2B-4bit vs 7B-4bit trio fairly represents the fixed ~16 GB consumer-GPU budget choice.
    Framing in §1 and Arm table in §3.1; other footprints, KV-cache costs, or 8-bit settings are not compared.
  • domain assumption Food-101 four-way multiple choice with unique-containment matching is an adequate proxy for degraded-image VLM decision quality.
    Data and matching in §3.3–3.6; limitations note open-ended VQA is untested.
  • standard math Bootstrap percentile intervals with B=2000 and exclusion of near-ceiling conditions with undefined AUROC yield trustworthy aggregate comparisons.
    Metrics §3.7; aggregates omit n/a conditions.
  • domain assumption Greedy decoding with a 32-token cap does not materially distort relative confidence-signal quality across arms.
    Inference setup §3.2.

pith-pipeline@v1.2.0-grok45-kimik3 · 14081 in / 3791 out tokens · 69599 ms · 2026-07-31T14:52:15.630748+00:00 · methodology

0 comments
read the original abstract

Vision-language models (VLMs) deployed on consumer hardware must decide when to answer and when to defer, and that decision depends on having a confidence signal that tracks correctness. A practitioner with a fixed memory budget faces a choice between a small model at full precision, the same small model quantized, and a larger model quantized into the same footprint -- three configurations that push the confidence signal in opposing directions. We measure, on identical inputs, how model scale and 4-bit quantization affect two confidence signals in the Qwen2-VL family: the confidence a model states in natural language, and its own mean token probability over the answer it generates. Across 5,700 predictions spanning six realistic photographic degradations at three severities, we find that scale sharply improves the model's internal uncertainty signal (mean error-detection AUROC 0.80 to 0.98 from 2B to 7B) while its verbalized confidence stays weak and often at chance (mean 0.61 to 0.69): the gap between what the model knows and what it says widens rather than closes with size. We find that 4-bit quantization is nearly free for accuracy (-1.6 points) but expensive for the confidence signal (internal AUROC 0.95 to 0.80, and the verbalized-confidence parse rate collapses from 99% to 64%). For a fixed memory budget the recommendation is therefore to prefer a larger quantized model over a smaller full-precision one: 7B-4bit gives both the best accuracy and the best uncertainty signal (internal AUROC 0.98) of the three configurations that fit. We frame the results as selective-prediction operating points so they translate directly into a deployment recommendation, and we argue that error-detection AUROC, not calibration error, is the metric that exposes the difference between the two signals.

Figures

Figures reproduced from arXiv: 2607.24440 by M M Asif Ferdous.

Figure 1
Figure 1. Figure 1: Error-detection AUROC per condition, 2B vs 7B (both 4-bit). The 7B internal signal (dark blue) [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Quantization effect at 2B (means over 19 conditions). Accuracy is nearly unchanged from fp16 to [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Risk–coverage by internal confidence. 7B-4bit (red) answers the most confident [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy under increasing low light. Scale softens but does not remove the collapse; all arms fall [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 6 linked inside Pith

  1. [1]

    Know what you do not know: Verbalized uncertainty estimation robustness on corrupted images in vision-language models.arXiv preprint arXiv:2504.03440, 2025

    Mirko Borszukovszki, Ivo Pascal de Jong, and Matias Valdenegro-Toro. Know what you do not know: Verbalized uncertainty estimation robustness on corrupted images in vision-language models.arXiv preprint arXiv:2504.03440, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Food-101 – mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. InComputer Vision – ECCV 2014, volume 8694 ofLecture Notes in Computer Science, pages 446–461. Springer, 2014. 11

    2014

  3. [3]

    Glenn W. Brier. Verification of forecasts expressed in terms of probability.Monthly Weather Review, 78 (1):1–3, 1950

    1950

  4. [4]

    Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models.arXiv preprint arXiv:2405.02917, 2024

    Tobias Groot and Matias Valdenegro-Toro. Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models.arXiv preprint arXiv:2405.02917, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InInternational Conference on Machine Learning (ICML), pages 1321–1330. PMLR, 2017

    2017

  6. [6]

    Benchmarking neural network robustness to common corrup- tions and perturbations.arXiv preprint arXiv:1903.12261, 2019

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations.arXiv preprint arXiv:1903.12261, 2019

    Pith/arXiv arXiv 1903

  7. [7]

    Ecker, Matthias Bethge, and Wieland Brendel

    Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming.arXiv preprint arXiv:1907.07484, 2019

    Pith/arXiv arXiv 1907

  8. [8]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning.Proceedings of the AAAI Conference on Artificial Intelligence, 29 (1):2901–2907, 2015

    2015

  9. [9]

    Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.Advances in Neural Information Processing Systems, 32, 2019

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.Advances in Neural Information Processing Systems, 32, 2019

    2019

  10. [10]

    Interpreting the effects of quantization on LLMs

    Manpreet Singh and Hassan Sajjad. Interpreting the effects of quantization on LLMs. InProceedings of the 2025 Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL), 2025. arXiv:2508.16785

    arXiv 2025

  11. [11]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.arXiv preprint arXiv:2305.14975, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. InThe Twelfth International Conference on Learning Representations (ICLR), 2024. 12

    2024