{"id":"81d43742-d640-4217-9922-030e83d1cf4f","arxiv_id":"2505.03788","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Combining self-consistency scores with a grounding model's confidence, scaled by a fitted power and offset, reduces expected calibration error for LLaVA and LLaVA-Med on VQAv2 and Slake.","lead":"This paper introduces a way to make uncertainty scores from multimodal AI models more trustworthy by asking a second vision model to check whether the text answer matches the image. The method reduces calibration error on two visual question answering datasets, but its experiments lack a fair comparison and public code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed grounding advantage is confounded by the extra offset C in Eq. (2); until a same-complexity two-parameter recalibration baseline is compared, the central claim is not established.","rationale":"The reader's weakest assumption is exactly the control for the extra parameter C in Eq. (2), and the paper's experimental comparison does not include a same-complexity baseline. This is load-bearing because the central claim is comparative: grounding should beat temperature scaling, not just improve over an uncalibrated baseline. The absence of a two-parameter grounding-free baseline means the mechanism behind the reported ECE reductions is unidentified. The concern is not that the method cannot work; the grounding confidence is input-dependent, so it could genuinely help. It is that the evidence as presented does not establish it. The paper has some independent support from reliability diagrams and five-split variance reporting, but no code release and no sample sizes, so the conditional verdict is appropriate. If the proposed concrete test fails, the paper would need major revision or a re-framed claim; if it passes, the central claim would be substantially strengthened. Since the reader already assigned CONDITIONAL with medium confidence, my read does not change the verdict.","tokens_in":13988,"tokens_out":7034,"duration_ms":76466,"concrete_test":"Re-run the Slake SemEnt row of Table 2 with a two-parameter affine recalibration a*Conf_baseline + b, fitting a and b on the same validation split used to select T and C for Eq. (2), and compute ECE on the held-out test split. If the affine baseline achieves ECE close to the 0.008 reported with BiomedCLIP rather than 0.222, the claimed advantage of grounding is unsupported; if it remains above roughly 0.1, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.4 defines the only grounding-free calibration baseline as Conf_baseline^{1/T}, a one-parameter monotone transform. The proposed Eq. (2) adds a second fitted constant C, selected on the same validation set. Appendix Tables 3 and 4 show many of the best grounding results use C = 0.2 or 0.5, so part of the ECE reduction could come from an additive shift rather than from cross-modal grounding. The paper claims in Section 5.5 that grounding 'is much more effective than calibrating these confidence scores via temperature scaling', but this is not an apples-to-apples comparison. A two-parameter recalibration without grounding, e.g. a*Conf_baseline + b or a*Conf_baseline^{1/T} + b, fitted on the same validation splits, must be reported before the ECE drops (e.g., Slake SemEnt 0.376 to 0.008 with BiomedCLIP) can be attributed specifically to the grounding signal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a post-hoc calibration method for self-consistency-based uncertainty quantification in multimodal LLMs. Given a baseline confidence Conf_baseline and a grounding model's confidence Conf_GM, Eq. (2) defines calibrated confidence as Conf_baseline × Conf_GM^(1/T) + C, with T and C fitted on a validation split. The method is evaluated on VQAv2 with LLaVA and on Slake with LLaVA-Med, using four self-consistency baselines (LexSim, NumSets, PredEnt, SemEnt) and several grounding models (GroundedSAM, CLIP, LLaMA3.2V, QwenVL, BiomedCLIP, Biomed-QwenVL). The authors report large ECE reductions relative to the raw baselines and claim that grounding is much more effective than temperature scaling alone.","tokens_in":14162,"tokens_out":6177,"duration_ms":61790,"significance":"If the grounding advantage survives a same-complexity baseline, the paper offers a simple, training-free, model-agnostic way to improve calibration for multimodal LLMs, and it directly targets the important failure mode in which self-consistency is high but accuracy is low. The paper does not ship code, proofs, or released artifacts, and its central empirical comparison currently does not isolate the effect of cross-modal grounding from the effect of an extra fitted offset. The direction is timely and the proposed pipeline is plausible, but the evidence as presented is not yet sufficient to support the headline claim.","major_comments":[{"comment":"The central claim in §5.5 that grounding is 'much more effective than calibrating these confidence scores via temperature scaling' is confounded by model complexity. Eq. (2) has two fitted parameters, T and C, while the 'Without Grounding' baseline Conf_baseline^(1/T) has only T. Appendix Tables 3 and 4 show that many of the strongest grounding results use C = 0.2 or 0.5, so part of the ECE reduction may come from the additive offset rather than from cross-modal grounding. Please report a two-parameter grounding-free recalibration of the baseline, e.g., a·Conf_baseline + b or a·Conf_baseline^(1/T) + b, fitted and evaluated on the same validation/test splits, before attributing ECE drops such as Slake SemEnt 0.376→0.008 with BiomedCLIP to the grounding signal.","section":"§5.4, Eq. (2), Tables 1–2"},{"comment":"The proposed confidence can exceed 1: with C = 0.5 and a product term above 0.5, Conf is greater than 1, and the paper does not state whether or how the result is clipped or renormalized to [0,1]. ECE and reliability diagrams are only well-defined for confidence in [0,1], so the exact post-processing must be specified. If clipping is used, the effective transform is piecewise, which should be reflected in the description and in the baseline comparison.","section":"§4, Eq. (2)"},{"comment":"The empirical claim of 'significantly improved calibration' is supported only by mean ECE and variance across five random splits. The paper gives no confidence intervals, bootstrap intervals, or significance tests, and it does not state the number of test examples per condition; from the Introduction, the Slake sample appears very small (75 manually verified cases). Without this information, reductions such as 0.376 to 0.008 cannot be assessed for statistical reliability, and the comparison becomes even harder to interpret once the baseline is given the same number of fitted parameters. Add per-condition confidence intervals or paired tests and report the test-set sizes.","section":"§5.5, Appendix A.4"}],"minor_comments":[{"comment":"The caption says 'Comparison of ECE over accuracy of LLaVA for VQA', but Table 2 reports Slake results with LLaVA-Med, BiomedCLIP, and Biomed-QwenVL; the caption should be corrected.","section":"Table 2 caption"},{"comment":"The table headers call C a 'confidence threshold (C)', but C in Eq. (2) is an additive offset, not a threshold; rename it to avoid confusion.","section":"Appendix A.4"},{"comment":"The sentence 'For our experiments, we use again use 80/20 percentage split' contains a grammatical repetition ('use again use'); please rephrase.","section":"§5.1"},{"comment":"The baseline transformation Conf_baseline^(1/T) is described as temperature scaling, but standard temperature scaling operates on logits; since the transformation here is a power transform of confidence values, clarify the relationship or rename the baseline to avoid overloading the term.","section":"§5.4"},{"comment":"The explanation for Biomed-QwenVL's poor performance invokes synthetic post-training data; this is presented as a hypothesis rather than a demonstrated cause, and the sentence should be softened or supported.","section":"§5.5"}],"recommendation":"major_revision","confidential_remarks":"The central idea is worth pursuing, but the comparison with a one-parameter baseline is a genuine confound that must be addressed with a same-complexity recalibration baseline. The paper also needs to clarify the [0,1] handling of Eq. (2) and provide uncertainty estimates for the ECE values. I do not see a circularity problem: tuning T and C on a validation split and evaluating on a test split is standard model selection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper proposes a simple post-hoc recipe to improve calibration of self-consistency UQ for multimodal LLMs by multiplying the baseline confidence with a temperature-scaled grounding-model confidence and adding an offset C. The idea is new in this application, the recipe is easy to run, and the reported ECE drops are large. But the main comparison is not apples-to-apples: Eq. (2) fits two parameters (T and C) while the temperature-scaled baseline fits only one (T). I agree with the stress-test note. The paper needs a same-complexity baseline — e.g., a*conf_baseline + b or a*conf_baseline^(1/T)+b — fitted on the same validation splits before we can attribute the improvement to the grounding signal. This is the load-bearing issue, and it is fixable.\n\nWhat is genuinely good: the framework is agnostic to the choice of grounding model (segmentation, CLIP-style, or foundation VLM), requires no retraining, and the validation/test split procedure is standard model selection, not circular reasoning. The reliability diagrams and the motivating observation that consistently wrong answers can be highly confident (Fig. 1) are well made. The paper also honestly reports three cases where grounding hurts (Biomed-QwenVL on Slake) and gives hyperparameter values in the appendix.\n\nSoft spots beyond the confound: sample sizes for the test/validation sets are missing; the reported variances are rounded to 0.00001 and appear to be across-split variances rather than uncertainties in ECE; no confidence intervals or significance tests are given. There are small presentation errors — Table 2's caption says VQA while the text says Slake, and the appendix table captions (Tables 3 and 4) repeat the same mistake. The coarse mapping of foundation-model responses (Yes→1, No/Not sure→0) is defensible but deserves discussion. No code release is a limitation for reproducibility.\n\nThis is a within-subfield contribution for people working on UQ and calibration of vision-language models. It will not reshape the field, but the recipe is simple enough that others will try it once the baseline question is resolved.\n\nRecommendation: send to peer review, but ask for the two-parameter baseline, sample sizes, and code/data release before acceptance. The core idea may well hold, but the evidence as written does not support the claim that grounding is \"much more effective\" than temperature scaling alone.","headline":"Grounding-based calibration is a plausible and useful idea, but the headline claim is not yet established because the comparison baseline has one fewer fitted parameter.","tokens_in":14759,"tokens_out":2296,"would_cite":false,"duration_ms":24247,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grounding a vision-language model's answers back to the input image, and combining that grounding confidence with self-consistency, calibrates its uncertainty far more effectively than temperature scaling alone.","keywords":["uncertainty quantification","multi-modal LLM","calibration","grounding","self-consistency","expected calibration error","vision-language model","temperature scaling"],"falsifier":"Fit a two-parameter recalibration of the baseline confidence alone, such as a linear mapping $a\\cdot\\mathrm{Conf}_{\\mathrm{baseline}} + b$, to the same validation sets with the same number of free parameters as Eq. (2); if its ECE on VQA and Slake matches or beats the grounding-based scores, the claim that cross-modal grounding improves calibration would be unsupported.","tokens_in":13761,"feed_emoji":"🎯","tokens_out":5603,"duration_ms":48171,"temperature":0.7,"pith_summary":"The paper claims that self-consistency-based uncertainty scores for multi-modal LLMs, such as semantic entropy, are poorly calibrated because a model can be consistently wrong yet report high confidence. It proposes fixing this by grounding each generated response to the input image with a separate grounding model, then combining the grounding confidence with the self-consistency confidence using a two-parameter formula. The observed calibration error (ECE) drops substantially, for example from 0.108 to 0.029 on VQA with a large vision-language grounding model, and from 0.376 to 0.008 on a medical QA dataset. A sympathetic reader would care because the method is training-free and operates at inference time, potentially making LLM confidence trustworthy in safety-critical settings like medical question answering.","feed_headline":"Grounding answers in images slashes visual-LLM calibration error","feed_subtitle":"Grounding-based calibration beats temperature scaling alone on VQA and medical QA, cutting ECE up to 96%.","key_machinery":"The load-bearing object is the multiplicative confidence formula $\\mathrm{Conf} = \\mathrm{Conf}_{\\mathrm{baseline}} \\times \\mathrm{Conf}_{\\mathrm{GM}}^{1/T} + C$ (Eq. 2). The grounding model's confidence $\\mathrm{Conf}_{\\mathrm{GM}}$ is the average over the sampled responses of how well each response is grounded to the image, measured by a bounding-box segmentation model, a text-image similarity model, or a foundation-model verdict of 'Yes'/'No'/'Not sure'. The exponent $1/T$ temperature-scales the grounding confidence to account for the grounding model's own uncertainty, and the constant $C$ offsets the shrinkage caused by multiplying two values in $[0,1]$; both $T$ and $C$ are fit on a validation set. This formula carries the argument because it converts cross-modal evidence into a calibrated confidence score that is then evaluated by expected calibration error.","core_discovery":"The central discovery is that cross-modal consistency provides a corrective signal that self-consistency alone lacks: if a textual answer cannot be grounded in the image, the model is likely wrong even when multiple sampled answers agree. The paper operationalizes this by multiplying the baseline self-consistency confidence $\\mathrm{Conf}_{\\mathrm{baseline}}$ with a temperature-scaled grounding-model confidence $\\mathrm{Conf}_{\\mathrm{GM}}^{1/T}$, plus a small constant $C$, with both $T$ and $C$ fit on a validation set. Across four self-consistency baselines (lexical similarity, predictive entropy, semantic entropy, and number of semantic sets) and several grounding models (segmentation-based, CLIP-style similarity, and visual foundation models), the proposed confidence score yields lower expected calibration error than the same baselines recalibrated by temperature scaling alone. The paper concludes that calibrating confidence scores with external grounding is substantially more effective than calibrating them via temperature scaling.","pith_inferences":["The comparison baseline uses only a power exponent $T$, so a concrete check the paper leaves implicit is whether a two-parameter recalibration of the baseline alone, such as a linear mapping $a\\cdot\\mathrm{Conf}_{\\mathrm{baseline}} + b$, achieves similar ECE; if it does, the grounding signal may be serving mainly as an extra recalibration dimension rather than genuinely new evidence.","The framework could be applied to pure-text LLMs by grounding answers against a retrieved corpus or knowledge base instead of an image, turning 'grounding' into a general external-consistency check that does not require vision.","A testable extension is to measure per-question-type ECE to see whether grounding helps most on questions that require visual evidence versus commonsense knowledge, since the paper's best grounding models differ by domain (LLaMA3.2V for general VQA, BiomedCLIP for medical).","The reported gains rely on mapping grounding-model outputs to binary or similarity confidence; using richer probability outputs from the grounding model, such as the full softmax distribution in a closed-form setting, could yield a more informative calibration signal than the binary Yes/No mapping used here."],"forward_implications":["If grounding-based calibration works as claimed, any vision-language system using self-consistency UQ can be made more trustworthy at inference time without retraining or fine-tuning.","The choice of grounding model matters: domain-matched grounding models (e.g., BiomedCLIP on medical data) give the best calibration, so practitioners should match the grounding model to the deployment domain.","The approach extends beyond image-text pairs to other modalities: the authors note future work on audio and video, suggesting the same product formula may apply whenever a response can be grounded in a second modality.","Because the method needs only a small validation set to set two parameters, it is practical for safety-critical applications such as medical QA where answers must be vetted.","The paper's observation that consistency does not imply accuracy implies that self-consistency scores alone should not be relied on as confidence signals in high-stakes settings."],"supporting_citations":[{"why":"Supplies the semantic entropy self-consistency baseline that the proposed approach calibrates, and the semantic-clustering method for uncertainty estimation.","marker":"Kuhn et al., 2023"},{"why":"Defines temperature scaling and the expected calibration error metric used to evaluate both the baseline and the proposed method.","marker":"Guo et al., 2017"},{"why":"Provides CLIP, the semantic-based grounding model that produces text-image similarity confidence for VQA.","marker":"Radford et al., 2021"},{"why":"Provides BiomedCLIP, the domain-specific grounding model that achieves the lowest ECE on the medical Slake dataset.","marker":"Zhang et al., 2023b"},{"why":"Supplies the VQA v2.0 dataset, one of the two evaluation benchmarks for the visual question answering task.","marker":"Goyal et al., 2017"},{"why":"Supplies the Slake medical QA dataset used to evaluate the approach on a safety-critical medical domain.","marker":"Liu et al., 2021"},{"why":"Provides LLaMA 3.2V, the foundation-model grounding agent that performs best on the general VQA task.","marker":"Grattafiori et al., 2024"},{"why":"Introduces self-consistency theory that motivates the baselines the paper calibrates.","marker":"Wang et al., 2022"}],"fun_headline_variants":["Grounding answers in images fixes LLM overconfidence","Cross-modal grounding beats temperature scaling for LLM calibration","Image grounding reduces visual-LLM calibration error","Tie text to images to calibrate multi-modal LLM confidence","Grounding improves uncertainty calibration in VQA and medical QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the observed ECE reduction is caused by the grounding signal rather than by the extra free parameter $C$ in Eq. (2).","fun_headline_variants_meta":{"raw":{"variants":["Grounding answers in images fixes LLM overconfidence","Cross-modal grounding beats temperature scaling for LLM calibration","Image grounding reduces visual-LLM calibration error","Tie text to images to calibrate multi-modal LLM confidence","Grounding improves uncertainty calibration in VQA and medical QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1665,"prompt_tokens":950,"completion_tokens":715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":636}},"tokens_in":566,"tokens_out":715,"duration_ms":7993,"temperature":1.0,"reasoning_tokens":636,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:50:17.142617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit a two-parameter recalibration of the baseline confidence alone, such as a linear mapping $a\\cdot\\mathrm{Conf}_{\\mathrm{baseline}} + b$, to the same validation sets with the same number of free parameters as Eq. (2); if its ECE on VQA and Slake matches or beats the grounding-based scores, the claim that cross-modal grounding improves calibration would be unsupported.","supporting_citations":[{"cited_title":"Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering","cited_arxiv_id":null,"evidence_quote":"Supplies the Slake medical QA dataset used to evaluate the approach on a safety-critical medical domain."}],"review_version":1}