{"id":"1d2b53f2-f773-453b-b82f-a458ef4490eb","arxiv_id":"2504.14848","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new training framework, CSP, uses semantic perturbation of object regions plus preference optimization to make VLMs verbalize confidence that better matches answer correctness.","lead":"This paper presents a training method that blurs or adds noise to objects in images and teaches vision-language models to state lower confidence when objects are harder to see. The method improves how well a model's stated confidence matches whether its answers are actually correct, without hurting its accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training labels are defined by a linear noise schedule, not by empirical correctness; the central calibration claim rests on an unvalidated proxy.","rationale":"The reader's weakest assumption correctly identifies the noise-to-confidence mapping as the fragile link, and I agree with that diagnosis. This is load-bearing because it connects the training signal to the evaluation metric: ECE and Brier score interpret verbalized confidence as a probability of correctness, but the supervision provides no such probability. A linear diffusion schedule is an arbitrary parametrization of visual degradation, and the paper gives no evidence that after Tc steps a fraction c/100 of such answers are correct. The construction also makes the mapping suspect: a fixed response's truth value is invariant to the perturbation, and RLAIF-V can contain preferred and dispreferred responses, so the c=100 target can be attached to wrong answers. The proposed diagnostic would settle this with a straightforward annotated sample. I keep the reader's CONDITIONAL verdict rather than escalating because the empirical tables might still survive if the diagnostic shows the schedule happens to be calibrated on hallucination benchmarks, and because the paper's other weaknesses, such as the missing candidate-answer protocol and the absence of error bars, are already captured by the conditional. The condition should explicitly require the label-validity diagnostic and a full description of how candidate answers and their confidence scores are elicited.","tokens_in":17679,"tokens_out":13519,"duration_ms":133480,"concrete_test":"Diagnostic on the constructed training data: take a random sample of pairs (v_perturbed, q_c, r_c) together with the original unperturbed image and the original response, and obtain ground-truth labels for whether the original response is correct by using POPE or AMBER-style annotations for the same image and question. Then group by the assigned label c, using bins at 0, 25, 50, 75, and 100, and compute the empirical accuracy of the quoted response. If P(correct | c) is not approximately c/100, in particular if the c=100 bin contains many incorrect responses or the c=0 bin contains many correct responses, the training labels are not calibrated probabilities of correctness and the central claim is unsupported. This test requires no retraining; if it fails, the authors would need to recalibrate the schedule or re-label by empirical correctness before the method can be evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 sets the ground-truth confidence label c by the diffusion-step schedule Tc = floor(Tmax*(1-c/100)) and attaches rc=c to every perturbed sample. The label is therefore a monotone function of injected pixel noise, not a measure of the probability that the quoted response is correct. Nothing in the construction filters or reweights the RLAIF-V response pairs by correctness: the same fixed answer is paired with all perturbation levels, so a factually correct answer remains correct under heavy noise, and a hallucinated answer remains wrong even at c=100%. For a yes/no object query, perturbing the object region does not change the truth value of a fixed candidate answer; the empirical frequency P(correct | c) is not controlled by the schedule. Thus the SFT objective teaches the model to emit low scores for degraded-looking images and high scores for clean images, rather than to verbalize a probability of correctness. Moreover, test images in POPE and AMBER are clean, so the learned noise-level cue is absent; the reported ECE/AUC gains cannot be attributed to the proposed mechanism without showing that the scores track correctness rather than image degradation. The SimPO rejected response 100%-c inherits the same issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Confidence Calibration through Semantic Perturbation (CSP), a two-stage training framework for object-level verbalized confidence in vision-language models. In the dataset construction stage, the authors localize query-referenced objects with GroundingDINO and SAM, inject Gaussian noise into those regions using a diffusion schedule, and assign a confidence label c determined by the noise step. They then fine-tune VLMs with supervised learning to output that label in a third-person confidence prompt, followed by SimPO preference optimization with the rejected confidence 100%-c. Evaluation is carried out on POPE and AMBER with metrics Acc, F1, ECE, AUC, and Brier score, and on MME for task performance; the paper reports consistent calibration gains across Qwen-VL, Qwen2-VL, InternVL2, and Phi-3.5, together with ablations and an analysis of internal/verbalized confidence correlation. The central claim is that CSP improves the alignment between verbalized confidence and response correctness while maintaining or enhancing task performance.","tokens_in":17881,"tokens_out":9095,"duration_ms":83419,"significance":"The paper addresses an important and timely problem: verbalized confidence calibration in VLMs, especially in the context of object hallucination. The idea of using semantic mask perturbation to create training data with controlled visual uncertainty is simple and potentially useful, and the authors test it on four model families and several benchmarks. The manuscript is clearly organized, and the limitation section honestly acknowledges the scope of the study. That said, the current evidence does not yet establish the central claim, because the training labels are a noise-level proxy rather than correctness-based labels, and because the evaluation protocol for the headline metrics is not described. With additional correctness-conditional analysis, a fully specified evaluation harness, and a fuller ablation table, the work could become a solid contribution.","major_comments":[{"comment":"The ground-truth confidence label is defined by the diffusion-step schedule Tc = floor(Tmax*(1-c/100)), so c is a monotone function of injected pixel noise and not a measure of P(correct | perturbed image). For a fixed query-response pair, perturbing the object region does not change the truth value of the answer: a correct answer remains correct even at c=0%, and a hallucinated answer remains wrong even at c=100%. The SFT objective therefore trains the model to verbalize low confidence for degraded-looking images and high confidence for clean images, and the SimPO rejected response 100%-c inherits the same artifact. Because the POPE and AMBER test images are clean, the reported ECE/AUC/Brier improvements could reflect a shift in score distributions rather than improved alignment with correctness. I request (i) a validation of the assumed noise-to-correctness mapping, for example by reporting the empirical correctness rate of the underlying answers at each noise level, and (ii) a correctness-conditional analysis on the held-out clean benchmarks, such as ECE and score distributions computed separately for correct and incorrect responses, to show that the learned scores track correctness rather than image degradation.","section":"Section 3.2"},{"comment":"The evaluation protocol is under-specified. The paper repeatedly refers to candidate answers {a_1,...,a_k} and to 'the prediction chosen by the highest confidence c(ŷ_i)', but it never states how the candidate set is constructed for POPE/AMBER, how many candidates are used, whether the correct answer is always among them, or how the verbalized response (e.g., '70.37%') is parsed into a numeric probability p_i. The reported baseline values, Acc=0.25 for Qwen-VL on POPE random and Acc=0.11 for Qwen2-VL on POPE random, are far below both chance for binary questions and standard POPE accuracy for these models; the table caption explicitly says these metrics do not measure dataset performance, but no alternative protocol is defined. Without a precise description of candidate construction, confidence parsing, and tie handling, the Acc/F1/AUC/ECE numbers are not reproducible, and the claimed improvements cannot be attributed to the method. Please provide a complete evaluation protocol in the appendix and ideally release the evaluation code.","section":"Sections 3.1, 4.2, Table 1"},{"comment":"The ablation study is reported only for one model/setting (Qwen2 on POPE adversarial) and without numerical values or uncertainty estimates. The text states that 'SimPO Only ... shows nearly no gains' and 'Original RLAIF ... does not have improvement', but no table presents these numbers across the four models and multiple datasets. This matters because these ablations are the main evidence that the semantic mask perturbation, rather than the extra SFT data or preference optimization, is responsible for the reported gains. Please add a full ablation table across models and datasets, with the same metrics as Table 1, and report repeated-run variability if feasible.","section":"Section 4.3.2, Figure 4"}],"minor_comments":[{"comment":"The first sentence of the Introduction contains a duplicated citation block: '(Achiam et al., 2023; Bubeck et al., 2023)' appears twice in a row.","section":"Introduction"},{"comment":"There are several typos: 'we begin with extract' should be 'we begin by extracting', 'designed to evaluation' should be 'designed to evaluate', 'donwloaded' should be 'downloaded' (twice in Appendix B.2), and Section 3.2 ends with a double period after 'recognition tasks..'.","section":"Sections 3.2 and 4.1, Appendix B.2"},{"comment":"The hyperparameters Tmax and gamma are never reported, and the distribution from which confidence labels are sampled is not specified; please add these values to the implementation details in Appendix B.1.","section":"Section 3.2, Appendix B.1"},{"comment":"The definition of ECE uses K bins of equal confidence range, but K and the binning strategy are not reported; please state the number of bins and whether they are equal-width or equal-frequency.","section":"Section 4.2"},{"comment":"The Spearman and Kendall correlations are reported without confidence intervals or significance tests, and several values are near zero (for example, Qwen-VL on AMBER relation has rho = -0.02); the claim of consistent improvement should be qualified accordingly.","section":"Table 2"},{"comment":"The MME results report a macro-averaged F1 score without defining how it is computed across the 14 subtasks; standard MME reporting uses perception and cognition accuracy scores, so please clarify the F1 aggregation.","section":"Table 5, Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising, but the manuscript currently has a serious validity gap: the training labels are noise-schedule values, not correctness probabilities, and the evaluation protocol for the main metrics is missing. A revision that adds a correctness-conditional analysis and a complete evaluation description would allow the claims to be assessed properly. I would also encourage the authors to release the evaluation code, since the numbers in Table 1 are otherwise difficult to reproduce."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper has a genuinely neat idea — perturb object regions to teach confidence — but as written the main evaluation protocol is missing, and the training labels are a self-defined proxy rather than measured correctness. The central claim is plausible but not yet supported.\n\nWhat's new: applying mask-based Gaussian noise to query-relevant object regions (GroundingDINO + SAM), then SFT+SimPO to align verbalized confidence. The ablations are the paper's best part: they show local mask perturbation matters more than global noise, and that SimPO without the perturbed data does nothing. The authors also checked MME for performance trade-offs and listed limitations honestly. Credit where due.\n\nSoft spots, in order of size. First, the evaluation protocol for the headline numbers is under-specified. The paper never says how candidate answers are constructed or how 'the answer with highest confidence' is selected, yet Table 1 reports Acc/F1/ECE for POPE and AMBER. Those numbers are not the standard benchmark accuracies, so readers cannot reconstruct what was measured. Without this, the main claim 'improves alignment' is hard to verify. Second, the confidence labels are assigned by the noise schedule: c=100% is clean, c=0% is maximal noise. The same fixed answer is paired with every perturbation level, and for a yes/no object query adding noise doesn't change whether that answer is correct. So the SFT target is a monotone function of pixel degradation, not of empirical correctness. The model may learn to say low confidence when the image looks noisy, and on clean test images that cue is absent. The paper needs to show the learned scores track correctness rather than degradation. Third, no error bars and no code/data, which matters for a method whose main evidence is a set of tables. Fourth, the appendix MME results show small F1 drops (e.g., InternVL from .8450 to .8384, Phi3.5 from .7537 to .7402), which qualifies the 'no performance loss' claim. Minor, but the phrasing should be softened.\n\nBottom line: this is a thoughtful recipe with solid ablations, but the evaluation protocol is under-specified and the proxy-label assumption is unvalidated. A serious referee should engage with it — the idea is new and the experiments are extensive — but the authors need to release code, specify the candidate-answer construction, and provide evidence that their noise-derived labels correspond to correctness. I'd send it to review, not desk reject, and expect major revision.","headline":"A clever but under-supported training recipe for verbalized confidence; the main evaluation protocol is missing and the labels are a noise-derived proxy.","tokens_in":18431,"tokens_out":3505,"would_cite":false,"duration_ms":32326,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training vision-language models on images whose key object regions are deliberately degraded makes the models' stated confidence track their actual correctness on object-level queries.","keywords":["verbalized confidence","confidence calibration","vision-language models","object hallucination","semantic perturbation","preference optimization","uncertainty estimation","multimodal trustworthiness"],"falsifier":"Take a held-out set of real images with genuine occlusion, blur, or low lighting, obtain oracle correctness labels for each object query, and compare expected calibration error before and after CSP training; if the improvement disappears, or the model merely becomes pessimistic about noisy-looking objects without better predicting correctness, the assumed noise-to-correctness mapping fails to generalize.","tokens_in":17469,"feed_emoji":"🎯","tokens_out":5545,"duration_ms":49724,"temperature":0.7,"pith_summary":"This paper sets out to show that a vision-language model's verbalized confidence, such as saying it is 80% sure a cat is present, can be made to match whether that object claim is actually correct. The proposed approach, Confidence Calibration through Semantic Perturbation (CSP), works by adding controlled Gaussian noise to the specific object region a question asks about, labeling each degraded image with a confidence value, and fine-tuning the model first with supervised learning and then with preference optimization. On object-hallucination benchmarks, the method reduces the gap between stated confidence and correctness while holding or improving task accuracy. The reason to care is that users and downstream systems could then treat a model's stated confidence as a trustworthy signal for accepting or rejecting object claims, rather than being misled by confident errors.","feed_headline":"Noisy-object training fixes overconfident vision-language models","feed_subtitle":"Gaussian blur on key regions plus preference tuning aligns stated confidence with actual answer correctness.","key_machinery":"The load-bearing mechanism is semantic perturbation: a linear schedule maps a target confidence label c to a diffusion step T_c = floor(T_max * (1 - c/100)), Gaussian noise is iteratively added to the masked object region, and the noised region is composited onto the unperturbed background. Each perturbed image becomes a training pair whose target verbalized confidence is exactly c. The training then uses supervised cross-entropy on those pairs, followed by the SimPO preference objective with c as the chosen response and 100% - c as the rejected response. This chain is what teaches the model to tie stated confidence to local visual reliability.","core_discovery":"The central claim is that object-level verbalized confidence in vision-language models can be calibrated by supervised training on images whose key object regions have been degraded to known noise levels, followed by preference optimization that rewards confidence statements matching the noise-derived label. The paper reports that this two-stage procedure lowers expected calibration error and Brier score, raises accuracy, F1, and AUC on POPE and AMBER across four different models, and preserves or improves performance on the MME benchmark. It also reports that internal token-level probabilities and verbalized confidence become more strongly correlated, even though the training never directly supervises internal probabilities.","pith_inferences":["A natural next test, not run in the paper, would replace the synthetic noise-derived confidence labels with human-annotated confidence on real occluded or low-light images; if CSP still improves calibration, the noise schedule is only a convenient surrogate rather than the essential ingredient.","Because the perturbation targets semantic object regions, the same construction could be applied to other uncertainty dimensions the paper leaves open, such as attribute, relation, or temporal ambiguity in video.","The reported gains suggest a cheap route to selective prediction: a deployment threshold on verbalized confidence could flag likely hallucinations in object-centric systems, although the paper does not evaluate such a rejection rule."],"forward_implications":["After CSP training, the answer a vision-language model selects by its highest stated confidence is correct more often, making verbalized confidence usable for answer selection and abstention.","Calibration gains appear in models with very different starting points, including both severely overconfident and already well-calibrated models.","Task accuracy on object-probing and hallucination benchmarks, and on the broader MME evaluation, stays flat or improves, so the calibration is not bought by degrading overall competence.","Internal token-level probabilities and verbalized confidence become more consistent even though the training objective never directly supervises internal probabilities."],"supporting_citations":[{"why":"Supplies the diffusion noising procedure used to map confidence labels to controlled noise levels.","marker":"Ho et al., 2020"},{"why":"GroundingDINO localizes the bounding boxes of objects referenced in the query before masking.","marker":"Liu et al., 2025"},{"why":"SAM converts the bounding boxes into pixel-wise masks so that only the key object regions are perturbed.","marker":"Kirillov et al., 2023"},{"why":"RLAIF-V provides the base image-query-response samples that the semantic perturbation dataset is built from.","marker":"Yu et al., 2024"},{"why":"SimPO is the preference-optimization objective that sharpens the alignment between verbalized confidence and correctness.","marker":"Meng et al., 2024"},{"why":"POPE supplies the object-presence evaluation benchmark with random, popular, and adversarial splits.","marker":"Li et al., 2023"},{"why":"AMBER supplies the attribute and relation hallucination evaluation used to measure calibration.","marker":"Wang et al., 2023"},{"why":"Motivates the third-person framing of confidence queries as a way to reduce sycophantic self-assessment.","marker":"Kumar et al., 2024"}],"fun_headline_variants":["Blur key objects to make vision-language models honest about uncertainty","Two-stage noise training makes VLM confidence match correctness","Add Gaussian noise to objects to calibrate VLM confidence","Training with blurred objects teaches VLMs to verbalize uncertainty"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic mapping from pixel noise to confidence label is treated as a valid proxy for the probability that the model's answer is actually correct, so that training on noisy objects transfers to real unperturbed images.","fun_headline_variants_meta":{"raw":{"variants":["Blur key objects to make vision-language models honest about uncertainty","Two-stage noise training makes VLM confidence match correctness","Add Gaussian noise to objects to calibrate VLM confidence","Training with blurred objects teaches VLMs to verbalize uncertainty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3024,"prompt_tokens":834,"completion_tokens":2190,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":2123}},"tokens_in":450,"tokens_out":2190,"duration_ms":12704,"temperature":1.0,"reasoning_tokens":2123,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:39:30.486393+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of real images with genuine occlusion, blur, or low lighting, obtain oracle correctness labels for each object query, and compare expected calibration error before and after CSP training; if the improvement disappears, or the model merely becomes pessimistic about noisy-looking objects without better predicting correctness, the assumed noise-to-correctness mapping fails to generalize.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GroundingDINO localizes the bounding boxes of objects referenced in the query before masking."}],"review_version":1}