{"id":"63a568fd-347d-4812-82d0-e5dea874fc40","arxiv_id":"2607.29240","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Selective Prior Calibration subtracts a model's text-only candidate preferences from image-conditioned scores with a learned per-instance strength and a conservative revision rule, substantially repairing counterfactual-image errors on CDH-Bench while leaving commonsense-image accuracy nearly unchan","lead":"Vision-language models often answer from commonsense habit, reporting a five-fingered hand even when the image clearly shows six fingers. This paper proposes Selective Prior Calibration, a post-hoc step that subtracts the model's no-image answer preference with an instance-dependent strength and changes the answer only when corrected scores clearly support an alternative.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Text-only prior proxy is the load-bearing assumption; RQ5 shows it breaks under question-form/claim-order shifts, so SPC's gains are conditional on prompt format.","rationale":"The reader identifies the text-only proxy as the weakest assumption, and the paper's RQ5 data corroborate this precisely: the proxy's reliability varies sharply with question form and claim order, and SPC's CF gains turn negative on atypical-first questions for LLaVA on unseen forms. This is a genuine load-bearing limitation, because both the empirical phenomenon (directed prior attraction) and the correction rule (Eq. 1) are defined through s̄P. However, the concern does not overturn the central MC result: the reader's strongest claim is specifically about the MC numbers, and those are supported by bootstrap CIs, positive repair/harm balances, component ablations, and external-transfer results on Visual CounterFact/HallusionBench/ConflictVIS. The failure mode is acknowledged in the paper's limitations section, and the balanced claim-order analysis is itself an honest stress test that the authors report. I therefore do not see grounds to move the verdict from CONDITIONAL to REJECT; the appropriate action is to keep CONDITIONAL and require the proposed MC-form/order check before the abstract's unqualified generality claim is accepted. The agreement is 'agree' because the reader's weakest_assumption and my load-bearing concern are the same: s̄P(y) is not a stable, content-independent estimate of the prior preference, and the method's success is contingent on its alignment with the deployed prompt.","tokens_in":36974,"tokens_out":5782,"duration_ms":70318,"concrete_test":"Port the RQ5 balanced claim-order/question-form protocol to the MC task: for all four templates (Contrastive, Instead, Choice, Reject second) and both claim orders, evaluate SPC on CDH-Bench MC with the same development data and hyperparameter selection, reporting ΔCF/ΔCS separately for each of the 16 model×form×order cells. If SPC maintains positive ΔCF and nonnegative ΔCS in every cell, the proxy concern is confined to binary QA and the current scope is defensible. If any cell shows negative ΔCF or substantial ΔCS loss, the abstract and conclusion should be re-scoped to 'works when the text-only prior is aligned with the ordinary-state answer under the deployed prompt format.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"SPC's mechanism equates the text-only log-probability s̄P(y) with the model's latent prior preference p(y); this identification is required by Eq. 1, by the directed-prior-attraction measurement, and by the stability analysis in Appendix B (Eq. A1). The paper's own RQ5 shows that this proxy is not stable: for LLaVA-1.6-34B, exchanging claim order reverses the text-only prediction on only 47.4% of Contrastive-form pairs, and on unseen question forms this drops to 14.3%, 5.7%, and 0.0% (Table A20). In those unseen forms, SPC's CF accuracy change is negative on atypical-first questions (−11.3, −13.9, and −43.0 points, respectively), and the corresponding CS changes are large and inconsistent. This is not a peripheral QA artifact: it is the central mechanism failing when the proxy misaligns. The headline MC results in Tables 1–2 use only the original question form and candidate order, so they do not test this fragility. The paper does acknowledge the binary-QA limitation in Section 6, but the abstract and conclusion claim general CF repair without that qualification. Thus the weakest link is not the revision rule itself (ablations and bootstrap CIs support it under matched prompts) but the assumption that s̄P measures an ordinary-state preference rather than a prompt-format or yes/no bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies commonsense-driven hallucination (CDH) in vision-language models, where a model answers according to its text-only commonsense prior even when the image supports an atypical state. It proposes Selective Prior Calibration (SPC): for each candidate answer, compute image-conditioned and text-only log-probabilities, mean-center them, subtract a scaled version of the text-only scores from the image-conditioned scores with an instance-dependent strength learned on paired CF-CS development data, and adopt the revised proposal only if it satisfies margin and support conditions. Experiments on CDH-Bench MC/QA, Visual CounterFact, HallusionBench, ConflictVIS, POPE, and POPEv2 use Qwen3-VL-32B and LLaVA-1.6-34B, with ablations, bootstrap intervals, and McNemar tests. The central claims are directed prior attraction, effective CF repair with CS retention, and generalization across categories, candidate-order permutations, and external conflict benchmarks.","tokens_in":37272,"tokens_out":8143,"duration_ms":84063,"significance":"If the result holds, SPC provides a simple, training-free-at-inference method with a clear mechanism and a favorable repair-retention trade-off. The paper's strengths are substantial: code release, exact McNemar tests and bootstrap intervals, a careful component analysis isolating candidate-specific prior subtraction, adaptive strength, and selective revision, plus transfer to external conflict benchmarks. The paper is also unusually honest in RQ5, showing where the method fails. However, the result is conditional on the text-only prior proxy remaining aligned with the ordinary-state preference, and the main text overstates the QA and generalization claims relative to the evidence in the supplementary material.","major_comments":[{"comment":"The NoLan baseline in Table 2 is not measured against the same reference as the other entries. Table 2 reports NoLan as +4.8/-1.7 (Qwen MC), +15.2/-0.9 (Qwen QA), +11.7/-3.5 (LLaVA MC), and +24.8/-8.7 (LLaVA QA). Appendix E states that these values correspond to a matched decoding loop with beta=0, while the changes relative to the common original predictions are +0.0/-4.8, +13.9/-0.9, +9.6/-2.6, and +24.8/-8.7. Since all other rows are measured against the original model, the 'SPC achieves the highest utility' claim is not based on a like-for-like comparison. The authors should report one consistent reference for all methods or justify why a different reference is appropriate for NoLan.","section":"Table 2 and Appendix E"},{"comment":"The binary-QA headline numbers are confounded by label imbalance. In the original QA form, 218 of 230 CF examples have ground truth 'no' and only 12 have 'yes.' SPC changes Qwen accuracy by +26.1 on the no side and -16.7 on the yes side; LLaVA changes by +32.6 and -16.7. The balanced claim-order test shows that the LLaVA atypical-first gain is not reliable: after fitting on the original form, SPC changes atypical-first CF by -1.7 (95% interval [-6.5, 3.0]). The paper acknowledges this only in the appendix and in a sentence in Section 6. The main text should not present the QA rows of Tables 1-2 as general evidence of CF repair; the balanced test should be reported in the main body, and the abstract/conclusion should be qualified accordingly.","section":"Appendix H, Table A17/A18; Section 5 QA rows"},{"comment":"The central mechanism requires the text-only log-probability s_P(y) to be a stable estimate of the model's latent preference for the ordinary state. RQ5 shows this assumption breaks under question-form and claim-order changes. For LLaVA, the text-only prediction reverses with claim order on only 47.4% of Contrastive pairs, and on 14.3%, 5.7%, and 0.0% of the three unseen forms; SPC's CF accuracy change on atypical-first questions is -11.3, -13.9, and -43.0 points, respectively, with large inconsistent CS changes. The abstract's unqualified claim that SPC 'substantially improves accuracy on CF images' is therefore too broad. The conclusion should state explicitly that the method is reliable when the prior proxy is aligned with the ordinary-state answer, and the atypical-first results should be summarized in the main text rather than only in the supplement.","section":"Section 5, Table 6 / Table A20; Eq. (1)"},{"comment":"The 'directed prior attraction' finding is close to definitional and should be framed more carefully. CDH errors are, by construction, cases in which the commonsense prior overrides visual evidence, and the text-only score is the same model's own commonsense answer preference. The empirical content is not the high attraction rate per se, but rather that the prior-favored candidate coincides with the paired CS ground truth; the paper does show this in Figure 2 and the appendix. Given that CDH-Bench is from the same research group (Chen et al. 2026), the discovery claim should be tempered and the external benchmarks emphasized as the stronger evidence for the mechanism.","section":"Section 3, Eq. (5); RQ1"}],"minor_comments":[{"comment":"CFAD, RPD, and CCR are used in the main text but defined only in the appendix (Table A2 caption). Define them at first use in the main paper.","section":"Section 4 / Table 1"},{"comment":"Several cells have very small n (e.g., n=1 or n=2). The text notes this for attribute cells, but the figure would be clearer if cells with n<3 were visually de-emphasized or marked.","section":"Figure 2"},{"comment":"The uncertainty-aware extension is used for generalization, category-holdout, and reduced-data results, but the main paper does not always state this. A sentence in Section 4 clarifying which results use the point-estimate SPC and which use the uncertainty-aware variant would improve reproducibility.","section":"Appendix C"},{"comment":"The notation 'Reported change' in Table 5 mixes CF/CS deltas and overall accuracy deltas without a uniform unit. This is understandable per benchmark, but the caption should state explicitly which metric each row uses.","section":"Section 5, RQ4 and Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent in the appendix, and the core MC result appears internally consistent. The main issues are presentation of the QA results, the inconsistent NoLan reference in Table 2, and overbroad abstract/conclusion claims relative to RQ5. These are fixable with re-analysis and rewording. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent paper on a real problem with a clear mechanism, solid MC evidence, and an unusually candid discussion of its own limits. The advertised generality is wider than the evidence supports, but the core contribution deserves a serious look.\n\nWhat is new: the directed-prior-attraction characterization (CF errors converge on the text-only-favored candidate) and the SPC method itself — learned instance-dependent subtraction strength plus a gating rule that only revises when the corrected proposal is well supported. The MC results are the load-bearing evidence: +7.8/+7.0 CF points for Qwen/LLaVA with CS changes inside the bootstrap intervals, repairs outnumber harms, ablations isolate each component, and transfer to Visual CounterFact and HallusionBench is positive. The paper also shows restraint by testing candidate-order permutations and category holdouts.\n\nSoft spots, in proportion. First, the text-only log-probability is treated as a proxy for the model's ordinary-state prior, and RQ5 shows that proxy is unstable: for LLaVA, the text-only prediction reverses with claim order on only 47% of Contrastive pairs and almost never on some unseen question forms, and SPC then loses CF accuracy on atypical-first questions. The paper acknowledges this in Section 6, but the abstract and conclusion still claim broad generalization, so the scope should be re-stated. Second, the QA gains in Tables 1-2 are confounded by the original form's yes/no imbalance — 218 of 230 CF answers are \"no\". The appendix admits this and shows SPC mostly moves answers toward \"no\", so the QA numbers are partly a bias correction. The MC results do not suffer this problem since candidate positions are balanced and rotations are tested. Third, the main benchmark is the authors' own CDH-Bench with overlapping authorship; the external transfer partly offsets this, but it should be disclosed. Minor: the reproducibility artifact is a bare GitHub URL without commit hash or data package.\n\nWho this is for: people working on training-free VLM hallucination mitigation, particularly candidate-scoring and contrastive-decoding methods. The MC contribution is useful and honestly evaluated; the QA claims need re-scoping.\n\nRecommendation: send to peer review. The MC evidence is solid, the mechanism is clear, and the RQ5 self-critique is exactly what a referee hopes to see. The paper needs revision on abstract and conclusion claims, and the QA framing should be corrected, but desk rejection would be a mistake.","headline":"A solid, honestly-scoped method for counterfactual-image MCQ correction; the QA claims are over-inflated and the prior proxy is fragile, but the MC evidence and self-critique earn it a real referee.","tokens_in":37808,"tokens_out":2300,"would_cite":true,"duration_ms":25324,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Errors on counterfactual images are pulled toward the model's no-image answer, and selectively subtracting that estimated prior repairs them while sparing commonsense cases.","keywords":["commonsense-driven hallucination","vision-language models","counterfactual images","prior calibration","hallucination mitigation","directed prior attraction","selective prior calibration","visual priors"],"falsifier":"Build a set of counterfactual images where the text-only scores do not favor the ordinary-state answer but the model still errs toward the ordinary answer; if SPC's correction then fails on those cases, directed prior attraction is not the error mechanism. The paper's own RQ5 approaches this condition for LLaVA under 'Instead,' 'Choice,' and 'Reject second' question forms, where the prior favors the ordinary answer in only 14.3%, 5.7%, and 0.0% of pairs and CF gains disappear or go negative on atypical-first questions.","tokens_in":57,"feed_emoji":"🖼️","tokens_out":5537,"duration_ms":98820,"temperature":0.7,"pith_summary":"Commonsense-driven hallucination is when a vision-language model overrides clear visual evidence of an atypical state and reports the ordinary state instead — a visibly six-fingered hand is reported as five-fingered. The paper shows these errors are directed: when a model gets a counterfactual image wrong, its answer usually coincides with the candidate it scores highest with the image removed. It then introduces Selective Prior Calibration (SPC), which subtracts these text-only prior-preference scores from image-conditioned candidate scores with a learned, instance-dependent strength and revises the prediction only when the corrected score pattern strongly supports an alternative. On a benchmark of 300 matched counterfactual/commonsense image pairs, SPC raises counterfactual accuracy by 7.8 and 7.0 points on two large models while changing commonsense accuracy by only +0.4 and −0.4 points. If the reported error direction is a general property of vision-language models, this offers a cheap, training-free way to reduce a specific class of hallucinations without retraining.","feed_headline":"Subtracting no-image bias lifts counterfactual accuracy 7.8 points","feed_subtitle":"The fix keeps commonsense answers intact: matched ordinary images lose less than 0.5 points.","key_machinery":"The load-bearing identity is sC(y) = s̄I(y) − λ s̄P(y): the corrected score of candidate y equals its mean-centered image-conditioned log-probability minus λ times its mean-centered text-only log-probability, the latter being the operational estimate of the model's prior preference. λ is a per-instance correction strength produced by a low-capacity softplus-linear model over 13 score-derived features, fit by maximum-a-posteriori estimation with a Gaussian prior on paired CF/CS development data. Around that sits a selective revision rule: the corrected proposal is adopted only if it differs from the original answer, its softmax margin over the runner-up exceeds a threshold, the candidate-set","core_discovery":"The central claim is directed prior attraction: incorrect predictions on counterfactual images systematically converge on the candidate preferred by the model's text-only scoring, and this candidate usually equals the ground-truth answer of the matched commonsense image. Suppressing this prior indiscriminately trades one error for another, but suppressing it with an instance-dependent strength — learned from paired counterfactual/commonsense development data — and gating the change behind a support condition repairs counterfactual errors while preserving commonsense accuracy. On the benchmark's multiple-choice task, corrected counterfactual accuracy rises by 7.8 points for Qwen3-VL-32B and 7","pith_inferences":["If directed prior attraction is a general failure mode, SPC suggests a diagnostic: comparing a vision-language model's image-conditioned and text-only answer distributions on any QA pair quantifies how much a given question leans on priors, independent of benchmark labels.","Using multiple rephrasings of the same question to estimate the prior (an ensemble over forms) might stabilize the correction where single-form text-only scores mislead — a testable extension of the paper's own RQ5 analysis.","Because SPC needs only candidate log-probabilities, it could be combined with decoding-time interventions rather than only post-hoc candidate rescoring; the paper's ablations show that hand-designed token-level contrastive rules change far more predictions on object-hallucination benchmarks, suggesting a calibration target for such methods.","If text-only score order reliably tracked the ordinary-state answer, SPC would amount to an unsupervised visual-fidelity check; the RQ5 results bound when that proxy is safe, for instance favoring 'contrastive' question forms over rephrased alternatives."],"forward_implications":["CDH errors are not random: on the benchmark, 70–92% of incorrect counterfactual answers select the text-only prior-favored candidate across models and tasks.","Candidate-level prior subtraction, not just re-ranking by image-conditioned scores, drives the gain; ablated variants without the subtraction lose or reverse the improvement.","Instance-dependent strength and gated revision are complementary: always adopting the corrected proposal repairs more counterfactual errors but harms matched commonsense images (e.g., LLaVA QA commonsense loss jumps to −13.9 points).","The correction transfers to held-out CDH categories, unseen candidate-order permutations, and other conflict benchmarks, while changing almost no predictions on object-hallucination benchmarks.","The text-only prior estimate is the fragile link: under rephrased questions and reversed claim orders, correction gains shrink or reverse, and for one model the text-only prediction reverses with claim order on only 0–47% of cases depending on question form."],"fun_headline_variants":["Prior calibration recovers 7.8 pts on counterfactual images","Counterfactual accuracy +7.8, commonsense -0.5","Selective prior subtraction corrects atypical visual errors","No-image bias subtraction boosts counterfactual reasoning","Fix six-finger errors without losing five-finger answers"],"cache_read_input_tokens":39040,"weakest_assumption_plain":"That the model's score for a candidate with the image removed (text-only log-probability) faithfully measures the latent commonsense prior that pulls errors toward the ordinary-state answer, even when question wording or claim order changes.","fun_headline_variants_meta":{"raw":{"variants":["Prior calibration recovers 7.8 pts on counterfactual images","Counterfactual accuracy +7.8, commonsense -0.5","Selective prior subtraction corrects atypical visual errors","No-image bias subtraction boosts counterfactual reasoning","Fix six-finger errors without losing five-finger answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000525,"raw_usage":{"total_tokens":2363,"prompt_tokens":728,"completion_tokens":1635,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1551}},"tokens_in":472,"tokens_out":1635,"duration_ms":13348,"temperature":1.0,"reasoning_tokens":1551,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:58:17.128951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a set of counterfactual images where the text-only scores do not favor the ordinary-state answer but the model still errs toward the ordinary answer; if SPC's correction then fails on those cases, directed prior attraction is not the error mechanism. The paper's own RQ5 approaches this condition for LLaVA under 'Instead,' 'Choice,' and 'Reject second' question forms, where the prior favors the ordinary answer in only 14.3%, 5.7%, and 0.0% of pairs and CF gains disappear or go negative on atypical-first questions.","supporting_citations":[],"review_version":1}