REVIEW 2 major objections 5 minor 21 references
This paper claims that accuracy on spatial yes/no benchmarks overstates image-grounded correctness, and that a label-free decision-level audit can decompose benchmark success into correctness, additional image support, and relation-consiste
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 10:08 UTC pith:UKQVWBP5
load-bearing objection Careful, mostly convincing label-free audit of image support in spatial yes/no benchmarks; headline C-U numbers hold under the paper's operationalization, with two fixable soft spots. the 2 major comments →
Visual Credit Audit for Multimodal Spatial Reasoning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that a single label-free event — whether the original image gives the model's declared answer more support than both a text-only prompt and a fixed gray blank — separates benchmark success into four exhaustive cells, and that a substantial share of correct answers fall in the "correct but uncredited" cell. In the reported runs, this share ranges from 12.73% to 26.25%, meaning ordinary accuracy overstates support-qualified correctness by that amount. The paper further shows that this gap is not an artifact of the specific controls: replacing the benchmark image with a deterministic same-split unrelated image lowers dependence-credited correctness by 21.25 to 47.80
What carries the argument
The central object is the prediction-aligned dependence event D_i = 1{r_{i,o} > max(r_{i,t}, r_{i,b})}, where r is the model's margin for its own declared answer under the original image, text-only, and blank contexts, with the event fixed before correctness labels are seen. This yields the four-cell accuracy-credit decomposition and the headline quantities D-CC and C-U. The companion relation axis uses the fixed-pixel true-false separation H_c and the interaction G_int = H_o − max(H_t, H_b), plus the Joint conjunction, to distinguish marginal image support from relation-consistent visual response. The matched same-split image permutation is the calibration mechanism that tests whether the o
Load-bearing premise
The audit's load-bearing premise is that a text-only prompt and a fixed mid-gray blank image are neutral stand-ins for "no benchmark image," so that any extra support the original image provides over both is genuinely due to the image's content; if the gray blank acts as an active distractor or as a hidden "objects absent" cue, the correct-but-uncredited share is overstated.
What would settle it
Run the same audit with a control that preserves scene structure but removes the queried objects, or with a set of unrelated natural images matched for low-level statistics; if D-CC rises to near accuracy — that is, if the 13–26% correct-but-uncredited mass largely disappears — then the C-U estimate is an artifact of the particular text-only/blank controls rather than a stable property of the models.
If this is right
- Benchmark reports would need to include D-CC (dependence-credited correctness) alongside accuracy, since two models with similar accuracy can differ by more than 20 points in support-qualified success.
- Because 12.73–26.25% of correct decisions are not better supported by the image than by text-only or blank controls, accuracy is an upper bound on image-grounded correctness, not a measure of it.
- Credit is a relative-support statement, not answer necessity: a decision can be uncredited even when the no-image controls do not flip the answer, so "no image support" should not be read as "image irrelevant."
- Correct-but-uncredited decisions are often still visually responsive: under controlled relation reversal, 81.57–100.00% of such decisions move in the correct direction and 32.11% change their answer, so a single audit number cannot capture both support and relation response.
- The audit is training- and label-free and needs only three cached contexts per item, so it can be applied to any frozen model and any fixed binary answer interface without retraining.
Where Pith is reading between the lines
- The same four-cell decomposition would likely transfer to other closed-form multimodal benchmarks (VQA, visual entailment, object presence), where a text-only or blank control can also be defined; the spatial setting may just be the easiest place to see it.
- Because the dependence event is label-free, it could serve as a data-filtering signal to remove benchmark items that reward answer priors, complementing label-based filtering methods.
- The deterministic same-split permutation could be replaced by a distribution over unrelated images to produce a tighter and more general null for "generic image" support; the paper tests one permutation per sample.
- The high directional response among uncredited decisions suggests that "uncredited" means redundant visual support rather than absent visual support; a useful next step would be to separate those two sub-cases inside the C-U cell.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Visual Credit Audit (VCA), a label-free, decision-level procedure for closed yes/no spatial benchmarks. For each query it caches a frozen MLLM's yes/no continuation margin under the original image, a text-only prompt, and a fixed mid-gray blank image, and defines a dependence event D = 1{r_o > max(r_t, r_b)}. After labels are applied, the paper obtains D-CC = E[κD] and C-U = E[κ(1−D)], and shows that on correct items the event reduces to a gold-aligned positive-gain statistic. Across four MLLMs and VSR/GSR-COCO, it reports 12.73–26.25% correct-but-uncredited decisions and a matched same-split image-permutation D-CC excess of 21.25–47.80 points. A relation axis (SRC, PairCredit, Joint) and a 3×3 evidence-source factorial are used to argue that marginal image-support advantage and fixed-pixel relation response are separate estimands. The paper concludes that benchmark accuracy can overstate image-grounded success.
Significance. The paper is unusually transparent and the accounting is sound: D is assigned before labels, the four-cell decomposition in Eq. (3) is complete, and the correct-item restriction in Eq. (4) exactly recovers gold-aligned gain. The paired-bootstrap intervals, deterministic same-split permutation calibration, threshold-mass analysis, multi-verbalizer checks, and direction-balanced sensitivities are all appropriate and mostly well reported. The paper ships code, deterministic data builders, cached margins, and deidentified annotations, which is a real strength. The two-estimand separation — relative image support versus relation response — is a useful corrective, and the factorial convincingly shows why null-control marginal support cannot identify relation response. If the central C-U estimate survives a neutral no-objects control, the finding that 12.73–26.25% of correct decisions lack control-relative image support is important for benchmark interpretation.
major comments (2)
- [§3.1, Eq. (1); Table 10; Tables 14 and 28] The headline C-U estimate treats the fixed mid-gray blank as a neutral 'no benchmark image' baseline. For a spatial yes/no probe, a gray canvas is a visible scene with no objects, which can directly license 'no'. The paper's strata show the signature: in Table 14, LLaVA VSR C-U is 36.94 for gold-no vs 2.10 for gold-yes; InternVL VSR is 34.78 for gold-yes vs 1.20 for gold-no. C-U is concentrated in the direction favored by no-image/blank priors. Table 28's strict T/B requirement collapses Dep (e.g., InternVL GSR 42.84→10.45), consistent with controls usually agreeing with the original answer rather than being neutral. The mean-color blank (Table 37) changes low-level color but still has no objects, so it does not remove the 'objects absent' cue. This is load-bearing for the 12.73–26.25% range. Please add a no-objects background control (or text-only-only reporting) to isolate the gray-bla
- [Abstract/Conclusion vs Table 15] The headline range is not direction-invariant. Prediction-balanced C-U is 11.73–43.36% while gold-balanced is 12.73–26.25%, and VSR orderings change. The supplement discloses this, but the main text (abstract, conclusion) states the range without the qualification. Since the C-U estimate is also sensitive to the blank-control concern in Comment 1, the main text should explicitly say that the 12.73–26.25% figure is benchmark-distribution-specific and should report the direction-balanced estimates in the main robustness section.
minor comments (5)
- [Abstract/§3.1] Throughout, 'no-image controls' is imprecise for the blank: a gray canvas is still an image. Suggest 'image-absent' or 'content-free image' to avoid implying that the blank contains no visual input.
- [Table 1] The last column header 'C-U∆ perm DCC' is hard to parse; split into 'C-U' and 'Δ perm D-CC [CI]' and explain in the caption that Δ is the paired original-minus-permutation D-CC difference.
- [§4.3/Figure 1] The 3×3 evidence-source factorial is central to the response-vs-support separation but is only described in text; a diagram of the nine cells (or at least the agreement and conflict cells) would make the design easier to verify.
- [Table 2/§4.2] PairCredit = 1{G_int > 0} can be positive even when the original true–false separation is negative if the controls are more negative. The caption's phrase 'image-specific semantic separation' should explicitly state that PairCredit does not require H_o > 0; PairCorrect plays that role.
- [Limitations] The Limitations section is honest about causal attribution but does not list the gray-blank neutrality assumption as a limitation; add it explicitly, together with the no-objects control that would test it.
Circularity Check
No significant circularity: the audit quantities are prespecified, the two definitional reductions are explicitly disclosed and non-load-bearing, and the headline empirical claims rest on independent control contrasts.
full rationale
The derivation chain is self-contained. D_i = 1{G_img_i > 0} with G_img_i = r_i,o - max(r_i,t, r_i,b) is a prespecified, label-free event; no parameter is fitted to labels, and the zero threshold is fixed before labels are consulted (Eqs. 1-2). The paper explicitly discloses that on correct items this event is algebraically identical to a gold-aligned gain: Eq. (4) states kappa_i D_i = kappa_i 1{s_i m_i,o > max(s_i m_i,t, s_i m_i,b)}, and the text says the event 'intentionally reduces exactly to a gold-aligned positive gain' but 'extends the same order event label-free to all decisions.' That is an honest characterization, not a hidden prediction. Likewise, the Joint => G_int > 0 implication is explicitly labeled 'a structural property' and 'definitional,' and the paper reports PairCredit and mean G_int as the empirical quantities outside Joint, so no derived result is being passed off as evidence. The headline C-U estimates are direct empirical frequencies from cached margins and benchmark labels, not outputs of a fitted model. The matched same-split permutation calibration changes only the image while holding the audited decision fixed; this is an independent empirical contrast, and the 21.25-47.80 point drops are not forced by the definition of D. The factorial and 108-edit natural-image checks provide external, non-self-referential validation. The main substantive threat identified by the skeptical review is whether the fixed mid-gray blank operationalizes 'no benchmark image' neutrally; but the paper states that 'dependence' means this control-relative support advantage, so any concern is about construct validity of the control, not circularity of the derivation. No load-bearing self-citations or imported uniqueness theorems appear.
Axiom & Free-Parameter Ledger
free parameters (2)
- Dependence threshold =
0
- Edit renderer constraints (center gap ≥0.05, new-box IoU ≤0.20, padding 4, inpaint radius 3) =
see supplement
axioms (6)
- domain assumption Next-token continuation-loss margins on a fixed forced-choice answer interface are a valid measure of the MLLM's support for a decision.
- domain assumption Text-only (no image) and blank mid-gray image are valid no-evidence controls representing absence of benchmark-image support.
- domain assumption Sampling the same relation pair with true and false statements cancels answer-surface offsets; H_c differences are comparable across contexts.
- domain assumption Procedural rendered scenes in the 3×3 factorial are informative about behavior on natural benchmark images.
- domain assumption The deterministic same-split image permutation is unrelated enough to act as a mismatch calibration.
- domain assumption Human majority consensus on 108 edits is a reliable ground truth for relation flips and object visibility.
read the original abstract
Closed yes/no spatial benchmarks can reward a correct answer even when the image adds little support beyond no-image contexts. Under a fixed forced-choice interface, Visual Credit Audit (VCA) separates two estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence. The first audit is training- and label-free and does not require an answer flip. Applying labels yields dependence-credited correctness (D-CC); on correct items, it equals same-control gold-aligned positive gain, while prediction alignment extends the audit to errors. Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited. Matched same-split image permutation reduces D-CC by 21.25-47.80 points, with every paired 95% interval above zero. Fixed-pixel relation contrasts and a 3x3 evidence-source factorial show why null controls cannot identify relation response. Among controlled correct-but-uncredited agreement decisions, response to relation reversal spans 81.57-100.00%, while 32.11% pooled change answer. Independently audited outcomes on 108 geometry-compatible edits provide a bounded natural-image correspondence check. VCA thereby decomposes benchmark success into correctness, additional image support, and relation-consistent response.
Figures
Reference graph
Works this paper leans on
-
[1]
Answer the user’s question with exactly yes or no,
Continuous Raw-Gap Distributions BinaryDeprecordsthegapsign;itsmagnitudedistinguishes near-zeroboundarycasesfromstrongper-itemevidence.Ta- ble17showsbroadoverlapacrossmodelsanddatasetsrather thanadegeneratemassatzero.Inparticular,InternVL/GSR- COCO has a negative median and mean but a positive upper quartile, explaining how its Dep rate can remain 42.84%....
2026
-
[4]
VisNec: Measuring and Leveraging Visual NecessityforMultimodalInstructionTuning.arXivpreprint arXiv:2603.01195. Fu, H. Y.; Yang, C.; Chang, T.-Y.; Wang, Z.; Wu, Z.; Zhou, J.; and Fu, D
-
[6]
InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2901–2910
CLEVR: A Diag- nostic Dataset for Compositional Language and Elementary Visual Reasoning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2901–2910. Kamath,A.;Hessel,J.;andChang,K.-W.2023. What’s“up” withVision-LanguageModels?InvestigatingTheirStruggle with Spatial Reasoning. InProceedings of the 2023 Confer- enceonEmpiric...
2023
-
[7]
From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA.arXiv preprint arXiv:2606.30220. Lan, Z.; Sun, L.; Walter, M. R.; and Zhou, J
-
[8]
Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L
Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?arXiv preprint arXiv:2605.22903. Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L
-
[11]
Ma, C.; Lu, K.; Cheng, T.-Y.; Trigoni, N.; and Markham, A
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning.arXiv preprint arXiv:2306.14565. Ma, C.; Lu, K.; Cheng, T.-Y.; Trigoni, N.; and Markham, A
-
[12]
SpatialPIN: Enhancing Spatial Reasoning Capa- bilities of Vision-Language Models through Prompting and Interacting 3D Priors.arXiv preprint arXiv:2403.13438. Mistral AI
-
[13]
Ministral 3.arXiv preprint arXiv:2601.08584. Naeini, M. P.; Cooper, G.; and Hauskrecht, M
-
[14]
Reich,D.;Putze,F.;andSchultz,T.2023.MeasuringFaithful and Plausible Visual Grounding in VQA
GSR-BENCH: A Bench- mark for Grounded Spatial Reasoning Evaluation via Multi- modal LLMs.arXiv preprint arXiv:2406.13246. Reich,D.;Putze,F.;andSchultz,T.2023.MeasuringFaithful and Plausible Visual Grounding in VQA. InFindings of the Association for Computational Linguistics: EMNLP 2023, 3129–3144. Rohrbach, A.; Hendricks, L. A.; Burns, K.; Darrell, T.; an...
Pith/arXiv arXiv 2023
-
[16]
Do Vision-Language Models See or Guess? Measuring and Re- ducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark.arXiv preprint arXiv:2606.10400. Sun, Z.; Shen, S.; Cao, S.; Liu, H.; Li, C.; Shen, Y.; Gan, C.; Gui, L.-Y.; Wang, Y.-X.; Yang, Y.; et al
-
[17]
arXiv preprint arXiv:2309.14525
Aligning LargeMultimodalModelswithFactuallyAugmentedRLHF. arXiv preprint arXiv:2309.14525. Thrush, T.; Jiang, R.; Bartolo, M.; Singh, A.; Williams, A.; Kiela, D.; and Ross, C
-
[19]
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 10006–10030
MMoE: Enhancing Multimodal ModelswithMixturesofMultimodalInteractionExperts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 10006–10030. Zhang, C.; Ding, W.; Liu, J.; Wu, M.; Wu, Q.; and Mooney, R
2024
-
[20]
Zhang, L.; Zhai, X.; Zhao, Z.; Zong, Y.; Wen, X.; and Zhao, B
Do Images Speak Louder than Words? Investigat- ing the Effect of Textual Misinformation in VLMs.arXiv preprint arXiv:2601.19202. Zhang, L.; Zhai, X.; Zhao, Z.; Zong, Y.; Wen, X.; and Zhao, B
-
[305]
Li, Y.; Tian, W.; Jiao, Y.; and Chen, J. 2024b. Eyes Can Deceive: Benchmarking Counterfactual Reasoning Capabil- ities of Multimodal Large Language Models.arXiv preprint arXiv:2404.12966. Liang, X.; Guo, X.; Jin, Z.; Pan, W.; Shang, P.; Cai, D.; Lin, B.;andYe,J.2025.EnhancingSpatialReasoningthroughVi- sualandTextualThinking.arXivpreprintarXiv:2507.20529. ...
Pith/arXiv arXiv 2025
-
[2017]
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6904–6913
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6904–6913. Guo,C.;Pleiss,G.;Sun,Y.;andWeinberger,K.Q.2017. On Calibration of Modern Neural Networks. InInternational Conference on Machine Learning, 1321–1330. Johnson, J.; Ha...
2017
-
[2018]
InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4035–4045
Object Hallucination in Image Caption- ing. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4035–4045. Singla, P.; Garg, S.; Singh, V.; and Chopra, P
2018
-
[2022]
InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5238–5248
Winoground: Probing Vision andLanguageModelsforVisio-LinguisticCompositionality. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5238–5248. Wang,W.;Gao,Z.;Gu,L.;Pu,H.;Cui,L.;Wei,X.;Liu,Z.; Jing,L.;Ye,S.;Shao,J.;etal.2025.InternVL3.5:Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency.arXiv...
Pith/arXiv arXiv 2025
-
[2023]
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, TextReading,andBeyond.arXivpreprintarXiv:2308.12966. Bai, S.; et al
-
[2024]
Mitigating Object Hallucinations in Large Vision-LanguageModelsthroughVisualContrastiveDecod- ing.ProceedingsoftheIEEE/CVFConferenceonComputer Vision and Pattern Recognition, 13872–13882. Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; and Li, C. 2024a. LLaVA-OneVision:EasyVisualTaskTransfer.arXiv preprint ...
Pith/arXiv arXiv 2023
-
[2025]
Qwen3-VL Technical Report.arXiv preprint arXiv:2511.21631. Bhat, S. D.; and Yamasaki, T
-
[2026]
arXiv preprint arXiv:2607.09068
OmniMapBench: Benchmark- ing Visual-Centric Reasoning on Diverse Map Documents. arXiv preprint arXiv:2607.09068. Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024b. In- ternVL: Scaling up Vision Foundation Models and Align- ing for Generic Visual-Linguistic Tasks. InProceedings of the IEEE/CVF Confe...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.