Pith. sign in

REVIEW 2 major objections 5 minor 21 references

This paper claims that accuracy on spatial yes/no benchmarks overstates image-grounded correctness, and that a label-free decision-level audit can decompose benchmark success into correctness, additional image support, and relation-consiste

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 10:08 UTC pith:UKQVWBP5

load-bearing objection Careful, mostly convincing label-free audit of image support in spatial yes/no benchmarks; headline C-U numbers hold under the paper's operationalization, with two fixable soft spots. the 2 major comments →

arxiv 2607.27069 v2 pith:UKQVWBP5 submitted 2026-07-29 cs.CV cs.AI

Visual Credit Audit for Multimodal Spatial Reasoning

classification cs.CV cs.AI
keywords Visual Credit Auditspatial reasoningmultimodal language modelsdependence-credited correctnesscorrect-but-uncredited decisionsbenchmark auditinglabel-free evaluationrelation consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Visual Credit Audit (VCA) tries to show that accuracy on closed yes/no spatial benchmarks can be high even when the benchmark image contributes little beyond text-only or blank prompts. For each decision, it compares how strongly the image supports the model's own answer against a text-only prompt and a blank gray canvas; only decisions with a positive "dependence gap" receive credit. Applying correctness labels afterwards yields a four-way split of all decisions, and across four open multimodal models and two benchmarks 12.73–26.25% of decisions are correct yet uncredited. A matched image-permutation control lowers dependence-credited correctness by 21.25–47.80 points, with every confidence interval above zero. If the claim holds, benchmark accuracy alone overstates image-grounded correctness, and reports should separate correctness from visual credit.

Core claim

The paper's central claim is that a single label-free event — whether the original image gives the model's declared answer more support than both a text-only prompt and a fixed gray blank — separates benchmark success into four exhaustive cells, and that a substantial share of correct answers fall in the "correct but uncredited" cell. In the reported runs, this share ranges from 12.73% to 26.25%, meaning ordinary accuracy overstates support-qualified correctness by that amount. The paper further shows that this gap is not an artifact of the specific controls: replacing the benchmark image with a deterministic same-split unrelated image lowers dependence-credited correctness by 21.25 to 47.80

What carries the argument

The central object is the prediction-aligned dependence event D_i = 1{r_{i,o} > max(r_{i,t}, r_{i,b})}, where r is the model's margin for its own declared answer under the original image, text-only, and blank contexts, with the event fixed before correctness labels are seen. This yields the four-cell accuracy-credit decomposition and the headline quantities D-CC and C-U. The companion relation axis uses the fixed-pixel true-false separation H_c and the interaction G_int = H_o − max(H_t, H_b), plus the Joint conjunction, to distinguish marginal image support from relation-consistent visual response. The matched same-split image permutation is the calibration mechanism that tests whether the o

Load-bearing premise

The audit's load-bearing premise is that a text-only prompt and a fixed mid-gray blank image are neutral stand-ins for "no benchmark image," so that any extra support the original image provides over both is genuinely due to the image's content; if the gray blank acts as an active distractor or as a hidden "objects absent" cue, the correct-but-uncredited share is overstated.

What would settle it

Run the same audit with a control that preserves scene structure but removes the queried objects, or with a set of unrelated natural images matched for low-level statistics; if D-CC rises to near accuracy — that is, if the 13–26% correct-but-uncredited mass largely disappears — then the C-U estimate is an artifact of the particular text-only/blank controls rather than a stable property of the models.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark reports would need to include D-CC (dependence-credited correctness) alongside accuracy, since two models with similar accuracy can differ by more than 20 points in support-qualified success.
  • Because 12.73–26.25% of correct decisions are not better supported by the image than by text-only or blank controls, accuracy is an upper bound on image-grounded correctness, not a measure of it.
  • Credit is a relative-support statement, not answer necessity: a decision can be uncredited even when the no-image controls do not flip the answer, so "no image support" should not be read as "image irrelevant."
  • Correct-but-uncredited decisions are often still visually responsive: under controlled relation reversal, 81.57–100.00% of such decisions move in the correct direction and 32.11% change their answer, so a single audit number cannot capture both support and relation response.
  • The audit is training- and label-free and needs only three cached contexts per item, so it can be applied to any frozen model and any fixed binary answer interface without retraining.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same four-cell decomposition would likely transfer to other closed-form multimodal benchmarks (VQA, visual entailment, object presence), where a text-only or blank control can also be defined; the spatial setting may just be the easiest place to see it.
  • Because the dependence event is label-free, it could serve as a data-filtering signal to remove benchmark items that reward answer priors, complementing label-based filtering methods.
  • The deterministic same-split permutation could be replaced by a distribution over unrelated images to produce a tighter and more general null for "generic image" support; the paper tests one permutation per sample.
  • The high directional response among uncredited decisions suggests that "uncredited" means redundant visual support rather than absent visual support; a useful next step would be to separate those two sub-cases inside the C-U cell.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Visual Credit Audit (VCA), a label-free, decision-level procedure for closed yes/no spatial benchmarks. For each query it caches a frozen MLLM's yes/no continuation margin under the original image, a text-only prompt, and a fixed mid-gray blank image, and defines a dependence event D = 1{r_o > max(r_t, r_b)}. After labels are applied, the paper obtains D-CC = E[κD] and C-U = E[κ(1−D)], and shows that on correct items the event reduces to a gold-aligned positive-gain statistic. Across four MLLMs and VSR/GSR-COCO, it reports 12.73–26.25% correct-but-uncredited decisions and a matched same-split image-permutation D-CC excess of 21.25–47.80 points. A relation axis (SRC, PairCredit, Joint) and a 3×3 evidence-source factorial are used to argue that marginal image-support advantage and fixed-pixel relation response are separate estimands. The paper concludes that benchmark accuracy can overstate image-grounded success.

Significance. The paper is unusually transparent and the accounting is sound: D is assigned before labels, the four-cell decomposition in Eq. (3) is complete, and the correct-item restriction in Eq. (4) exactly recovers gold-aligned gain. The paired-bootstrap intervals, deterministic same-split permutation calibration, threshold-mass analysis, multi-verbalizer checks, and direction-balanced sensitivities are all appropriate and mostly well reported. The paper ships code, deterministic data builders, cached margins, and deidentified annotations, which is a real strength. The two-estimand separation — relative image support versus relation response — is a useful corrective, and the factorial convincingly shows why null-control marginal support cannot identify relation response. If the central C-U estimate survives a neutral no-objects control, the finding that 12.73–26.25% of correct decisions lack control-relative image support is important for benchmark interpretation.

major comments (2)
  1. [§3.1, Eq. (1); Table 10; Tables 14 and 28] The headline C-U estimate treats the fixed mid-gray blank as a neutral 'no benchmark image' baseline. For a spatial yes/no probe, a gray canvas is a visible scene with no objects, which can directly license 'no'. The paper's strata show the signature: in Table 14, LLaVA VSR C-U is 36.94 for gold-no vs 2.10 for gold-yes; InternVL VSR is 34.78 for gold-yes vs 1.20 for gold-no. C-U is concentrated in the direction favored by no-image/blank priors. Table 28's strict T/B requirement collapses Dep (e.g., InternVL GSR 42.84→10.45), consistent with controls usually agreeing with the original answer rather than being neutral. The mean-color blank (Table 37) changes low-level color but still has no objects, so it does not remove the 'objects absent' cue. This is load-bearing for the 12.73–26.25% range. Please add a no-objects background control (or text-only-only reporting) to isolate the gray-bla
  2. [Abstract/Conclusion vs Table 15] The headline range is not direction-invariant. Prediction-balanced C-U is 11.73–43.36% while gold-balanced is 12.73–26.25%, and VSR orderings change. The supplement discloses this, but the main text (abstract, conclusion) states the range without the qualification. Since the C-U estimate is also sensitive to the blank-control concern in Comment 1, the main text should explicitly say that the 12.73–26.25% figure is benchmark-distribution-specific and should report the direction-balanced estimates in the main robustness section.
minor comments (5)
  1. [Abstract/§3.1] Throughout, 'no-image controls' is imprecise for the blank: a gray canvas is still an image. Suggest 'image-absent' or 'content-free image' to avoid implying that the blank contains no visual input.
  2. [Table 1] The last column header 'C-U∆ perm DCC' is hard to parse; split into 'C-U' and 'Δ perm D-CC [CI]' and explain in the caption that Δ is the paired original-minus-permutation D-CC difference.
  3. [§4.3/Figure 1] The 3×3 evidence-source factorial is central to the response-vs-support separation but is only described in text; a diagram of the nine cells (or at least the agreement and conflict cells) would make the design easier to verify.
  4. [Table 2/§4.2] PairCredit = 1{G_int > 0} can be positive even when the original true–false separation is negative if the controls are more negative. The caption's phrase 'image-specific semantic separation' should explicitly state that PairCredit does not require H_o > 0; PairCorrect plays that role.
  5. [Limitations] The Limitations section is honest about causal attribution but does not list the gray-blank neutrality assumption as a limitation; add it explicitly, together with the no-objects control that would test it.

Circularity Check

0 steps flagged

No significant circularity: the audit quantities are prespecified, the two definitional reductions are explicitly disclosed and non-load-bearing, and the headline empirical claims rest on independent control contrasts.

full rationale

The derivation chain is self-contained. D_i = 1{G_img_i > 0} with G_img_i = r_i,o - max(r_i,t, r_i,b) is a prespecified, label-free event; no parameter is fitted to labels, and the zero threshold is fixed before labels are consulted (Eqs. 1-2). The paper explicitly discloses that on correct items this event is algebraically identical to a gold-aligned gain: Eq. (4) states kappa_i D_i = kappa_i 1{s_i m_i,o > max(s_i m_i,t, s_i m_i,b)}, and the text says the event 'intentionally reduces exactly to a gold-aligned positive gain' but 'extends the same order event label-free to all decisions.' That is an honest characterization, not a hidden prediction. Likewise, the Joint => G_int > 0 implication is explicitly labeled 'a structural property' and 'definitional,' and the paper reports PairCredit and mean G_int as the empirical quantities outside Joint, so no derived result is being passed off as evidence. The headline C-U estimates are direct empirical frequencies from cached margins and benchmark labels, not outputs of a fitted model. The matched same-split permutation calibration changes only the image while holding the audited decision fixed; this is an independent empirical contrast, and the 21.25-47.80 point drops are not forced by the definition of D. The factorial and 108-edit natural-image checks provide external, non-self-referential validation. The main substantive threat identified by the skeptical review is whether the fixed mid-gray blank operationalizes 'no benchmark image' neutrally; but the paper states that 'dependence' means this control-relative support advantage, so any concern is about construct validity of the control, not circularity of the derivation. No load-bearing self-citations or imported uniqueness theorems appear.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The central claim has no fitted parameters: the zero threshold is prespecified and non-fitted, and the main result is robust to near-zero boundary mass. The key axioms are the validity of no-image controls and the transfer from procedural factorial scenes to natural benchmark behavior.

free parameters (2)
  • Dependence threshold = 0
    Prespecified zero boundary of G_img; not fitted to labels, but it is a hand-set number. Sensitivity shows only 0.08–0.72% of items have |G_img|≤0.01.
  • Edit renderer constraints (center gap ≥0.05, new-box IoU ≤0.20, padding 4, inpaint radius 3) = see supplement
    Hand-set thresholds for object-swap feasibility; they determine which 114/440 pairs enter the natural-image edit check.
axioms (6)
  • domain assumption Next-token continuation-loss margins on a fixed forced-choice answer interface are a valid measure of the MLLM's support for a decision.
    Used throughout Eq. (1)–(2) and all tables; the paper limits the claim to this interface, but the entire audit rests on this representational assumption.
  • domain assumption Text-only (no image) and blank mid-gray image are valid no-evidence controls representing absence of benchmark-image support.
    Primary D-CC/C-U decomposition in Eq. (1)–(3) depends on max(r_t, r_b) as the null; if blank is an active distractor, C-U is misestimated. Supplement Table 37 checks mean-color blank.
  • domain assumption Sampling the same relation pair with true and false statements cancels answer-surface offsets; H_c differences are comparable across contexts.
    Relation axis Eq. (5) and pair-centered contrasts rely on additivity of query offsets.
  • domain assumption Procedural rendered scenes in the 3×3 factorial are informative about behavior on natural benchmark images.
    Factorial in Experiments; the paper uses 108 human-audited edited natural images as correspondence check, acknowledging transfer is partial.
  • domain assumption The deterministic same-split image permutation is unrelated enough to act as a mismatch calibration.
    Tables 1 and 29; object absence can make a statement false, so mismatch is deliberately supplementary.
  • domain assumption Human majority consensus on 108 edits is a reliable ground truth for relation flips and object visibility.
    Edit set construction; high inter-annotator agreement is reported, but it remains a human judgment.

pith-pipeline@v1.3.0-daily-deepseek · 29851 in / 20735 out tokens · 162434 ms · 2026-08-01T10:08:01.139263+00:00 · methodology

0 comments
read the original abstract

Closed yes/no spatial benchmarks can reward a correct answer even when the image adds little support beyond no-image contexts. Under a fixed forced-choice interface, Visual Credit Audit (VCA) separates two estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence. The first audit is training- and label-free and does not require an answer flip. Applying labels yields dependence-credited correctness (D-CC); on correct items, it equals same-control gold-aligned positive gain, while prediction alignment extends the audit to errors. Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited. Matched same-split image permutation reduces D-CC by 21.25-47.80 points, with every paired 95% interval above zero. Fixed-pixel relation contrasts and a 3x3 evidence-source factorial show why null controls cannot identify relation response. Among controlled correct-but-uncredited agreement decisions, response to relation reversal spans 81.57-100.00%, while 32.11% pooled change answer. Independently audited outcomes on 108 geometry-compatible edits provide a bounded natural-image correspondence check. VCA thereby decomposes benchmark success into correctness, additional image support, and relation-consistent response.

Figures

Figures reproduced from arXiv: 2607.27069 by Feixiang Liu, Huawei Shen, Lanbo Sun, Nan Wei, Qiang Qiu, Xueqi Cheng.

Figure 1
Figure 1. Figure 1: VCA separates relative image support from image–relation interaction. The support audit compares original-image [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: VCA decomposes correctness and control-relative support; same-image relation consistency and paired dependence [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 15 linked inside Pith

  1. [1]

    Answer the user’s question with exactly yes or no,

    Continuous Raw-Gap Distributions BinaryDeprecordsthegapsign;itsmagnitudedistinguishes near-zeroboundarycasesfromstrongper-itemevidence.Ta- ble17showsbroadoverlapacrossmodelsanddatasetsrather thanadegeneratemassatzero.Inparticular,InternVL/GSR- COCO has a negative median and mean but a positive upper quartile, explaining how its Dep rate can remain 42.84%....

  2. [4]

    VisNec: Measuring and Leveraging Visual NecessityforMultimodalInstructionTuning.arXivpreprint arXiv:2603.01195. Fu, H. Y.; Yang, C.; Chang, T.-Y.; Wang, Z.; Wu, Z.; Zhou, J.; and Fu, D

  3. [6]

    InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2901–2910

    CLEVR: A Diag- nostic Dataset for Compositional Language and Elementary Visual Reasoning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2901–2910. Kamath,A.;Hessel,J.;andChang,K.-W.2023. What’s“up” withVision-LanguageModels?InvestigatingTheirStruggle with Spatial Reasoning. InProceedings of the 2023 Confer- enceonEmpiric...

  4. [7]

    Lan, Z.; Sun, L.; Walter, M

    From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA.arXiv preprint arXiv:2606.30220. Lan, Z.; Sun, L.; Walter, M. R.; and Zhou, J

  5. [8]

    Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L

    Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?arXiv preprint arXiv:2605.22903. Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L

  6. [11]

    Ma, C.; Lu, K.; Cheng, T.-Y.; Trigoni, N.; and Markham, A

    Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning.arXiv preprint arXiv:2306.14565. Ma, C.; Lu, K.; Cheng, T.-Y.; Trigoni, N.; and Markham, A

  7. [12]

    Mistral AI

    SpatialPIN: Enhancing Spatial Reasoning Capa- bilities of Vision-Language Models through Prompting and Interacting 3D Priors.arXiv preprint arXiv:2403.13438. Mistral AI

  8. [13]

    Naeini, M

    Ministral 3.arXiv preprint arXiv:2601.08584. Naeini, M. P.; Cooper, G.; and Hauskrecht, M

  9. [14]

    Reich,D.;Putze,F.;andSchultz,T.2023.MeasuringFaithful and Plausible Visual Grounding in VQA

    GSR-BENCH: A Bench- mark for Grounded Spatial Reasoning Evaluation via Multi- modal LLMs.arXiv preprint arXiv:2406.13246. Reich,D.;Putze,F.;andSchultz,T.2023.MeasuringFaithful and Plausible Visual Grounding in VQA. InFindings of the Association for Computational Linguistics: EMNLP 2023, 3129–3144. Rohrbach, A.; Hendricks, L. A.; Burns, K.; Darrell, T.; an...

  10. [16]

    Sun, Z.; Shen, S.; Cao, S.; Liu, H.; Li, C.; Shen, Y.; Gan, C.; Gui, L.-Y.; Wang, Y.-X.; Yang, Y.; et al

    Do Vision-Language Models See or Guess? Measuring and Re- ducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark.arXiv preprint arXiv:2606.10400. Sun, Z.; Shen, S.; Cao, S.; Liu, H.; Li, C.; Shen, Y.; Gan, C.; Gui, L.-Y.; Wang, Y.-X.; Yang, Y.; et al

  11. [17]

    arXiv preprint arXiv:2309.14525

    Aligning LargeMultimodalModelswithFactuallyAugmentedRLHF. arXiv preprint arXiv:2309.14525. Thrush, T.; Jiang, R.; Bartolo, M.; Singh, A.; Williams, A.; Kiela, D.; and Ross, C

  12. [19]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 10006–10030

    MMoE: Enhancing Multimodal ModelswithMixturesofMultimodalInteractionExperts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 10006–10030. Zhang, C.; Ding, W.; Liu, J.; Wu, M.; Wu, Q.; and Mooney, R

  13. [20]

    Zhang, L.; Zhai, X.; Zhao, Z.; Zong, Y.; Wen, X.; and Zhao, B

    Do Images Speak Louder than Words? Investigat- ing the Effect of Textual Misinformation in VLMs.arXiv preprint arXiv:2601.19202. Zhang, L.; Zhai, X.; Zhao, Z.; Zong, Y.; Wen, X.; and Zhao, B

  14. [305]

    Li, Y.; Tian, W.; Jiao, Y.; and Chen, J. 2024b. Eyes Can Deceive: Benchmarking Counterfactual Reasoning Capabil- ities of Multimodal Large Language Models.arXiv preprint arXiv:2404.12966. Liang, X.; Guo, X.; Jin, Z.; Pan, W.; Shang, P.; Cai, D.; Lin, B.;andYe,J.2025.EnhancingSpatialReasoningthroughVi- sualandTextualThinking.arXivpreprintarXiv:2507.20529. ...

  15. [2017]

    In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6904–6913

    Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6904–6913. Guo,C.;Pleiss,G.;Sun,Y.;andWeinberger,K.Q.2017. On Calibration of Modern Neural Networks. InInternational Conference on Machine Learning, 1321–1330. Johnson, J.; Ha...

  16. [2018]

    InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4035–4045

    Object Hallucination in Image Caption- ing. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 4035–4045. Singla, P.; Garg, S.; Singh, V.; and Chopra, P

  17. [2022]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5238–5248

    Winoground: Probing Vision andLanguageModelsforVisio-LinguisticCompositionality. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5238–5248. Wang,W.;Gao,Z.;Gu,L.;Pu,H.;Cui,L.;Wei,X.;Liu,Z.; Jing,L.;Ye,S.;Shao,J.;etal.2025.InternVL3.5:Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency.arXiv...

  18. [2023]

    Bai, S.; et al

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, TextReading,andBeyond.arXivpreprintarXiv:2308.12966. Bai, S.; et al

  19. [2024]

    Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; and Li, C

    Mitigating Object Hallucinations in Large Vision-LanguageModelsthroughVisualContrastiveDecod- ing.ProceedingsoftheIEEE/CVFConferenceonComputer Vision and Pattern Recognition, 13872–13882. Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; and Li, C. 2024a. LLaVA-OneVision:EasyVisualTaskTransfer.arXiv preprint ...

  20. [2025]

    Qwen3-VL Technical Report.arXiv preprint arXiv:2511.21631. Bhat, S. D.; and Yamasaki, T

  21. [2026]

    arXiv preprint arXiv:2607.09068

    OmniMapBench: Benchmark- ing Visual-Centric Reasoning on Diverse Map Documents. arXiv preprint arXiv:2607.09068. Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024b. In- ternVL: Scaling up Vision Foundation Models and Align- ing for Generic Visual-Linguistic Tasks. InProceedings of the IEEE/CVF Confe...