Pith. sign in

REVIEW 2 major objections 2 minor 20 references

Ill-Posed by Design: Probing Evidence Use in VLMs

T0 review · 2 major / 2 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models rely on target identity rather than scene geometry when estimating object sizes from single images.

desk verdict The ill-posed size probe is a clean framing move, but the six-channel counterfactual claims rest on unshown isolation details that the stress-test concern rightly flags. read the letter →

arxiv 2606.24335 v1 pith:HJZZBNXV submitted 2026-06-23 cs.CV

Boaz Meivar , Shaked Perek , Shani Shvartzman , Eli Schwartz , Shai Avidan This is my paper · ORCID
classification cs.CV
keywords vision-languagemodelsmetricobjectsizeestimationcounterfactualanalysisscenegeometryevidenceselectionill-posedtasksmonocularvisionmodelprobing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper treats monocular metric object-size estimation as an ill-posed diagnostic task because physical size cannot be recovered from one uncalibrated image, forcing models to pick among imperfect cues. It assembles Metric VQA with over ten thousand queries and runs counterfactual edits on six evidence channels to measure each channel's contribution. Even the largest VLMs fall short of a text-only frontier model on in-the-wild scenes, with target identity emerging as the dominant cue and global geometry largely ignored. Fine-tuning raises accuracy without inducing geometry use. The design therefore exposes systematic patterns in how current models select visual evidence.

What carries the argument

Counterfactual decomposition of six evidence channels inside the ill-posed monocular metric object-size estimation task.

What would settle it

A VLM that matches or exceeds the text-only baseline on the in-the-wild split while ablation tests show measurable sensitivity to scene-geometry edits would falsify the central claim.

Watch

Extended reading notes

Core claim

Because physical size is indeterminate from a single image, models must choose among imperfect cues including category priors, target appearance, local context, apparent image size, scene geometry, and language. Counterfactual analysis across twelve VLMs reveals that target identity is the strongest cue, scene geometry is largely ignored, and frontier models still trail text-only baselines on in-the-wild data. The task proves learnable under LoRA but geometry use does not emerge.

Load-bearing premise

Counterfactual modifications to the six evidence channels cleanly isolate each channel's contribution without introducing unintended side effects on model processing.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes monocular metric object-size estimation as an ill-posed diagnostic task to study evidence selection in VLMs, since physical size cannot be recovered from a single uncalibrated image. The authors assemble the Metric VQA dataset (10,813 Objectron queries + 331 tape-measured in-the-wild scenes) and evaluate 12 open-weight VLMs (3–397 B parameters) via counterfactual analysis that decomposes six evidence channels (category priors, target appearance, local context, apparent image size, scene geometry, language). Headline results: even the largest VLMs trail a text-only frontier LLM on the in-the-wild split; target identity is the most load-bearing cue while global scene geometry is largely unused. LoRA fine-tuning is examined as a task-specific intervention.

Significance. If the counterfactuals cleanly isolate the six channels, the work supplies a concrete, falsifiable probe for how VLMs combine visual and linguistic cues on genuinely ill-posed problems and documents a concrete limitation (under-use of scene geometry) that survives scale. The dataset, the in-the-wild split, and the comparison against a text-only LLM are concrete strengths that would make the findings useful for the community studying multimodal evidence use.

major comments (2)
  1. [Methods / Counterfactual Analysis] Methods / Counterfactual Analysis section: the manuscript provides no implementation details on how the six evidence channels were isolated (e.g., exact image-editing operations for scene-geometry edits, prompt templates for language ablation, or controls ensuring that an edit to one channel does not alter statistics of another). Without these, the attribution of performance deltas to specific channels cannot be verified and the central diagnostic claims remain only partially supported.
  2. [Results / In-the-wild split] Results on in-the-wild split (Table X and associated text): the claim that target identity is the dominant cue and scene geometry is unused rests on the assumption that the 331-scene counterfactuals cleanly manipulate only the intended channel. The paper does not report checks for unintended side-effects (e.g., whether depth/perspective edits also shift apparent size or local context statistics), which directly undermines the load-bearing conclusions.
minor comments (2)
  1. [Abstract] The abstract states dataset sizes and model counts but omits any mention of how error bars or statistical significance were computed; adding one sentence would improve clarity.
  2. [Figures] Figure captions for the counterfactual examples should explicitly list which channel is being ablated in each panel.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which highlight important areas for improving the clarity and rigor of our counterfactual analysis. We address each point below.

read point-by-point responses

  1. Referee: [Methods / Counterfactual Analysis] Methods / Counterfactual Analysis section: the manuscript provides no implementation details on how the six evidence channels were isolated (e.g., exact image-editing operations for scene-geometry edits, prompt templates for language ablation, or controls ensuring that an edit to one channel does not alter statistics of another). Without these, the attribution of performance deltas to specific channels cannot be verified and the central diagnostic claims remain only partially supported.

    Authors: We agree that the current manuscript lacks sufficient implementation details for full reproducibility and verification of the channel isolations. In the revised version, we will add a dedicated subsection in Methods detailing the exact operations used for each evidence channel, including image-editing procedures for visual ablations (such as depth and perspective manipulations for scene geometry), the full set of prompt templates for language and other ablations, and any statistical controls or sanity checks performed to ensure edits primarily affect the intended channel. This will allow readers to verify the attribution of performance changes. revision: yes

  2. Referee: [Results / In-the-wild split] Results on in-the-wild split (Table X and associated text): the claim that target identity is the dominant cue and scene geometry is unused rests on the assumption that the 331-scene counterfactuals cleanly manipulate only the intended channel. The paper does not report checks for unintended side-effects (e.g., whether depth/perspective edits also shift apparent size or local context statistics), which directly undermines the load-bearing conclusions.

    Authors: The referee correctly identifies a gap in our reporting. While the counterfactuals were designed to target specific channels, we did not include explicit side-effect analyses in the original submission. We will perform and report additional checks in the revision, such as measuring changes in apparent size and local context statistics before and after geometry edits on the 331 scenes. If any significant unintended effects are found, we will discuss their implications for the conclusions regarding target identity and scene geometry usage. This will strengthen the support for our diagnostic claims. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: purely empirical evaluation with direct measurements

full rationale

The paper conducts an empirical evaluation of VLMs on a constructed Metric VQA dataset using counterfactual modifications to six evidence channels. No derivations, equations, fitted parameters renamed as predictions, or self-citation chains appear in the load-bearing claims. All conclusions rest on observed model outputs on held-out splits (Objectron and in-the-wild), which are independently falsifiable and do not reduce to the paper's own inputs by construction.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the domain assumption that monocular size estimation is ill-posed and that counterfactual edits isolate cue contributions; no free parameters or new entities are introduced.

assumptions (1)
  • domain assumption Physical size cannot be determined from a single uncalibrated image, forcing reliance on imperfect cues
    Explicitly stated in the abstract as the reason the task serves as an ill-posed diagnostic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ill-Posed by Design: Probing Evidence Use in VLMs." pith.science (2026). https://pith.science/paper/HJZZBNXV

@misc{pith2026260624335,
  author       = {Pith},
  title        = {Pith review of: Ill-Posed by Design: Probing Evidence Use in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJZZBNXV}},
  note         = {Machine review of arXiv:2606.24335}
}
abstract

Counterfactual analysis is widely used to study evidence use in vision-language models, but its diagnostic value is limited on well-posed tasks: when several cues independently support the same answer, removing one may not change the prediction. We propose monocular metric object-size estimation as an ill-posed diagnostic setting for evidence selection: because physical size cannot be determined from a single uncalibrated image, models must rely on imperfect cues category priors, target appearance, local context, apparent image size, and scene geometry. We assemble Metric VQA ($10{,}813$ dimension queries from Objectron and $331$ tape-measured in-the-wild scenes) and evaluate $12$ open-weight VLMs ($3$--$397$\,B parameters) with counterfactual analysis decomposing six visual and language evidence channels. Even the largest VLMs tested (Qwen3-VL-235B, Qwen3.5-397B, InternVL3.5-241B) trail a text-only frontier LLM on the in-the-wild split. The diagnostic analysis shows: target identity is the most load-bearing cue, target pixels and local context help only some models, apparent size shifts predictions without a directional readout, and global scene geometry is largely unused. We analyze LoRA fine-tuning as an actionable intervention specific to metric estimation: while the task is learnable, the models do not learn to leverage scene geometry.

Figures

Figures reproduced from arXiv: 2606.24335 by the authors.

Figure 1
Figure 1. Ill-posed metric estimation as a diagnostic for evidence use in VLMs. We apply six image-level interventions (object-only mask, wrong-object substitution, context ring, zoom 0.6×/1.4×, lens distortion) and a text-only ablation, holding the query fixed and re-querying 12 open-weight VLMs spanning 3–397B parameters. Each panel pairs the intervention with the headline finding across the pool; full per-model signatures … view at source ↗
Figure 2
Figure 2. Per-VLM intervention signature on the in-the-wild split. Each panel is one VLM; axes are the interventions defined in Section 4. The solid black polygon traces Acc@10% on each intervention. The orange dashed hexagon marks the model’s full-image Acc@10% (anchor for no-image / object_ring / lens dist.); the green dashed circle marks its object_only Acc@10% (anchor for wrong_object / zoom-in / zoom-out). A vertex insid… view at source ↗
Figure 3
Figure 3. Per-model base-vs-LoRA intervention signatures. Each panel overlays base (orange) and LoRA￾finetuned (blue) Acc@10% on the intervention axes of [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Object-class composition of the two splits. Each slice is one distinct object class, sized by query count and colored by mean GT on a shared log-cm scale. Left: in-the-wild (n=331) contains 99 distinct objects. Right: Objectron (n=10,482) contains 9. 0 5 10 15 20 25 30…
Figure 5
Figure 5. Figure 5: Image-conditioned versus language-prior Acc@10% on the in-the-wild split. x: each model’s language-prior Acc (text-only, or gray-blank fallback). y: same model’s Acc with the original image. Points above the diagonal: image helped. Dotted purple line: ChatGPT 5.5 text-…
Figure 6
Figure 6. Figure 6: Cumulative accuracy as a function of the relative-error tolerance k. x: tolerance k ∈ [0, 50]%. y: fraction of queries with |yˆ − y|/|y| ≤ k. Dotted vertical at k=10% is the headline metric. Dashed lines: ChatGPT 5.5 text-only (both splits); human reference (in-the-wil…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 2 canonical work pages

  1. [1]

    Li, Puyin and Xiang, Tiange and Mao, Ella and Wei, Shirley and Chen, Xinye and Masood, Adnan and Li, Fei-Fei and Adeli, Ehsan , journal =

  2. [2]

    Cai, Zhipeng and Yeh, Ching-Feng and Xu, Hu and Liu, Zhuang and Meyer, Gregory and Lei, Xinjie and Zhao, Changsheng and Li, Shang-Wen and Chandra, Vikas and Shi, Yangyang , journal =

  3. [3]

    Advances in Neural Information Processing Systems , year =

    Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , author =. Advances in Neural Information Processing Systems , year =

  4. [4]

    Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei , booktitle =

  5. [5]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year =

    Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models , author =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year =

    2024

  6. [6]

    Making the

    Goyal, Yash and Khot, Tejas and Summers-Stay, Douglas and Batra, Dhruv and Parikh, Devi , booktitle =. Making the

  7. [7]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  8. [8]

    Hsieh, Cheng-Yu and Zhang, Jieyu and Ma, Zixian and Kembhavi, Aniruddha and Krishna, Ranjay , booktitle =

Show all 20 references
  1. [9]

    Zhao, Tiancheng and Zhang, Tianqi and Zhu, Mingwei and Shen, Haozhan and Lee, Kyusong and Lu, Xiaopeng and Yin, Jianwei , journal =

  2. [10]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , year =

    Interpretable Explanations of Black Boxes by Meaningful Perturbation , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , year =

  3. [11]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild With Pose Annotations , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  4. [12]

    Masry, Ahmed and Long, Do Xuan and Tan, Jia Qing and Joty, Shafiq and Hoque, Enamul , booktitle =

  5. [13]

    2024 , howpublished =

    2024

  6. [14]

    arXiv preprint arXiv:2502.13923 , year =

    Qwen2.5-VL Technical Report , author =. arXiv preprint arXiv:2502.13923 , year =

    arXiv

  7. [15]

    Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and others , booktitle =

  8. [16]

    Advances in Neural Information Processing Systems , year =

    Visual Instruction Tuning , author =. Advances in Neural Information Processing Systems , year =

  9. [17]

    Deitke, Matt and Clark, Christopher and Lee, Sangho and others , booktitle =

  10. [18]

    arXiv preprint arXiv:2504.05299 , year =

    Marafioti, Andr. arXiv preprint arXiv:2504.05299 , year =

    arXiv

  11. [19]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle =

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle =

  12. [20]

    2025 , journal =

    Carion, Nicolas and Gustafson, Laura and Hu, Yuan-Ting and Debnath, Shoubhik and Hu, Ronghang and Suris, Didac and Ryali, Chaitanya and Alwala, Kalyan Vasudev and Khedr, Haitham and Huang, Andrew and Lei, Jie and Ma, Tengyu and Guo, Baishan and Kalla, Arpit and Marks, Markus a...

    2025

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.