Pith. sign in

REVIEW 3 major objections 4 minor 44 references

Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning on fully controlled synthetic images improves VLM spatial reasoning on real photos, outperforming fine-tuning on the full COCO training set.

desk verdict Solid controlled synthetic-data study with a real transfer signal for Molmo and Qwen2-VL, but the 'synthetic beats full COCO' headline rests on a collapsed fine-tuning baseline, not a data-quality result. read the letter →

arxiv 2511.11440 v3 pith:A6G7TRL7 submitted 2025-11-14 cs.CV cs.CL

Massimo Rizzoli , Simone Alghisi , Seyed Mahed Mousavi , Giuseppe Riccardi This is my paper · ORCID
classification cs.CVcs.CL
keywords syntheticdatavision-languagemodelsfine-tuningspatialreasoningabsolutepositiondatasetbiasdistributionbalancetransferlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the quality, balance, and control of fine-tuning data matter more than sheer quantity for vision-language models. Using a synthetic dataset of simple shapes on black backgrounds, exhaustively varied by color, shape, size, and grid position, the authors fine-tune several state-of-the-art VLMs on a 3×3 absolute-position task. They find that with as few as 130 synthetic samples, models reach near-perfect accuracy on matched synthetic tests and gain 13–21 percentage points on the real-world COCO benchmark, while fine-tuning on the full 161k-image COCO training set collapses accuracy to near zero. The paper concludes that carefully constructed, bias-free synthetic stimuli can serve as an efficient and reliable alternative to conventional real-data fine-tuning for spatial reasoning.

What carries the argument

The load-bearing mechanism is a deterministic synthetic data-generation pipeline (referred to as CIVET) that renders single objects on uniform backgrounds while exhaustively varying color, shape, size, and position across a 9×9 cell grid. This produces a balanced, annotation-error-free dataset for the Absolute Position task, formulated as visual question answering over a 3×3 grid. Models are fine-tuned with LoRA adapters on attention layers, and the controlled data allow the authors to isolate spatial reasoning from dataset biases and measure transfer to the real-world COCO Absolute Position benchmark.

What would settle it

Fine-tune the same VLMs on the full COCO Absolute Position set with careful hyperparameter tuning (e.g., lower learning rate, stronger regularization, or generation constraints) and measure COCO accuracy; if the models recover above 60% instead of collapsing to near zero, the paper's central advantage of synthetic over full real-data training would be substantially weakened or reversed.

Watch

Extended reading notes

Core claim

The central claim is that fine-tuning a VLM on a small, balanced synthetic dataset — where object attributes and positions are sampled exhaustively and annotations are error-free — yields uniform spatial reasoning across the visual scene and transfers to real-world images better than fine-tuning on large, imbalanced real datasets. In the authors' experiments, synthetic fine-tuning improved Molmo from 37% to 58% and Qwen2-VL from 39% to 60% on COCO, whereas fine-tuning on the complete COCO training set dropped both models to 6% and 20%, respectively. A balanced COCO subset of the same size as the synthetic set performs comparably or better on COCO for some models, but the synthetic approach d

Load-bearing premise

The claim that synthetic data outperforms real data assumes that the near-zero accuracy of models fine-tuned on the full COCO set reflects a property of the data itself, not a fine-tuning failure — since the paper reports those models stopped producing valid answers.

Editorial extensions

If this is right

  • If the central claim holds, fine-tuning on balanced synthetic data can replace large-scale real-data collection for spatial reasoning tasks, cutting annotation cost and time dramatically.
  • The finding that 130 synthetic samples achieve most of the transfer gain suggests a path toward extremely sample-efficient fine-tuning for other visual reasoning skills.
  • The observed uniform cell-level accuracy after synthetic fine-tuning implies that models can be trained to reduce positional bias, potentially improving robustness in real-world deployment where objects appear off-center.
  • The near-collapse of models fine-tuned on the full COCO set warns that naive fine-tuning on large imbalanced real datasets can hurt rather than help reasoning, pointing to the need for data-balancing strategies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The stark failure of full-COCO fine-tuning may stem from optimization instability rather than intrinsic data quality; the paper itself reports that models stopped generating valid answers. A carefully tuned full-COCO baseline with adjusted learning rates or early stopping might recover nontrivial accuracy, narrowing the claimed advantage of synthetic data.
  • The balanced COCO subset actually outperforms synthetic fine-tuning on COCO for LLaVA-NeXT (67% vs 43%), suggesting that real data of matched size may be superior when balance is enforced — a comparison the headline glosses over.
  • The distractor experiments hint at a domain-randomization effect: moderate scene clutter improves transfer. This suggests that controllably increasing synthetic scene complexity could approximate real-world distribution shift more closely, a testable extension for other reasoning tasks.
  • The method's reliance on synthetic shapes and colors may not extend to tasks requiring rich semantic understanding (e.g., counting, relations), where real-world object identity matters; applying the same control to semantic attributes is an open question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies whether fully controlled synthetic data can improve VLM spatial reasoning on the absolute-position task (a 3×3 grid over the image). Using the authors' CIVET framework, it constructs balanced synthetic datasets that vary color, shape, size, and position, then fine-tunes four VLMs (LLaVA-NeXT, Molmo, Qwen2-VL, CLIP) with LoRA. RQ1 reports near-perfect accuracy on synthetic evaluation with as few as 1,296 training samples; RQ2 reports transfer gains on a COCO-derived absolute-position test set and claims that synthetic fine-tuning outperforms fine-tuning on the full COCO train set. The paper also includes distractor ablations and layer-wise probing analyses.

Significance. If the central comparison were valid, the paper would be a strong practical result: 1.3k controlled synthetic samples improving real-world spatial reasoning more than 161k real images would be highly relevant to data-efficient VLM adaptation. The strengths are the exhaustively balanced synthetic design, evaluation on an external real-world benchmark (COCO), five-run standard deviations, and the promise of released materials. However, the headline 'synthetic beats full COCO' claim rests on degenerate full-COCO baselines that are acknowledged to be fine-tuning collapses, and the matched balanced-COCO baseline actually matches or exceeds synthetic transfer for two of three multimodal models. The core Molmo/Qwen transfer effect is real and interesting, but the paper's broader 'quality over quantity' conclusion is not yet established.

major comments (3)
  1. [Table 2 and §III.1] The headline comparison 'synthetic fine-tuning outperforms models fine-tuned on the full COCO train set' rests entirely on the COCO Complete (161k) rows of Table 2 (LLaVA 0, Molmo 6, Qwen 20). §III.1 attributes these to output degeneration: LLaVA stops generating outputs, Molmo loops over {center, right, left}, and Qwen emits valid-but-wrong answers. These are textbook signs of fine-tuning instability, not evidence about dataset quality. Because a single LoRA configuration (rank 32, α=64, lr=1e-4, up to 10 epochs, §I) is applied to every dataset, the full-COCO baseline is not a fair representative of real-data fine-tuning. Moreover, the balanced COCO subset (1.3k) already gives LLaVA 67 and Qwen 61 on COCO, matching or exceeding synthetic transfer (43 and 60). The claim that 'quality, balance, and control outweigh sheer quantity' is therefore unproven; a properly tuned full-COCO baseline
  2. [Table 4 and Abstract] The abstract's '13% improvement on COCO' corresponds to LLaVA-NeXT, whose Table 4 result is 43±17 versus a 30 baseline. With a standard deviation of 17 across five runs, the +13-point gain is not statistically meaningful. The robust transfer effects are Molmo (+21±5) and Qwen2-VL (+21±4). The paper should either report confidence intervals and emphasize the consistent models, or stop presenting LLaVA's noisy gain as a headline. This is compounded by Fig. 3, where LLaVA's transfer peaks at 130 samples and declines with more data, suggesting overfitting rather than a uniform transfer benefit.
  3. [Section 4.A / RQ2 framing] The paper introduces two 'matched' settings without making the comparison target explicit. In RQ2, the matched setting is the full COCO training set, but Table 2 also includes a balanced COCO subset that is matched in size to the synthetic set. The conclusion that 'fine-tuning on real-world data from the same distribution ... does not consistently achieve the robustness achieved through our synthetic approach' is based on the full-COCO collapse, not on the balanced subset, which actually achieves comparable or better COCO accuracy for LLaVA and Qwen. The narrative should distinguish the size-matched comparison from the degenerate full-COCO comparison.
minor comments (4)
  1. [Table 3] The training-set column header reads 'Synthetic (13k)' while the text and Table 2 consistently use 1,296 / 1.3k samples. Correct this typo.
  2. [Fig. 1B caption] The caption says 'color-code the cells according to the most frequently predicted position' but the figure appears to render predicted region labels with a colormap. Clarify the mapping and whether the majority vote is over all object variations per cell.
  3. [Section 5.A] The claim that moderate distractors 'align synthetic and real-world scene statistics' is asserted without a quantitative comparison of scene complexity (e.g., object counts or positional distributions). Either add such a comparison or soften the claim.
  4. [Section 3.2] The terms 'unmatched' and 'matched' are used for both the synthetic evaluation and the COCO evaluation. Define them explicitly at first use to avoid ambiguity (e.g., matched = train/test from same distribution; unmatched = synthetic train, real test).

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the transfer claim is tested on external COCO data, and the only notable self-citation (CIVET) is not central to the derivation.

full rationale

The paper's derivation is essentially self-contained. The synthetic data are generated deterministically by the authors' CIVET framework, which is a prior self-citation, but the paper describes the construction explicitly: attributes (color, shape, size, position) are systematically varied, training and synthetic test use disjoint color-shape combinations, and the data balance is a property of the generation procedure rather than an imported, unverified result. The central transfer claim -- that synthetic fine-tuning improves real-world COCO performance -- is evaluated on an external benchmark: COCO images with ground-truth labels derived from COCO annotations, independent of the synthetic training distribution. The matched COCO-subset comparison is also built from external COCO data. No prediction is fitted from the COCO test set, and no uniqueness theorem is invoked to force the authors' choice. The main vulnerability noted by the skeptic, namely the near-zero full-COCO baseline (LLaVA 0, Molmo 6, Qwen 20; Table 2 and §III.1), is attributed by the paper to output degeneration (LLaVA stops generating, Molmo loops over {center, right, left}) and is best characterized as a baseline-comparison / optimization-stability concern rather than a circular step: it does not make the synthetic-transfer result equivalent to its inputs by construction. The only self-citation to CIVET appears for image resolution (672x672) and for the 'Answer with as few words as possible' prompt; neither is load-bearing for the paper's main finding. Therefore no specific reduction or fitted-input-renamed-as-prediction can be exhibited, and the appropriate score is 2 for minor, non-load-bearing self-citation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. It relies on the authors' synthetic generator (CIVET) and on several dataset-construction choices. Key free choices are the 3x3 grid resolution and the LoRA recipe, both disclosed but not independently varied. The single-instance COCO filter and the bbox-center labeling are data assumptions whose influence is not quantified.

free parameters (2)
  • 3x3 answer grid + 9x9 positional cells = 3x3 grid for answers; 9x9 subcells for generated object positions
    Chosen by hand to define the absolute-position task (Sec 3.1). Results depend on this discretization, but it is not fitted to data.
  • LoRA hyperparameters = rank=32, alpha=64, lr=1e-4, up to 10 epochs, early stopping patience 2
    Disclosed in Supplementary §I. Not fitted against COCO transfer, so they do not inflate the transfer numbers, but the full-COCO collapse may be sensitive to these choices.
assumptions (4)
  • domain assumption CIVET synthetic generation is controlled, exhaustive, balanced, and error-free as claimed in the authors' prior work.
    Sec 3.2 states this; the synthetic training and matched-eval sets are built on it. If the generated position labels are wrong, the matched results are meaningless.
  • domain assumption Bounding-box center is a sufficient ground-truth for absolute position in the COCO-derived task.
    Sec 3.2: target object position is computed as the center of its bbox and assigned to one of nine grid regions. Objects spanning cell borders get a single hard label, which can be noisy.
  • domain assumption The COCO-derived train/eval partition and the 'balanced COCO subset' (1.3k) are representative matched real-world settings.
    The balanced subset is constructed by the authors; its balance differs from the natural COCO distribution, and the single-instance filter removes common ambiguous cases (Sec 3.2).
  • domain assumption Accuracy on closed-ended absolute-position questions is a meaningful measure of spatial reasoning.
    The task is a nine-way multiple choice on grid cells; performance could reflect simple edge/center heuristics rather than a general spatial model. This is the construct the paper aims to measure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation." pith.science (2026). https://pith.science/paper/A6G7TRL7

@misc{pith2026251111440,
  author       = {Pith},
  title        = {Pith review of: Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6G7TRL7}},
  note         = {Machine review of arXiv:2511.11440}
}
read the original abstract

Performance gains of Vision Language Models (VLMs) obtained by fine-tuning are generally based on ad hoc data collection and annotation of real-world scenes. Despite the improvements, this process is often prone to biases, errors, and distribution imbalance, resulting in overfitting and imbalanced performance. Although a few studies have explored synthetic data generation, they typically lack control over data distribution and annotation quality. In this work, we re-evaluate the potential of model fine-tuning by exploring a fully controlled data generation and annotation pipeline, obtaining bias-free data with balanced distribution and clean annotations. Using the spatial reasoning task of identifying the absolute position of an object as a use case, we fine-tune state-of-the-art VLMs and conduct exhaustive evaluations on both synthetic and real-world benchmarks, including transferability to real-world scenes. Our experiments reveal two key findings: 1) fine-tuning on balanced data yields uniform performance across the visual scene and mitigates common biases with as few as 130 samples; and 2) fine-tuning on synthetic stimuli improves performance by 13% on real-world data (COCO), outperforming models fine-tuned on the full COCO train set.

Figures

Figures reproduced from arXiv: 2511.11440 by the authors.

Figure 1
Figure 1. A) Cell-Level Accuracy and B) Spatial Prediction of VLMs. A) Accuracy averaged over object variations across a 9 × 9 grid shows pronounced positional biases before fine-tuning. All models perform best in upper regions, with consistently low accuracy in center-left and center-right cells; CLIP exhibits an extreme central bias, failing elsewhere. B) Majority-vote predictions reveal how models internally distort spatia… view at source ↗
Figure 2
Figure 2. Effect of synthetic dataset scale on synthetic test. Ac￾curacy on the Absolute Position task as a function of the number of synthetic training stimuli. All VLMs rapidly improve with in￾creasing data and reach near-optimal accuracy using 10% of the full dataset, demonstrating the sample efficiency of our balanced synthetic training set [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Effect of synthetic training scale on COCO Absolute Position (unmatched). Accuracy on real-world evaluation as a function of the number of synthetic training stimuli. tively spatial reasoning learned from synthetic data trans￾fers to real-world images. Each model is fine-tuned on the synthetic dataset of 1,296 balanced stimuli and sub￾sequently tested both on the synthetic and COCO bench￾marks. To compare the perfor… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Model predictions on the COCO Absolute Position test set after fine-tuning on different data sources (by majority voting); A) Models fine-tuned on synthetic data; and, B) Models fine-tuned on COCO. performing best in the upper and lower regions while strug￾gling in the…
Figure 6
Figure 6. Figure 6: shows the layer-wise probing accuracy for Qwen2-VL 7B (results for LLaVA-NeXT and Molmo are reported in §III.4). On synthetic data, accuracy rapidly in￾creases for all models in early layers and saturates in the upper-middle layers. In contrast, on COCO the same trend …
Figure 8
Figure 8. Figure 8: Cell-level accuracy of Molmo 7B before and after fine-tuning on synthetic data, on both the synthetic and COCO Absolute Position test sets. Base Model Fine-Tuned Synthetic COCO [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Cell-level accuracy of CLIP before and after fine￾tuning on synthetic data, on both the synthetic and COCO Abso￾lute Position test sets. increases. When tested with five distractors, Molmo and Qwen2-VL show little to no benefit from fine-tuning with distractors, while …
Figure 10
Figure 10. Figure 10: Layer-wise probing accuracy of LLaVA-NeXT 7B before (blue) and after (orange) fine-tuning on the synthetic dataset, evaluated on Synthetic (top) and COCO (bottom). Error bars represent standard deviation across fine-tuning runs. 0 5 10 15 20 25 30 Layer Number 40 60 8…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 2 linked inside Pith

  1. [1]

    Tal- lyqa: Answering complex counting questions.Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 8076–8084, 2019

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tal- lyqa: Answering complex counting questions.Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 8076–8084, 2019. 1

    2019

  2. [2]

    Lipton, and J

    Sumukh K Aithal, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. Understanding hallucinations in diffusion models through mode interpolation. InAdvances in Neu- ral Information Processing Systems, pages 134614–134644. Curran Associates, Inc., 2024. 2

    2024

  3. [3]

    [De|Re] constructing VLMs’ Reasoning in Counting.arXiv preprint arXiv:2510.19555, 2025

    Simone Alghisi, Gabriel Roccabruna, Massimo Rizzoli, Seyed Mahed Mousavi, and Giuseppe Riccardi. [De|Re] constructing VLMs’ Reasoning in Counting.arXiv preprint arXiv:2510.19555, 2025. 8

    arXiv 2025

  4. [4]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 14455– 14465, 2024. 1

    2024

  5. [5]

    Are we on the right way for eval- uating large vision-language models? InAdvances in Neural Information Processing Systems, pages 27056–27087

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for eval- uating large vision-language models? InAdvances in Neural Information Processing Systems, pages 27056–27087. Cur- ran Associates, Inc., 2024. 1

    2024

  6. [6]

    Why is spatial reasoning hard for VLMs? an attention mechanism perspective on focus ar- eas

    Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for VLMs? an attention mechanism perspective on focus ar- eas. InForty-second International Conference on Machine Learning, 2025. 1

    2025

  7. [7]

    From the least to the most: Building a plug-and-play visual rea- soner via data synthesis

    Chuanqi Cheng, Jian Guan, Wei Wu, and Rui Yan. From the least to the most: Building a plug-and-play visual rea- soner via data synthesis. InProceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, pages 4941–4957, Miami, Florida, USA, 2024. Association for Computational Linguistics. 1, 2

    2024

  8. [8]

    Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison- Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Y...

    2025

Show all 44 references
  1. [9]

    If CLIP could talk: Understanding vision-language model representations through their preferred concept descriptions

    Reza Esfandiarpoor, Cristina Menghini, and Stephen Bach. If CLIP could talk: Understanding vision-language model representations through their preferred concept descriptions. InProceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pages 9797...

    2024

  2. [10]

    Hidden in plain sight: VLMs overlook their visual rep- resentations

    Stephanie Fu, tyler bonnen, Devin Guillory, and Trevor Dar- rell. Hidden in plain sight: VLMs overlook their visual rep- resentations. InSecond Conference on Language Modeling,

  3. [11]

    Generate then select: Open- ended visual question answering guided by world knowl- edge

    Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, Alexander Hanbo Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, and Bing Xiang. Generate then select: Open- ended visual question answering guided by world knowl- e...

    2023

  4. [12]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InComputer Vision – ECCV 2024, pages 148–166, Cham, 2025. Springer Nature Switzerland. 1

    2024

  5. [13]

    G-LLaV A: Solving geometric problem with multi-modal large language model

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing HONG, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-LLaV A: Solving geometric problem with multi-modal large language model. InThe Thirteenth International Conference on Learning Repre...

    2025

  6. [14]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 1

    2017

  7. [15]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations, 2022. 4, 1

    2022

  8. [16]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elemen- tary visual reasoning. InProceedings of the IEEE Confer- ence on Computer Vision and Pattern Recogn...

  9. [17]

    What’s “up” with vision-language models? investigating their strug- gle with spatial reasoning

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? investigating their strug- gle with spatial reasoning. InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Pro- cessing, pages 9161–9175, Singapore, 2023. Associa...

    2023

  10. [18]

    Seunghoi Kim, Chen Jin, Tom Diethe, Matteo Figini, Henry F. J. Tregidgo, Asher Mullokandov, Philip Teare, and Daniel C. Alexander. Tackling structural hallucination in im- age translation with local diffusion. InComputer Vision – ECCV 2024, pages 87–103, Cham, 2025. Springer N...

    2024

  11. [19]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. InProceedings of the IEEE/CVF International Conference on Computer Vision...

    2023

  12. [20]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123(1):32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer...

    2017

  13. [21]

    Enhancing vision-language com- positional understanding with multimodal synthetic data

    Haoxin Li and Boyang Li. Enhancing vision-language com- positional understanding with multimodal synthetic data. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 24849–24861,

  14. [22]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, pages 740–755, Cham,

    2014

  15. [23]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 3

    2024

  16. [24]

    Enhanc- ing spatial reasoning in multimodal large language mod- els through reasoning-based segmentation.arXiv preprint arXiv:2506.23120, 2025

    Zhenhua Ning, Zhuotao Tian, Shaoshuai Shi, Guangming Lu, Daojing He, Wenjie Pei, and Li Jiang. Enhanc- ing spatial reasoning in multimodal large language mod- els through reasoning-based segmentation.arXiv preprint arXiv:2506.23120, 2025. 2

    2025 arXiv

  17. [25]

    SpaRE: Enhancing spatial reasoning in vision-language models with synthetic data

    Michael Ogezi and Freda Shi. SpaRE: Enhancing spatial reasoning in vision-language models with synthetic data. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7855–7875, Vienna, Austria, 2025. Associatio...

    2025

  18. [26]

    Teaching clip to count to ten

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In2023 IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 3147–3157, 2023. 1

    2023

  19. [27]

    Synthetic visual genome

    Jae Sung Park, Zixian Ma, Linjie Li, Chenhao Zheng, Cheng-Yu Hsieh, Ximing Lu, Khyathi Chandu, Quan Kong, Norimasa Kobori, Ali Farhadi, Yejin Choi, and Ranjay Kr- ishna. Synthetic visual genome. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2025

  20. [28]

    Synthesize diagnose and optimize: Towards fine- grained vision-language understanding

    Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zux- uan Wu. Synthesize diagnose and optimize: Towards fine- grained vision-language understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13279–13288, 2024. 1, 2

    2024

  21. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the...

    2021

  22. [30]

    Vision language models are blind

    Pooyan Rahmanzadehgervi, Logan Bolton, Moham- mad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. InProceedings of the Asian Conference on Computer Vision (ACCV), pages 18–34, 2024. 1, 2

    2024

  23. [31]

    CIVET: Systematic evaluation of understanding in VLMs

    Massimo Rizzoli, Simone Alghisi, Olha Khomyn, Gabriel Roccabruna, Seyed Mahed Mousavi, and Giuseppe Riccardi. CIVET: Systematic evaluation of understanding in VLMs. InFindings of the Association for Computational Linguis- tics: EMNLP 2025, pages 4462–4480, Suzhou, China, 2025....

    2025

  24. [32]

    Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata

    Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with ran- dom words and broad concepts. In2023 IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 15700–157...

    2023

  25. [33]

    Forgotten polygons: Multimodal large language models are shape-blind

    William Rudman, Michal Golovanevsky, Amir Bar, Vedant Palit, Yann LeCun, Carsten Eickhoff, and Ritambhara Singh. Forgotten polygons: Multimodal large language models are shape-blind. InFindings of the Association for Computa- tional Linguistics: ACL 2025, pages 11983–11998, Vi...

    2025

  26. [34]

    Laion- 400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Robert Kaczmarczyk, Aran Komat- suzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Jenia Jitsev, Theo Coombes, and Clayton Mullis. Laion- 400m: Open dataset of clip-filtered 400 million image-text pairs. InNeurIPS Workshop Datacentric AI, number FZJ- 20...

    2022

  27. [35]

    Math- LLaV A: Bootstrapping mathematical reasoning for multi- modal large language models

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math- LLaV A: Bootstrapping mathematical reasoning for multi- modal large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 4663...

    2024

  28. [36]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 3

    2024 arXiv

  29. [37]

    Embodied scene understanding for vi- sion language models via metavqa

    Weizhen Wang, Chenda Duan, Zhenghao Peng, Yuxin Liu, and Bolei Zhou. Embodied scene understanding for vi- sion language models via metavqa. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22453–22464, 2025. 2

    2025

  30. [38]

    Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Ren- liang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A...

    2024

  31. [39]

    When and why vision- language models behave like bags-of-words, and what to 10 do about it? InThe Eleventh International Conference on Learning Representations, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to 10 do about it? InThe Eleventh International Conference on Learning Representations, 2023. 1

    2023

  32. [40]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InComputer Vision – ECCV 2024, pages 169–186, Cham, 2025

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InComputer Vision – ECCV 2024, pages 169–1...

    2024

  33. [41]

    MA VIS: Mathe- matical visual instruction tuning with an automatic data en- gine

    Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Shang- hang Zhang, Peng Gao, and Hongsheng Li. MA VIS: Mathe- matical visual instruction tuning with an automatic data en- gine. InThe Thirteenth International Conference on...

    2025

  34. [42]

    Answer with as few words as possible

    Fangrui Zhu, Jianwei Yang, and Huaizu Jiang. Towards flex- ible visual relationship segmentation. InAdvances in Neu- ral Information Processing Systems, pages 107633–107661. Curran Associates, Inc., 2024. 2 11 From Synthetic Scenes to Real Performance: Enhancing Spatial Reason...

    2024

  35. [2014]

    Springer International Publishing. 2, 3

  36. [2023]

    Association for Computational Linguistics. 1

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.