REVIEW 11 cited by
Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Vision-Language Models (VLMs) have recently emerged as powerful tools, excelling in tasks that integrate visual and textual comprehension, such as image captioning, visual question answering, and image-text retrieval. However, existing benchmarks for VLMs include spatial components, which often fail to isolate spatial reasoning from related tasks such as object detection or semantic comprehension. In this paper, we address these deficiencies with a multi-faceted approach towards understanding spatial reasoning. Informed by the diverse and multi-dimensional nature of human spatial reasoning abilities, we present a detailed analysis that first delineates the core elements of spatial reasoning: spatial relations, orientation and navigation, mental rotation, and spatial visualization, and then assesses the performance of these models in both synthetic and real-world images, bridging controlled and naturalistic contexts. We analyze 13 state-of-the-art Vision-Language Models, uncovering pivotal insights into their spatial reasoning performance. Our results reveal profound shortcomings in current VLMs, with average accuracy across the 13 models approximating random chance, highlighting spatial reasoning as a persistent obstacle. This work not only exposes the pressing need to advance spatial reasoning within VLMs but also establishes a solid platform for future exploration. Code available on GitHub (https://github.com/stogiannidis/srbench) and dataset available on HuggingFace (https://huggingface.co/datasets/stogiannidis/srbench).
Forward citations
Cited by 11 Pith papers
-
A Deep Learning Model of Mental Rotation Informed by Interactive VR Experiments
A deep neural model with equivariant, symbolic, and agentic components reproduces human accuracy and action counts on Shepard-Metzler mental rotation, but not full reaction-time behavior.
-
Foveated Probes Recover Localized Binding Information in Vision Foundation Models
A foveated attention readout over frozen patch tokens recovers most of the localized binding signal that global pooling discards, so apparent spatial blindness can be readout-level rather than representational.
-
Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text
Image-generation models become competitive on spatial benchmarks when answers are expressed as protocol-constrained pixels, while text-output VLMs retain an edge on compositional reasoning — with a parser-sensitivity ...
-
It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Models
TickTockVQA (12k real-world clocks) plus Swap-DPO lifts Llama-3.2-11B full-time analog clock accuracy from 1.41% to 46.22%, far above synthetic-data baselines.
-
The N-Body Problem: Parallel Execution from Single-Person Egocentric Video
Structured prompting for a VLM, fed with ground-truth spatial zone schedules, can generate more feasible multi-agent parallel executions from single-person egocentric videos.
-
11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis
A new spatial reasoning benchmark shows current multimodal models lag humans badly and lack the item-level predictability humans show.
-
AntiGrounding: Lifting Robotic Actions into VLM Representation Space for Decision Making
AntiGrounding lifts candidate robot trajectories into the VLM's visual space via multi-view rendering and structured VQA, and reports 57.5% average success across eight manipulation tasks, beating three intermediate-r...
-
BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Models
A Blender-based diagnostic toolkit that tests VLMs on fine-grained visual skills by varying one visual attribute at a time, exposing failure modes that coarse benchmarks miss.
-
Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models
Current vision-language models fall far short of humans on spatial reasoning, especially when they must generate answers directly instead of choosing from options.
-
OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignment
OrientSAM injects Fourier-encoded object orientation into a vision-language model and uses curriculum training, improving reference-centric spatial reasoning and reducing camera-centric shortcut behavior.
-
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
A 235-item multimodal stress-test shows frontier closed models outpace open-weight peers by ~10% and leaves shared failures on counting, spatial, and character-level tasks.
Discussion (0). Continue with ORCID to comment.