Pith. sign in

REVIEW 3 major objections 4 minor 4 cited by

11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current multimodal large language models show early signs of spatial cognition, but their per-item responses remain largely unpredictable, while human correctness is systematically shaped by pattern complexity.

desk verdict A genuinely useful new spatial-reasoning benchmark with careful annotations, but the headline claim that MLLMs are 'random' while humans are 'highly predictable' is not supported by the reported AUCs and null results. read the letter →

arxiv 2508.20068 v1 pith:HHKJUPYO submitted 2025-08-27 cs.CL cs.CVcs.LG

Chengzu Li , Wenshan Wu , Huanyu Zhang , Qingtao Li , Zeyu Gao , Yan Xia , José Hernández-Orallo , Ivan Vulić
show 1 more author
Furu Wei
This is my paper
classification cs.CLcs.CVcs.LG
keywords spatialreasoningmultimodallargelanguagemodelscognitiveprofilesbenchmarkinstance-levelpredictabilitypsychometrictestsfeatureannotationcognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current multimodal large language models have begun to show a faint, aggregate form of spatial cognition while lacking any reliable item-level reasoning structure. To show this, it builds 11Plus-Bench, a set of 915 spatial aptitude problems taken from 11+ standardized tests, each annotated by experts for visual pattern complexity and for the sequence of reasoning operations needed. On 14 models and three human raters, the paper finds that model accuracy rises on problems humans find easy, and that model “thinking length” tracks reasoning complexity as human response time does. But when the same annotated features are used to predict which specific items a model will get right, model correctness is statistically indistinguishable from random, while human correctness is predictable from pattern complexity. The paper takes this contrast as evidence that model spatial reasoning is emergent but shallow, more sensitive to low-level visual cues than to the abstract structure that organizes human performance.

What carries the argument

The load-bearing instrument is the annotation scheme that assigns each item a perceptual load (number of atomic components in question and answer patterns, plus image resolution) and a reasoning profile (counts of four atomic operations: pattern matching, spatial relation analysis, spatial manipulation, logical deduction). These seven features feed two predictive analyses: random-forest classifiers that try to predict item correctness and linear regressions that predict human response time and model token counts. The contrast between what predicts correctness and what predicts effort—pattern complexity for human accuracy, reasoning steps for human effort, and image resolution and spatial rel

What would settle it

Take the same 402 human-evaluated items and add formal problem features (rotation angle, number of distractors, mirror axis, folding depth, distractor layout) to the seven annotated features, then retrain the correctness classifiers for each MLLM. If any model's per-item correctness becomes significantly predictable (e.g., AUC clearly above chance with p<0.01), the paper's “random instance-level behavior” claim is refuted as an artifact of feature selection.

Watch

Extended reading notes

Core claim

The paper's central claim is that human spatial reasoning is structured and predictable at the level of individual test items, while current MLLMs, despite aggregate signs of competence, are not. The vehicle is 11Plus-Bench: 915 expert-annotated items spanning three psychometric capabilities—Spatial Relation and Orientation, Spatial Visualization, and Flexibility of Closure—each tagged with seven cognitive features: three perceptual (question pattern complexity, option pattern complexity, image resolution) and four reasoning-step counts (pattern matching, spatial relation analysis, spatial manipulation, logical deduction). Aggregate model accuracy correlates with human-rated difficulty, and

Load-bearing premise

The analysis assumes that the seven annotated cognitive features—three perceptual and four reasoning-step counts—are enough to describe what makes an item hard; if models err for reasons outside this feature space, the “near-random instance-level behavior” conclusion would not follow.

Editorial extensions

If this is right

  • Evaluations of MLLM spatial reasoning should report separate-image and composite-image settings separately; the composite format depresses accuracy by about 4% on average for GPT-series models and can make genuine spatial ability look like a visual parsing deficit.
  • Per-item reliability claims are not yet warranted: since model correctness is unpredictable from cognitive features, applications that depend on knowing whether a specific spatial answer is trustworthy cannot use current MLLMs.
  • Human-perceived item difficulty, especially answer-option pattern complexity, is a usable aggregate-level probe for MLLMs: models do better on items humans call easy, so difficulty-stratified sampling can create more informative evaluations.
  • Model “thinking length” should not be read as effort or confidence: unlike human response time, it has near-zero correlation with accuracy (0.021 for Gemini 2.5 Pro), so longer reasoning traces do not imply better spatial reasoning.
  • Improving MLLM spatial reasoning likely requires reducing sensitivity to low-level visual cues such as resolution and spatial layout, and strengthening the abstract pattern-complexity signal that organizes human performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's near-random result is only as strong as its seven-feature space; a natural test is whether an expanded feature set (rotation angle, number of distractors, symmetry axes, folding depth) makes model correctness predictable. If it does, the right conclusion would be “unexplained by human-inspired features,” not “random.”
  • The human baseline rests on three raters, so the high predictability of human correctness may partly reflect rater-specific habits; replicating with a larger and more diverse sample would determine whether the human-model divergence is a general property of spatial cognition or an artifact of this comparison group.
  • The benchmark's contamination-control design—no golden answers for over 50% of public items plus a private test-center split—offers a template for other benchmarks; if widely adopted, it could lower inflated model performance from memorized test items.
  • A direct test implied by the paper: adversarially perturbing image resolution, distractor layout, or option complexity while holding the spatial operation fixed should swing model accuracy more than human accuracy, which would confirm that models rely on low-level cues rather than abstract spatial structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces 11Plus-Bench, a benchmark of 824 public and 91 private items derived from standardized 11+ spatial aptitude tests, annotated by three experts with fine-grained cognitive features: perceptual complexity (question/option pattern complexity, image resolution) and reasoning-process features (counts of Pattern Matching, Spatial Relation Analysis, Spatial Manipulation, and Logical Deduction steps). The authors evaluate 14 MLLMs under two input formats (single composite image vs. separate cropped images) and collect human performance data (accuracy and response time) from three participants on a 402-item subset. The main empirical claims are: (1) current MLLMs show early signs of spatial cognition because their accuracy correlates with human-rated difficulty; (2) MLLMs' cognitive profiles partially resemble humans' in that reasoning-related complexity correlates with response length; and (3) instance-level MLLM correctness is largely random, whereas human correctness is highly predictable and driven by abstract pattern complexity. The paper also reports SHAP analyses and linear regressions to characterize which features drive correctness and cognitive effort in humans and models.

Significance. If the central asymmetry claim were established, the paper would make a valuable contribution by moving beyond aggregate accuracy to instance-level cognitive profiling of MLLMs relative to humans. The benchmark design has notable strengths: items from realistic psychometric tests, three spatial capability axes with task mappings, expert annotations with reported inter-annotator agreement, a private held-out split from commercial test providers for contamination control, and human response-time data as a cognitive-load proxy. The fine-grained annotation scheme is a useful resource even independent of the conclusions. However, the key asymmetry—'human correctness is highly predictable ... instance-level performance in MLLMs remains largely random'—is not supported by the reported statistics, which show human AUCs of only 0.58–0.64 and model AUCs near 0.5 with only sparse nominally significant F1 values. The claim that MLLM behavior is random is an overinterpretation of a null result from a small hand-selected feature set. The benchmark itself remains credible; the interpretive claims need substantial revision.

major comments (3)
  1. [§4.3, Figures 6–7] The central asymmetry claim is not established by the reported evidence. Human per-participant AUCs are 0.579, 0.643, and 0.621 (Fig. 6), which are only modestly above chance; 'highly predictable' overstates these numbers. The MLLM classifiers achieve AUCs around 0.5, with only a few nominally significant weighted-F1 values (e.g., GPT-o3 p=0.0138, Gemini 2.5 Pro p=0.0002, Fig. 7) while AUC remains near chance. More importantly, the conclusion that MLLM performance is 'largely random' is an inference from a null result over the seven features defined in §4.3 (question/option pattern complexity, discretized image resolution, and four reasoning-step counts). Systematic predictors outside this set—exact distractor similarity, transformation-type failures, option ordering, response-format parsing failures (Tables 6–7), or image-encoding artifacts—would produce the same null. With n=402, ~30%
  2. [§4.3, Fig. 7 / 'MLLMs show partial alignment...'] SHAP attributions are computed from random-forest classifiers that, by the paper's own significance criterion, mostly do not beat chance (orange highlights in Fig. 7). Feature-importance rankings from chance-level classifiers are not interpretable. The statements that 'Option Pattern Complexity is a shared influential feature' and that MLLMs are 'disproportionately influenced by low-level visual cues' (Section 4.3) are therefore unsupported. Please restrict SHAP analysis to models whose classifiers show significant predictive performance, or explicitly label the SHAP values as exploratory and remove the causal-sounding claims about model sensitivity to low-level cues.
  3. [§4.2, Figure 3(b)] The claim that MLLMs 'exhibit early signs of spatial cognition' rests on the positive slope of model accuracy against aggregated human accuracy, reported as 'statistically significant tests against a random baseline' without giving the test statistic, effect size, or how the random baseline was generated. Human-item correlations can be confounded by item-level factors such as image resolution, visual clarity, or ambiguity, which are themselves part of the feature set. Please provide details of the permutation test and, ideally, partial correlations controlling for the seven cognitive features. Without this, the 'early signs' assertion is not quantitatively supported.
minor comments (4)
  1. [§4.2] Typo: 'P earson' should be 'Pearson'.
  2. [Table 7] The rows for Gemini 2.5 Flash and Gemini 2.5 Pro are incomplete or misaligned; the parsing-failure counts are unclear. Please fix the table so each row is interpretable.
  3. [Throughout] The benchmark name is typeset inconsistently as '11P LUS -BENCH', '11Plus-Bench', and '11P LUS -B ENCH'. Use a single consistent notation.
  4. [Appendix A / §3.3] The description of the annotation protocol says annotators choose among four reasoning-operation categories, but the main text (Section 3.2) states that the general reasoning process is 'subjectively profound' and depends on the annotator. Clarify how the four-category scheme handles inter-annotator variation and whether the 'reasoning steps' counts are averaged or adjudicated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark and the cognitive-profile analyses use distinct outcome variables, and the central claims rest on statistical fits rather than on definitions or self-citations that force the conclusions.

full rationale

The paper's central claims are empirical: per-item human correctness is predictable from seven annotated cognitive features, while MLLM correctness is not. No derived quantity is defined in terms of a target claim. The benchmark is an external testbed (public crawl plus a held-out private set from test centers), and the outcome variables used in the analyses—human response time, human correctness, MLLM correctness, and response token counts—are distinct from the input feature set. The only self-referential element is that the authors' own expert annotations are used both to validate the benchmark (§3.3, predicting response time) and as inputs to the cognitive-profile classifiers (§4.3, predicting correctness and effort). This is a shared feature set, not a circular reduction: the conclusions are statistical fits over distinct labels, not consequences of the annotation scheme by construction. The 'largely random' MLLM claim rests on a null random-forest result; unmeasured systematic error sources or low power could weaken that inference, but that is a validity and power concern, not an equivalence between input and output. No load-bearing self-citation or imported uniqueness theorem is invoked; citations to the authors' prior work are contextual (e.g., [39], [79], [82]) and are not used to justify the predictive analyses. The manuscript contains no derivation chain whose conclusion is identical to its premises, so no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central comparative conclusions rest on domain assumptions about what the chosen tasks measure, whether response time reflects cognitive load, and whether the hand-selected annotation features span item difficulty. These are reasonable but unproven assumptions, not standard mathematical axioms. No free parameters or invented entities are introduced.

assumptions (4)
  • domain assumption The three selected capabilities (Spatial Relation and Orientation, Spatial Visualization, Flexibility of Closure) and the 11+ tasks chosen are a valid operationalization of human spatial reasoning.
    Stated in Section 3.1; if these tasks do not measure spatial reasoning, the benchmark's validity fails.
  • domain assumption Response time is a valid proxy for cognitive load in humans.
    Invoked in Section 4.1 and used for cognitive-effort regressions; supported by citations but remains an assumption.
  • domain assumption The expert annotations (pattern complexity as atomic component counts, reasoning step categories) capture the perceptual and reasoning difficulty of items.
    Defined in Section 3.2 and Appendix A; the entire predictability analysis depends on this feature space.
  • domain assumption Random forest classifiers trained on these features can reveal the presence or absence of structured cognitive profiles.
    Central to the inference from predictiveness to 'structured cognition'; a null result is interpreted as absence of structure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis." pith.science (2026). https://pith.science/paper/HHKJUPYO

@misc{pith2026250820068,
  author       = {Pith},
  title        = {Pith review of: 11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HHKJUPYO}},
  note         = {Machine review of arXiv:2508.20068}
}
read the original abstract

For human cognitive process, spatial reasoning and perception are closely entangled, yet the nature of this interplay remains underexplored in the evaluation of multimodal large language models (MLLMs). While recent MLLM advancements show impressive performance on reasoning, their capacity for human-like spatial cognition remains an open question. In this work, we introduce a systematic evaluation framework to assess the spatial reasoning abilities of state-of-the-art MLLMs relative to human performance. Central to our work is 11Plus-Bench, a high-quality benchmark derived from realistic standardized spatial aptitude tests. 11Plus-Bench also features fine-grained expert annotations of both perceptual complexity and reasoning process, enabling detailed instance-level analysis of model behavior. Through extensive experiments across 14 MLLMs and human evaluation, we find that current MLLMs exhibit early signs of spatial cognition. Despite a large performance gap compared to humans, MLLMs' cognitive profiles resemble those of humans in that cognitive effort correlates strongly with reasoning-related complexity. However, instance-level performance in MLLMs remains largely random, whereas human correctness is highly predictable and shaped by abstract pattern complexity. These findings highlight both emerging capabilities and limitations in current MLLMs' spatial reasoning capabilities and provide actionable insights for advancing model design.

Figures

Figures reproduced from arXiv: 2508.20068 by the authors.

Figure 1
Figure 1. Overview of evaluation framework with 11PLUS-BENCH, including fine-grained annotations of cognitive features across diverse tasks targeting three core spatial capabilities. These annotations enable predictive modeling of correctness for both humans and MLLMs, followed by cognitive profile analysis to identify key features that influence accuracy and cognitive load. Carroll’s Three-Stratum Theory of Intelligence [8, … view at source ↗
Figure 2
Figure 2. Quality analysis of expert data collection. Expert annotations achieve high accuracy on private data with golden answers and exhibit strong agreement across both subjective and objective annotation fields. Benchmark Quality Analysis The fine-grained annotated bench￾mark contains 824 data points in the public set and 91 data points in the private set after filtering, all annotated by 3 domain experts. The annotations… view at source ↗
Figure 3
Figure 3. (a) Models perform better with multiple separate images as input compared to a single image. With multiple-image input, most closed-source models pass the significance test (p < 0.05) over random guess, whereas still all open-sourced models fail. (b) MLLM performance correlates with human accuracy (0–3 correct responses across all participants), achieving higher accuracy on instances rated as easier by human evaluat… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Cognitive profile analysis using SHAP values for correctness prediction and linear regression [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Data distributions over lengths of reasoning process and golden options. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Feature Relevance in the Cognitive Profiles of Individual Human Participants and Aggre [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Feature Relevance in the Cognitive Profiles of Different Model Variants. [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing

    cs.CV 2026-02 unverdicted novelty 7.0 of 10

    VIBE benchmark evaluates visual instruction following in image editing models across deictic, morphological, and causal levels, finding proprietary models lead but all degrade on harder tasks.

  2. Do multimodal models imagine electric sheep?

    cs.CV 2026-05 conditional novelty 6.0 of 10

    Fine-tuning VLMs to output action sequences for puzzles causes emergent internal visual representations that improve performance when integrated into reasoning.

  3. Multimodal Language Models Cannot Spot Spatial Inconsistencies

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Multimodal LLMs significantly underperform humans at spotting objects that break 3D consistency in multi-view image pairs.

  4. PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    PointVG-R is a new MLLM that reaches SOTA on pointing localization by 15.86 mIoU points via a geometric reasoning pipeline, EgoPoint-CoT dataset, SFT, RL, and variance-based reward weighting.

Reference graph

Works this paper leans on

88 extracted references · 55 canonical work pages · cited by 4 Pith papers

  1. [1]

    M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al

    Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. URL https://doi.org/10.48550/arXiv.2305.10403

    arXiv doi:10.48550/arxiv.2305.10403 2023

  2. [2]

    Program synthesis with large language models

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  3. [3]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    Time and cognitive load in working memory

    Barrouillet, P., Bernardin, S., Portrat, S., Vergauwe, E., and Camos, V . Time and cognitive load in working memory. Journal of Experimental Psychology: Learning, Memory, and Cognition , 33(3):570–585, 2007. doi: 10.1037/0278-7393.33.3.570

    work page doi:10.1037/0278-7393.33.3.570 2007

  5. [5]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford,...

    2020

  6. [6]

    Inferring capabilities from task performance with bayesian triangulation

    Burden, J., V oudouris, K., Burnell, R., Rutar, D., Cheke, L., and Hernández-Orallo, J. Inferring capabilities from task performance with bayesian triangulation. arXiv preprint arXiv:2309.11975, 2023

    work page arXiv 2023

  7. [7]

    Spatial cognition and the brain

    Burgess, N. Spatial cognition and the brain. Annals of the New York Academy of Sciences , 1124 (1):77–97, 2008. doi: 10.1196/annals.1440.002

    work page doi:10.1196/annals.1440.002 2008

  8. [8]

    Carroll, J. B. Human Cognitive Abilities: A Survey of Factor-Analytic Studies . Cambridge Uni- versity Press, Cambridge, UK, 1993. ISBN 9780521387125. doi: 10.1017/CBO9780511571312

    doi:10.1017/cbo9780511571312 1993

Show all 88 references
  1. [9]

    Carroll, J. B. The three-stratum theory of cognitive abilities. In Flanagan, D. P., Genshaft, J. L., and Harrison, P. L. (eds.), Contemporary Intellectual Assessment: Theories, Tests, and Issues , pp. 122–130. The Guilford Press, New York, NY , 1997

    1997

  2. [10]

    and Mix, K

    Cheng, Y .-L. and Mix, K. S. Spatial training improves children’s mathematics ability.Journal of Cognition and Development, 15(1):2–11, 2014. doi: 10.1080/15248372.2012.725186

    2014

  3. [11]

    Arc prize 2024: Technical report

    Chollet, F., Knoop, M., Kamradt, G., and Landers, B. Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604, 2024

    2024 arXiv

  4. [12]

    Arc-agi-2: A new challenge for frontier ai reasoning systems

    Chollet, F., Knoop, M., Kamradt, G., Landers, B., and Pinkard, H. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831, 2025

    2025 arXiv

  5. [13]

    Comsa, I. M. and Narayanan, S. A benchmark for reasoning with spatial prepositions. In Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 16328–16335, 2023

    2023

  6. [14]

    Cooper, L. A. Mental rotation of random two-dimensional shapes. Cognitive Psychology, 7(1): 20–43, 1975. doi: 10.1016/0010-0285(75)90003-1

    1975 doi

  7. [15]

    J., Penke, L., and Johnson, W

    Deary, I. J., Penke, L., and Johnson, W. The neuroscience of human intelligence differences. Nature Reviews Neuroscience, 11:201–211, 2010. doi: 10.1038/nrn2793

    2010 doi

  8. [16]

    Ekstrom, A. D. and Hill, P. F. Spatial navigation and memory: A review of the similarities and differences relevant to brain models and age. Neuron, 111(7):1037–1049, 2023

    2023

  9. [17]

    Ekstrom, R. B. and Harman, H. H. Manual for Kit of Factor-Referenced Cognitive Tests . Educational Testing Service, Princeton, NJ, 1976

    1976

  10. [18]

    Garey, M. R. and Johnson, D. S. Computers and intractability, volume 29. wh freeman New York, 2002

    2002

  11. [19]

    Gemini 2.5: Our most intelligent AI model

    Gemini. Gemini 2.5: Our most intelligent AI model. March

  12. [20]

    Spatial perspective taking: Effects of social, directional, and interactive cues

    Gunalp, P., Moossaian, T., and Hegarty, M. Spatial perspective taking: Effects of social, directional, and interactive cues. Memory & cognition, 47:1031–1043, 2019

    2019

  13. [21]

    Spatial ability, skills, reasoning or thinking: What does it mean for mathematics? In Leong, Y ., Kaur, B., Choy, B., Yeo, J., and Wong, L

    Harris, D. Spatial ability, skills, reasoning or thinking: What does it mean for mathematics? In Leong, Y ., Kaur, B., Choy, B., Yeo, J., and Wong, L. (eds.), Excellence in Mathematics Education: F oundations and Pathway, pp. 219–226. Mathematics Education Research Group of Au...

    2021

  14. [22]

    S., and Hirsh-Pasek, K

    Harris, J., Newcombe, N. S., and Hirsh-Pasek, K. A new twist on studying the development of dynamic spatial transformations: Mental paper folding in young children. Mind, Brain, and Education, 7(1):49–55, 2013. 10

    2013

  15. [23]

    Harvey, T. J. The correlation between mechanical reasoning and spatial ability for first year secondary school boys and girls — a research note. Journal of Further and Higher Education , 9(2):77–80, 1985. doi: 10.1080/0309877850090208. URL https://doi.org/10.1080/ 0309877850090208

    1985 doi

  16. [24]

    and Waller, D

    Hegarty, M. and Waller, D. A dissociation between mental rotation and perspective-taking spatial abilities. Intelligence, 32(2):175–191, 2004. doi: 10.1016/j.intell.2003.12.001

    2004 doi

  17. [25]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  18. [26]

    A., Tolmie, A

    Hodgkiss, A., Gilligan, K. A., Tolmie, A. K., Thomas, M. S., and Farran, E. K. Spatial cognition and science achievement: The contribution of intrinsic and extrinsic spatial skills from 7 to 11 years. British Journal of Educational Psychology , 88(4):675–697, 2018

    2018

  19. [27]

    A., Tolmie, A

    Hodgkiss, A., Gilligan, K. A., Tolmie, A. K., Thomas, M. S. C., and Farran, E. K. Spatial cognition and science achievement: The contribution of intrinsic and extrinsic spatial skills from 7 to 11 years. British Journal of Educational Psychology , 88(4):675–697, 2018. doi: htt...

    2018 doi

  20. [28]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  21. [29]

    and Nachev, P

    Husain, M. and Nachev, P. Space and the parietal cortex. Trends in Cognitive Sciences, 11(1): 30–36, 2007. doi: 10.1016/j.tics.2006.10.011

    2007 doi

  22. [30]

    Jirout, J. J. and Newcombe, N. S. Building blocks for developing spatial skills: Evidence from a large, representative us sample. Psychological science, 26(3):302–310, 2015

    2015

  23. [31]

    L., and Girshick, R

    Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 1...

    1988

  24. [32]

    F., Barron, L

    Johnson, J. F., Barron, L. G., Carretta, T. R., and Rose, M. R. Predictive validity of spatial ability and perceptual speed tests for aviator training. The International Journal of Aerospace Psychology, 27(3-4):109–120, 2017

    2017

  25. [33]

    Johnson-Laird, P. N. Mental models and human reasoning. Proceedings of the National Academy of Sciences, 107(43):18243–18250, 2010

    2010

  26. [34]

    Spatial versus object visualizers: A new characterization of visual cognitive style

    Kozhevnikov, M., Kosslyn, S., and Shephard, J. Spatial versus object visualizers: A new characterization of visual cognitive style. Memory & cognition, 33(4):710–726, 2005

    2005

  27. [35]

    Kyllonen, P. C. and Zu, J. Use of response time for measuring cognitive ability. Journal of Intelligence , 4(4), 2016. ISSN 2079-3200. doi: 10.3390/jintelligence4040014. URL https://www.mdpi.com/2079-3200/4/4/14

    2016 doi

  28. [36]

    Ds-1000: A natural and reliable benchmark for data science code generation

    Lai, Y ., Li, C., Wang, Y ., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-t., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning , pp. 18319–18345. PMLR, 2023

    2023

  29. [37]

    S., and Stoyanchev, S

    Li, C., Zhang, C., Teufel, S., Doddipatla, R. S., and Stoyanchev, S. Semantic map-based generation of navigation instructions. In Calzolari, N., Kan, M.-Y ., Hoste, V ., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computa...

    2024

  30. [38]

    TopViewRS: Vision-language models as top-view spatial reasoners

    Li, C., Zhang, C., Zhou, H., Collier, N., Korhonen, A., and Vuli ´c, I. TopViewRS: Vision-language models as top-view spatial reasoners. In Al-Onaizan, Y ., Bansal, M., and 11 Chen, Y .-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Pr...

    2024 doi

  31. [39]

    Imagine while reasoning in space: Multimodal visualization-of-thought

    Li, C., Wu, W., Zhang, H., Xia, Y ., Mao, S., Dong, L., Vuli´c, I., and Wei, F. Imagine while reasoning in space: Multimodal visualization-of-thought. arXiv preprint arXiv:2501.07542, 2025

    2025 arXiv

  32. [40]

    Large language models are miscal- ibrated in-context learners

    Li, C., Zhou, H., Glavaš, G., Korhonen, A., and Vuli´c, I. Large language models are miscal- ibrated in-context learners. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.), Findings of the Association for Computational Linguistics: ACL 2025 , pp. 11575–11596, Vi...

    2025 doi

  33. [41]

    Linn, M. C. and Petersen, A. C. Emergence and characterization of sex differences in spatial ability: A meta-analysis. Child Development, 56(6):1479–1498, 1985. doi: 10.2307/1130467

    1985 doi

  34. [42]

    and Forbus, K

    Lovett, A. and Forbus, K. Modeling spatial ability in mental rotation and paper-folding. In Proceedings of the Annual Meeting of the Cognitive Science Society , volume 35, 2013

    2013

  35. [43]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR) , 2024

    2024

  36. [44]

    McGee, M. G. Human spatial abilities: Psychometric studies and environmental, genetic, hormonal, and neurological influences. Psychological Bulletin, 86(5):889–918, 1979. doi: 10.1037/0033-2909.86.5.889

    1979 doi

  37. [45]

    B., Guilford, J

    Michael, W. B., Guilford, J. P., Fruchter, B., and Zimmerman, W. S. The description of spatial-visualization abilities. Educational and psychological measurement , 17(2):185–199, 1957

    1957

  38. [46]

    Moulton, S. T. and Kosslyn, S. M. Imagining predictions: mental imagery as mental emulation. Philosophical Transactions of the Royal Society B: Biological Sciences , 364(1521):1273–1280,

  39. [47]

    Newcombe, N. S. Spatial Cognition. MIT Press, jul 24 2024. https://oecs.mit.edu/pub/or750iar

    2024

  40. [48]

    Newcombe, N. S. and Learmonth, A. E. Development of spatial competence. The Cambridge handbook of visuospatial thinking , pp. 213–256, 2005

    2005

  41. [49]

    Introducing OpenAI o1

    OpenAI. Introducing OpenAI o1. September 2024. URL https://openai.com/o1/. Ac- cessed: 2025-06-08

    2024

  42. [50]

    Introducing GPT-4.1 in the API

    OpenAI. Introducing GPT-4.1 in the API. April 2025. URL https://openai.com/index/ gpt-4-1/. Accessed: 2025-06-08

    2025

  43. [51]

    Introducing OpenAI o3 and o4-mini

    OpenAI. Introducing OpenAI o3 and o4-mini. April 2025. URL https://openai.com/ index/introducing-o3-and-o4-mini/ . Accessed: 2025-06-08

    2025

  44. [52]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730–27744, 2022

    2022

  45. [53]

    and Cutts, Q

    Parkinson, J. and Cutts, Q. Investigating the relationship between spatial skills and computer science. In Proceedings of the 2018 ACM Conference on International Computing Education Research, pp. 106–114, 2018

    2018

  46. [54]

    C., DeLange, N., and Hazelrigg, M

    Presson, C. C., DeLange, N., and Hazelrigg, M. D. Orientation-specificity in kinesthetic spatial learning: The role of multiple orientations. Memory & Cognition, 15(3):225–229, 1987

    1987

  47. [55]

    and Gandevia, S

    Proske, U. and Gandevia, S. C. The kinaesthetic senses. The Journal of physiology, 587(17): 4139–4146, 2009. 12

    2009

  48. [56]

    K., Wijmans, E., Krähenbühl, P., and Koltun, V

    Ramakrishnan, S. K., Wijmans, E., Krähenbühl, P., and Koltun, V . Does spatial cognition emerge in frontier models? In Proceedings of the International Conference on Learning Representations (ICLR), 2025. Poster

    2025

  49. [57]

    and others

    Ray, A. and others. Sat: Spatial aptitude training for multimodal language models. arXiv preprint arXiv:2408.01234, 2024

    2024 arXiv

  50. [58]

    P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A

    Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T. P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A. M., Millican, K., Dyer, E., Glaese, M., Sottiaux, T., Lee, B., Viola, F., Reynolds, M.,...

    2024 doi arXiv

  51. [59]

    Shepard, R. N. and Metzler, J. Mental rotation of three-dimensional objects. Science, 171 (3972):701–703, 1971. doi: 10.1126/science.171.3972.701

    1971 doi

  52. [60]

    G., Yu, X., Haffari, G., and Li, Y .-F

    Shiri, F., Guo, X.-Y ., Far, M. G., Yu, X., Haffari, G., and Li, Y .-F. An empirical analysis on spatial reasoning capabilities of large multimodal models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 21440–21455, 2024

    2024

  53. [61]

    Stogiannidis, I., McDonagh, S., and Tsaftaris, S. A. Mind the gap: Benchmarking spatial reasoning in vision–language models. arXiv preprint arXiv:2503.19707, 2025. Under review

    2025 arXiv

  54. [62]

    A corpus for reasoning about natural language grounded in photographs

    Suhr, A., Zhou, Y ., Zhang, Z., Bai, H., Cho, K., and Artzi, Y . A corpus for reasoning about natural language grounded in photographs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , pp. 6418–6428, 2019

    2019

  55. [63]

    Gemma 3 technical report

    Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  56. [64]

    H., Meadow, N

    Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, L., and Newcombe, N. S. The malleability of spatial skills: A meta-analysis of training studies. Psychological Bulletin, 139(2):352–402, 2013. doi: 10.1037/a0028446

    2013 doi

  57. [65]

    H., McKee, K., Simms, N., Hegarty, M., and Newcombe, N

    Uttal, D. H., McKee, K., Simms, N., Hegarty, M., and Newcombe, N. S. How can we best assess spatial skills? practical and conceptual challenges. Journal of Intelligence, 12(1):8, 2024

    2024

  58. [66]

    Wai, J., Lubinski, D., and Benbow, C. P. Spatial ability for stem domains: Aligning over 50 years of cumulative psychological knowledge solidifies its importance. Journal of Educational Psychology, 101:817–835, 2009. URL https://api.semanticscholar.org/CorpusID: 17233758

    2009

  59. [67]

    Is a picture worth a thousand words? delving into spatial reasoning for vision–language models

    Wang, J., Ming, Y ., Shi, Z., Vineet, V ., Wang, X., Li, Y ., and Joshi, N. Is a picture worth a thousand words? delving into spatial reasoning for vision–language models. arXiv preprint arXiv:2409.12345, 2024

    2024 arXiv

  60. [68]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Wang, K., Pan, J., Shi, W., Lu, Z., Ren, H., Zhou, A., Zhan, M., and Li, H. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2024. URL https://openrevie...

    2024

  61. [69]

    Mathcoder-VL: Bridging vision and code for enhanced multimodal mathematical reasoning

    Wang, K., Pan, J., Wei, L., Zhou, A., Shi, W., Lu, Z., Xiao, H., Yang, Y ., Ren, H., Zhan, M., and Li, H. Mathcoder-VL: Bridging vision and code for enhanced multimodal mathematical reasoning. In The 63rd Annual Meeting of the Association for Computational Linguistics , 2025. ...

    2025

  62. [70]

    X., Anson, E

    Wei, E. X., Anson, E. R., Resnick, S. M., and Agrawal, Y . Psychometric tests and spatial navigation: Data from the baltimore longitudinal study of aging. Frontiers in neurology, 11: 484, 2020

    2020

  63. [71]

    J., Reuel, A., Roberts-Gaal, X., Calcott, R., Coxon, E., and Deshpande, C

    Wei, K., Paskov, P., Dev, S., Byun, M. J., Reuel, A., Roberts-Gaal, X., Calcott, R., Coxon, E., and Deshpande, C. Position: Human baselines in model evaluations need rigor and transparency (with recommendations & reporting checklist). In F orty-second International Conference ...

    2025

  64. [72]

    Defining and evaluating visual language models’ basic spatial abilities: A perspective from psychometrics

    Xu, W., Lyu, D., Wang, W., Feng, J., Gao, C., and Li, Y . Defining and evaluating visual language models’ basic spatial abilities: A perspective from psychometrics. arXiv preprint arXiv:2502.11859, 2025

    2025 arXiv

  65. [73]

    Visual planning: Let’s think only with images

    Xu, Y ., Li, C., Zhou, H., Wan, X., Zhang, C., Korhonen, A., and Vuli´c, I. Visual planning: Let’s think only with images. arXiv preprint arXiv:2505.11409, 2025

    2025

  66. [74]

    W., Han, R., Fei-Fei, L., and Xie, S

    Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., and Xie, S. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  67. [75]

    Spatialsense: An adversarially crowdsourced bench- mark for spatial relation recognition

    Yang, K., Russakovsky, O., and Deng, J. Spatialsense: An adversarially crowdsourced bench- mark for spatial relation recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2041–2050, 2019

  68. [76]

    WikiQA: A challenge dataset for open-domain question answering

    Yang, Y ., Yih, W.-t., and Meek, C. WikiQA: A challenge dataset for open-domain question answering. In Màrquez, L., Callison-Burch, C., and Su, J. (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pp. 2013–2018, Lisbon, Portugal, ...

    2015 doi

  69. [77]

    B., and Kohli, P

    Yi, K., Gan, C., Li, Y ., Wu, J., Kushman, N., Tenenbaum, J. B., and Kohli, P. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations (ICLR), 2020

    2020

  70. [78]

    Yılmaz, H. B. On the development and measurement of spatial ability. International Electronic Journal of Elementary Education, 1(2):83–96, 2009

    2009

  71. [79]

    Scaling and beyond: Advancing spatial reasoning in mllms requires new recipes

    Zhang, H., Li, C., Wu, W., Mao, S., Zhang, Y ., Tian, H., Vuli´c, I., Zhang, Z., Wang, L., Tan, T., et al. Scaling and beyond: Advancing spatial reasoning in mllms requires new recipes. arXiv preprint arXiv:2504.15037, 2025

    2025 arXiv

  72. [80]

    Zhang, Y ., Zhang, H., Tian, H., Fu, C., Zhang, S., Wu, J., Li, F., Wang, K., Wen, Q., Zhang, Z., et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? In The Thirteenth International Conference on Learnin...

  73. [81]

    A., Martínez-Plumed, F., Burden, J., Burnell, R., Cheke, L., Ferri, C., Marcoci, A., Mehrbakhsh, B., Moros-Daval, Y ., et al

    Zhou, L., Moreno-Casares, P. A., Martínez-Plumed, F., Burden, J., Burnell, R., Cheke, L., Ferri, C., Marcoci, A., Mehrbakhsh, B., Moros-Daval, Y ., et al. Predictable artificial intelligence. arXiv preprint arXiv:2310.06167, 2023

    2023 arXiv

  74. [82]

    M., Moros-Daval, Y ., Zhang, S., Zhao, Q., Huang, Y ., Sun, L., Prunty, J

    Zhou, L., Pacchiardi, L., Martínez-Plumed, F., Collins, K. M., Moros-Daval, Y ., Zhang, S., Zhao, Q., Huang, Y ., Sun, L., Prunty, J. E., et al. General scales unlock ai evaluation with explanatory and predictive power. arXiv preprint arXiv:2503.06378, 2025

    2025 arXiv

  75. [83]

    G., Plomin, R., and Malanchini, M

    Zhou, Q., Wang, Z., Rimfeld, K., Allegrini, A. G., Plomin, R., and Malanchini, M. Exploring the specific predictive ability of multiple domains of spatial ability on stem educational out- comes. bioRxiv, 2024. doi: 10.1101/2024.12.20.629833. URL https://www.biorxiv.org/ conten...

    2024 doi

  76. [86]

    Disentangling Cognitive Features (§3.2). Previous benchmarks often conflate distinct cognitive features that affect model accuracy in spatial reasoning tasks, such as perceptual difficulty and inherent reasoning complexity. Ignoring these features undermines evaluation validit...

  77. [87]

    Instance-Wise Evaluation with Predictive Power (§4.2). Typical average-based benchmark scores (e.g., accuracy) primarily represent overall performance, making it difficult to anticipate whether a model will correctly answer a new question. Inspired by Zhou et al. [82], our fra...

  78. [88]

    Part X of A points toward C

    Parallel Analysis with Human Cognitive Profiles (§4.2). Despite drawing inspiration from human cognitive tests, previous work lacks direct comparison with human cognition. We bridge this gap by incorporating human evaluation with response time for each question as a proxy for ...

  79. [2009]

    doi: 10.1098/rstb.2008.0314

    2008

  80. [2025]

    Accessed: 2025-05-09

    URL https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ . Accessed: 2025-05-09

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.