Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

This paper claims that self-generated visual imagery currently provides no benefit for multi-step visual reasoning, and that models fail to use even correct visual aids as evidence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:20 UTC pith:I7K5D3PY

load-bearing objection Solid, well-scoped negative result on visual thought, but the interpretation-error decomposition is weaker than the headline and the oracle-visual format (block-appended images) confounds the deepest claim. the 3 major comments →

arxiv 2602.02465 v2 pith:I7K5D3PY submitted 2026-02-02 cs.AI cs.CVcs.LG

MentisOculi: Revealing the Limits of Reasoning with Mental Imagery

Jana Zeller , Thadd\"aus Wiedemer , Fanfei Li , Thomas Klein , Prasanna Mayilvahanan , Matthias Bethge , Felix Wichmann , Ryan Cotterell
show 1 more author
Wieland Brendel
This is my paper
classification cs.AI cs.CVcs.LG
keywords mental imageryvisual reasoningunified multimodal modelsvisual chain-of-thoughtbenchmarkspatial reasoninggeneration errorsinterpretation errors
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that machine mental imagery—letting models generate intermediate images or videos as visual 'thoughts'—does not currently help multi-step reasoning about visual problems. To test this, it introduces MentisOculi, a procedurally generated suite of five spatial puzzles with five difficulty levels and ground-truth step-by-step visual solutions. On these tasks, models that generate interleaved images or video traces perform no better, and often worse, than text-only baselines. The paper argues the bottleneck is not raw reasoning: given a verbose textual transcript of the same puzzle, some models solve it. Instead, it decomposes the failure into generation errors (wrong images) and interpretation errors (correct images not used as evidence).

Core claim

On the paper's own terms, the central discovery is that explicit visual thought is currently ineffective: across five tasks and multiple model families, no self-generated imagery—latent tokens, interleaved images from unified multimodal models, or video rollouts—reliably outperforms text-only reasoning. A control using ground-truth visual chains of thought shows performance can rise on some tasks (notably Form Board), but models still often fail to turn correct images into actionable evidence. The paper concludes that frontier models possess the textual competence and sometimes the generation capacity, but cannot couple image generation with reliable multi-step reasoning.

What carries the argument

The central object is MentisOculi, a procedural benchmark whose five tasks (Form Board, Hinge Folding, Paper Fold, Rush Hour, Sliding Puzzle) are designed to be hard to textualize and best solved visually, with difficulty stratified by minimum number of operations. Its load-bearing mechanism is the ground-truth visual chain-of-thought: a step-aligned sequence of intermediate state images provided by the generator. Replacing a model's self-generated images with these oracle visuals lets the paper separate generation errors (wrong images) from interpretation errors (correct images not used), which is the diagnostic that carries the argument.

Load-bearing premise

The load-bearing premise is that presenting ground-truth visual chain-of-thought images as one appended block is a fair test of a model's ability to use correct visuals; if that format itself undermines performance, the paper's 'interpretation error' conclusion would overstate the model's deficit.

What would settle it

Take a fixed set of Rush Hour instances and compare a unified multimodal model with ground-truth images appended as one block (the paper's protocol) versus the same images interleaved one at a time immediately before the reasoning step that uses them. If step-aligned presentation lifts accuracy to match the text-transcript competence level, the paper's claim that models fail to leverage ground-truth visualizations would be weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, current unified multimodal models cannot be relied on to reason through multi-step spatial problems by generating images; text-only inference is at least as good.
  • The competence result implies that visual puzzles like Rush Hour are not inherently beyond model reasoning—failure is in perception or representation, not logic—so targeted training on visual state-tracking could unlock gains.
  • Because performance rises on some tasks when oracle visuals replace generated ones, improving image-generation consistency is a necessary (though not sufficient) condition for visual thought to help.
  • The benchmark's procedural generation and stratification provide a reusable tool for tracking progress: as models improve, harder generated instances can be released without contamination.
  • Established text-reasoning enhancements (in-context learning, prompt optimization, higher reasoning budget, tool use) yield no systematic visual-reasoning gains, suggesting the bottleneck is specific to visual-state manipulation rather than general reasoning effort.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A reader should be cautious about the 'interpretation error' label: in the oracle control, ground-truth images are appended as one block rather than interleaved at the exact step where they belong, so part of the failure could be an input-format limitation rather than a fundamental inability to use visual evidence.
  • A stronger test would interleave ground-truth images one at a time just before each reasoning step; if performance then matches text-level competence, the paper's deepest claim would need refinement.
  • The benchmark's design could also serve as a training curriculum: using freshly generated instances with ground-truth visual traces may teach models to treat self-generated images as state trackers rather than decorative outputs.
  • The human comparison suggests models lack adaptive effort allocation, which may be a separate failure from generation or interpretation; interventions that make models spend more compute on harder visual states might help.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MENTISOCULI, a procedurally generated, stratified benchmark of five multi-step visual reasoning tasks (Form Board, Hinge Folding, Paper Fold, Rush Hour, Sliding Puzzle) with ground-truth visual chain-of-thought solutions. It evaluates text-only MLLMs, latent-visual-reasoning models (Mirage), unified multimodal models (UMMs; Gemini 2.5-I, Gemini 3-I) prompted to interleave self-generated images, and a video model (Veo 3.1). The headline finding is that explicit visual thoughts—whether latent, interleaved-image, or video-based—do not improve text-only reasoning, with performance often at or below chance at the hardest levels. The paper further claims a dual failure for UMMs: generation errors (incorrect intermediate images) and interpretation errors (failure to use even ground-truth visualizations as evidence). Secondary contributions include negative results for ICL, prompt optimization, increased reasoning budget, and tool use, plus a small human psychophysics study showing humans allocate more time to harder puzzles while Gemini 3 does not.

Significance. If the headline result holds, this is a timely and important negative result for the emerging 'thinking with images' paradigm. The benchmark itself is a valuable community asset: it is procedural, stratified, includes ground-truth visual chain-of-thought for fine-grained diagnosis, avoids low-information grid-world shortcuts, and the authors ship generators and evaluation code. The text-only competence control (Figure 4) is a well-designed check showing that the tasks are not beyond frontier models' reasoning ability. The negative results for ICL, prompt optimization, and reasoning budget are also useful. However, the deepest mechanistic claim—that UMMs 'fail to leverage even ground-truth visualizations'—depends on an oracle-visual protocol that appends all ground-truth images as a single block, which may measure input-format limitations rather than a fundamental interpretation deficit. Because this claim appears in the abstract, the conclusion, and Takeaway 3, it needs to be supported by a stronger control before the paper's full conclusions are justified.

major comments (3)
  1. [Section 4.3 and Appendix H.5] The oracle-visual condition appends the entire ground-truth chain of thought as one undifferentiated image block at the end of the prompt (H.5: 'The following images correspond to intermediate images in the reasoning process. You must use them to obtain your answer'). This differs structurally from the interleaved self-generation condition (H.2), where images are generated after each move and are immediately followed by textual state updates. A UMM may process a long appended block poorly—due to attention, resolution, or missing per-step textual anchors—so low oracle accuracy could reflect an input-format artifact rather than an inability to interpret visual state as evidence. This confound is load-bearing for the 'interpretation error' claim in Section 4.3, the abstract, and Takeaway 3. Please add an interleaved oracle condition that places each ground-truth image at the corresponding r
  2. [Section 4.3 and Figure 5] The interpretation-error decomposition is computed for a single model, Gemini 2.5-I, yet the abstract and Takeaway 3 state that 'UMMs' generally fail to leverage ground-truth visualizations. One model family instance is too narrow to support a plural mechanistic claim. Either run the oracle-visual condition on at least Gemini 3-I (and ideally another UMM class) or restrict the claim to 'the tested Gemini 2.5-I.' The paper already reports Gemini 3-I data for the interleaved condition, so the oracle experiment could be extended at moderate cost.
  3. [Section 4.3 / Figure 5] The relative contributions of generation and interpretation errors are stated qualitatively ('Generation errors are evidently a problem' and 'UMMs also suffer from interpretation errors') but no explicit decomposition metric is defined. The gap between interleaved and oracle performance measures the joint effect of generation errors and prompt-format differences, while the gap between oracle and text-only MLLM measures interpretation. These are not separately quantified. Please define concrete quantities (e.g., per-task oracle accuracy, generation fidelity rate, and a combined error decomposition) and report them with confidence intervals, since the current figures do not show variance across the three query repetitions.
minor comments (6)
  1. [Appendix H.2] The first 'HINGEFOLDING' prompt under H.2 is actually the FORMBOARD prompt ('target shape outlined in black and five pieces labeled A through E'), a copy-paste error. This will confuse readers and should be corrected.
  2. [Appendix G.2] Typo: 'eaach' should be 'each'.
  3. [Appendix E.1] Heading 'Quantitive Results' should be 'Quantitative Results'.
  4. [Figure 10 caption] The caption uses 'R = 0.98' while the text reports 'R2 ≥ 0.98'; please standardize (and clarify whether the plotted values are R or R²).
  5. [Section 3.4] The chance-performance computation for the planning tasks is only described verbally ('accounting for (limited) backtracking'). Please provide the formula or a pseudocode description so the reported chance baselines are reproducible.
  6. [Section 4.5 / Figure 7] The human comparison is based on n=5 participants and a time-threshold simulation. The paper cites a small-sample design defense, but the claim 'Gemini 3 performs like humans at 5–10 s' would benefit from confidence intervals around the human time-accuracy curves.

Circularity Check

0 steps flagged

No significant circularity; the paper's conclusions are empirical comparisons against independent text-only baselines, not derivations from fitted inputs.

full rationale

MENTISOCULI is an empirical benchmark paper. Its central claims—that self-generated imagery does not improve text-only reasoning and that UMMs fail to leverage ground-truth visualizations—are established by querying frontier models on procedurally generated instances and comparing accuracy against text-only baselines on the same instances. No parameter is fitted to the outcome and then renamed as a prediction; the oracle-visual condition injects ground-truth images as an experimental manipulation, not as a fitted input. The paper's self-citations (e.g., Wiedemer et al. for the video auto-rater, Mayilvahanan et al. in the discussion) are ancillary tools or speculative pointers rather than load-bearing derivations. The most plausible concern—that appending oracle images as a single block (Appendix H.5) may confound interpretation ability with input-format limitations—is a validity threat to a specific mechanistic conclusion, but it is not a definitional or construction-level circularity: the headline negative result does not reduce to that manipulation. There is no exhibited equation or fitted parameter equivalent to the paper's predictions, so no circular step is identified.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper is empirical; the 'free parameters' are hand-chosen design constants (chance baseline length, sample size, query budget, time thresholds). No numbers were fitted to data to produce the conclusion. The axioms listed are the unverified assumptions behind the two main decompositions: generation vs. interpretation error, and textual competence vs. visual failure. No new physical or mathematical entities are postulated; the contribution is an evaluation suite and protocol.

free parameters (4)
  • chance_sequence_length = 6 steps
    Chance for Rush Hour and Sliding Puzzle is defined as the probability that a random six-step action sequence reaches the goal (Section 3.4). The length 6 is hand-chosen and directly sets the chance baseline used to interpret 'below chance' performance.
  • samples_per_level_per_task = 30
    30 instances per level per task (Section 2, Datasheet). Hand-chosen; with 30 samples the binomial standard error is about ±9 percentage points at 50% accuracy, yet main figures plot point accuracies without confidence intervals.
  • query_budget = 3
    Models are queried up to three times and the highest reasoning budget is used (Section 3.1). This affects reported accuracies and is a hand-chosen protocol constant.
  • human_time_threshold = 5–30 s
    Figure 7 simulates time-constrained human accuracy by thresholding response times at arbitrary cutoffs; used for the claim that Gemini 3 performs like humans at 5–10 s.
axioms (4)
  • domain assumption Appending all oracle chain-of-thought images as a block (Appendix H.5) is a faithful test of a model's ability to use visual aids; any resulting failure reflects interpretation, not input-format or attention limitations.
    Load-bearing for the 'interpretation error' conclusion in Section 4.3 (Figure 5). If the block-format itself degrades multi-image grounding, the conclusion is confounded.
  • domain assumption Procedurally generated instances have not been memorized during pretraining; no contamination check is performed.
    Section 2 claims procedural generation provides a mechanism to address contamination 'in the future,' but the paper does not run a leakage test. 'Below chance at Level 5' assumes models are solving rather than recalling near-identical puzzles.
  • domain assumption The verbose text transcription of Rush Hour (Section G.1) is lossless with respect to the simulator state, so solving from it counts as purely textual reasoning.
    Used in Figure 4 to conclude that models 'possess the competence' to solve Rush Hour textually. The transcription is an unusual high-precision format, so high text performance could partly reflect format-specific pattern matching.
  • domain assumption The lenient video auto-rater heuristic (first move per vehicle, relative order, ignoring minor artifacts) recovers the intended action sequence from Veo 3.1 rollouts.
    Section 3.2. If the heuristic accepts near-arbitrary motion, video-model scores are overestimates; this is conservative for the negative headline conclusion but weakens the per-model comparison.

pith-pipeline@v1.3.0-alltime-deepseek · 30822 in / 16142 out tokens · 156613 ms · 2026-08-03T05:20:58.465263+00:00 · methodology

0 comments
read the original abstract

Frontier models are transitioning from multimodal large language models (MLLMs) that merely ingest visual information to unified multimodal models (UMMs) capable of native interleaved generation. This shift has sparked interest in using intermediate visualizations as a reasoning aid, akin to human mental imagery. Central to this idea is the ability to form, maintain, and manipulate visual representations in a goal-oriented manner. To evaluate and probe this capability, we develop MentisOculi, a procedural, stratified suite of multi-step reasoning problems amenable to visual solution, tuned to challenge frontier models. Evaluating visual strategies ranging from latent tokens to explicit generated imagery, we find they generally fail to improve performance. Analysis of UMMs specifically exposes a critical limitation: While they possess the textual reasoning capacity to solve a task and can sometimes generate correct visuals, they suffer from compounding generation errors and fail to leverage even ground-truth visualizations. Our findings suggest that despite their inherent appeal, visual thoughts do not yet benefit model reasoning. MentisOculi establishes the necessary foundation to analyze and close this gap across diverse model families.

Figures

Figures reproduced from arXiv: 2602.02465 by Fanfei Li, Felix Wichmann, Jana Zeller, Matthias Bethge, Prasanna Mayilvahanan, Ryan Cotterell, Thadd\"aus Wiedemer, Thomas Klein, Wieland Brendel.

Figure 1
Figure 1. Figure 1: MENTISOCULI comprises five visual reasoning tasks designed to be best-solved with mental imagery. Collectively, the tasks require models to solve multi-step reasoning problems with geometric constraints. Success hinges on the ability to maintain a visual representation with high fidelity and consistent geometry under affine transformations. Each task is procedurally generated across five difficulty levels,… view at source ↗
Figure 2
Figure 2. Figure 2: MLLMs and UMMs display similar failure patterns across tasks: Performance degrades noticeably with difficulty and falls below chance at Level 5, indicating that visual reasoning limitations are task-agnostic. Data for all levels in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Different kinds of mental imagery do not greatly improve multi-step reasoning on RUSH HOUR: Compared to MLLMs, the latent visual reasoning model Mirage that is fine-tuned to generate interleaved visual latent tokens shows some improvement (especially considering its relatively weak base model), but with diminishing returns at harder levels. In contrast, UMMs that interleave generated images and texts gener… view at source ↗
Figure 4
Figure 4. Figure 4: MLLMs have the competence to solve RUSH HOUR when prompted with a transcription of the task. Gemini 3 and GPT-5.1 even perform on par with humans, even though the text-only RUSH HOUR requires mathematically solving for possible collisions. 1 3 5 Level 0 0.2 0.4 0.6 0.8 1.0 Accuracy Form Board 1 3 5 Level Hinge Folding 1 3 5 Level Paper Fold 1 3 5 Level Rush Hour 1 3 5 Level Sliding Puzzle Chance Gemini 2.5… view at source ↗
Figure 5
Figure 5. Figure 5: UMM performance faces a dual issue: Generation errors are ubiquitous—performance on all tasks increases with oracle visualizations. However, on most tasks, UMMs fail to utilize even correct visuals to aid their reasoning, which we term interpretation errors. Data for all levels in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Techniques that improve language-based reasoning fail to benefit visual reasoning: In-context learning (ICL), prompt optimization, increased reasoning budget, and tool use yield no consistent gains, especially at higher levels. The tool use and prompt optimization experiments were conducted with low reasoning. we replace self-generated imagery with oracle visuals (see Section G.2) in Gemini 2.5-I’s chain-o… view at source ↗
Figure 7
Figure 7. Figure 7: Gemini 3 performs like humans at 5 − 10s. We plot average human performance at each difficulty level, while simulating different thinking time cutoffs (5 − 30s). ( [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Human subjects perform similarly We plot the difficulty levels against performance for all our human subjects. Evidently, differences between humans present themselves only at the hardest difficulty level. Overall, our subjects perform similarly and, crucially, on par with the authors, demonstrating that we successfully investigated subjects close to the performance ceiling. only during the practice phase.… view at source ↗
Figure 10
Figure 10. Figure 10: Visual CoT induces linear scaling between model compute and human response time, yet alignment is not a proxy for performance We find that Gemini 2.5-I equipped with oracle visual CoT (R 2 = 0.99) and Qwen3-VL with in context learning examples containing a visual CoT (R = 0.98)—exhibit near-perfect linear scaling, where token expenditure is directly proportional to human cognitive load. However, this alig… view at source ↗
Figure 11
Figure 11. Figure 11: Performance across all difficulty levels for MLLMs and UMMs While [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance across all difficulty levels of Gemini 2.5 and Gemini 2.5-I While [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Higher reasoning budget does not reliably improve accuracy Low vs. high budget results for Gemini 3 and GPT-5.1 across all tasks and levels in MENTISOCULI; any gains are small, inconsistent, and largely disappear at higher difficulty. E. Generated Images & Videos E.1. Unified Multimodal Models Qualitative Results Qualitative inspection of the visual rollouts from Gemini 2.5-I reveals a pervasive lack of s… view at source ↗
Figure 14
Figure 14. Figure 14: Gemini 2.5-I image rollouts are strongly task-dependent and often drift from valid state updates Random qualitative samples from instances where the model generated intermediate images (levels 1, 3, and 5), illustrating frequent rule violations and hallucinated state changes in several tasks. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Gemini 3-I produces clean, coherent intermediate states in lower levels, but still hallucinates at higher difficulty Random qualitative samples from instances with generated intermediate images; two samples per RUSH HOUR level, highlighting improved visual consistency on easier levels and compounding errors on harder ones. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Gemini 2.5-I does not reliably match rollout length to the expected number of visual CoT steps Joint distributions of expected intermediate images (x-axis; implied by ground-truth CoT steps) vs. images generated by the model (y-axis). The dashed line indicates x = y (perfect alignment). not explain downstream performance; in addition, even when image counts align, the qualitative analysis shows that the i… view at source ↗
Figure 17
Figure 17. Figure 17: Gemini 2.5-I shows uneven coupling between action proposals and explicit visual state updates across tasks. Joint distributions of proposed actions (x-axis) vs. generated images (y-axis). The dashed line indicates x = y; values above the diagonal suggest extra images (e.g., backtracking), while off-diagonal spread indicates inconsistent per-action state tracking. 0 1 2 3 4 5 6 7 8 Proposed Actions 0 1 2 3… view at source ↗
Figure 18
Figure 18. Figure 18: Gemini 3-I more consistently generates one image per action, but still struggles to predict how many images a problem will require Same diagnostics as Figures 16 and 17. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Video frames generated by multiple video models from the same prompt and initial image Following an initial qualitative comparison, we zoom in on results from Veo-3.1 for more detailed analysis. Level 1 Level 2 Level 3 Level 4 Level 5 [PITH_FULL_IMAGE:figures/full_fig_p022_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Qualitative results from Veo 3.1 on RUSH HOUR across five difficulty levels. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Text descriptions are verbose simulator states, not compact natural-language prompts Example RUSH HOUR instance (left) and its deterministic state specification (right), which uses continuous-valued geometry and explicit motion axes. H. Prompts & Instructions H.1. MLLM Standard Prompts FORM BOARD Look at the image: It is showing from left to right, a target shape outlined in black and five pieces labeled … view at source ↗
Figure 22
Figure 22. Figure 22: Ground-truth visual CoTs render the simulator state after every action, providing step-aligned supervision for multi-step imagery Random samples from levels 1, 3, and 5 across tasks; each example shows the initial instance (left) and the corresponding sequence of intermediate rendered states along the reference solution trajectory (right). 25 [PITH_FULL_IMAGE:figures/full_fig_p025_22.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Visual prompt engineering for video models

    cs.CV 2026-07 conditional novelty 6.0

    Automatically converting task images to photorealistic variants (visual prompt engineering) improves video-model reasoning performance, often beating text prompt engineering and test-time scaling.

  2. Do multimodal models imagine electric sheep?

    cs.CV 2026-05 conditional novelty 6.0

    Fine-tuning VLMs to output action sequences for puzzles causes emergent internal visual representations that improve performance when integrated into reasoning.

Reference graph

Works this paper leans on

112 extracted references · 2 canonical work pages · cited by 2 Pith papers

  1. [1]

    Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y ., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y ., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., and Lin, J. Qwen2.5-VL technical report.arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., and Ruan, C. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Thinking with generated images

    Chern, E., Hu, Z., Chern, S., Kou, S., Su, J., Ma, Y ., Deng, Z., and Liu, P. Thinking with generated images. arXiv preprint arXiv:2505.22525, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    Chollet, F. On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    Pith/arXiv arXiv 1911

  5. [5]

    T.De Oratore

    Cicero, M. T.De Oratore. Harper & Brothers, New York, -55. Citation from Book III, Chapter XLI, Sec- tion 163. Cited from English edition edited and trans- lated by J. S. Watson, 1875

  6. [6]

    Cui, Y ., Chen, H., Deng, H., Huang, X., Li, X., Liu, J., Liu, Y ., Luo, Z., Wang, J., Wang, W., et al. Emu3. 5: Native multimodal models are world learners.arXiv preprint arXiv:2510.26583, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    L., Martins, L

    de Oliveira, B. L., Martins, L. G., Brand˜ao, B., da Luz, M. L., Soares, T. W. d. L., and Melo, L. C. Sliding puzzles gym: A scalable benchmark for state represen- tation in visual reinforcement learning.arXiv preprint arXiv:2410.14038, 2024

    arXiv 2024

  8. [8]

    Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., et al. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    ImageNet: A large-scale hier- archical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hier- archical image database. In2009 IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 248–255. IEEE, 6 2009. doi: 10.1109/cvpr. 2009.5206848. URL http://dx.doi.org/10. 1109/cvpr.2009.5206848

    arXiv 2009

  10. [10]

    Ekstrom, R. B. and Harman, H. H.Manual for kit of factor-referenced cognitive tests, 1976. Educational testing service, 1976

    1976

  11. [11]

    J., Guan, X., and Wang, X

    Fan, Y ., He, X., Yang, D., Zheng, K., Kuo, C.-C., Zheng, Y ., Narayanaraju, S. J., Guan, X., and Wang, X. E. GRIT: Teaching MLLMs to think with images. arXiv preprint arXiv:2505.15879, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    W., Wallach, H., III, H

    Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., and Crawford, K. Datasheets for datasets.Communications of the ACM, 64(12):86–92, 11 2021. ISSN 1557-7317. doi: 10.1145/3458723. URL http://dx.doi.org/ 10.1145/3458723. 9 Revealing the Limits of Reasoning with Mental Imagery

    doi:10.1145/3458723 2021

  13. [13]

    Veo 3 model card

    Google DeepMind. Veo 3 model card. https://storage.googleapis.com/ deepmind-media/Model-Cards/ Veo-3-Model-Card.pdf , 2025. Accessed: 2026-01-20

    2025

  14. [14]

    Gemini 2.5 Flash and native capabilities – audio & image model card

    Google DeepMind. Gemini 2.5 Flash and native capabilities – audio & image model card. https://storage.googleapis. com/deepmind-media/Model-Cards/ Gemini-2-5-Flash-Model-Card.pdf ,

  15. [15]

    Gemini 3 pro image model card

    Google DeepMind. Gemini 3 pro image model card. https://storage.googleapis. com/deepmind-media/Model-Cards/ Gemini-3-Pro-Image-Model-Card.pdf ,

  16. [16]

    Gemini 3 pro model card

    Google DeepMind. Gemini 3 pro model card. https://storage.googleapis. com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf , 2025. Accessed: 2026-01-09

    2025

  17. [17]

    Accessed: 2026-01-09

    2026

  18. [18]

    Courier Corporation, 1954

    Hadamard, J.An essay on the psychology of invention in the mathematical field. Courier Corporation, 1954

    1954

  19. [19]

    LTX-2: Efficient joint audio-visual foundation model,

    HaCohen, Y ., Brazowski, B., Chiprut, N., Bitterman, Y ., Kvochko, A., Berkowitz, A., Shalem, D., Lifschitz, D., Moshe, D., Porat, E., Richardson, E., Shiran, G., Chachy, I., Chetboun, J., Finkelson, M., Kupchick, M., Zabari, N., Guetta, N., Kotler, N., Bibi, O., Gordon, O., Panet, P., Benita, R., Armon, S., Kulikov, V ., In- ger, Y ., Shiftan, Y ., Melum...

  20. [20]

    Visfac- tor: Benchmarking fundamental visual cognition in multimodal large language models.arXiv preprint arXiv:2502.16435, 2025

    Huang, J.-T., Dai, D., Huang, J.-Y ., Yuan, Y ., Liu, X., Wang, W., Jiao, W., He, P., and Tu, Z. Visfac- tor: Benchmarking fundamental visual cognition in multimodal large language models.arXiv preprint arXiv:2502.16435, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    HunyuanVideo: A systematic framework for large video generative models, 2025

    Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., Wu, K., Lin, Q., Yuan, J., Long, Y ., Wang, A., Wang, A., Li, C., Huang, D., Yang, F., Tan, H., Wang, H., Song, J., Bai, J., Wu, J., Xue, J., Wang, J., Wang, K., Liu, M., Li, P., Li, S., Wang, W., Yu, W., Deng, X., Li, Y ., Chen, Y ., Cui, Y ., Peng, Y ., Yu, ...

    Pith/arXiv arXiv 2025

  22. [22]

    W., Li, L., Yang, Z., Wang, L., and Cheng, Y

    Hao, Y ., Gu, J., Wang, H. W., Li, L., Yang, Z., Wang, L., and Cheng, Y . Can mllms reason in multimodality? emma: An enhanced multimodal reasoning bench- mark.arXiv preprint arXiv:2501.05444, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Unfolding spatial cognition: Evaluating multimodal models on visual simulations

    Li, L., Bigverdi, M., Gu, J., Ma, Z., Yang, Y ., Li, Z., Choi, Y ., and Krishna, R. Unfolding spatial cognition: Evaluating multimodal models on visual simulations. arXiv preprint arXiv:2506.04633, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    ROVER: Benchmarking reciprocal cross-modal rea- soning for omnimodal generation.arXiv preprint arXiv:2511.01163, 2025

    Liang, Y ., Chow, W., Li, F., Ma, Z., Wang, X., Mao, J., Chen, J., Gu, J., Wang, Y ., and Huang, F. ROVER: Benchmarking reciprocal cross-modal rea- soning for omnimodal generation.arXiv preprint arXiv:2511.01163, 2025

    arXiv 2025

  25. [25]

    B., Liu, O., Guo, P., Neiswanger, W., Huang, F., et al

    Li, A., Wang, C., Fu, D., Yue, K., Cai, Z., Zhu, W. B., Liu, O., Guo, P., Neiswanger, W., Huang, F., et al. Zebra-cot: A dataset for interleaved vision language reasoning.arXiv preprint arXiv:2507.16746, 2025

    arXiv 2025

  26. [26]

    TUNA: Taming unified visual representations for native unified multimodal models.arXiv preprint arXiv:2512.02014, 2025

    Liu, Z., Ren, W., Liu, H., Zhou, Z., Chen, S., Qiu, H., Huang, X., An, Z., Yang, F., Patel, A., et al. TUNA: Taming unified visual representations for native unified multimodal models.arXiv preprint arXiv:2512.02014, 2025

    arXiv 2025

  27. [27]

    Jigsaw-puzzles: From seeing to under- standing to reasoning in vision-language models

    Lyu, Z., Zhang, D., Ye, W., Li, F., Jiang, Z., and Yang, Y . Jigsaw-puzzles: From seeing to under- standing to reasoning in vision-language models. InProceedings of the 2025 Conference on Empir- ical Methods in Natural Language Processing, pp. 26003–26014. Association for Computational Lin- guistics, 2025. doi: 10.18653/v1/2025.emnlp-main

    doi:10.18653/v1/2025.emnlp-main 2025

  28. [28]

    Sora: A review on background, technology, limitations, and opportunities of large vision models,

    Liu, Y ., Zhang, K., Li, Y ., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y ., Sun, H., Gao, J., He, L., and Sun, L. Sora: A review on background, technology, limitations, and opportunities of large vision models,

  29. [29]

    and Morales, J

    McCarty, M. and Morales, J. Artificial phantasia: Evidence for propositional reasoning-based mental imagery in large language models.arXiv preprint arXiv:2509.23108, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Mi, Z., Wang, K.-C., Qian, G., Ye, H., Liu, R., Tulyakov, S., Aberman, K., and Xu, D. I think, 10 Revealing the Limits of Reasoning with Mental Imagery therefore i diffuse: Enabling multimodal in-context reasoning in diffusion models.arXiv preprint arXiv:2502.10458, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Oxford University Press, Oxford, 2023

    Nanay, B.Mental Imagery. Oxford University Press, Oxford, 2023. ISBN 978-0-19-880950-0. doi: 10. 1093/oso/9780198809500.001.0001

    arXiv 2023

  32. [32]

    GPT-5.1 model documentation

    OpenAI. GPT-5.1 model documentation. https://platform.openai.com/docs/ models/gpt-5.1, 2026. Accessed: 2026-01-20

    2026

  33. [33]

    Llms on the line: Data de- termines loss-to-loss scaling laws.arXiv preprint arXiv:2502.12120, 2025

    Mayilvahanan, P., Wiedemer, T., Mallick, S., Bethge, M., and Brendel, W. Llms on the line: Data de- termines loss-to-loss scaling laws.arXiv preprint arXiv:2502.12120, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    K., Wijmans, E., Kraehenbuehl, P., and Koltun, V

    Ramakrishnan, S. K., Wijmans, E., Kraehenbuehl, P., and Koltun, V . Does spatial cognition emerge in fron- tier models?arXiv preprint arXiv:2410.06468, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Richardson, A.Defining Mental Imagery, pp. 1–12. Springer Berlin Heidelberg, 1969. ISBN 9783662378175. doi: 10.1007/978-3-662-37817-5

    work page doi:10.1007/978-3-662-37817-5 1969

  36. [36]

    Seedance 1.5 pro: A native audio- visual joint generation foundation model, 2025

    Seedance, T., Chen, H., Chen, S., Chen, X., Chen, Y ., Chen, Y ., Chen, Z., Cheng, F., Cheng, T., Cheng, X., Chi, X., Cong, J., Cui, J., Cui, Q., Dong, Q., Fan, J., Fang, J., Fang, Z., Feng, C., Feng, H., Gao, M., Gao, Y ., Guo, D., Guo, Q., Hao, B., Hao, Q., He, B., He, Q., Hoang, T., Hu, R., Hu, X., Huang, W., Huang, Z., Huang, Z., Ji, D., Jiang, S., Ji...

    Pith/arXiv arXiv 2025

  37. [37]

    S., Tinaz, B., Fabian, Z., and Soltanolkotabi, M

    Sepehri, M. S., Tinaz, B., Fabian, Z., and Soltanolkotabi, M. Hyperphantasia: A benchmark for evaluating the mental visualization capabilities of multimodal LLMs.arXiv preprint arXiv:2507.11932, 2025

    arXiv 2025

  38. [38]

    K., Yuan, Z., and Wu, X

    Qu, L., Zhang, H., Liu, Y ., Wang, X., Jiang, Y ., Gao, Y ., Ye, H., Du, D. K., Yuan, Z., and Wu, X. Token- Flow: Unified image tokenizer for multimodal under- standing and generation. In2025 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 2545–2555. IEEE, 6 2025. doi: 10.1109/ cvpr52734.2025.00243. URL http://dx.doi. org/10.1...

    arXiv 2025

  39. [39]

    Smith, P. L. and Little, D. R. Small is beau- tiful: In defense of the small-n design.Psy- chonomic Bulletin & Review, 25(6):2083–2101, 3 2018. ISSN 1531-5320. doi: 10.3758/ s13423-018-1451-8. URL http://dx.doi.org/ 10.3758/s13423-018-1451-8

    work page doi:10.3758/s13423-018-1451-8 2083

  40. [40]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Team, C. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    URL http://dx.doi.org/10.1007/ 978-3-662-37817-5_1

  42. [42]

    Thinking with video: Video generation as a promis- ing multimodal reasoning paradigm.arXiv preprint arXiv:2511.04570, 2025

    Tong, J., Mou, Y ., Li, H., Li, M., Yang, Y ., Zhang, M., Chen, Q., Liang, T., Hu, X., Zheng, Y ., et al. Thinking with video: Video generation as a promis- ing multimodal reasoning paradigm.arXiv preprint arXiv:2511.04570, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    S., Matarese, N., Swersky, K., Kim, B., Jaini, P., and Geirhos, R

    Wiedemer, T., Li, Y ., Vicol, P., Gu, S. S., Matarese, N., Swersky, K., Kim, B., Jaini, P., and Geirhos, R. Video models are zero-shot learners and reasoners.arXiv preprint arXiv:2509.20328, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    OpenEvolve: an open-source evolutionary coding agent, 2025

    Sharma, A. OpenEvolve: an open-source evolutionary coding agent, 2025. URL https://github. com/algorithmicsuperintelligence/ openevolve

    2025

  45. [45]

    Xie, J., Yang, Z., and Shou, M. Z. Show-o2: Improved native unified multimodal models.arXiv preprint arXiv:2506.15564, 2025. 11 Revealing the Limits of Reasoning with Mental Imagery

    Pith/arXiv arXiv 2025

  46. [46]

    Vi- sulogic: A benchmark for evaluating visual reasoning in multi-modal large language models.arXiv preprint arXiv:2504.15279, 2025

    Xu, W., Wang, J., Wang, W., Chen, Z., Zhou, W., Yang, A., Lu, L., Li, H., Wang, X., Zhu, X., et al. Vi- sulogic: A benchmark for evaluating visual reasoning in multi-modal large language models.arXiv preprint arXiv:2504.15279, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Qwen3 technical report, 2025

    Team, Q. Qwen3 technical report, 2025. URLhttps: //arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  48. [48]

    Machine mental imagery: Empower multimodal reasoning with latent visual tokens.arXiv preprint arXiv:2506.17218, 2025

    Yang, Z., Yu, X., Chen, D., Shen, M., and Gan, C. Machine mental imagery: Empower multimodal reasoning with latent visual tokens.arXiv preprint arXiv:2506.17218, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    DeepEyes: Incentiviz- ing” thinking with images” via reinforcement learning

    Zheng, Z., Yang, M., Hong, J., Zhao, C., Xu, G., Yang, L., Shen, C., and Yu, X. DeepEyes: Incentiviz- ing” thinking with images” via reinforcement learning. arXiv preprint arXiv:2505.14362, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Y ., Zhang, Y ., and Chang, S

    Wu, Q., Zhao, H., Saxon, M., Bui, T., Wang, W. Y ., Zhang, Y ., and Chang, S. Vsp: Assessing the dual challenges of perception and reasoning in spatial plan- ning tasks for vlms.arXiv preprint arXiv:2407.01863, 2024

    Pith/arXiv arXiv 2024

  51. [53]

    Video as the new language for real-world decision making

    Yang, S., Walker, J., Parker-Holder, J., Du, Y ., Bruce, J., Barreto, A., Abbeel, P., and Schuurmans, D. Video as the new language for real-world decision making. arXiv preprint arXiv:2402.17139, 2024

    Pith/arXiv arXiv 2024

  52. [56]

    When visualizing is the first step to reason- ing: MIRA, a benchmark for visual chain-of-thought

    Zhou, Y ., Tu, H., Wang, Z., Wang, Z., Muennighoff, N., Nie, F., Choi, Y ., Zou, J., Deng, C., Yan, S., et al. When visualizing is the first step to reason- ing: MIRA, a benchmark for visual chain-of-thought. arXiv preprint arXiv:2511.02779, 2025. 12 Revealing the Limits of Reasoning with Mental Imagery A. Automatic Puzzle Generation To construct MENTISOC...

    arXiv 2025

  53. [60]

    answer":

    Pieces are shown in their correct orientation and size (no rotation or scaling needed). Task: Determine the subset of pieces from{A, B, C, D, E}necessary to assemble the target shape. Output: Respond in JSON format as follows: {"answer": "A C E"} List only the letter labels of the pieces needed, separated by space. 24 Revealing the Limits of Reasoning wit...

  54. [61]

    Shapes are connected in a kinematic chain; each hinge connects two adjacent shapes

  55. [65]

    answer":

    The goal is to find the sequence of hinge rotations that transforms the initial configuration into the target. Task: Determine the rotation angle (in degrees) for each numbered hinge to achieve the target configuration. Output: Respond in JSON format as follows: {"answer": "A 90, B 90, C 180"} Each pair specifies the hinge label and its rotation angle in ...

  56. [72]

    Track where the hole is punched through all folded layers

  57. [73]

    Mentally unfold the paper with the punched hole step-by-step in reverse order

  58. [74]

    answer":

    Determine which unfolded pattern (A, B, C, D, or E) matches your mental result. Output: Respond in JSON format as follows: {"answer": "C"} RUSHHOUR Look at the image: It shows the initial configuration of a congested parking lot. Each colored rectangle with a letter and arrow represents a vehicle. Black rectangles without a letter represent immovable obst...

  59. [78]

    answer":

    The red vehicle must reach the exit on the edge of the grid. Task: Plan the minimal sequence of moves needed to free the red car and allow it to exit the parking lot. Each move should specify which vehicle moves and in which direction (forward or backward). Output: Respond in JSON format as follows: {"answer": "A forward, C backward, E forward, R forward"...

  60. [83]

    answer":

    The goal is to reconstruct the original, coherent image by rearranging the scrambled tiles. Task: Determine the shortest sequence of moves needed to solve the puzzle and restore the original image. Output: Respond in JSON format as follows: {"answer": "up right down left up"} Each word specifies a direction to move the blank tile. You may guess the most p...

  61. [84]

    The target shape can be assembled using 1 to 5 of the given pieces

  62. [85]

    Pieces must fit together perfectly with no gaps or overlaps

  63. [86]

    Some pieces are distractors and are not needed

  64. [87]

    answer":

    Pieces are shown in their correct orientation and size (no rotation or scaling needed). Task: Move one piece at a time of{A, B, C, D, E}from the right into the outlined target shape on the left. Generate a new image for each move. If you notice a mistake, you may also return a piece from the outlined target shape back to the candidate shapes. Also generat...

  65. [88]

    Shapes are connected in a kinematic chain — each hinge connects two adjacent shapes

  66. [89]

    All shapes are connected; all shapes to the right of the rotated shape rotate with it

    Rotating hingeNcauses the shape on the right side of the hinge in the original configuration to rotate anti-clockwise. All shapes are connected; all shapes to the right of the rotated shape rotate with it. All shapes to the left of the hinge remain fixed

  67. [90]

    Rotations must be multiples of45 ◦ (i.e.,45 ◦,90 ◦,135 ◦,180 ◦,225 ◦,270 ◦,315 ◦)

  68. [91]

    Shapes maintain their connections throughout all rotations

  69. [92]

    answer":

    The goal is to find the sequence of hinge rotations that transforms the initial configuration into the target. Task: Determine the rotation angle (in degrees) for each numbered hinge to achieve the target configuration. After each move, generate a new image in which you update the left side of the image to reflect your proposed hinge rotation. Once the ou...

  70. [93]

    The paper starts as a flat square

  71. [94]

    Each step shows the paper being folded along a line (horizontal, vertical, or diagonal)

  72. [95]

    After all folds are complete, one hole is punched through all layers at the marked positions

  73. [96]

    When the paper is unfolded completely, holes appear at multiple positions due to the layering

  74. [97]

    One of the five options (A, B, C, D, E) shows the correct hole pattern. Task:

  75. [98]

    Mentally follow each fold in sequence as shown in the first row

  76. [99]

    Track where holes are punched through all folded layers

  77. [100]

    Unfold the paper with the punched hole step-by-step in reverse order

  78. [101]

    After each unfolding move, generate an image of the (partially) unfolded paper with hole(s) in the correct positions

  79. [102]

    answer":

    Determine which unfolded pattern (A, B, C, D, or E) matches your generated result. Output: First, reason through the moves and generate images with the progressively more unfolded paper sheet. Make sure that these updated images are generated by following the rules specified above. Generate one image after each unfold. Finally, respond in JSON format as f...

  80. [106]

    answer":

    The red vehicle must reach the exit on the edge of the grid. Task: Plan the minimal sequence of moves needed to free the red car and allow it to exit the parking lot. Each move should specify which vehicle moves and in which direction (forward or backward). After each move, generate an image showing the updated puzzle state. Output: First, reason through ...

Showing first 80 references.