Pith. sign in

REVIEW 3 major objections 3 minor 42 references

Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read New jigsaw benchmark shows best vision-language model trails humans by 19 points.

desk verdict Useful spatial-reasoning benchmark with a plausible headline result, but the 30% Order Generation figure needs a documented parser before I'd trust it. read the letter →

arxiv 2505.20728 v4 pith:4J3HJ2TJ submitted 2025-05-27 cs.AI

Zesen Lyu , Dandan Zhang , Wei Ye , Fangdi Li , Zhihang Jiang , Yao Yang This is my paper
classification cs.AI
keywords spatialreasoningvision-languagemodelsbenchmarkjigsawpuzzlesmulti-steporderrestorationhumanperformancebaselinereal-worldimageevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Jigsaw-Puzzles, a benchmark of 1,100 real-world images that turns jigsaw-style tasks into five evaluation levels: selecting a missing piece, localizing pieces, verifying adjacency, detecting anomalies, and restoring shuffled order. The authors claim that these tasks trace the human cognitive path from seeing to understanding to reasoning, and that current vision-language models fail at the reasoning end. Across 24 models, the strongest tested model reaches 77.14% overall accuracy while human participants average 96.36%; on the open-ended Order Generation task the same model drops to 30.00%, against 94.09% for humans. The paper argues that existing spatial-reasoning benchmarks rely on synthetic or simple scenes, and that a real-image, automatically constructed benchmark is needed to expose this gap and guide progress.

What carries the argument

The load-bearing mechanism is the five-task cognitive ladder itself, built from a fully automated template pipeline that produces deterministic question-answer pairs from real photographs. Each task is designed to isolate one stage, from perception (Missing Piece Selection) through single-step structural reasoning (Piece Localization, Connection Verification, Anomaly Detection) to multi-step reconstruction (Order Restoration), so that the accuracy drop between adjacent rungs diagnoses where spatial reasoning breaks. A second mechanism, the Jigsaw-Puzzles-Lite subset with an Order Generation task that removes answer options, turns the benchmark from a perception test into a test of autonomous reasoning.

What would settle it

Recruit at least twenty participants with no prior exposure to Jigsaw-Puzzles and have them complete the full Lite subset under the same no-tools conditions; if their average accuracy lands near the VLM level rather than 96%, the claimed human-VLM gap does not hold.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that VLM spatial reasoning is not uniformly weak but collapses as the cognitive demand rises: models approach human accuracy on missing-piece selection, show emerging competence on single-step tasks such as localization and connection verification, and fall far behind on multi-step order restoration. The gap is not a perception failure, since the strongest model reaches 97.18% on the hard missing-piece condition, but a failure to compose local spatial cues into a reconstructed whole. The paper also finds that reasoning-enhanced models improve most on multi-step tasks, apparently because answer choices let them re-evaluate and self-correct, and that performance correlates with model scale and with basic spatial-understanding ability. When the multiple-choice scaffold is removed in Order Generation, the best model drops from 73.64% to 30.00%, which the paper reads as evidence that current VLMs cannot autonomously build a coherent spatial reasoning chain.

Load-bearing premise

The human upper bound rests on three participants solving a 220-image subset, and the paper does not say whether they were naive to the benchmark or involved in building it.

Editorial extensions

If this is right

  • If the gap is real, claims of robust VLM spatial intelligence should be treated as unsupported until a model substantially closes the 96% human baseline on this benchmark.
  • The correlation between Missing Piece Selection and reasoning tasks implies that improving basic spatial understanding may be the most direct route to better multi-step VLM reasoning.
  • Benchmark builders should prefer real-image, automatically generated tasks over synthetic grids, since the failure pattern is only visible under high visual complexity.
  • Multiple-choice formats can mask reasoning failures, so open-ended variants such as Order Generation should become standard diagnostic checks.
  • Model scaling alone is unlikely to close the gap, since even the largest proprietary models remain far behind humans on multi-step tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My reading: the benchmark may partly measure a model's ability to exploit low-level cues such as patch borders, seams, and color continuity rather than true spatial structure; a controlled version with seamless cuts would test this.
  • My reading: the three-participant human baseline makes the headline 19-point gap fragile; a larger naive sample is needed before treating 96% as the upper bound.
  • My reading: the same template pipeline could be extended to 3D scenes, temporal sequences, or video jigsaws, giving a direct test of whether the reasoning deficit is specific to static 2D layout.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces Jigsaw-Puzzles, a new benchmark for evaluating spatial reasoning in vision-language models, built from 1,100 real-world images with five automatically generated tasks (Missing Piece Selection, Piece Localization, Connection Verification, Anomaly Detection, Order Restoration) and an open-ended Order Generation variant on a 220-image subset. The authors evaluate 24 proprietary and open-source VLMs, report human performance from three participants on the subset, and find that the best model (Gemini-2.5-Pro) reaches 77.14% overall but only 30.00% on Order Generation, far below the human performance above 90%. They also analyze task correlations, scaling with model size, and the effect of reasoning-enhanced models.

Significance. The benchmark addresses a real gap: existing spatial reasoning benchmarks are largely synthetic or multiple-choice, and the paper provides a scalable, automated pipeline with visual complexity from real images. The task hierarchy from perception to multi-step reasoning is well motivated, and the evaluation spans a wide range of current VLMs. The Order Generation task is a valuable attempt to move beyond option-constrained evaluation. The main results would be significant if confirmed, because they quantify a persistent VLM-human gap in open-ended spatial reasoning. However, the headline figures depend on evaluation details (answer parsing and human baseline) that are not currently documented.

major comments (3)
  1. [§4.1, §4.2, Appendix B (Fig. 18)] The Order Generation result of 30.00% (Gemini-2.5-Pro) in Figure 8 is the paper's headline evidence for a VLM-human gap in open-ended spatial reasoning, but the paper specifies only 'exact match accuracy (%)' as the metric and does not describe the answer parser or normalization rules. For a free-form sequence task, correct answers can be expressed as 'A B C D', 'A, B, C, D', '1 2 3 4', or with piece names/descriptions; naive exact-string matching would mark all but one of these equivalent answers wrong. The authors need to provide the parsing and normalization procedure, release the evaluation code, or show example model outputs with their scoring outcomes. As written, 30.00% is a lower bound on demonstrated competence rather than a measured accuracy, and this directly affects the central comparative claim.
  2. [§4.1 (Human Performance), Table 3] The human baseline is derived from only three participants on a 220-image subset, and the paper does not state whether the participants were naive or involved in dataset curation. If the participants are the authors or otherwise familiar with the design, the 96.36% overall and 97.73% Order Restoration accuracies could overstate the human-VLM gap. In addition, no inter-participant variance or confidence intervals are reported, so the precision of the human upper bound is unknown. The authors should describe participant recruitment and naivety status and report per-participant scores.
  3. [§4.2 (Main Results), Tables 2-3, Figures 5-6] All VLM scores are reported as point estimates from a single run without error bars or confidence intervals, and the correlation and scaling analyses in Figures 5 and 6 use only 24 model-level points without significance tests or confidence bounds. Because several model differences in Tables 2 and 3 are only a few percentage points on samples of 1,100 or 220 items, some of the qualitative conclusions (e.g., the ordering of top-performing models, or the strength of task correlations) may not be robust. The paper should provide at least bootstrap confidence intervals for the main accuracies and p-values for the correlations.
minor comments (3)
  1. [§4.2] The sentence 'As shown in Tables 2,' should be 'As shown in Table 2,' and 'Table 2, 3 report' should be 'Tables 2 and 3 report' for grammatical correctness.
  2. [Figure 8 caption] The caption says 'Evaluation of Order Restoration and Order Generation tasks on Jigsaw-Puzzles-Lite'; since Order Generation is introduced only on the Lite subset, clarify that Order Restoration results are also on Lite in this figure, whereas Table 2 reports the full set.
  3. [§4.1 (Baselines)] The 'p-value-based critical value' is defined as the minimum accuracy to outperform random guessing at p=0.05, but the formula or exact computation is not shown; include the calculation so readers can reproduce the 32.56% overall threshold.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Jigsaw-Puzzles is an externally grounded empirical benchmark with deterministic task construction; model and human scores are measured against independently defined ground truths, not against fitted parameters or self-cited claims.

full rationale

The paper's claims are empirical evaluations, not derivations. Each task's ground truth is determined by the automated construction pipeline from real images: e.g., Missing Piece Selection identifies the true patch by construction, Order Restoration's correct order is the original spatial layout, and Order Generation asks the model to produce that same layout without options. There are no fitted parameters, no equations, and no 'predictions' that are defined in terms of the models' outputs. The human baseline is an external measurement, not an input to the benchmark definition; the small sample size (three participants on 220 images) is a limitation in statistical strength, but it does not make the human-model comparison circular. The paper's use of self-citations is limited to related-work positioning and standard methodology references (e.g., CLIP for distractor selection); none of these citations carries the load of the main results. The exact-match metric for Order Generation is a plausible correctness risk because free-form sequence outputs may need normalization, but that is an evaluation robustness concern, not circularity: the ground-truth sequence is still defined independently of the model's answer. Accordingly, no concrete circular step satisfying the evidence standard was found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity depends on subjective image curation and a small human baseline, both treated as unexamined ground truth. No fitted parameters are used. The benchmark itself is a dataset, not a new theoretical entity.

assumptions (4)
  • domain assumption The expert-defined filtering rules yield images with unambiguous ground truth for all puzzle tasks.
    Section 3.2 describes two human experts iteratively filtering images, but no inter-annotator agreement or validation of ambiguity is reported. If some tasks have multiple valid answers, model errors may not reflect spatial reasoning deficits.
  • domain assumption Exact-match accuracy on a single generated answer is a fair metric for VLM performance.
    The paper states exact match is used, but does not describe how free-form model outputs are parsed or standardized. Different models may format answers differently, potentially underestimating some models' accuracy.
  • domain assumption The performance of three human participants on a 220-image subset is a representative upper bound for general human spatial reasoning.
    Section 4.1 states three participants complete all tasks, but does not disclose their background or whether they were naive to the dataset. The small, possibly author-involved sample may inflate the human baseline.
  • domain assumption CC3M images filtered by the authors provide a diverse and unbiased sample of real-world spatial scenes.
    Image selection starts from CC3M and applies subjective expert rules, which may introduce selection bias toward images humans find solvable, affecting the benchmark's difficulty and representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models." pith.science (2026). https://pith.science/paper/4J3HJ2TJ

@misc{pith2026250520728,
  author       = {Pith},
  title        = {Pith review of: Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4J3HJ2TJ}},
  note         = {Machine review of arXiv:2505.20728}
}
read the original abstract

Spatial reasoning is a core component of human cognition, enabling individuals to perceive, comprehend, and interact with the physical world. It relies on a nuanced understanding of spatial structures and inter-object relationships, serving as the foundation for complex reasoning and decision-making. To investigate whether current vision-language models (VLMs) exhibit similar capability, we introduce Jigsaw-Puzzles, a novel benchmark consisting of 1,100 carefully curated real-world images with high spatial complexity. Based on this dataset, we design five tasks to rigorously evaluate VLMs' spatial perception, structural understanding, and reasoning capabilities, while deliberately minimizing reliance on domain-specific knowledge to better isolate and assess the general spatial reasoning capability. We conduct a comprehensive evaluation across 24 state-of-the-art VLMs. The results show that even the strongest model, Gemini-2.5-Pro, achieves only 77.14% overall accuracy and performs particularly poorly on the Order Generation task, with only 30.00% accuracy, far below the performance exceeding 90% achieved by human participants. This persistent gap underscores the need for continued progress, positioning Jigsaw-Puzzles as a challenging and diagnostic benchmark for advancing spatial reasoning research in VLMs. Our project page is at https://zesen01.github.io/jigsaw-puzzles.

Figures

Figures reproduced from arXiv: 2505.20728 by the authors.

Figure 1
Figure 1. Jigsaw-Puzzles example. While human partic [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Evaluation of VLMs on Jigsaw-Puzzles. The [PITH_FULL_IMAGE:figures/full_fig_p001_2.png] view at source ↗
Figure 3
Figure 3. Task examples of Jigsaw-Puzzles. Note: the questions above are slightly simplified for clarity and brevity, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Dataset curation pipeline. Step 1 filters candidate images through expert-defined rules to build a spatial [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Task Similarity Heatmap. The heatmap il [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Relationship between VLM size and perfor [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Evaluation of [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 7
Figure 7. Figure 7: An example of self-correction. Red shows the initial incorrect answer generated by Gemini-2.5-Pro; Blue indicates the ground-truth answer; Green illustrates the model’s self-correction process. lies (e.g., InternVL3, Qwen2.5-VL). This positive correlation suggests that…
Figure 11
Figure 11. Figure 11: Top: examples of images rejected by expert [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 9
Figure 9. Figure 9: An example of Mind the Gap ( [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: An example of LEGO-Puzzles (Tang et al., 2025). B Dataset Curation Rules Pool [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 13
Figure 13. Figure 13: Template of Piece Localization (Easy) [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: Template of Piece Localization (Hard) [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]
Figure 15
Figure 15. Figure 15: Template of Connection Verification [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]
Figure 16
Figure 16. Figure 16: Template of Anomaly Detection [PITH_FULL_IMAGE:figures/full_fig_p012_16.png]
Figure 17
Figure 17. Figure 17: Template of Order Restoration. Note: <op￾tion list> serves as a placeholder for the answer choices. The text in parentheses is an example and should be removed in actual use. One option is the correct answer, while the remaining three are randomly drawn from the other…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 11 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, and 1 others. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743

    arXiv 2025

  4. [4]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  5. [5]

    Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    arXiv 2023

  6. [6]

    Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku

    2024

  7. [7]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  8. [8]

    Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. 2023. Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2616--2627

    work page 2023

Show all 42 references
  1. [9]

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. 2024. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455--14465

    2024

  2. [10]

    Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter Beller-Morales, and 1 others. 2025. Aya vision: Advancing the frontier of multilingual multimodality. arXiv preprint arXiv:2505.08751

    2025 arXiv

  3. [11]

    Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, and 1 others. 2025. Kimi-vl technical report. arXiv preprint arXiv:2504.07491

    2025 arXiv

  4. [12]

    Patrick Fissler, Olivia Caroline K \"u ster, Daria Laptinskaya, Laura Sophia Loy, Christine AF Von Arnim, and Iris-Tatjana Kolassa. 2018. Jigsaw puzzling taps multiple cognitive abilities and is a potential protective factor for cognitive aging. Frontiers in aging neuroscience...

    2018

  5. [13]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. https://arxiv.org/abs/2306.13394 Mme: A comprehensive evaluation benchmark for multimodal large language models . Preprint, ...

    2024 arXiv

  6. [14]

    Grok. 2024. Bringing grok to everyone

    2024

  7. [15]

    Toru Ishikawa and Nora S Newcombe. 2021. Why spatial is special in education, learning, and everyday activities

    2021

  8. [16]

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, and 1 others. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal...

    2017

  9. [17]

    Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253

    2017

  10. [18]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299--13308

    2024

  11. [19]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, ...

    2014

  12. [20]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, and 1 others. 2024. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer

    2024

  13. [21]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195--3204

    2019

  14. [22]

    Mistral. 2025. Mistral small 3.1

    2025

  15. [23]

    Mehdi Noroozi and Paolo Favaro. 2016. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 69--84. Springer

    2016

  16. [24]

    Atin Pothiraj, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. 2025. Capture: Evaluating spatial reasoning in vision language models via occluded object counting. arXiv preprint arXiv:2504.15485

    2025 arXiv

  17. [25]

    Qwen. 2024. https://qwenlm.github.io/blog/qvq-72b-preview/ Qvq: To see the world with wisdom

    2024

  18. [26]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine...

    2021

  19. [27]

    Yufan Ren, Konstantinos Tertikas, Shalini Maiti, Junlin Han, Tong Zhang, Sabine S \"u sstrunk, and Filippos Kokkinos. 2025. Vgrp-bench: Visual grid reasoning puzzle benchmark for large vision-language models. arXiv preprint arXiv:2503.23064

    2025 arXiv

  20. [28]

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114

    2021 arXiv

  21. [29]

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2018

  22. [30]

    Liheng Shen, Tetsu Matsukawa, and Einoshin Suzuki. 2024. Satjip: Spatial and augmented temporal jigsaw puzzles for video anomaly detection. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 27--40. Springer

    2024

  23. [31]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326

    2019

  24. [32]

    Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. 2025. Visualpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge. arXiv preprint arXiv:2504.10342

    2025 arXiv

  25. [33]

    Ilias Stogiannidis, Steven McDonagh, and Sotirios A Tsaftaris. 2025. Mind the gap: Benchmarking spatial reasoning in vision-language models. arXiv preprint arXiv:2503.19707

    2025 arXiv

  26. [34]

    Kexian Tang, Junyao Gao, Yanhong Zeng, Haodong Duan, Yanan Sun, Zhening Xing, Wenran Liu, Kaifeng Lyu, and Kai Chen. 2025. Lego-puzzles: How good are mllms at multi-step spatial reasoning? arXiv preprint arXiv:2503.19990

    2025 arXiv

  27. [35]

    Ashish V Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut. 2022. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. arXiv preprint arXiv:2205.12522

    2022 arXiv

  28. [36]

    Guodong Wang, Yunhong Wang, Jie Qin, Dongming Zhang, Xiuguo Bao, and Di Huang. 2022. Video anomaly detection by solving decoupled spatio-temporal jigsaw puzzles. In European Conference on Computer Vision, pages 494--511. Springer

    2022

  29. [37]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024 a . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  30. [38]

    Xinglin Wang, Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Boyuan Pan, Heda Wang, Yao Hu, and Kan Li. 2024 b . Coglm: Tracking cognitive development of large language models. arXiv preprint arXiv:2408.09150

    2024 arXiv

  31. [39]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67--78

    2014

  32. [40]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Co...

    2024

  33. [41]

    Zicheng Zhang, Xiangyu Zhao, Xinyu Fang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Haodong Duan, Kai Chen, and Guangtao Zhai. 2025. Redundancy principles for mllms benchmarks. arXiv preprint arXiv:2501.13953

    2025 arXiv

  34. [42]

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, and 1 others. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.