Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

The paper claims that current vision-language models fail at constrained-manifold spatial reasoning, with the best model scoring 33.6% while humans score 91.6%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A new human-curated benchmark of 1,000 ranking questions on real-world engineering structures shows the best vision-language model reaches 33.6% accuracy where humans reach 91.6%.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A solid, well-scoped VLM spatial-reasoning benchmark whose main weakness is that the answer key rests on annotator consensus — release the data and the claim becomes checkable. the 4 major comments →

arxiv 2602.07864 v2 pith:OMSW2CIN submitted 2026-02-08 cs.CV

Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces

Chen Yang , Guanxin Lin , Youquan He , Peiyao Chen , Guanghe Liu , Yufan Mo , Zhouyuan Xu , Linhao Wang
show 5 more authors
Guohui Zhang Zihang Zhang Shenxiang Zeng Chen Wang Jiansheng Fan
This is my paper
classification cs.CV
keywords constrained-manifold spatial reasoningvision-language modelsspatial intelligence benchmark3D structure understandingranking questionsgeometric reasoningtopological reasoningstructural grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that spatial intelligence in vision-language models is far weaker than everyday benchmarks suggest, once scenes are governed by strong structural constraints. It introduces SSI-Bench, a set of 1,000 ranking questions over real-world engineering structures, where correct answers require recovering a 3D interpretation consistent with geometric, topological, and physical feasibility. Across 31 models, the best proprietary model reaches 33.6% taskwise accuracy, the best open-source model 22.2%, and random guessing 12.85%, while humans reach 91.6%. Chain-of-thought reasoning improves accuracy only marginally, and the paper attributes the gap to failures in structural grounding and globally consistent 3D reasoning rather than simple perception errors. If the benchmark is valid, it shows that current VLM spatial performance is partly an artifact of unconstrained scenes that allow 2D shortcuts.

Core claim

The central claim is that constrained-manifold spatial reasoning (CMSR) is a distinct and largely unsolved capability. The paper formalizes a structural scene as a graph with geometry and attributes, restricts feasible configurations to a manifold defined by equality and inequality constraints, and turns spatial intelligence into a ranking problem: order candidate members or groups under an explicit geometric or topological criterion. Evaluated on this benchmark, even the strongest models stay far below human performance, and explicit thinking tokens do not close the gap. Error analysis identifies four recurrent failure modes: misestimating member extent under occlusion, misrecognizing compo

What carries the argument

The constrained-manifold formulation is the central mechanism: each scene is represented by nodes, members, a connectivity graph, geometric degrees of freedom, and discrete attributes, with admissible states restricted to a feasible set M where equality constraints encode geometric compatibility and connectivity and inequality constraints encode non-intersection, support, and physical feasibility. The ranking objective then orders candidates by a task-defined criterion. The paper's key argument is that strong constraints narrow the space of plausible 3D interpretations, making target relations more determinate from visual evidence and therefore making the benchmark a cleaner test of genuine

Load-bearing premise

The load-bearing premise is that each ranking question has a single correct order determined by the actual 3D structure; if annotator consensus rather than external geometric truth is the only authority, then the benchmark may be measuring agreement among humans rather than constraint-consistent spatial reasoning.

What would settle it

For a random sample of SSI-Bench questions, reconstruct the underlying structures in 3D with photogrammetry or CAD models and compute the stated geometric or topological criterion from the recovered geometry. If a non-trivial fraction of questions admit two or more candidate orderings that are both consistent with the reconstruction, the claimed uniqueness of the ground truth fails; if the human-annotated rankings always match the 3D-derived ordering, the benchmark's premise is verified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If SSI-Bench measures what it claims, then strong performance on everyday spatial benchmarks does not imply constraint-consistent 3D understanding; the human-model gap here is roughly 58 percentage points.
  • Chain-of-thought scaling alone will not solve CMSR: thinking tokens give only incremental gains and can even hurt tasks that require global 3D consistency such as multi-view reasoning and volume estimation.
  • Improvements should target structural grounding—accurate member extent, orientation, and identity under occlusion and clutter—along with globally coherent 3D reconstruction.
  • The gap between taskwise and pairwise accuracy indicates models often make individual comparisons correctly but fail to compose them into a globally consistent ordering.
  • Because current models are far from saturation on this benchmark, it can serve as a meaningful evaluation instrument for future spatial-reasoning advances.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's ground truth rests on annotator judgment rather than external geometric measurement; a natural extension would be to verify rankings against CAD models or laser-scan reconstructions, which would independently test whether the correct ordering is uniquely determined by the 3D structure.
  • The constrained-manifold design could transfer to other domains with strong physical constraints, such as mechanical assemblies, anatomical structures, or molecular geometry, where rankings are determinate from structure and 2D shortcuts are hard to exploit.
  • A testable prediction of the paper's error analysis is that models with explicit 3D outputs—depth, normals, or keypoints—should improve substantially on SSI-Bench relative to pure language-reasoning models; if they do not, the bottleneck lies deeper in representation rather than perception.
  • The non-monotonic relationship between thinking-token usage and accuracy suggests that confidence-calibrated reasoning, rather than simply more deliberation, is a more promising direction for constrained spatial tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SSI-Bench, a 1,000-question VQA benchmark for spatial reasoning on 'constrained manifolds' — real-world engineering structures whose 3D configurations are governed by geometric, topological, and physical constraints. The benchmark uses a ranking formulation (Eq. 3) over 3–4 candidates and covers ten sub-categories in geometric and topological task families, plus a multi-view subset. Construction is human-centered: ten researchers spent 400+ hours curating images and annotating rankings with tie-handling and independent review. The paper evaluates 31 VLMs with temperature 0 and reports a large human–model gap: best proprietary Gemini-3-Flash at 33.60%, best open-source GLM-4.6V at 22.20%, humans at 91.60% (Table 1). It further reports that chain-of-thought/thinking modes yield only marginal gains (Section 4.3) and presents an error analysis with four failure modes (Section 4.4).

Significance. If the benchmark's answer keys are valid — that is, if the ground-truth permutations are uniquely entailed by the images and the stated structural constraints — SSI-Bench would be a valuable contribution: it targets a distinct regime (constrained-manifold spatial reasoning) where 2D shortcuts are purportedly less effective, and it provides a strict ranking metric, a broad model sweep, and a plausible error taxonomy. The formalization in Section 3.1 is clean, the ranking objective is well defined, and the two-metric evaluation (taskwise and pairwise) is sensible. The large human–model gap and the modest gains from explicit thinking are potentially important findings. However, the central claim depends entirely on answer-key validity, which is not established. The dataset is not released in the manuscript, no inter-annotator agreement is reported, and there is no external geometric verification of the ground-truth rankings. Until that load-bearing premise is checked, the reported gap and the CoT findings cannot be interpreted as measuring constraint-consistent 3D reasoning rather than agreement with annotator conventions.

major comments (4)
  1. [§3.3, §3.2, Eq. (3), App. D.2–D.3, App. H] Answer-key validity is the load-bearing premise. The paper asserts each question 'is curated to have a unique correct answer' (§3.2) and describes a human pipeline (annotator ranking, checker attempt, third-reviewer adjudication; §3.3, App. D.2) as the evidence for uniqueness. There is no external geometric ground truth (CAD model, reconstruction, or formal derivation from the image), no inter-annotator agreement statistic, no tie-rate report, and no adjudication outcome summary. If the 'correct' ordering is not uniquely determined by the 3D structure and the image, then the ranking problem in Eq. (3) is not well-posed, and the low VLM scores in Table 1 may reflect mismatches with human interpretation conventions rather than failures of constraint-consistent spatial reasoning. The Limitations appendix (App. H) discusses scalability but not this validity gap. This must be addressed before
  2. [§4.1, App. E.4, Table 1] Human performance is a single point estimate of 91.60% from six evaluators, with no confidence interval, no per-participant breakdown, and an ambiguity in the protocol: the text says the six participants 'collectively completed all 1,000 benchmark questions,' which suggests each participant answered only a subset. If the 91.60% averages over different question sets per participant, it is not a stable estimate of human accuracy. The gap to the best model (33.60%) is large, but a lower-bound human estimate (e.g., per-question majority or worst participant) would materially change the interpretation. Please report per-participant accuracy, a confidence interval, and the exact allocation of questions to participants.
  3. [§4.3, Table 5, App. E.6] The 'thinking yields only marginal gains' claim rests on comparisons of single temperature-0 runs: Gemini-3-Pro HIGH (29.5%) vs LOW (27.1%) and Qwen3-VL-30B-A3B THINKING (22.5%) vs INSTRUCT (20.6%). With 1,000 questions, a 2.4-percentage-point difference has a standard error of roughly 1.4 points at these accuracy levels, so the difference is not clearly significant. The token-bucket analysis in Figure 4 (Left) is also presented without error bars or significance tests. To support the conclusion that explicit thinking is only marginally beneficial, the authors should report multiple runs and/or binomial confidence intervals, and ideally a paired analysis across the same questions.
  4. [§4.1, App. E.3, release status] The manuscript does not release the dataset, annotation metadata, or evaluation code. The project page is mentioned but no URL content is provided in the preprint. For a benchmark paper, this makes the core results (Table 1, Table 4, Table 5) uncheckable by readers. The answer-key validity concern in my first comment cannot be independently assessed without at least a sample of instances and the annotation guidelines, if not the full dataset. Please include a release plan or provide the data with the revision.
minor comments (6)
  1. [Figure 15 caption] Typo: 'ralative distance' should be 'relative distance'.
  2. [§3.3 and App. D heading] 'Benchmark Construction Progress' appears to be a misspelling for 'Process'; the same word is used in App. D. This is likely a wording error, not substantive.
  3. [Table 1 and Table 4] The column labeled 'M-View' appears twice — once under Geometric and once under Topological. Rename them 'M-View (Geo.)' and 'M-View (Topo.)' to avoid ambiguity, matching Appendix C.
  4. [App. D.5] Difficulty labeling uses human lead time as the sole proxy (plus a human-error flag). Lead time on an annotation interface conflates interface familiarity and annotation speed with question difficulty. This is a reasonable coarse proxy, but should be acknowledged as such.
  5. [§4.2, Table 1] The random baseline row is correct but uneven: 4.17% for 4-candidate tasks and 16.67% for 3-candidate tasks. The average 12.85% is fine, but it would help to state this explicitly in the text rather than only in the table.
  6. [Figure 4] The left panel's x-axis ('fraction of maximum thinking-token count') and the non-monotonic pattern are described qualitatively. Adding the number of questions per bucket and error bars would strengthen the visual claim.

Circularity Check

0 steps flagged

No significant circularity: SSI-Bench reports benchmark measurements built from a human annotation pipeline; no derivation or prediction reduces to its own inputs.

full rationale

The paper's load-bearing claims are empirical benchmark results, not derivations from fitted parameters. The formalization in Section 3.1 defines the ground-truth ranking as an ordering induced by a criterion function (Eq. 3); this is a definition, not a circular inference. The benchmark's gold answers are produced by a fully human annotation and quality-control pipeline (Section 3.3, Appendix D), with independent checkers and adjudication. This is a standard way to construct a benchmark, and while it raises a legitimate validity concern about whether human annotator judgments uniquely determine the correct ordering from the images, that concern is about benchmark validity, not about circularity in the paper's derivation chain: the authors do not fit any parameter to the VLM results and then re-predict those results. The reported human performance (91.6%) is an independent measurement on the same task, not the source of the answer keys. No uniqueness theorem from prior work by the same authors is invoked to force the benchmark design; no ansatz is smuggled in via self-citation; no known result is merely renamed. Appendix H appropriately acknowledges the scalability limitation of manual curation, but that limitation does not constitute a circular step. The central claim—that current VLMs score far below humans on SSI-Bench—is an external, falsifiable empirical observation conditional on the benchmark's ground truth. Under the specified definitions of circularity, no load-bearing reduction of the paper's conclusions to its inputs is exhibited, so the score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The benchmark's numerical results are free of fitted physics parameters, but the central claim depends on domain assumptions about constraint-governed structures and on human judgment as the source of ground truth. The only hand-chosen numbers (K, difficulty thresholds, image size) affect evaluation difficulty but not the existence of the gap.

free parameters (3)
  • Candidate count per question K = 3 or 4
    Hand-chosen design parameter in Section 3.2; determines random baselines (4.17% and 16.67%) and the exact-match difficulty, so it shapes all reported accuracies.
  • Difficulty thresholds (lead time) = 150 s, 270 s, 390 s
    Derived in Appendix D.5 from IQR of annotation lead times; used only for Easy/Medium/Hard labels, not for the central model-vs-human gap.
  • Input image resize = longer side ≤ 512 px
    Evaluation preprocessing choice in Section 4.1; applied uniformly but could affect all VLM scores.
axioms (3)
  • domain assumption Real-world engineering structures can be faithfully represented as constrained feasible sets M = {s : c(s)=0, h(s)≤0}, and these constraints make queried relations stable under plausible interpretations.
    Introduced in Section 3.1 (Eq. 2); the entire claim that ranking answers are determinate and 2D shortcuts are blocked depends on this modeling premise.
  • domain assumption Human annotators and reviewers can correctly recover the unique ordering from the image, and their consensus is ground truth.
    Used in Section 3.3 and Appendix D.2; no external geometric ground truth (CAD, scan, or formal model) verifies the answers, so the benchmark measures human judgment as truth.
  • domain assumption The curated candidate sets actually prevent reliable 2D-pixel-level shortcut solutions.
    Stated as curation intent in Section 3.3 and Appendix D.3, but no quantitative verification is provided for each question.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces." pith.science (2026). https://pith.science/paper/OMSW2CIN

@misc{pith2026260207864,
  author       = {Pith},
  title        = {Pith review of: Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OMSW2CIN}},
  note         = {Machine review of arXiv:2602.07864}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Spatial intelligence is crucial for vision--language models (VLMs), yet many scene-centric benchmarks evaluate unconstrained environments where a single image may admit multiple plausible 3D interpretations. We introduce SSI-Bench, a VQA benchmark for Structure-Centric Spatial Reasoning (SCSR) in constraint-governed spaces. Built from complex real-world 3D structures, it uses structural constraints from geometry, topology, and physical feasibility to make component relations more determinate from visual evidence. The benchmark contains 1,000 ranking questions spanning geometric and topological reasoning, where correct ordering requires resolving all candidate-wise 3D relations, imposing stronger demands on spatial understanding. It is created through a fully human-centered pipeline with over 400 researcher-hours of image curation, component annotation, and question design. Evaluating 31 VLMs reveals a large gap to humans: the best open-source model achieves 22.2% accuracy and the strongest closed-source model reaches 33.6%, while humans score 91.6%. Further results show that chain-of-thought reasoning brings only marginal gains, and error analysis reveals fundamental limitations in current models' spatial understanding within constraint-governed spaces. Project page: https://ssi-bench.github.io.

Figures

Figures reproduced from arXiv: 2602.07864 by Chen Wang, Chen Yang, Guanghe Liu, Guanxin Lin, Guohui Zhang, Jiansheng Fan, Linhao Wang, Peiyao Chen, Shenxiang Zeng, Youquan He, Yufan Mo, Zhouyuan Xu, Zihang Zhang.

Figure 1
Figure 1. Figure 1: SSI-Bench is a diverse, human-annotated, and challenging benchmark, designed to evaluate models’ constrained-manifold spatial reasoning on complex real-world 3D structures. The bar chart on the right illustrates the significant performance gap between state-of-the-art VLMs and human performance on SSI-Bench. Abstract Spatial intelligence is crucial for vision–language models (VLMs) in the physical world, y… view at source ↗
Figure 2
Figure 2. Figure 2: Representative SSI-Bench samples from each category. For visualization, we overlay all candidates in one image; the benchmark provides separately annotated images per option. Ties use the smaller index first. Full questions are in Appendix F. unconstrained Euclidean space. SSI-Bench is a structure￾centric benchmark designed to evaluate VLMs under these constraints. SSI-Bench adopts a structural view of eac… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the SSI-Bench construction pipeline. cess: ten researchers with interdisciplinary expertise in AI and structural engineering devoted over 400 hours to its con￾struction. Additional details are provided in Appendix D. Data collection. To target constrained-manifold spatial reasoning, we curate real-world images of 3D engineer￾ing structures with strong geometric and topological con￾straints.… view at source ↗
Figure 4
Figure 4. Figure 4: (Left) Relationship between thinking-token usage and accuracy; (Right) Sub-category level effects of thinking on CMSR. to 2D shortcut cues. Even strong VLMs remain far from human performance. The best proprietary model, Gemini-3- Flash, reaches 33.60% average Taskwise Accuracy, and the best open-source model, GLM-4.6V, reaches 22.20%, while humans achieve 91.60%. The random-ranking baseline is 12.85%, and … view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of four error types identified in VLM spatial reasoning on SSI-Bench. that hinge on globally consistent 3D reconstruction, particu￾larly the Multi-View setting and Volume. Overall, these pat￾terns suggest that additional deliberation helps mainly when the evidence supporting the task criterion is stable, but it can amplify errors when structural grounding or cross-view correspondence is uncert… view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of categories in SSI-Bench. compares either member groups or node groups under an explicit geometric or topological criterion and requires selecting the correct permutation among 3 or 4 candidates (cf [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Screenshot of the data annotation interface, where annotators select members/groups, the task category, and specify equality relations. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Screenshot of the manual review interface used during question generation, where annotators verify whether the annotated colors and text placements are appropriate. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Screenshot of the independent review interface used for quality control, where annotators attempt the problem and provide their answers. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Distribution of human lead time for question annotation and the corresponding difficulty categorization. • Medium. Instances with lead time in [150, 270) seconds (319 instances). These questions generally require additional verification, such as careful disambiguation under mild occlusion or more subtle comparisons. • Hard. Instances that satisfy either of the following conditions (74 instances): (i) lead… view at source ↗
Figure 11
Figure 11. Figure 11: Distributions of the number of tokens used for explicit thinking for different models [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Full example for geometric-ground height category. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Full example for geometric-ground angle category [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Full example for geometric-dimension category. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Full example for geometric-ralative distance category [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Full example for geometric-area category. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Full example for geometric-volume category [PITH_FULL_IMAGE:figures/full_fig_p028_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Full example for geometric-multi view category. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Full example for topological-hop distance category [PITH_FULL_IMAGE:figures/full_fig_p029_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Full example for topological-cycle length category. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Full example for topological-multi view-1 category [PITH_FULL_IMAGE:figures/full_fig_p030_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Full example for topological-multi view-2 category. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Illustration of 3D spatial-logic errors (highlighted in blue). 31 [PITH_FULL_IMAGE:figures/full_fig_p031_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Illustration of 3D spatial-logic errors (highlighted in blue), Member-extent errors (highlighted in orange), Object-recognition errors (highlighted in red). 32 [PITH_FULL_IMAGE:figures/full_fig_p032_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Illustration of 3D spatial-logic errors (highlighted in blue), Computational and comparison-logic errors (highlighted in green). 33 [PITH_FULL_IMAGE:figures/full_fig_p033_25.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. WildRoadBench: A Wild Aerial Road-Damage Grounding Benchmark for Vision-Language Models and Autonomous Agents

    cs.CV 2026-05 accept novelty 7.0

    WildRoadBench provides a professionally annotated UAV corpus and dual-track protocol showing frontier VLMs and LLM agents achieve limited performance on wild aerial road-damage grounding under unified metrics.

  2. WildRoadBench: A Wild Aerial Road-Damage Grounding Benchmark for Vision-Language Models and Autonomous Agents

    cs.CV 2026-05 unverdicted novelty 6.0

    WildRoadBench is a new dual-track benchmark on professionally annotated wild UAV road-damage images showing closed-source VLMs lead but leave over half the AP_50 metric on the table while agents lag and open-source mo...

Reference graph

Works this paper leans on

15 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [2]

    Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., and Xia, F

    Accessed: 2026-01-10. Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., and Xia, F. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455–14465,

    2026

  2. [3]

    Accessed: 2026-01-10

    https://blog.google/products-and-pla tforms/products/gemini/gemini-3/ , 2025b. Accessed: 2026-01-10. He, Y ., Huang, Y ., Chen, G., Pei, B., Xu, J., Lu, T., and Pang, J. Egoexobench: A benchmark for first-and third- person view video understanding in mllms.arXiv preprint arXiv:2507.18342,

    Pith/arXiv arXiv 2026

  3. [4]

    Gemini 2.5: Our most intelligent ai model

    Google DeepMind. Gemini 2.5: Our most intelligent ai model. https://blog.google/innovation-a nd-ai/models-and-research/google-dee pmind/gemini-model-thinking-updates-m arch-2025/, 2025a. Accessed: 2026-01-10. Google DeepMind. A new era of intelligence with gemini

    2025

  4. [6]

    Viewspatial-bench: Evaluating multi-perspective spatial localization in vision-language models.arXiv preprint arXiv:2505.21500, 2025a

    Li, D., Li, H., Wang, Z., Yan, Y ., Zhang, H., Chen, S., Hou, G., Jiang, S., Zhang, W., Shen, Y ., et al. Viewspatial-bench: Evaluating multi-perspective spatial localization in vision-language models.arXiv preprint arXiv:2505.21500, 2025a. Li, Y ., Zhang, Y ., Lin, T., Liu, X., Cai, W., Liu, Z., and Zhao, B. Sti-bench: Are mllms ready for precise spatial...

    arXiv

  5. [7]

    Mo, K., Zhu, S., Chang, A

    Accessed: 2026-01-10. Mo, K., Zhu, S., Chang, A. X., Yi, L., Tripathi, S., Guibas, L. J., and Su, H. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object under- standing. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 909–918,

    2026

  6. [8]

    Accessed: 2026-01-10. OpenAI. Introducing gpt-4.1 in the api. https://op enai.com/index/gpt-4-1/ , 2025a. Accessed: 2026-01-10. OpenAI. Introducing gpt-5. https://openai.com/i ndex/introducing-gpt-5/ , 2025b. Accessed: 2026-01-10. OpenAI. Introducing gpt-5.2. https://openai.com /index/introducing-gpt-5-2/ , 2025c. Ac- cessed: 2026-01-10. Peng, J., Ning, C...

    2026

  7. [9]

    Pexels. Pexels. https://www.pexels.com/ . Ac- cessed: 2026-01-10. Pixabay. Pixabay. https://pixabay.com/ . Ac- cessed: 2026-01-10. Qi, L., Bai, J., Wu, H., Xu, G., Xiong, H., and Yang, Y . The first engineering application of 10mn cfrp cables in cable-stayed bridge in china. InStructures, volume 68, pp. 107199. Elsevier,

    2026

  8. [10]

    Unsplash

    Unsplash. Unsplash. https://unsplash.com/ . Accessed: 2026-01-10. V¨ollmecke, L., Krenzer, A., and Seim, W. Assessment of nailed connections in existing timber trusses.Construc- tion and Building Materials, 491:142359,

    2026

  9. [11]

    Xu, R., Wang, W., Tang, H., Chen, X., Wang, X., Chu, F.-J., Lin, D., Feiszli, M., and Liang, K. J. Multi-spatialmllm: Multi-frame spatial understanding with multi-modal large language models.arXiv preprint arXiv:2505.17015,

    Pith/arXiv arXiv

  10. [13]

    From flatland to space: Teaching vision-language models to per- ceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025a

    Zhang, J., Chen, Y ., Zhou, Y ., Xu, Y ., Huang, Z., Mei, J., Chen, J., Yuan, Y .-J., Cai, X., Huang, G., et al. From flatland to space: Teaching vision-language models to per- ceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025a. Zhang, Z., Wang, Z., Zhang, G., Dai, W., Xia, Y ., Yan, Z., Hong, M., and Zhao, Z. Dsi-bench: A bench- mark for dynam...

    arXiv

  11. [14]

    Candidates

    evaluates alignment across egocentric/exocentric views, and MMSI-Video (Lin et al., 2025a) extends multi-video reasoning with human annotation and additional task coverage (e.g., planning, cross-video reasoning). These datasets raise the difficulty through cross-episode generalization, yet their underlying environments are typically not governed by strong...

    1920

  12. [15]

    (Meta AI, 2025), Google’s Gemma-3 family (27B/12B/4B) (Kamath et al., 2025), and LLaV A-OneVision in 72B and 7B settings (Li et al., 2024). E.3. Implementation Details for Model Evaluation Execution setup.Proprietary models are accessed via their official APIs to ensure standardized inference behavior and fair comparison across providers. In contrast, ope...

    2025

  13. [2022]

    W., Han, R., Fei-Fei, L., and Xie, S

    Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., and Xie, S. Thinking in space: How multimodal large lan- guage models see, remember, and recall spaces. InPro- ceedings of the Computer Vision and Pattern Recognition Conference, pp. 10632–10643, 2025a. Yang, S., Xu, R., Xie, Y ., Yang, S., Li, M., Lin, J., Zhu, C., Chen, X., Duan, H., Yue, X., et al...

    Pith/arXiv arXiv

  14. [2024]

    Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning

    Chen, J., Tang, J., Qin, J., Liang, X., Liu, L., Xing, E., and Lin, L. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. In Findings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pp. 513–523,

    2021

  15. [2025]

    Accessed: 2026-01-10. Bai, S., Cai, Y ., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y ., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., S...

    2026

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.