REVIEW 4 major objections 5 minor 32 references
Do large language vision models understand 3D shapes?
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Large vision-language models partially understand 3D shapes, but their accuracy collapses when both orientation and texture change, far below human performance.
desk verdict A controlled synthetic benchmark with a clear two-factor failure, but the missing error bars, undocumented human baseline, and open displacement-map question keep it from being a finished result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a synthetic four-panel forced-choice shape-matching test. For each question the model sees one query object (panel A) and three candidate panels, one of which is the same 3D mesh rendered with a controlled subset of transformations—orientation, PBR (physically based rendering) texture/material, and HDRI background varied or fixed—while the other two panels are different objects. An automatic CGI pipeline built on a large 3D object repository, 60,000 textures, and 600 backgrounds generates 1,000 unique trials per condition; varying which factors are held constant isolates what the model actually uses to match shape. The 'identical 3D shape' wording forces reliance on geometry rather than texture or 2D projection, and the specific ablation conditions—especially both orientation and texture changed—reveal the combined-variation failure.
What would settle it
Re-run the benchmark with distractor panels matched to the target on coarse shape attributes (all panels elongated, flat, dish-like, or same object class, with similar size). If model accuracy in the single-factor conditions collapses toward chance, the paper's central claim overstates shape understanding: the models would be passing via coarse features, and the combined-change drop would be an artifact of distractor selection rather than an integration failure. Conversely, if models reach human-like accuracy when given two views or depth information of each object, that would falsify the claim that they lack combined orientation-texture invariance.
Extended reading notes
Core claim
The paper's central claim is that LVLMs have acquired some genuine 3D shape understanding, but not the integrated invariant representation humans use. On its benchmark, every model recognizes an identical shape across a change in orientation alone or a change in texture alone, and changing background or illumination has almost no effect. When orientation and texture are varied together, however, all models drop dramatically—Gemini and GPT variants fall to the low-to-mid 80s, Claude to the 70s, and LLaMA to mid-40s—whereas humans remain near 98%. The paper reads this as evidence that the models can use either 2D projection or material cues as a crutch, but cannot jointly abstract away both pose and appearance at once; the explanations models give, such as citing rotations or coarse shape words like 'flat' and 'dishlike,' support that the answers are not grounded in a precise geometric transform.
Load-bearing premise
The load-bearing assumption is that the four-panel forced-choice task with randomly selected distractors measures 3D shape understanding, so the accuracy drop in the both-changed condition reflects a failure of shape abstraction rather than rendering artifacts, prompt wording, distractor selection, or use of coarse category-level features like 'elongated' or 'flat' instead of exact shape matching.
Editorial extensions
If this is right
- Current LVLMs should not be treated as reliable for tasks that require identifying the same object across combined pose and appearance changes, such as robotic pick-and-place or visual navigation.
- Improving 3D shape understanding will likely require training data that varies orientation and texture jointly, not just more images with one factor changed at a time.
- Because background and illumination changes had the smallest effect, object-scene separation is not the bottleneck; the bottleneck is composing geometric and material transformations.
- The same CGI pipeline can generate unlimited shape-matching examples, so the benchmark doubles as a recipe for synthetic data for training or fine-tuning.
- Models' confident verbal explanations often mention transformations that did not occur such as mirroring or resizing, so self-reported 3D reasoning should not be taken as evidence of reliable 3D representation.
Reading between the lines
- If the single-factor successes are supported by coarse shape features (e.g., 'elongated', 'flat') or by 2D silhouette matching, then a version of this benchmark with distractors matched on those coarse attributes would likely expose much weaker true 3D understanding than the headline numbers suggest.
- The pattern suggests an integration bottleneck: the models can register one transformation at a time but cannot compose two independent transformations into a single invariant representation; testing intermediate conditions, such as orientation changed with a subtle, non-distinctive texture change, could map exactly where the composition fails.
- The benchmark could be adapted to video or multi-view input to test whether motion and viewpoint continuity close the gap, and to separate true 3D understanding from static-image shortcut learning.
- A practical consequence, if the paper is right, is that downstream applications that depend on durable object identity—inventory tracking, 3D asset matching, or embodied AI—should expect current LVLMs to fail precisely in realistic situations where appearance and viewpoint vary together.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a procedurally generated CGI benchmark for testing whether large vision-language models (LVLMs) can match objects of identical 3D shape under changes in orientation, material/texture, background, and their combinations. Seven test conditions are defined, and six LVLMs plus a human baseline are evaluated on 1000 trials per condition. The central empirical claim is that models are well above chance but significantly below humans, and that accuracy drops sharply when both orientation and material/texture are varied. The authors argue this indicates a partial but incomplete abstract understanding of 3D shape. Code and benchmark data are stated to be publicly available.
Significance. If the benchmark is valid, it is a useful, scalable synthetic evaluation resource for a fundamental perceptual capability, and the main finding—LVLMs partially abstract 3D shape but lag behind humans, especially when multiple visual factors change—is practically relevant for model development. The paper's strengths include its large-scale procedural generation, systematic variation of individual factors, testing of six current model families, and release of code and data. The inclusion of model explanation examples is also valuable, as it provides qualitative evidence about the strategies models use. However, several methodological details are missing or ambiguous, and these bear directly on whether the reported accuracies measure exact 3D shape matching as claimed. The central result is plausible but needs additional validation before it can be fully accepted.
major comments (4)
- [Section 2 (Generating test images)] The paper does not specify whether the PBR materials from Vastexture include displacement or bump maps and whether Blender applies them during rendering. If displacement maps are applied, replacing the material changes the rendered surface geometry, so the 'identical 3D shape' condition is violated in the texture-varied tests. This directly affects the both-changed condition in Table 1, because the accuracy drop could then reflect material-induced geometric deformation rather than a failure of 3D shape abstraction. Please state explicitly that only albedo/color or other non-geometric maps are used, or verify geometric invariance by rendering depth/normal maps and comparing the underlying mesh.
- [Section 3 (Testing the models) and Table 1] The four-panel tests are constructed by randomly sampling images, so the two distractor objects are typically very different from the target, often from different object classes. This makes the task solvable with coarse shape cues such as 'elongated', 'flat', or even object-class identity, rather than exact 3D shape matching. The paper's own appendix shows models citing exactly such features (e.g., 'flat or planar structure', 'elongated, complex structures'). To support the claim that models match 'identical 3D shapes', the benchmark needs a hard-negative condition in which distractors share coarse shape features or category membership with the target, or a stratified analysis over trials where such cues are controlled. Without this, the reported accuracies may overestimate fine-grained 3D shape understanding.
- [Section 3 and Table 1] No confidence intervals, error bars, or significance tests are reported despite N=1000 trials per cell. For example, GPT-4o's 82% versus 85% in the last two columns may be within sampling noise. Please report binomial confidence intervals or repeated-run variability. In addition, the human baseline is presented as a single row with no description of the number of participants, their selection criteria, viewing conditions, or whether they saw the same image sets; to support the headline comparison, this information is necessary.
- [Section 3.1 (Prompt selection)] The paper reports that different prompts can lead to up to 8% accuracy differences and that results are shown for the 'best prompts', but it does not state how many prompts were tried per model, on which data they were selected, or whether the same test trials were used during prompt selection. The repeated-question and 'increased assertiveness' policy for non-answers is also not quantified. These are adjustable protocol choices that can shift reported accuracies; please describe the exact selection procedure and report a sensitivity analysis, including how often the repetition policy was invoked.
minor comments (5)
- [Table 1] The row labels 'Keep Original Texture' and the V/x notation are ambiguous: in the first columns, V seems to mean 'kept', while in later columns it seems to mean 'varied'. Please make the legend explicit and consistent, for example by using separate rows for 'original texture kept' and 'texture varied'.
- [Appendix] The appendix is not referenced from the main text; please add a cross-reference in Section 4 where the explanations are discussed.
- [References] Reference 16 appears to have an incorrect author list; the entry should be checked against the original paper. Several references also lack full author lists or page numbers.
- [Text and figures] There are typos such as 'Querry' instead of 'Query' and inconsistent capitalization like 'LLama' and 'GPT 4o'. The code/data links are shown as placeholders 'URL: 1, 2, 3, 4'; please provide actual URLs in the final version.
- [Section ordering] The reference section is numbered 6 and appears before the supporting materials section numbered 7; this order should be corrected.
Circularity Check
No circularity: the paper is an empirical benchmark evaluation with no fitted parameters or derivation whose conclusion is forced by construction.
full rationale
The paper reports a controlled CGI benchmark: it generates four-panel forced-choice tasks from Objaverse objects, Vastexture PBR materials, and HDRI backgrounds, then measures LVLM accuracy under different combinations of changed orientation, texture, and background. The central claim is an empirical finding about model performance (e.g., GPT-4o drops from 97% to 82% when both orientation and texture change), not a derived quantity. No equation in the paper relates the reported accuracies to a fitted parameter, to the benchmark construction, or to the author's prior datasets in a way that would make the conclusion equivalent to its inputs. The self-citations (Vastextures, refs. [22,23]) serve only as a source of PBR textures; the benchmark's validity does not depend on any theorem or claimed result from those papers. The skeptical concern about displacement maps affecting rendered geometry is a potential threat to benchmark validity, not circularity: it questions whether the ground-truth condition 'identical 3D shape' holds, but it does not show that any result is assumed by construction. Therefore the paper is self-contained as an empirical evaluation and receives a circularity score of 0.
Assumptions & free parameters
free parameters (2)
- Best prompt selection =
Short prompt: 'Which of the panels contains an object with an identical 3D shape to the object in panel A.
- Answer repetition policy =
Uns specified; questions repeated 'several times' with 'increased assertiveness'
assumptions (3)
- domain assumption Rendered CGI images provide a valid proxy for real-world visual scenes
- domain assumption The Objaverse, Vastextures, and HDRI Haven pools are diverse enough to avoid systematic bias in distractor difficulty
- domain assumption Model API responses reflect visual understanding rather than instruction-following quirks
Cite this review
Pith. "Pith review of Do large language vision models understand 3D shapes?." pith.science (2026). https://pith.science/paper/NKPQWU5F
@misc{pith2026241210908,
author = {Pith},
title = {Pith review of: Do large language vision models understand 3D shapes?},
year = {2026},
howpublished = {\url{https://pith.science/paper/NKPQWU5F}},
note = {Machine review of arXiv:2412.10908}
}
read the original abstract
Large vision language models (LVLM) are the leading A.I approach for achieving a general visual understanding of the world. Models such as GPT, Claude, Gemini, and LLama can use images to understand and analyze complex visual scenes. 3D objects and shapes are the basic building blocks of the world, recognizing them is a fundamental part of human perception. The goal of this work is to test whether LVLMs truly understand 3D shapes by testing the models ability to identify and match objects of the exact same 3D shapes but with different orientations and materials/textures. A large number of test images were created using CGI with a huge number of highly diverse objects, materials, and scenes. The results of this test show that the ability of such models to match 3D shapes is significantly below humans but much higher than random guesses. Suggesting that the models have gained some abstract understanding of 3D shapes but still trail far beyond humans in this task. Mainly it seems that the models can easily identify the same object with a different orientation as well as matching identical 3D shapes of the same orientation but with different materials and textures. However, when both the object material and orientation are changed, all models perform poorly relative to humans. Code and benchmark are available.
Reference graph
Works this paper leans on
-
[1]
Introduction A general visual understanding of the world is fundamental for any intelligent system that needs to interact and act autonomously in the physical world. Creating such a system is one of the main goals of computer vision and can enable autonomous robots, cars, and numerous other applications[6]. Large vision language models (LVLM) such as GPT,...
-
[2]
Generating test images The images for the test were generated using the Blender 4.3 Computer graphic program. The use of CGI allows for massive amounts of object materials and environments as well as easy replacement of object materials. Which allows for the generation of a large number of highly diverse images. A large set of 3D objects were downloaded f...
-
[3]
Match objects with their original textures and the same background illumination, but different orientations (Figure 2)
-
[4]
Match objects with the same material/texture for all objects in the test and the same background illumination, but different orientations (Figure 3)
-
[5]
Match objects with the same orientation and illumination but different texture/material for each image (Figure 4)
-
[6]
Same environment but different orientation and texture for each image (Figure 5) 5) Same orientation and materials, but different background and illumination (figure 6) 6) Different everything: orientation, texture, and environment (Figure 7) 7) Same everything (basically a set of identical images for each object). Note that tests 2,4,6 force the model to...
-
[7]
Testing the models To test the model's ability to recognize 3D shapes we create a four-panel image (Figure 1-6). Two panels contain the same 3D shape but with some variation (orientation, texture, environment, Figures 2-6) and the model asks to find which panel contains an object with 3D shape identical to the object in panel A. The test images (Figure 1-...
-
[8]
2) Has a different orientation compared to the object in panel A
Has the exact same 3d shape as the object in panel A. 2) Has a different orientation compared to the object in panel A. 3)Has a different texture compared to the object in panel A. Respond with ONLY the letter of the panel (B, C, or D) that meets all these criteria.” Specifically, pointing out the various possible changes seems to have negative effects. A...
Show all 32 references
-
[9]
Gemini and GPT 4o clearly excel in this task but all models show some level of understanding (Table 1)
Results The results in Table 1 clearly show that all the models grasped the tasks and have some understanding of 3D shapes which is way above random (33%), even for cases where the object material is replaced or when viewed from a different angle. Gemini and GPT 4o clearly exc...
-
[10]
These results are consistent with previous works which show that despite their impressive performance Vision Language Models (VLM) often miss basic aspects of reality[8-12]
Conclusion The results of this work show that large vision models have gained an abstract understanding of 3D shapes, but still trail far beyond humans in this basic task. These results are consistent with previous works which show that despite their impressive performance Vis...
-
[11]
Code used to evaluate the models on the benchmark available at this URL
Supporting materials The 3D shape Matching Dataset images are available at these URLs: GITHUB, ZENODO Code used to generate the dataset is available at this URL. Code used to evaluate the models on the benchmark available at this URL
-
[12]
Flamingo: a visual language model for few-shot learning
Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in neural information processing systems 35 (2022): 23716-23736
2022
-
[13]
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Meta. LLama3 model. URL: https://ai.meta.com/blog/meta-llama-3/ 3. Anthropic, The Claude 3 Model Family: Opus, Sonnet, Haiku. URL: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf 4. OpenAI. Gpt-4 technical report. arXiv, pages 230...
2024 arXiv
-
[14]
Real-world robot applications of foundation models: A review
Kawaharazuka, Kento, et al. "Real-world robot applications of foundation models: A review." Advanced Robotics 38.18 (2024): 1232-1254
2024
-
[15]
Approaching human 3D shape perception with neurally mappable models
O'Connell, Thomas P., et al. "Approaching human 3D shape perception with neurally mappable models." arXiv preprint arXiv:2308.11300 (2023)
2023 arXiv
-
[16]
Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models
Xu, Peng, et al. "Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
2024
-
[17]
VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information
Kamoi, Ryo, et al. "VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information." arXiv preprint arXiv:2412.00947 (2024)
2024 arXiv
-
[18]
Are Vision Language Models Texture or Shape Biased and Can We Steer Them?
Gavrikov, Paul, et al. "Are Vision Language Models Texture or Shape Biased and Can We Steer Them?." arXiv preprint arXiv:2403.09193 (2024)
2024 arXiv
-
[19]
Exploring the frontier of vision-language models: A survey of current methodologies and future directions
Ghosh, Akash, et al. "Exploring the frontier of vision-language models: A survey of current methodologies and future directions." arXiv preprint arXiv:2404.07214 (2024)
2024
-
[20]
Blink: Multimodal large language models can see but not perceive
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390, 2024
2024 arXiv
-
[21]
Can 3D Vision-Language Models Truly Understand Natural Language?
Deng, Weipeng, et al. "Can 3D Vision-Language Models Truly Understand Natural Language?." arXiv preprint arXiv:2403.14760 (2024)
2024 arXiv
-
[22]
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017
2017
-
[23]
GPT4Vis: what can GPT-4 do for zero-shot visual recognition?
Wu, Wenhao, et al. "GPT4Vis: what can GPT-4 do for zero-shot visual recognition?." arXiv preprint arXiv:2311.15732 (2023)
2023 arXiv
-
[24]
A general protocol to probe large vision models for 3d physical understanding
Zhan, Guanqi, et al. "A general protocol to probe large vision models for 3d physical understanding." The Thirty-eighth Annual Conference on Neural Information Processing Systems . 2023
2023
-
[25]
A survey on hallucination in large vision-language models
Liu, Hanchao, et al. "A survey on hallucination in large vision-language models." arXiv preprint arXiv:2402.00253 (2024)
2024 arXiv
-
[26]
When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
Ma, Xianzheng, et al. "When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models." arXiv preprint arXiv:2405.10255 (2024)
2024
-
[27]
Language-Image Models with 3D Understanding
Cho, Jang Hyun, et al. "Language-Image Models with 3D Understanding." arXiv preprint arXiv:2405.03685 (2024)
2024 arXiv
-
[28]
Shapellm: Universal 3d object understanding for embodied interaction
Qi, Zekun, et al. "Shapellm: Universal 3d object understanding for embodied interaction." European Conference on Computer Vision . Springer, Cham, 2025
2025
-
[29]
Objaverse: A universe of annotated 3d objects
Deitke, Matt, et al. "Objaverse: A universe of annotated 3d objects." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2023
2023
-
[30]
Vastextures: Vast repository of textures and PBR materials extracted from real-world images using unsupervised methods
Eppel, Sagi. "Vastextures: Vast repository of textures and PBR materials extracted from real-world images using unsupervised methods." arXiv preprint arXiv:2406.17146 (2024)
2024 arXiv
-
[31]
Infusing Synthetic Data with Real-World Patterns for Zero-Shot Material State Segmentation
Eppel, Sagi, et al. "Infusing Synthetic Data with Real-World Patterns for Zero-Shot Material State Segmentation." The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track
-
[32]
Which panel contains an object that has identical 3D shape to the object in panel A, but different in orientation and texture. Explain
Polyhaven free HDRI repository. https://polyhaven.com Appendix: Sampled explanation given by models to justify their answers. Each response refers to the image above it. Query: “Which panel contains an object that has identical 3D shape to the object in panel A, but different ...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.