Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that a vision-language model understands a scene only if it can recreate that scene as an executable 3D program, and its benchmark shows current models name objects and attributes but place them imprecisely.

desk verdict A well-designed benchmark for testing VLM scene understanding through agentic inverse rendering, undercut by an empirical section whose numbers do not line up with its own table. read the letter →

arxiv 2506.23329 v1 pith:5OZDLVNE submitted 2025-06-29 cs.CV

Parker Liu , Chenxin Li , Zhengxin Li , Yipeng Wu , Wuyang Li , Zhiqin Yang , Zhenyuan Zhang , Yunlong Lin
show 2 more authors
Sirui Han Brandon Y. Feng
This is my paper
classification cs.CV
keywords agenticinverserenderingvision-languagemodelssceneunderstandinganalysis-by-synthesistool-usingagents3DreconstructionbenchmarkCLEVRBlender
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IR3D-Bench proposes that a vision-language model has really understood a scene only when it can recreate that scene from a single image, and builds a benchmark around that "understanding-by-creating" test. The agent sees one image and a fixed camera, writes a structured JSON description of the objects in the scene, and that description is rendered in Blender and compared with the original across localization, appearance, and layout scores. On 1,500 controlled CLEVR scenes, current models recognize object attributes and manage the tools, but their reconstructions are spatially imprecise: positions, sizes, and inter-object relations come out wrong more often than colors or materials. The paper concludes that the dominant bottleneck is visual precision, the ability to compare a rendering with the target image, rather than tool usage or output syntax. If the benchmark works, it gives the field a generative test of understanding that descriptive fluency alone cannot pass.

What carries the argument

The central object is the executable scene program: a JSON scene graph containing camera, lighting, and a list of objects with position, size parameters, shape, and material, which Blender renders into an image. The design instantiates the analysis-by-synthesis paradigm, perception as generating and refining hypotheses, by making the hypothesis an executable program. The argument is carried by an evaluation chain: project predicted 3D centers to 2D with fixed camera intrinsics; match predicted objects to ground-truth objects via CLIP text-similarity and the Hungarian algorithm with a threshold; prompt SAM at projected centers to obtain masks; then score pixel distance, count accuracy, bounding-box edge similarity, relation accuracy, mask IoU/DICE, CLIP attribute similarity, and GPT-4o layout and appearance ratings. This chain converts an agent's internal world model into externally comparable numbers, making "understanding" mean "reconstructable".

What would settle it

Take one model's predicted JSON, render it, and recompute the per-object mask scores twice: once with SAM prompted at the predicted projected center and once with SAM prompted at the ground-truth center for the same matched object; if IoU and DICE rise substantially under ground-truth prompting, the reported appearance scores are contaminated by localization error and the paper's central bottleneck claim would need to be re-derived on a localization-free score.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that understanding-by-creating is a measurable and operational test, and that current vision-language agents fail it in a specific way: they can say what is in a scene and emit valid tool calls, yet their reconstructed 3D layouts are imprecise. Even with a fixed camera and a tightly constrained JSON schema, models misplace object centers, misestimate scale, and break pairwise spatial relations, while attribute-level color, material, and shape scores remain high. The paper also shows that iterative refinement against rendered feedback and careful prompt decomposition improve reconstruction quality, which it reads as evidence that the limiting factor is the fidelity of the visual comparison, not programming syntax or tool access.

Load-bearing premise

The whole benchmarking result stands on the assumption that the matching and mask-extraction steps, CLIP text-similarity matching with an unreported threshold and then SAM segmenting at the projected center of each predicted object, assign each predicted object to the right ground-truth object, because systematic bias in those steps would shift every score without changing any model.

Editorial extensions

If this is right

  • Descriptive benchmarks may overstate understanding, because captioning or VQA scores could stay high while reconstruction-based scores expose missing 3D precision.
  • Training effort for tool-using vision-language models should shift from syntax scaffolding and instruction tuning toward sharper visual representations and reliable self-comparison of rendered output with the input image.
  • Iterative refinement with rendered feedback is a working lever, since ten steps bring a mid-tier model's output close to a top proprietary model, so test-time compute can partially substitute for base capabilities.
  • The released benchmark provides structured input-output pairs that can be used for supervised fine-tuning or chain-of-thought training on compositional 3D reasoning.
  • Benchmark scores separate models that follow the output format from models that actually place objects, making template-copying and degenerate outputs visible as failures rather than as partial credit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the bottleneck diagnosis holds, a promising next experiment is to give the agent a fast renderer loop that shows side-by-side image differences and measure whether reconstruction accuracy improves more than increasing model scale would.
  • The metric chain conflates localization with appearance: because SAM receives the predicted center as a prompt, an object placed in the wrong spot will be segmented at the wrong location, so IoU and DICE partly punish positioning error and may understate how well the model captured true appearance; testing masks grounded at ground-truth centers would disentangle these.
  • The use of CLIP text-text similarity over a small closed vocabulary of CLEVR attribute names likely makes matching easier than in the real world, so extending the protocol to open-vocabulary or real images will require validating that the matching threshold and CLIP embeddings do not silently reorder model rankings.
  • The same understanding-by-creating protocol could be applied to temporal or multi-view scenes, where a reconstruction that must explain several views would test consistency of the internal 3D world model rather than single-image heuristics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces IR3D-Bench, an agentic inverse rendering benchmark in which a vision-language agent must reconstruct a CLEVR scene from a single RGB image by producing a structured JSON scene description that is rendered in Blender. Reconstruction quality is measured with localization metrics (pixel distance, count accuracy, bounding-box score, relation accuracy), mask-level and semantic-level appearance metrics (SAM-based IoU/DICE and CLIP similarity), and GPT-4o-based language-aligned semantics scores. The authors evaluate more than twenty proprietary and open-source VLMs, report that attribute recognition and tool-use are relatively strong while spatial precision and mask overlap are low, and conclude that the dominant bottleneck is visual precision rather than tool use. The paper also reports an iterative refinement experiment and prompt-design ablations, and it releases the benchmark data and evaluation protocols.

Significance. If the measurement pipeline is valid, IR3D-Bench is a useful complement to descriptive scene-understanding benchmarks because it tests understanding through generation, uses CLEVR's exact annotations for ground-truth geometry and attributes, and provides a broad multi-metric comparison of current VLMs. The paper is candid about scope limitations in Appendix C.1, and the iterative-refinement study is a valuable practical result. However, the headline claim that the bottleneck is visual precision rather than tool use depends on mask-level and object-matching metrics whose construction is currently under-specified and whose reported numbers are internally inconsistent, so the main empirical conclusion is not yet established.

major comments (4)
  1. [Sec. 5.2, Table 1] The prose numbers in Sec. 5.2 contradict Table 1. The text reports GPT-4o Pixel Distance 0.0004 and Gemini-2.5-pro 0.0003, while Table 1 lists 0.5528 and 0.3791; the text reports GPT-4o IoU 0.43 and Claude-3.7 IoU 0.40, while Table 1 lists 0.07 and 0.09; the model-specific analysis reports Gemini IoU/DICE 0.41/0.55 and Qwen 0.28/0.36, while Table 1 lists 0.11/0.18 and 0.08/0.13. Because Table 1 is the only quantitative support for the central claim that spatial precision is the dominant bottleneck, these discrepancies must be resolved (by correcting the table, the text, or the units and metric definitions) before the conclusion can be assessed.
  2. [Sec. 4.2, Per-Object Mask Evaluation] The IoU and DICE metrics are computed from SAM masks prompted with the projected 2D center of each predicted 3D object. When a predicted center is wrong, which is precisely the failure mode the benchmark targets, SAM will tend to segment a different region of the image, so IoU/DICE conflate localization error with silhouette and scale error. The manuscript does not state whether SAM is applied to the input image or to a rendering of the predicted scene, and CLEVR already provides exact instance masks, making the unvalidated SAM step unnecessary. Please recompute mask-level metrics from true rendered masks of the predicted and ground-truth scenes (or validate point-prompted SAM on small, overlapping, and off-center cases) and report the resulting numbers; this is load-bearing for the Sec. 6 conclusion that the dominant bottleneck is a lack of visual precision.
  3. [Sec. 4.2, Object Matching] The similarity threshold tau in the matching step is never reported, and objects with S_i,phi*(i) < tau are assigned Match(i) = -1 and silently dropped from pixel distance, bounding-box, IoU/DICE, and related object-level metrics. A different tau changes which errors are counted and can reorder Table 1 without any model output changing. Please report the chosen tau and provide a sensitivity analysis (for example, a sweep over tau values) to show that the ranking and the reported performance gaps are robust. This also applies to the use of unvalidated CLIP text-text similarities for semantic matching.
  4. [Sec. 4.3.3, Language-Aligned Semantics] GPT-4o is both an evaluated model in Table 1 and the judge that produces the Language-Aligned Semantics scores in Sec. 4.3.3. This makes the LLM-score columns potentially self-referential for GPT-4o and risks systematic bias in its favor. Please report the exact judge configuration, validate the LLM scores against human ratings or a second non-GPT-4o judge on a sample, and disclose the overlap between evaluated models and judge models; if agreement is weak, these columns should be treated as exploratory rather than headline metrics.
minor comments (5)
  1. [Table 1] The rows for DeepSeek-VL2, Llama-3.2-11B-Vision, and H2OVL show '×Failed' with no metrics; please clarify whether these models are excluded entirely or scored as zeros, and report the failure rate explicitly.
  2. [References] Reference [76] attributes the Gemini model family to Anthropic, but it is a Google DeepMind model; reference [85] has the author name 'Laurenccon', which should be 'Laurençon'.
  3. [Fig. 6 and Table 1] The y-axis label 'Pixel Distance (x10^3)' in Fig. 6 conflicts with the Pixel Distance values in Table 1, which appear to be in different units; please state the units of Pixel Distance consistently in the table, text, and figures.
  4. [Sec. 2 and Fig. 2] The task is described as writing a Blender Python script, but the evaluated pipeline asks the VLM to output JSON that is then rendered; please clarify the relationship between the programmatic formulation in Sec. 2 and the JSON-based interface in Fig. 2, including where the Blender script is generated.
  5. [Table 5] The prompt-ablation table contains malformed entries (for example, '0.197.1598' and '2.91.94') and should be reformatted so every cell is legible and corresponds to a single reported value.

Circularity Check

1 steps flagged · score 6.0 of 10

The Semantic-level CLIP appearance score is the same objective used to fit object correspondences, so that sub-score is partly forced by construction; the spatial metrics remain independent.

  1. fitted input called prediction [Sec. 4.2 (Object Matching) and Sec. 4.3.2 (Visual Appearance, Semantic-level)]
    "For each attribute dimension k in color, size, material, and shape, we compute the semantic similarity using a CLIP text encoder ... We convert it into a cost matrix C = 1 − S, and solve the assignment using the Hungarian algorithm: ϕ∗ = arg maxϕ Σ_i S_i,ϕ(i). ... These are embedded using the CLIP model, and cosine similarity is computed between the predicted and reference embeddings. Attribute-wise scores are reported along with an Overall Appearance Score obtained by averaging across all annotated attributes."

    The Hungarian matching is explicitly chosen to maximize the total CLIP text-text similarity over predicted-GT pairs. The Semantic-level appearance score uses the same CLIP cosine similarity, averaged over exactly the pairs selected by that maximization. The reported Color, Size, Material, Shape, and Overall CLIP scores are therefore the objective function used to fit correspondences, not an independent measurement of attribute fidelity. A different matching rule (or a different similarity function) would change the appearance scores even with identical model outputs, so the paper's evidence that VLMs 'grasp high-level object attributes' is in part statistically forced by the matching construction.

full rationale

Most of IR3D-Bench is self-contained against external ground truth: CLEVR provides independent 3D coordinates, counts, boxes, and relation labels, and the geometric metrics (Pixel Distance, Count Accuracy, BBox Edge Score, Relation Accuracy, mask IoU/DICE) are computed from those annotations rather than from the evaluated models' own outputs. I found no load-bearing self-citations and no imported uniqueness theorems; the iterative-refinement study is an empirical experiment, not a circular derivation. The one genuine circularity is the CLIP-based object matching feeding the CLIP-based Semantic-level appearance score: Sec. 4.2's Hungarian objective and Sec. 4.3.2's Overall Appearance Score are the same quantity by construction, so that particular metric is a fitted objective presented as a measured result. This weakens the 'strong attribute recognition' claim but does not invalidate the spatial-precision conclusion, which rests on the independent IoU/DICE and Relation Accuracy metrics; hence a partial-circularity score of 6 rather than higher. Separately, the mask IoU/DICE conflates localization error with silhouette error because SAM masks are prompted by predicted centers, but this is a validity concern, not a reduction by construction, so it is not counted in the circularity score.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The benchmark introduces no new physical entities. Its free parameters are the unstated matching threshold tau and the unspecified 1,500-image subset selection, both of which directly shape the reported scores. The core premises are the trustworthiness of CLEVR GT annotations, SAM masks, CLIP text-text similarities, and GPT-4o judgments.

free parameters (2)
  • semantic matching threshold tau = not reported
    Sec. 4.2 gates all object matches with S(i,phi*(i)) >= tau and discards predicted objects below it. The value of tau is never stated, yet it determines which predictions contribute to every per-object metric.
  • evaluation subset of 1,500 CLEVR images = 1,500 of 15,000 validation images
    Appendix A.1 states all models are evaluated on a representative subset but does not specify the selection rule. Any non-random selection changes model rankings and the reported conclusions.
assumptions (5)
  • domain assumption CLEVR ground-truth scene graphs (6D poses, materials, colors) are treated as accurate references for scene understanding.
    Sec. 4.1 adopts the CLEVR validation split and its annotations as GT without independent verification; all benchmark scores inherit CLEVR's correctness.
  • domain assumption SAM, prompted by a projected object center, returns a reliable instance mask of the intended object.
    Sec. 4.2 uses SAM zero-shot masks for all IoU, DICE, and bounding-box metrics. A mistargeted prompt yields the neighboring object's mask, making localization errors double-counted.
  • domain assumption CLIP text-text cosine similarity is a valid measure of semantic attribute agreement for color, size, material, and shape.
    Secs. 4.2 and 4.3.2 score object matching and appearance using CLIP embeddings of text strings only, with no validation that CLIP rankings correlate with visual fidelity.
  • domain assumption GPT-4o provides calibrated, bias-free 0-5 ratings of scene layout and appearance from JSON descriptions.
    Sec. 4.3.3 uses GPT-4o as the judge for Language-Aligned Semantics scores; no calibration against human ratings is reported.
  • standard math The pinhole projection with fixed intrinsics K and extrinsics E exactly reproduces CLEVR render geometry.
    Sec. 4.2 defines projection pi(K,E,P3D) = K[R|t]P3D. If the fixed camera differs from CLEVR's true render camera, all pixel-space metrics inherit a systematic bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering." pith.science (2026). https://pith.science/paper/5OZDLVNE

@misc{pith2026250623329,
  author       = {Pith},
  title        = {Pith review of: IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5OZDLVNE}},
  note         = {Machine review of arXiv:2506.23329}
}
read the original abstract

Vision-language models (VLMs) excel at descriptive tasks, but whether they truly understand scenes from visual observations remains uncertain. We introduce IR3D-Bench, a benchmark challenging VLMs to demonstrate understanding through active creation rather than passive recognition. Grounded in the analysis-by-synthesis paradigm, IR3D-Bench tasks Vision-Language Agents (VLAs) with actively using programming and rendering tools to recreate the underlying 3D structure of an input image, achieving agentic inverse rendering through tool use. This "understanding-by-creating" approach probes the tool-using generative capacity of VLAs, moving beyond the descriptive or conversational capacity measured by traditional scene understanding benchmarks. We provide a comprehensive suite of metrics to evaluate geometric accuracy, spatial relations, appearance attributes, and overall plausibility. Initial experiments on agentic inverse rendering powered by various state-of-the-art VLMs highlight current limitations, particularly in visual precision rather than basic tool usage. IR3D-Bench, including data and evaluation protocols, is released to facilitate systematic study and development of tool-using VLAs towards genuine scene understanding by creating.

Figures

Figures reproduced from arXiv: 2506.23329 by the authors.

Figure 1
Figure 1. Understanding by Creating. (A) Humans demonstrate understanding by constructing [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the IR3D-Bench Pipeline. The benchmark consists of two stages: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visual Results with Selected VLMs. Gemini-2.5-pro demonstrates strong understanding of [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Holistic comparison over 14 metrics. In this section, we evaluate the agentic inverse rendering capabilities of VLMs as they inter￾act with programming and rendering tools to demonstrate their understanding by creating. 5.1 Benchmark Setup Models We evaluate more than …
Figure 5
Figure 5. Figure 5: Qualitative results illustrating the effect of [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Prompt Design and Error Analysis. (A) Inverse rendering prompt includes structured [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 6
Figure 6. Figure 6: Understanding scales with more refinements. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: More Visual Results with Selected VLMs on IR3D-Bench [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Failure output of selected models on IR3D-Bench [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning

    cs.CV 2026-07 accept novelty 7.0 of 10

    Current VLMs top out at 62.1/100 on holistic single-image 3D indoor layout prediction, with strong recognition but weak geometric regression, and mid-tier rankings that shift relative to QA and primitive-reconstructio...

Reference graph

Works this paper leans on

89 extracted references · 42 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    arXiv 2024

  2. [2]

    GPT-4o System Card

    OpenAI et al. GPT-4o System Card. arXiv:2410.21276, 2024

    arXiv 2024

  3. [3]

    The Claude 3 Model Family: Opus, Sonnet, Haiku

    Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. https://www.anthropic. com/news/claude-3-family

  4. [4]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [6]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024

    2024

  7. [7]

    Perceptions as hypotheses

    Richard Langton Gregory. Perceptions as hypotheses. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, 290(1038):181–197, 1980

    1980

  8. [8]

    Yuille and Daniel Kersten

    Alan L. Yuille and Daniel Kersten. Vision as bayesian inference: analysis by synthesis? Trends in Cognitive Sciences, 10:301–308, 2006

    2006

Show all 89 references
  1. [9]

    Efficient and robust analysis-by-synthesis in vision: A computational framework, behavioral tests, and modeling neuronal representations

    Ilker Yildirim, Tejas D Kulkarni, Winrich A Freiwald, and Joshua B Tenenbaum. Efficient and robust analysis-by-synthesis in vision: A computational framework, behavioral tests, and modeling neuronal representations. In Annual Conference of the Cognitive Science Society , 2015

    2015

  2. [10]

    Bever and David Poeppel

    Thomas G. Bever and David Poeppel. Analysis by synthesis: A (re-)emerging program of research for language and vision. Biolinguistics, 2010

    2010

  3. [11]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539– 68551, 2023

    2023

  4. [12]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14953–14962, 2022

    2023

  5. [13]

    Vipergpt: Visual inference via python execution for reasoning

    Didac Suris, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11854–11864, 2023

    2023

  6. [14]

    Gorilla: Large language model connected with massive apis

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems , 37:126544–126565, 2024

    2024

  7. [15]

    Blender - a 3D modelling and rendering package, 2016

    Blender Online Community. Blender - a 3D modelling and rendering package, 2016

    2016

  8. [16]

    https://github.com/modelcontextprotocol, 2024

    Model Context Protocol. https://github.com/modelcontextprotocol, 2024. 10

    2024

  9. [17]

    Blendermcp

    Siddharth Ahuja. Blendermcp. https://github.com/ahujasid/blender-mcp, 2025

    2025

  10. [18]

    Embodied question answering

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10, 2018

    2018

  11. [19]

    Multi-target embodied question answering

    Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L Berg, and Dhruv Batra. Multi-target embodied question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6309–6318, 2019

    2019

  12. [20]

    3d concept learning and reasoning from multi-view images

    Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9202–9212, 2023

    2023

  13. [21]

    One step at a time: Long-horizon vision-and-language navigation with milestones

    Chan Hee Song, Jihyung Kil, Tai-Yu Pan, Brian M Sadler, Wei-Lun Chao, and Yu Su. One step at a time: Long-horizon vision-and-language navigation with milestones. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15482–15491, 2022

    2022

  14. [22]

    Embodied bert: A transformer model for embodied, language-guided visual task completion

    Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Embodied bert: A transformer model for embodied, language-guided visual task completion. arXiv preprint arXiv:2108.04927, 2021

    2021 arXiv

  15. [23]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognitio...

    2017

  16. [24]

    Barrow and Jay M

    Harry G. Barrow and Jay M. Tenenbaum. Recovering intrinsic scene characteristics from images. In Computer Vision Systems, 1978

    1978

  17. [25]

    Soft rasterizer: A differentiable renderer for image-based 3d reasoning

    Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7708–7717, 2019

    2019

  18. [26]

    Carr, Jonathan Ragan-Kelley, and Frédo Durand

    Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan A. Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. Interna- tional Conference on Learning Representations (ICLR), 2020

    2020

  19. [27]

    Differentiable vector graphics rasterization for editing and learning

    Tzu-Mao Li, Michal Luká ˇc, Michaël Gharbi, and Jonathan Ragan-Kelley. Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Graphics (TOG), 39(6):1–15, 2020

    2020

  20. [28]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020

    2020

  21. [29]

    Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction

    Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, 2021

    2021

  22. [30]

    V olume rendering of neural implicit surfaces

    Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. V olume rendering of neural implicit surfaces. In NeurIPS, 2021

    2021

  23. [31]

    Synsin: End-to-end view synthesis from a single image

    Olivia Wiles, Georgia Gkioxari, and Noah Snavely. Synsin: End-to-end view synthesis from a single image. In CVPR, 2020

    2020

  24. [32]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42(4), 2023

    2023

  25. [33]

    Path-space differentiable rendering

    Cheng Zhang, Yihang Guo, Zexiang Dong, Ravi Ramamoorthi, and Manmohan Chandraker. Path-space differentiable rendering. In SIGGRAPH Asia, 2020

    2020

  26. [34]

    Inst3d-lmm: Instance-aware 3d scene understanding with multi-modal instruction tuning, 2025

    Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. Inst3d-lmm: Instance-aware 3d scene understanding with multi-modal instruction tuning, 2025. 11

    2025

  27. [35]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    2024 arXiv

  28. [36]

    Scanqa: 3d question answering for spatial scene understanding

    Daich Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19107–19117, 2021

    2022

  29. [37]

    3d-vista: Pre- trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre- trained transformer for 3d vision and text alignment. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2899–2909, 2023

    2023

  30. [38]

    Chang, and Matthias Nießner

    Dave Zhenyu Chen, Angel X. Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. ArXiv, abs/1912.08830, 2019

    1912 arXiv

  31. [39]

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas J. Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference on Computer Vision, 2020

    2020

  32. [40]

    Dave Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X. Chang. Scan2cap: Context- aware dense captioning in rgb-d scans. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3192–3202, 2020

    2021

  33. [41]

    Scalable 3d captioning with pretrained models

    Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems , 36:75307–75337, 2023

    2023

  34. [42]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent-Pierre Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv...

    2024

  35. [43]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CV...

    2020

  36. [44]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision, pages 289–310. Springer, 2024

  37. [45]

    Spatialrgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. Advances in Neural Information Processing Systems, 37:135062–135093, 2025

    2025

  38. [46]

    Conceptfusion: Open-set multimodal 3d mapping

    Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241, 2023

    2023 arXiv

  39. [47]

    Context-aware entity grounding with open-vocabulary 3d scene graphs

    Haonan Chang, Kowndinya Boyalakuntla, Shiyang Lu, Siwei Cai, Eric Jing, Shreesh Keskar, Shijie Geng, Adeeb Abbas, Lifeng Zhou, Kostas Bekris, et al. Context-aware entity grounding with open-vocabulary 3d scene graphs. In CoRL, 2023

    2023

  40. [48]

    Tenenbaum, Antonio Torralba, Florian Shkurti, and Liam Paull

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio Torralba, Florian Shkurti, and Liam Paull. Conceptgraphs...

    2024

  41. [49]

    3d-llm: Injecting the 3d world into large language models

    Yujie Hong, Huajun Zhen, Peixi Chen, et al. 3d-llm: Injecting the 3d world into large language models. arXiv:2307.12981, 2023. 12

    2023 arXiv

  42. [50]

    Agent3d-zero: An agent for zero-shot 3d understanding

    Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, and Yanyong Zhang. Agent3d-zero: An agent for zero-shot 3d understanding. arXiv:2403.11835, 2024

    2024 arXiv

  43. [51]

    3dmit: 3d multi-modal instruction tuning for scene understanding

    Zeju Li, Chao Zhang, Xiaoyan Wang, et al. 3dmit: 3d multi-modal instruction tuning for scene understanding. arXiv:2401.03201, 2024

    2024 arXiv

  44. [52]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In CVPR, 2024

    2024

  45. [53]

    Kulkarni, Pushmeet Kohli, Joshua B

    Tejas D. Kulkarni, Pushmeet Kohli, Joshua B. Tenenbaum, and Vikash K. Mansinghka. Pic- ture: A probabilistic programming language for scene perception. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4390–4399, 2015

    2015

  46. [54]

    Learning to infer graphics programs from hand-drawn images

    Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphics programs from hand-drawn images. Advances in neural information processing systems, 31, 2018

    2018

  47. [55]

    Learning to infer and execute 3d shape programs

    Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T Freeman, Joshua B Tenen- baum, and Jiajun Wu. Learning to infer and execute 3d shape programs. In 7th International Conference on Learning Representations, ICLR 2019, 2019

    2019

  48. [56]

    Shapeassembly: Learning to generate programs for 3d shape structure synthesis

    R Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J Mitra, and Daniel Ritchie. Shapeassembly: Learning to generate programs for 3d shape structure synthesis. ACM Transactions on Graphics (TOG), 39(6):1–20, 2020

    2020

  49. [57]

    Scenecraft: An llm agent for synthesizing 3d scenes as blender code

    Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An llm agent for synthesizing 3d scenes as blender code. In Forty-first International Conference on Machine Learning, 2024

    2024

  50. [58]

    The scene language: Representing scenes with programs, words, and embeddings

    Yunzhi Zhang, Zizhang Li, Matt Zhou, Shangzhe Wu, and Jiajun Wu. The scene language: Representing scenes with programs, words, and embeddings. ArXiv, abs/2410.16770, 2024

    2024 arXiv

  51. [59]

    3d-gpt: Procedural 3d modeling with large language models

    Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, and Stephen Gould. 3d-gpt: Procedural 3d modeling with large language models. arXiv preprint arXiv:2310.12945, 2023

    2023 arXiv

  52. [60]

    Scenemotifcoder: Example-driven visual program learning for generating 3d object arrange- ments

    Hou In Ivan Tam, Hou In Derek Pun, Austin T Wang, Angel X Chang, and Manolis Savva. Scenemotifcoder: Example-driven visual program learning for generating 3d object arrange- ments. arXiv preprint arXiv:2408.02211, 2024

    2024 arXiv

  53. [61]

    L3go: Language agents with chain-of-3d-thoughts for generating unconventional objects

    Yutaro Yamada, Khyathi Chandu, Yuchen Lin, Jack Hessel, Ilker Yildirim, and Yejin Choi. L3go: Language agents with chain-of-3d-thoughts for generating unconventional objects. arXiv preprint arXiv:2402.09052, 2024

    2024 arXiv

  54. [62]

    Creative agents: Empowering agents with imagination for creative tasks

    Chi Zhang, Penglin Cai, Yuhui Fu, Haoqi Yuan, and Zongqing Lu. Creative agents: Empowering agents with imagination for creative tasks. arXiv preprint arXiv:2312.02519, 2023

    2023

  55. [63]

    Scenex: Procedural controllable large-scale scene generation via large-language models

    Mengqi Zhou, Jun Hou, Chuanchen Luo, Yuxi Wang, Zhaoxiang Zhang, and Junran Peng. Scenex: Procedural controllable large-scale scene generation via large-language models. arXiv e-prints, pages arXiv–2403, 2024

    2024

  56. [64]

    Program-guided image manipulators

    Jiayuan Mao, Xiuming Zhang, Yikai Li, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Program-guided image manipulators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4030–4039, 2019

    2019

  57. [65]

    Layoutvlm: Differentiable optimization of 3d layout via vision-language models

    Fan-Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, and Jiajun Wu. Layoutvlm: Differentiable optimization of 3d layout via vision-language models. ArXiv, abs/2412.02193, 2024

    2024 arXiv

  58. [66]

    Virtualhome: Simulating household activities via programs

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502, 2018. 13

    2018

  59. [67]

    Xia, Peng Xu, Karol Hausman, Brian Ichter, Peter R

    Jacky Liang, Wenlong Huang, F. Xia, Peng Xu, Karol Hausman, Brian Ichter, Peter R. Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500, 2022

    2023

  60. [68]

    Monet: Unsupervised scene decomposition and representa- tion

    Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv:1901.11390, 2019

    1901 arXiv

  61. [69]

    Genesis: Generative scene inference and sampling with object-centric latent representations

    Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. arXiv:1907.13052, 2019

    1907 arXiv

  62. [70]

    Giraffe: Representing scenes as compositional genera- tive neural feature fields

    Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11453–11464, 2021

    2021

  63. [71]

    A simple neural network module for relational reasoning

    Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Advances in neural information processing systems, 30, 2017

    2017

  64. [72]

    Compositional attention networks for machine reasoning

    Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. arXiv:1803.03067, 2018

    2018 arXiv

  65. [73]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  66. [74]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  67. [75]

    GPT-4 Technical Report

    OpenAI et al. GPT-4 Technical Report. arXiv:2303.08774, 2023

    2023 arXiv

  68. [76]

    The Gemini 2 Model Family: Google Deepmind

    Anthropic. The Gemini 2 Model Family: Google Deepmind. https://gemini.google.com/

  69. [77]

    The Grok Model Family: xAI

    Anthropic. The Grok Model Family: xAI. https://grok.com/

  70. [78]

    Deepseek-vl2: Mixture-of-experts vision- language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision- language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  71. [79]

    Llava-next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong Jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, 2024. Accessed on 2024-05-06

    2024

  72. [80]

    The Llama 3 Herd of Models

    Meta AI. The Llama 3 Herd of Models. arXiv:2407.21783, 2024

    2024 arXiv

  73. [81]

    Galib, Shanshan Wang, Guanshuo Xu, Pascal Pfeiffer, Ryan Chesler, Mark Landry, and SriSatish Ambati

    Shaikat M. Galib, Shanshan Wang, Guanshuo Xu, Pascal Pfeiffer, Ryan Chesler, Mark Landry, and SriSatish Ambati. H2ovl-mississippi vision language models technical report. ArXiv, abs/2410.13611, 2024

    2024 arXiv

  74. [82]

    Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

    Phi-4 Research Team. Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs. arXiv:2503.01743, 2025

    2025 arXiv

  75. [83]

    Pixtral 12B

    Pravesh Agrawal et al. Pixtral 12B. arXiv:2410.07073, 2024

    2024 arXiv

  76. [84]

    Aria: An open multimodal native mixture-of-experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv:2410.05993, 2024. 14

    2024 arXiv

  77. [85]

    Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh

    Hugo Laurenccon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelisc: An open web-scale filtered dataset of interleaved image-text documents. arXi...

    2023 arXiv

  78. [86]

    Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling. arXiv:2412.05271, 2024

    2024 arXiv

  79. [87]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv:2504.10479, 2025

    2025 arXiv

  80. [88]

    Qwen2.5 technical report

    Qwen An Yang, Baosong Yang, and Beichen Zhang et al. Qwen2.5 technical report. arXiv:2412.15115, 2024

    2024 arXiv

  81. [90]

    ‘json

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.