Pith. sign in

REVIEW 3 major objections 6 minor 85 references

VLMs can name the furniture in a room, but they still cannot measure it: best score on holistic 3D layout is only 62.1/100.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:11 UTC pith:5VQIKDM5

load-bearing objection Solid diagnostic benchmark: holistic monocular layout on re-renderable indoor scenes, with a clear recognition–geometry gap that QA suites miss. the 3 major comments →

arxiv 2607.03614 v1 pith:5VQIKDM5 submitted 2026-07-03 cs.CV

IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning

Yuening Cai , Junwei Zhou , Youran Qu , Yu-Wing Tai This is my paper
classification cs.CV
keywords vision-language models3D layout inferencespatial intelligenceindoor scene understandingrender-and-compare evaluationholistic layout predictiongeometric accuracysynthetic indoor dataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that spatial question answering, the usual way to test whether vision-language models understand space, leaves a complementary skill untested: the ability to output a complete, structured 3D layout of every visible object from one image. The authors introduce IDEAL-Bench, built on a new set of 1,000 re-renderable photorealistic indoor scenes across ten room types, and require models to predict each object's category, center, size, and yaw. Predictions are scored both with numerical geometric metrics and by re-rendering the predicted layout and comparing it to the original image. Across 15 leading models the task is still largely unsolved: the strongest reaches only 62.1 out of 100. Recognition and formatting are strong, but position, scale, and rotation accuracy collapse. Model rankings also shift relative to QA and primitive-reconstruction benchmarks, especially in the mid-tier. The claim is that genuine spatial intelligence must include this measurable layout skill, not only fluent answers about who is next to what.

Core claim

Holistic single-image 3D layout inference on photorealistic indoor scenes remains substantially unsolved for current vision-language models. The best of 15 models scores only 62.1/100 overall; every model shows a sharp split between strong object recognition and weak geometric regression; and mid-tier rankings diverge from those on QA-based and primitive-reconstruction spatial benchmarks, so the new suite surfaces geometric failures that linguistic spatial tests miss.

What carries the argument

IDEAL-Bench: a dual evaluation protocol that forces a structured layout prediction (room type, room size, and per-object category, 3D center, dimensions, and yaw) and then scores it both by direct numerical comparison to ground truth along five dimensions and by asset-substituted re-rendering from the original camera for perceptual render-and-compare.

Load-bearing premise

That controlled synthetic single-room rectangular scenes with fixed lighting, a provided category list, mostly cardinal object orientations, and camera yaw snapped to the nearest cardinal direction isolate pure monocular spatial reasoning so that measured errors can be blamed on the model rather than on generation artifacts or alignment choices.

What would settle it

If a model family that currently scores near the top, when evaluated under the same fixed prompt and parsing rules, substantially closed the gap between recognition rates and position/IoU/rotation accuracy on the released 1,000-scene set (or on an expanded multi-view or real-captured extension), the claim that geometric regression is a universal bottleneck would be undermined.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Spatial intelligence claims for VLMs should be checked against holistic layout metrics, not only against relational QA accuracy.
  • Training and evaluation will need explicit supervision for continuous pose and extent, not only for category labels and language answers.
  • Mid-tier model rankings on layout tasks can reorder relative to QA and primitive-reconstruction leaderboards, so benchmark choice changes which systems look competitive.
  • Render-and-compare reconstruction becomes a practical, human-correlated way to audit whether a predicted layout actually matches a scene.
  • Future progress can be measured by whether the recognition-geometry asymmetry shrinks rather than by further gains on already-saturated parse and recognition rates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the recognition-geometry split is architectural rather than merely a data gap, token-based generation of continuous metric quantities may remain a ceiling until continuous spatial heads or hybrid 3D modules are added.
  • The same protocol could be turned into a training signal: re-render loss or layout-consistency rewards might push models past description toward measurement.
  • Grid rooms expose a distinct failure mode (regularity without world-frame anchoring) that pure per-object matching would have missed; similar structural metrics may be needed for other repetitive environments such as warehouses or parking lots.
  • Because the dataset ships with re-renderable Blender sources, the same scenes can later stress-test lighting, multi-view, or open-vocabulary variants without regenerating geometry.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes holistic single-image 3D layout inference as a complementary evaluation axis for VLM spatial competence and instantiates it as IDEAL-Bench on IDEAL-Scenes: 1,000 re-renderable photorealistic indoor Blender scenes across 10 room types with programmatically exact ground-truth layouts. Models must emit structured room type, room size, and per-object category/position/dimensions/yaw; predictions are scored with five numerical dimensions (scene validity, physical plausibility, geometric accuracy, object recognition, grid layout; 11 rate metrics) plus a render-and-compare Judge-VLM perceptual protocol validated against human MOS (ρ=0.79). Evaluating 15 VLMs, the authors report three main findings: the task remains substantially unsolved (best overall 62.1/100), all models show a sharp recognition–geometry asymmetry (strong D4, weak D3/D5), and mid-tier rankings partially diverge from QA-based and primitive-reconstruction spatial benchmarks while top-tier consensus holds.

Significance. If the empirical results hold, IDEAL-Bench is a useful diagnostic complement to dominant spatial-QA protocols: it makes global metric layout competence falsifiable via both numerical comparison and re-renderable asset substitution under shared camera/lighting/assets. Strengths that should be credited explicitly include the full 1,000-scene evaluation of 15 models, transparent metric definitions and applicability rules for grid vs non-grid rooms, bootstrap 95% CIs, failure analysis for parse-broken models, release of re-renderable Blender sources, and a human-correlated perceptual protocol. The recognition-versus-measurement asymmetry and the low PA/GCR ceilings are concrete, actionable signals for next-generation VLM training and evaluation.

major comments (3)
  1. Table 2 and Eq. (2): the headline Overall (best 62.1/100) is an unweighted mean of 11 rate metrics that includes near-saturated D1 (PSR/RTA) and strong D4 (RR/NHR) for most competitive models. Because D1 is essentially a schema/room-type gate and D4 is recognition under a provided category list, the scalar can overstate geometric competence relative to the paper’s central diagnostic claim. The separate Geo column already shows the harder signal (≈40 for the leader). The main text should either (i) lead with Geo (or a D2+D3+D5 composite) as the primary ranking scalar, or (ii) report Overall with and without D1/D4 and state explicitly how much of the 62.1 is carried by saturated non-geometric terms.
  2. Sec. 3 and Sec. 5.1 (Vertical Position Convention): models are required to predict full 3D centers, but for floor-standing objects predicted z is overwritten by GT in both numerical D3 and reconstruction. This is a reasonable control against dimension–center coupling, yet it means the evaluated task is not fully holistic 3D pose regression for the majority of objects. The abstract/intro claim of predicting “every visible object’s pose and extent” should be qualified in the main evaluation narrative (not only Limitations E.1), and a short ablation or diagnostic of raw predicted-z error (even if not used in Overall) would show whether the bottleneck is purely horizontal layout or also vertical measurement.
  3. Sec. 6.4 / Appendix D.5: cross-bench rank comparisons rest on very small intersections (n=3–4) and are labeled suggestive, but the abstract still states that “model rankings partially diverge” as one of three headline findings. With such n, ordinal shifts are fragile. Either restrict the abstract claim to the better-supported recognition–geometry asymmetry and unsolved ceiling, or add a clearer quantitative statement (e.g., which specific mid-tier swaps survive leave-one-out / bootstrap over the shared model set) so the divergence claim is not over-weighted relative to its evidence.
minor comments (6)
  1. Fig. 1 and the abstract both state the best score as 62.1; ensure the same primary scalar is used consistently in the teaser radar, Table 2, and abstract so readers do not mix Overall with Geo or perceptual means.
  2. Sec. 5.1 Object Matching: Hungarian matching without a distance cutoff can assign far-off predictions and inflate PE/PA; a brief sensitivity check with a cutoff (or reporting unmatched-by-distance rates) would help readers interpret low PA.
  3. Appendix B.3 thresholds (PA@0.3 m, IoU@0.15, RA@30°, NOR IoU 0.03, GSR τ_cl/τ_hit) are free parameters. A short sensitivity table in the appendix would strengthen the claim that the recognition–geometry gap is not an artifact of a single operating point.
  4. Table 4 vs Table 2: GPT-4o ranks high numerically but drops perceptually, while GPT-5.4 rises; a short qualitative discussion of which failure modes (global coherence vs local slot plausibility) drive this split would improve interpretability of the dual protocol.
  5. Typos/clarity: “s.” appears as a stray fragment after the GPT-5.4 perceptual discussion in Sec. 6.3; “dimen-sion” line break in Sec. 5.2; standardize model name spelling (Gemini-2.5-pro vs Gemini 2.5 Pro) across tables.
  6. Related Work / Table 1: IR3D-Bench is the closest prior; a one-paragraph side-by-side of what changes when moving from fixed-camera primitives to multi-viewpoint indoor assets would help non-specialist readers place the contribution.

Circularity Check

0 steps flagged

No significant circularity: IDEAL-Bench is an empirical evaluation suite whose scores are measured against independent ground truth, not derived by construction from fitted inputs or self-citation chains.

full rationale

The paper’s load-bearing claims are empirical measurements of 15 VLMs on a new structured 3D layout task, not first-principles derivations. Overall scores are unweighted means of pre-specified rate metrics (PSR, RTA, NOR, IBR, PA, Prec., RA, RR, NHR, GCR, GSR) computed against programmatically extracted GT layouts from re-renderable Blender scenes (Eq. 2; Sec. 5.2; Appendix B.3–B.4). Predictions are model outputs under a fixed prompt; they are not fitted parameters renamed as predictions. The recognition–geometry asymmetry (strong D4 vs. weak D3/D5, PA@0.3m ceiling ≤12.8% with bootstrap CIs) and the 62.1/100 ceiling are falsifiable against held-out GT and a separate perceptual render-and-compare protocol validated by human MOS (ρ=0.79). Design choices that soften evaluation (category list in prompt, cardinal yaw prior, z fixed to GT for floor-standing objects at scoring, T_align snap) are explicit scope/limitations (Sec. 5.1, E.1; Appendix D.2.2), not circular reductions of the target claim to its inputs. Self-citations (e.g., Zhou & Tai layout/generation works) appear only as related-work context and do not justify the central empirical results. Cross-bench rank shifts are labeled suggestive given small intersections. No step reduces a claimed prediction or uniqueness result to a fitted constant or an unverified self-citation by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

As a benchmark paper the load-bearing content is design choices and empirical measurements, not a physical derivation. Free parameters are the metric thresholds and filtering constants that define pass rates. Axioms are standard CV/graphics conventions plus domain assumptions that synthetic controlled rooms and structured JSON output measure 'spatial intelligence.' Invented entities are the benchmark constructs themselves; they have independent handles via released scenes and re-rendering.

free parameters (6)
  • PA position threshold = 0.3 m
    Pass rate uses τ_PE = 0.3 m on XY error; chosen as coarser than typical furniture half-extents and directly affects D3 and Overall.
  • Precision IoU threshold = 0.15
    Prec. pass rate uses τ_IoU = 0.15; permissive calibration against furniture aspect ratios that strongly influences geometric scores.
  • Rotation accuracy threshold = 30°
    RA uses τ_RE = 30° after category-aware symmetry folding.
  • Collision IoU for NOR = 0.03
    Pairwise collision flag at IoU3D ≥ 0.03 for Non-Overlap Rate.
  • In-view pixel / aspect / truncation filters = 300 px; ratio 12; 12.5% border / 3% area rules
    300-pixel mask threshold, aspect ratio >12 rejection, and truncation rules define which objects enter evaluation and thus all object-level rates.
  • GSR clustering and hit tolerances = 0.3 m / 0.1 m
    1-D linkage τ_cl = 0.3 m and hit τ_hit = 0.1 m define Grid Self-consistency Rate.
axioms (5)
  • domain assumption Manhattan-world rooms with mostly cardinal object yaws are an acceptable default for testing VLM layout inference, with only a minority of oblique placements as probes.
    Stated in task formulation and system prompt; discretizes rotation and simplifies alignment.
  • domain assumption Providing the candidate category list and room-type set isolates geometric regression from open-vocabulary detection, so remaining errors reflect spatial measurement.
    Explicit design choice in Sec. 3 and Limitations; shapes the interpretation of 'describing vs measuring'.
  • ad hoc to paper Snapping camera yaw to the nearest cardinal and fixing floor-object z to GT still yields a valid comparison of layout competence.
    Coordinate alignment and vertical convention in Sec. 5.1; authors argue non-uniform metric degradation but treat residual mismatch as minor.
  • ad hoc to paper Unweighted mean of the 11 applicable rate metrics is a fair Overall scalar for ranking models.
    Eq. 2 and Appendix B.4; alternative weightings could reorder mid-tier models.
  • standard math Standard graphics and matching tools (Hungarian assignment, AABB IoU, Blender Boolean) correctly operationalize physical plausibility and geometric accuracy.
    Used throughout D2–D5 definitions without novel mathematical claims.
invented entities (3)
  • IDEAL-Scenes dataset independent evidence
    purpose: 1,000 re-renderable Blender indoor scenes with programmatic GT layouts across 10 room types.
    Core resource enabling dual numerical and perceptual evaluation; independent handle via released bundles and generation pipeline.
  • IDEAL-Bench five-dimension metric suite (D1–D5) including GCR/GSR independent evidence
    purpose: Decompose layout competence into validity, plausibility, geometry, recognition, and grid structure.
    New scoring axes defined for this protocol; falsifiable by re-running on released scenes.
  • Judge-VLM perceptual render-and-compare protocol independent evidence
    purpose: Scene-level spatial similarity scores and rankings from re-rendered predictions vs GT images.
    Complements object-level metrics; validated with human MOS correlation on a 200-scene subset.

pith-pipeline@v1.1.0-grok45 · 43366 in / 3662 out tokens · 41254 ms · 2026-07-12T01:11:09.317117+00:00 · methodology

0 comments
read the original abstract

Spatial question answering is the dominant paradigm for evaluating spatial intelligence in Vision-Language Models (VLMs), but it leaves a complementary axis of spatial competence under-evaluated: holistic 3D layout inference, which predicts every visible object's pose and extent from a single image in a structured form. To this end, we introduce IDEAL-Bench, an evaluation suite that requires VLMs to predict structured 3D layouts on photorealistic indoor scenes across 10 room types, scored along five numerical dimensions and a perceptual render-and-compare protocol. By operating on semantically realistic scenes with full asset substitution under controlled lighting and viewpoint, IDEAL-Bench moves beyond CLEVR-style simple geometric primitives so that any image-space discrepancy reflects spatial reasoning alone. The benchmark is built on IDEAL-Scenes, a procedurally generated dataset of 1,000 re-renderable Blender environments with ground-truth layouts. Evaluating 15 prominent VLMs reveals three findings: the task remains substantially unsolved, with the strongest model reaching only 62.1/100 overall; all models exhibit a sharp asymmetry between object recognition and geometric regression, indicating that current VLMs are trained to describe scenes rather than to measure them; model rankings partially diverge from those on QA-based and primitive-reconstruction benchmarks: top-tier consensus holds, but mid-tier rankings shift substantially. Collectively, these findings establish IDEAL-Bench as a diagnostic suite, targeting the geometric and structural competencies that QA-based evaluation cannot surface, and paving the way towards more rigorous evaluation of spatial intelligence in next-generation VLMs. Together, these findings position IDEAL-Bench as a principled diagnostic for whether future VLMs achieve genuine spatial understanding rather than linguistic approximations of it.

Figures

Figures reproduced from arXiv: 2607.03614 by Junwei Zhou, Youran Qu, Yuening Cai, Yu-Wing Tai.

Figure 1
Figure 1. Figure 1: Towards better evaluation of spatial intelligence in VLMs. While existing 3D QA benchmarks probe relational understanding, IDEAL-Bench requires models to estimate every visible object’s 3D pose and extent, evaluated via numerical metrics and a render-and-compare protocol. The task demands genuine spatial reasoning and remains substantially unsolved, with the strongest model reaching only 62.1/100 overall. … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of IDEAL-Bench. Four stages. (1) Dataset synthesis: we modified InfiniGen to produce preprocessed, self-contained scenes across 10 room types. (2) Model inference: a VLM predicts a structured layout from an image, a system prompt and a user prompt containing a room type, and a category list. (3) Parse & reconstruct: predicted pose and position are applied to GT assets and re-rendered from the orig… view at source ↗
Figure 3
Figure 3. Figure 3: Overall score by room type. The grid illusion: high scores, weak anchor￾ing. We partition the 1,000 scenes into five density bins by in-view object count (Tab. 3), and reveal that complexity does not degrade the overall score monotonically: most models drop from Bin 1 to Bin 5, but GPT-4o and Claude Sonnet 4.6 improve on dense scenes. The same pattern shows up by room type ( [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figure 4
Figure 4. Figure 4: Scene reconstruction visualizations for four representative models (Gemini 2.5 Pro, Claude [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Image quality distribution across all scenes. Object counts. The mean number of furniture in￾stances per scene is 15.0 in the full scene graph and 11.3 after in-view filtering. Counts vary substantially across room types (Figure 6a): classrooms have the highest mean in-view count (29.3 ± 9.2), bathrooms the fewest (3.4 ± 1.2). The gap between full-graph and in-view counts is largest for homestudio and libr… view at source ↗
Figure 6
Figure 6. Figure 6: IDEAL-Scenes statistics. (a) Total and in-view object counts per room type (bar = mean, whiskers = min–max, tick = median); classrooms are the densest in-view (29.3 avg.) and bathrooms the sparsest (3.4), with the largest total–in-view gap in homestudio and library, where peripheral objects fall outside the camera view. (b) Floor area per room type, log scale, from ∼6 m2 (bathroom) to a median of 162 m2 fo… view at source ↗
Figure 7
Figure 7. Figure 7: Grid layout scenes perceptual reconstruction. [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: D3/D5 metrics across camera viewpoints, per model. Each line shows a model’s absolute score (PA, Prec., RA, GCR, or GSR; %) as a function of d, the angular distance between camera yaw and the nearest cardinal direction. Proprietary models are individually colored; the 9 open-source models are shown in light blue (several overlap near the ceiling in (e) GSR) [PITH_FULL_IMAGE:figures/full_fig_p033_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative examples of Judge-VLM perceptual scoring across three room categories. Each [PITH_FULL_IMAGE:figures/full_fig_p034_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

85 extracted references · 11 linked inside Pith

  1. [1]

    How to enable llm with 3d capacity? a survey of spatial reasoning in llm, 2025

    Jirong Zha, Yuxuan Fan, Xiao Yang, Chen Gao, and Xinlei Chen. How to enable llm with 3d capacity? a survey of spatial reasoning in llm, 2025

    2025

  2. [2]

    Spatialreasoner: Towards explicit and generalizable 3d spatial reasoning, 2025

    Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso de Melo, Jianwen Xie, and Alan Yuille. Spatialreasoner: Towards explicit and generalizable 3d spatial reasoning, 2025

    2025

  3. [3]

    Do 3d large language models really understand 3d spatial relationships?, 2026

    Xianzheng Ma, Tao Sun, Shuai Chen, Yash Bhalgat, Jindong Gu, Angel X Chang, Iro Armeni, Iro Laina, Songyou Peng, and Victor Adrian Prisacariu. Do 3d large language models really understand 3d spatial relationships?, 2026

    2026

  4. [4]

    Ir3d-bench: Evaluating vision-language model scene understanding as agentic inverse rendering.arXiv preprint arXiv:2506.23329, 2025

    Parker Liu, Chenxin Li, Zhengxin Li, Yipeng Wu, Wuyang Li, Zhiqin Yang, Zhenyuan Zhang, Yunlong Lin, Sirui Han, and Brandon Y Feng. Ir3d-bench: Evaluating vision-language model scene understanding as agentic inverse rendering.arXiv preprint arXiv:2506.23329, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces, 2025

    2025

  6. [6]

    From flatland to space: Teaching vision-language models to perceive and reason in 3d, 2026

    Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei, Junhui Chen, Yu-Jie Yuan, Xinyue Cai, Guowei Huang, Xingyue Quan, Hang Xu, and Li Zhang. From flatland to space: Teaching vision-language models to perceive and reason in 3d, 2026

    2026

  7. [7]

    Spatialrgpt: Grounded spatial reasoning in vision language models, 2024

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models, 2024

    2024

  8. [8]

    Omnispatial: Towards comprehensive spatial reasoning benchmark for vision language models

    Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. Omnispatial: Towards comprehensive spatial reasoning benchmark for vision language models. In International Conference on Learning Representations (ICLR), 2026

    2026

  9. [9]

    Gemini: A family of highly capable multimodal models, 2025

    Gemini Team. Gemini: A family of highly capable multimodal models, 2025

    2025

  10. [10]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025

    2025

  11. [11]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Openai gpt-5 system card, 2026

    Aaditya Singh, Adam Fry, Adam Perelman, et al. Openai gpt-5 system card, 2026

    2026

  14. [14]

    Claude 4.6 Sonnet System Card, 2026

    Anthropic. Claude 4.6 Sonnet System Card, 2026. Accessed: 2026-06-29

    2026

  15. [15]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Qwen2.5 technical report, 2025

    Qwen Team. Qwen2.5 technical report, 2025

    2025

  17. [17]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Qwen Team. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable rein- forcement learning, 2026

    V Team. Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable rein- forcement learning, 2026

    2026

  19. [19]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024

    2024

  20. [20]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023

    2023

  21. [21]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

    2025

  22. [22]

    Navitrace: Evaluating embodied navigation of vision-language models

    Tim Windecker, Manthan Patel, Moritz Reuss, Richard Schwarzkopf, Cesar Cadena, Rudolf Lioutikov, Marco Hutter, and Jonas Frey. Navitrace: Evaluating embodied navigation of vision-language models. arXiv preprint arXiv:2510.26909, 2025. 12

    arXiv 2025

  23. [23]

    Spatial Intelligence in Vision-Language Models: A Comprehensive Survey.TechRxiv preprint, 2025

    Disheng Liu, Tuo Liang, Zhe Hu, Jierui Peng, Yiren Lu, Yi Xu, Yun Fu, and Yu Yin. Spatial Intelligence in Vision-Language Models: A Comprehensive Survey.TechRxiv preprint, 2025

    2025

  24. [24]

    Advances in 3d generation: A survey.arXiv preprint arXiv:2401.17807, 2024

    Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, and Ying Shan. Advances in 3d generation: A survey.arXiv preprint arXiv:2401.17807, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    3d scene generation: A survey

    Beichen Wen, Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu. 3d scene generation: A survey. arXiv preprint arXiv:2505.05474, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    A comprehensive survey on 3d content generation, 2024

    Jian Liu, Xiaoshui Huang, Tianyu Huang, Lu Chen, Yuenan Hou, Shixiang Tang, Ziwei Liu, Wanli Ouyang, Wangmeng Zuo, Junjun Jiang, and Xianming Liu. A comprehensive survey on 3d content generation, 2024

    2024

  27. [27]

    Gena3d: Generative amodal 3d modeling by bridging 2d priors and 3d coherence, 2026

    Junwei Zhou and Yu-Wing Tai. Gena3d: Generative amodal 3d modeling by bridging 2d priors and 3d coherence, 2026

    2026

  28. [28]

    Layout-your-3d: Controllable and precise 3d generation with 2d blueprint

    Junwei Zhou, Xueting Li, Lu Qi, and Ming-Hsuan Yang. Layout-your-3d: Controllable and precise 3d generation with 2d blueprint. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  29. [29]

    Sam 3d: 3dfy anything in images.arXiv preprint arXiv:2511.16624, 2025

    Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J Liang, Alexander Sax, Hao Tang, Weiyao Wang, Michelle Guo, Thibaut Hardin, Xiang Li, et al. Sam 3d: 3dfy anything in images.arXiv preprint arXiv:2511.16624, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Physgaussian: Physics-integrated 3d gaussians for generative dynamics

    Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4389–4398, 2024

    2024

  31. [31]

    Pat3d: Physics-augmented text-to-3d scene generation.arXiv preprint arXiv:2511.21978, 2025

    Guying Lin, Kemeng Huang, Michael Liu, Ruihan Gao, Hanke Chen, Lyuhao Chen, Beijia Lu, Taku Komura, Yuan Liu, Jun-Yan Zhu, et al. Pat3d: Physics-augmented text-to-3d scene generation.arXiv preprint arXiv:2511.21978, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Coco4d: Comprehensive and complex 4d scene generation, 2025

    Junwei Zhou, Xueting Li, Lu Qi, and Ming-Hsuan Yang. Coco4d: Comprehensive and complex 4d scene generation, 2025

    2025

  33. [33]

    Tabletopgen: Instance-level interactive 3d tabletop scene generation from text or single image

    Ziqian Wang, Yonghao He, Licheng Yang, Wei Zou, Hongxuan Ma, Liu Liu, Wei Sui, Yuxin Guo, and Hu Su. Tabletopgen: Instance-level interactive 3d tabletop scene generation from text or single image. arXiv preprint arXiv:2512.01204, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Scenethesis: A language and vision agentic framework for 3d scene generation, 2025

    Lu Ling, Chen-Hsuan Lin, Tsung-Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng, Yunhao Ge, Ming-Yu Liu, Aniket Bera, and Zhaoshuo Li. Scenethesis: A language and vision agentic framework for 3d scene generation, 2025

    2025

  35. [35]

    Perceive-then-plan: Layout-as-policy for monocular 3d scene layout estimation, 2026

    Junwei Zhou and Yu-Wing Tai. Perceive-then-plan: Layout-as-policy for monocular 3d scene layout estimation, 2026

    2026

  36. [36]

    Artiscene: Language-driven artistic 3d scene generation through image intermediary

    Zeqi Gu, Yin Cui, Zhaoshuo Li, Fangyin Wei, Yunhao Ge, Jinwei Gu, Ming-Yu Liu, Abe Davis, and Yifan Ding. Artiscene: Language-driven artistic 3d scene generation through image intermediary. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2891–2901, June 2025

    2025

  37. [37]

    Layoutgpt: Compositional visual planning and generation with large language models.Advances in Neural Information Processing Systems, 36:18225–18250, 2023

    Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models.Advances in Neural Information Processing Systems, 36:18225–18250, 2023

    2023

  38. [38]

    Holodeck: Language guided generation of 3d embodied ai environments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, Chris Callison-Burch, Mark Yatskar, Aniruddha Kembhavi, and Christopher Clark. Holodeck: Language guided generation of 3d embodied ai environments. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  39. [39]

    Worldcraft: Photo-realistic 3d world creation and customization via llm agents.arXiv preprint arXiv:2502.15601, 2025

    Xinhang Liu, Chi-Keung Tang, and Yu-Wing Tai. Worldcraft: Photo-realistic 3d world creation and customization via llm agents.arXiv preprint arXiv:2502.15601, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities, 2024

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities, 2024. 13

    2024

  41. [41]

    Spatialbench: Benchmarking multimodal large language models for spatial cognition, 2026

    Peiran Xu, Sudong Wang, Yao Zhu, Jianing Li, Gege Qi, and Yunjian Zhang. Spatialbench: Benchmarking multimodal large language models for spatial cognition, 2026

    2026

  42. [42]

    Space-10: A comprehensive benchmark for multimodal large language models in compositional spatial intelligence, 2025

    Ziyang Gong, Wenhao Li, Oliver Ma, Songyuan Li, Zhaokai Wang, Songyuan Li, Jiayi Ji, Xue Yang, Gen Luo, Junchi Yan, and Rongrong Ji. Space-10: A comprehensive benchmark for multimodal large language models in compositional spatial intelligence, 2025

    2025

  43. [43]

    Chang, and Matthias Nießner

    Dave Zhenyu Chen, Angel X. Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. InEuropean Conference on Computer Vision (ECCV), 2020

    2020

  44. [44]

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas J. Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. InEuropean Conference on Computer Vision (ECCV), 2020

    2020

  45. [45]

    Space3D- Bench: Spatial 3D Question Answering Benchmark

    Emilia Szymanska, Mihai Dusmanu, Jan-Willem Buurlage, Mahdi Rad, and Marc Pollefeys. Space3D- Bench: Spatial 3D Question Answering Benchmark. InEuropean Conference on Computer Vision (ECCV) Workshops, 2024

    2024

  46. [46]

    Do vision-language models represent space and how? evaluating spatial frame of reference under ambiguities, 2025

    Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, and Ziqiao Ma. Do vision-language models represent space and how? evaluating spatial frame of reference under ambiguities, 2025

    2025

  47. [47]

    Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes, 2017

    2017

  48. [48]

    Matterport3d: Learning from rgb-d data in indoor environments, 2017

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments, 2017

    2017

  49. [49]

    Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data, 2022

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data, 2022

    2022

  50. [50]

    SUN3D: A database of big spaces reconstructed using SfM and object labels

    Jianxiong Xiao, Andrew Owens, and Antonio Torralba. SUN3D: A database of big spaces reconstructed using SfM and object labels. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1625–1632, 2013

    2013

  51. [51]

    Lichtenberg, and Jianxiong Xiao

    Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. SUN RGB-D: A RGB-D scene understanding benchmark suite. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 567–576, 2015

    2015

  52. [52]

    SceneNN: A scene meshes dataset with aNNotations

    Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung. SceneNN: A scene meshes dataset with aNNotations. InProceedings of the International Conference on 3D Vision (3DV), pages 92–101, 2016

    2016

  53. [53]

    John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J. Davison. Scenenet rgb-d: 5m photorealistic images of synthetic indoor trajectories with ground truth, 2017

    2017

  54. [54]

    Interiornet: Mega-scale multi-sensor photo-realistic indoor scenes dataset, 2018

    Wenbin Li, Sajad Saeedi, John McCormac, Ronald Clark, Dimos Tzoumanikas, Qing Ye, Yuzhong Huang, Rui Tang, and Stefan Leutenegger. Interiornet: Mega-scale multi-sensor photo-realistic indoor scenes dataset, 2018

    2018

  55. [55]

    Structured3d: A large photo-realistic dataset for structured 3d modeling, 2020

    Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling, 2020

    2020

  56. [56]

    Susskind

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding, 2021

    2021

  57. [57]

    3d-front: 3d furnished rooms with layouts and semantics, 2021

    Huan Fu, Bowen Cai, Lin Gao, Lingxiao Zhang, Jiaming Wang Cao Li, Zengqi Xun, Chengyue Sun, Rongfei Jia, Binqiang Zhao, and Hao Zhang. 3d-front: 3d furnished rooms with layouts and semantics, 2021

    2021

  58. [58]

    Scenescript: Reconstructing scenes with an autoregressive structured language model, 2024

    Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, Jakob Engel, Edward Miller, Richard Newcombe, and Vasileios Balntas. Scenescript: Reconstructing scenes with an autoregressive structured language model, 2024. 14

    2024

  59. [59]

    M3dlayout: A multi-source dataset of 3d indoor layouts and structured descriptions for 3d generation, 2026

    Yiheng Zhang, Zhuojiang Cai, Mingdao Wang, Meitong Guo, Tianxiao Li, Li Lin, and Yuwang Wang. M3dlayout: A multi-source dataset of 3d indoor layouts and structured descriptions for 3d generation, 2026

    2026

  60. [60]

    Infinigen indoors: Photorealistic indoor scenes using procedural generation, 2024

    Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, Zeyu Ma, and Jia Deng. Infinigen indoors: Photorealistic indoor scenes using procedural generation, 2024

    2024

  61. [61]

    What makes for good visual tokenizers for large language models?, 2023

    Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan. What makes for good visual tokenizers for large language models?, 2023. 15 A Dataset Details A.1 Scene Generation IDEAL-Scenes is built on InfiniGen Indoors [ 60], a Blender-based procedural scene-generation engine that combines a procedural asset library with a constraint-based arr...

    2023

  62. [62]

    Category gating.Categories present on only one side are routed directly to the unmatched pool: GT-only categories contribute missed detections (RR, Appendix B.3), prediction-only categories contribute hallucinations (NHR, Appendix B.3)

  63. [63]

    When|O c| ̸=|P c|, the surplus side is routed to the same unmatched pool as stage 1

    Per-category assignment.For each category present on both sides, we build the cost matrix C(c) ij =∥p (g) i −p (p) j ∥2 from world-frame XY centers and solve the Hungarian assignment. When|O c| ̸=|P c|, the surplus side is routed to the same unmatched pool as stage 1. Grid scenes.Classroom, library, and open-office scenes contain dominant categories (stud...

  64. [64]

    Round each xi to two decimals, deduplicate, and sort to obtain column coordinates col_xs (Cvalues); analogously fory i to obtain row coordinatesrow_ys(Rvalues)

  65. [65]

    A singleton axis (R= 1orC= 1) borrows the other axis’s pitch

    Estimate the row pitch as the median of consecutive differences in row_ys, and the column pitch analogously. A singleton axis (R= 1orC= 1) borrows the other axis’s pitch

  66. [66]

    This procedure assumes axis-aligned arrangements parallel to the world X/Y axes, which the dataset generator guarantees for the supported room types

    Form the R×C lattice; each cell is the axis-aligned rectangle with width equal to the column pitch and height equal to the row pitch, centerd on its lattice point and abutting its neighbours with no gap. This procedure assumes axis-aligned arrangements parallel to the world X/Y axes, which the dataset generator guarantees for the supported room types. A c...

  67. [67]

    2.Frame transform.Apply Eq

    In-view filter.Intersect GT.json object ids with objects_in_view.json to obtain Oiv, as used by the numerical metrics. 2.Frame transform.Apply Eq. 8 to bring the prediction intoW. 3.Match.Run the matcher of § B.2 to obtainM

  68. [68]

    Read original asset state.For each matched GT object, record its obj.location (oorig) andobj.dimensions((W o, Do, Ho)) viabpy

  69. [69]

    Inverse pose.Solve for the Blender pose realising the predicted center, orientation, and dimensions (below)

  70. [70]

    Compose render JSON.Per object: (a) an overridden pose for matched GT, (b) a hide flag for unmatched in-view GT, or (c) a hallucination marker (solid red sphere) for unmatched predictions

  71. [71]

    image_id

    Re-render.Apply the overrides and re-render from the original camera pose, yielding Ipred. Inverse pose.The Blender state per-object comprises obj.location (root-mesh origin, gener- allynotthe geometric center), obj.rotation_euler.z (θb) and obj.dimensions in local axes. Given the prediction’s world-frame center ppred, canonical (Wp, Dp, Hp), and yaw θ(p)...

  72. [72]

    Use the image to decide what is actually visible

  73. [73]

    The assets_list does not guarantee that every listed item appears in the scene, nor that each item appears only once

  74. [74]

    Do not output categories outside assets_list, and do not invent instances you do not see

    Only include objects you see. Do not output categories outside assets_list, and do not invent instances you do not see

  75. [75]

    A scene may contain multiple instances of the same category; distinguish them with different id values (e.g

    Output one instance per visible object. A scene may contain multiple instances of the same category; distinguish them with different id values (e.g. floor_lamp_01, floor_lamp_02)

  76. [76]

    When multiple instances of the same category are arranged in a regular pattern (e.g., rows or a grid), reflect this regularity in their positions, and account for perspective distortion. ### Special Assets and Edge Cases If you see a piece of furniture comprising multiple components that serve a single function, please treat them as a single integrated ob...

  77. [77]

    Assume the room is a perfectly orthogonal 3D box

    Orthogonal grid. Assume the room is a perfectly orthogonal 3D box. All walls meet at 90 degrees angles

  78. [78]

    Most objects in the scene are aligned with this grid

    Discrete rotation. Most objects in the scene are aligned with this grid. Unless there is overwhelming visual evidence of diagonal placement, every object's rotation_yaw must be exactly one of [0, 90, 180, 270]. Snap all orientations to the nearest 90 degree axis. ## Axes Definition - Origin (0, 0, 0): Geometric center of the room floor; Z = 0 is ground le...

  79. [79]

    No explanation, no markdown fences

    Output valid JSON only. No explanation, no markdown fences

  80. [80]

    scene": {

    You should follow strictly the json schema. Here is an example to describe a bedroom with one floor lamp: { "scene": { "room": { "type": "bedroom", "size": [W, D, H], "asset_count": 1 }, "objects": [ { "id": "floor_lamp_01", "category": "floor_lamp", "position": [x, y, z], "dimensions": [W, D, H], "rotation_yaw": null } ] } }

Showing first 80 references.