Pith. sign in

REVIEW 4 major objections 5 minor 84 references

Video Perception Models for 3D Scene Synthesis

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VIPScene shows that a first-person video generated from a text or image prompt can be reconstructed, decomposed into objects, and reassembled into a coherent 3D scene, and that evaluating scenes from first-person views tracks human…

desk verdict A novel pipeline and evaluation protocol worth knowing about, but the paper never isolates the video prior that its headline claim rests on. read the letter →

arxiv 2506.20601 v1 pith:CR6AQ4NZ submitted 2025-06-25 cs.CV

Rui Huang , Guangyao Zhai , Zuria Bauer , Marc Pollefeys , Federico Tombari , Leonidas Guibas , Gao Huang , Francis Engelmann This is my paper
classification cs.CV
keywords 3Dscenesynthesisvideogenerationpriorsfirst-personviewreconstructionopen-vocabularydetectionassetretrievallayoutoptimizationMLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that video generation models carry stronger spatial commonsense about real rooms than text-only LLMs or single-image generators, and that their first-person videos can serve as blueprints for full 3D scenes. VIPScene turns a prompt into a walkthrough video, reconstructs that video into a metric 3D point cloud, detects and tracks objects across frames, swaps in clean 3D assets, and optimizes poses so objects do not collide. Evaluated on living rooms, bedrooms, kitchens, and bathrooms, the resulting scenes beat the baselines on every reported metric and win a user study. The paper also proposes FPVScore, which feeds 360-degree first-person panoramas to a multimodal LLM; it agrees with human raters better than top-down CLIPScore, BLIPScore, VQAScore, or GPT-4o ranking.

What carries the argument

The generative backbone is a conditional video diffusion model that outputs a first-person scene walkthrough; this single artifact supplies viewpoint coverage and layout priors. The paper then uses a feedforward multi-view reconstructor to fuse frames into a metric point cloud, open-vocabulary 2D segmentation with cross-frame 3D tracking to isolate objects, adaptive erosion to denoise object masks, PCA- and ICP-based asset retrieval with a rigid-alignment objective, and a gradient-based refinement of bounding-box positions. For evaluation, the key instrument is FPVScore: a virtual camera rotating at 30-degree steps yields twelve concatenated first-person frames, and these stacks for all methods are ranked jointly by a multimodal LLM under a structured prompt with criteria for semantic correctness, layout correctness, and overall preference.

What would settle it

Generate a first-person video of a room, reconstruct the scene, and render a second camera path that loops through the reconstructed 3D geometry from a different trajectory; if the reconstructed layout shows walls, ceilings, or objects that contradict the video's parallax across those views, or if swapping the video condition to a different room type leaves the same object layout, the central claim would collapse.

Watch

Extended reading notes

Core claim

Video-grounded priors yield 3D scene layouts that are both semantically faithful and spatially coherent, because the video model jointly reasons about viewpoint, object placement, and room structure. The same first-person perspective that makes generation possible also makes evaluation reliable: FPVScore aligns with human ratings (Kendall's tau = 0.39) much better than top-down metrics (tau at or below 0.27), so the paper concludes that first-person MLLM-based scoring should replace top-down image-based metrics for judging 3D scene synthesis.

Load-bearing premise

The load-bearing premise is that a web-trained video generation model produces a first-person walkthrough whose layout is a faithful, physically plausible 3D scene, and that feedforward reconstruction plus 2D tracking recovers that layout accurately enough for object retrieval and collision-free assembly.

Editorial extensions

If this is right

  • Layouts inherit the video model's commonsense, so scene generation no longer depends on an LLM's explicit spatial reasoning.
  • A single generated video serves as both content and supervision: the reconstruction step yields editable object-level scenes without manual annotation.
  • Evaluation now has a protocol whose scores track human ratings better than top-down metrics, enabling cheaper and more scalable quality comparisons.
  • The pipeline is prompt- and input-agnostic: image-conditioned and multimodal prompts produce coherent scenes beyond the input field of view.
  • Replacing retrieved assets with generated or PBR assets is a compatible extension, as the paper notes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because FPVScore uses the same first-person viewing condition that the pipeline generates from, the metric may partly reward the video prior itself rather than the final assembled scene; a test comparing two scenes with identical first-person video but different hidden geometry would isolate this.
  • Beyond the paper: the framework suggests a general recipe, so any conditional video model with robust 3D-consistent priors could replace the specific generative backbone used here for layout tasks in robotics and embodied AI.
  • Beyond the paper: if video priors are indeed 3D-consistent, one could use the same walkthrough video to bootstrap per-object 6-DoF poses for asset insertion without ICP, reducing the retrieval stage to a nearest-neighbour search in feature space.
  • Beyond the paper: the adaptive erosion and tracking stages assume the generated video depicts a static occupied room; applying VIPScene to videos with moving agents or camera roll would require a dynamic-object detection stage the paper does not include.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VIPScene, a modular pipeline for 3D scene synthesis from text or image prompts. The pipeline generates a first-person video with a video diffusion model (Cosmos), reconstructs scene geometry with Fast3R, detects and tracks objects with Grounded-SAM and MASt3R, retrieves 3D assets from Objaverse, and refines object poses with a collision-avoidance optimization. The paper also introduces FPVScore, an automatic evaluation metric that renders 360-degree first-person views of a scene and uses an MLLM (GPT-4o or Gemini) to rank scenes against each other. Experiments compare VIPScene against Holodeck and Architect, reporting user-study scores, top-down metrics, and FPVScore. The paper claims that video-grounded priors yield more coherent layouts and that FPVScore aligns better with human judgment than top-down metrics (Kendall's tau = 0.39).

Significance. If the central claim holds, the idea of using video generation models as commonsense 3D layout priors is a meaningful step beyond LLM- and single-image-based scene synthesis, and the first-person evaluation protocol is a practical contribution to a field that currently relies on poorly aligned top-down metrics. The paper is also commendable for including a user study, ablations of core pipeline components, and a plan to release code. However, the evidence as presented does not isolate the video prior from the other strong components in the pipeline, and the quantitative support is statistically thin. The contribution is therefore promising but not yet established.

major comments (4)
  1. [Sec. 3.1, Sec. 5.4] The central claim that video generation provides commonsense layout priors is not directly tested. The ablations in Fig. 6 remove noise reduction, remove object pose refinement, and replace the 2D detector with Mask3D, but none of these experiments replaces Cosmos with a non-video prior, shuffles the video frames, or otherwise breaks temporal/3D consistency. As a result, the reported gains over Holodeck and Architect could plausibly come from the asset-retrieval and pose-refinement stages rather than from video-grounded layout knowledge. An additional control that isolates the video prior is load-bearing for the paper's main message.
  2. [Sec. 3.1] The pipeline assumes that the generated video provides 'diverse views of the scene' and that Fast3R plus MASt3R-based tracking reconstructs and associates object geometry accurately enough for downstream retrieval and assembly. No experiment measures reconstruction fidelity against the source video, a known scene, or a ground-truth layout. Without such a check, it is unclear whether layout coherence survives the video-to-point-cloud-to-asset chain or is instead introduced by the optimization in Sec. 3.2. A simple quantitative test, such as reprojection error of detected objects onto the video frames or a comparison of object bounding boxes before and after asset replacement, would strengthen the claim.
  3. [Sec. 5.1, Sec. 5.3, Table 1, Table 2] The quantitative evidence is not statistically supported. Table 1 reports no error bars, confidence intervals, or significance tests, and the user study uses only thirty participants with no reported inter-rater agreement. In Table 2, the headline FPVScore correlation is Kendall's tau = 0.39, which is modest, and no confidence interval or significance test is given, so it is unclear whether 0.39 is reliably better than the top-down GPT-4o value of 0.27. The abstract's phrase 'significantly outperforms' is therefore stronger than the evidence justifies.
  4. [Sec. 5.3, Appendix D] The validation of FPVScore may be confounded by the viewing condition shared with the human study. Participants in the user study were shown a 360-degree video from the center of each scene plus a top-down image, while FPVScore presents MLLMs with first-person frames from a 360-degree rotation. It is not clear whether the metric agrees with humans because it captures scene quality or because both the metric and the human raters are influenced by the same first-person stimulus. The paper should clarify whether the human judgments used for Table 2 are independent of the FPVScore inputs, or should report a control using human ratings collected from a different rendering condition.
minor comments (5)
  1. [Sec. 3] The object representation is written as oi = (ci, si, pi, θi) but the text then says 'orientation oi in R around the gravity axis'; the symbol oi is reused for the object and its orientation, which is confusing. Use a distinct symbol for orientation, e.g., θi.
  2. [Table 1, Sec. 5.1] The table would be much more informative with per-method standard deviations or confidence intervals, especially because the differences between Holodeck and VIPScene for CLIP and VQAScore are very small (29.17 vs. 29.98 and 81.43 vs. 82.13).
  3. [Figure 6] The caption does not explain whether the bars show the VIPScene win ratio, the variant win ratio, or a stacked breakdown. The numbers 45/40/10 and 55/60/90 suggest stacked segments, but this should be stated explicitly.
  4. [Appendix E] The limitation section mentions only object-texture quality and future object-generation plans. It should also acknowledge that the contribution of the video prior itself is not isolated by the current experiments and that reconstruction fidelity from generated video remains untested.
  5. [Throughout] The method name is inconsistently written as 'FPVScore' and 'FPVSCORE'; please use one spelling consistently.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity found: the central comparisons are against external baselines and independent human ratings; existing self-citations are contextual, and the missing video-prior control is a support gap, not a definitional reduction.

full rationale

The derivation chain does not collapse into its inputs. VIPScene's synthesis pipeline (Sec. 3) takes a text/image prompt, generates a video with Cosmos, reconstructs with Fast3R/UniDepth, detects and tracks with Grounded-SAM/MASt3R, and assembles assets via ICP-based retrieval plus pose refinement; none of these stages is defined in terms of the final coherence score. The main quantitative evidence (Tab. 1) compares VIPScene against Holodeck and Architect on 100 GPT-4o-written prompts, and the user study is an external human rating, not a function of the method's own parameters. FPVScore is not fitted to VIPScene outputs: its Kendall's tau = 0.39 (Tab. 2) is computed against human evaluator rankings, so its alignment claim is independently testable. Self-citations do exist (CommonScenes [71], EchoScene [73], Sg-Bot [72], MMGDreamer [70], LayoutVLM [48], Mask3D [46]) but they are used only as related-work context or as an ablation baseline, not as the justification for the paper's main claim. The skeptical concern is better framed as a missing control: Sec. 5.4 (Fig. 6) ablates noise reduction, object pose refinement, and 2D-vs-3D detection, but never replaces or corrupts the video generator, so the causal role of the video prior is not isolated; Appendix E acknowledges only texture-level limitations and not this omission. That is an experimental-support weakness, not a circularity, because the headline result still faces external benchmarks and human judgment. No equation in the paper defines a predicted quantity in terms of the fitted parameters or renames a fitted value as a prediction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several unproved behavioral assumptions about off-the-shelf models (video generator, reconstructor, tracker, MLLM judge) and on two hand-set loss weights. No new physical entities are postulated.

free parameters (2)
  • lambda_o, lambda_b = 10
    Hand-set weights in the pose refinement loss (Eq. 2) balancing position, overlap, and boundary terms; the collision-free claim depends on them.
  • Adaptive erosion strength
    Heuristic in Sec. 3.1 scaling morphological erosion with object size; no explicit values or scaling law given, affecting object point cloud quality.
assumptions (4)
  • domain assumption Cosmos video model encodes accurate commonsense 3D layout priors.
    Core premise of Sec. 3.1: the generated video is treated as a faithful walkthrough of a plausible scene.
  • domain assumption Fast3R and MASt3R work reliably on Cosmos videos.
    Sec. 3.1 uses them without validation on this video distribution; distribution shift could break reconstruction and tracking.
  • domain assumption Objaverse contains assets geometrically compatible with detected objects.
    Asset retrieval via ICP (Eq. 1) requires close geometric match between noisy object point clouds and database assets.
  • domain assumption MLLM first-person rankings approximate human preference.
    FPVScore's validity is supported only by one small user study with tau=0.39 (Tab. 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video Perception Models for 3D Scene Synthesis." pith.science (2026). https://pith.science/paper/CR6AQ4NZ

@misc{pith2026250620601,
  author       = {Pith},
  title        = {Pith review of: Video Perception Models for 3D Scene Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CR6AQ4NZ}},
  note         = {Machine review of arXiv:2506.20601}
}
read the original abstract

Traditionally, 3D scene synthesis requires expert knowledge and significant manual effort. Automating this process could greatly benefit fields such as architectural design, robotics simulation, virtual reality, and gaming. Recent approaches to 3D scene synthesis often rely on the commonsense reasoning of large language models (LLMs) or strong visual priors of modern image generation models. However, current LLMs demonstrate limited 3D spatial reasoning ability, which restricts their ability to generate realistic and coherent 3D scenes. Meanwhile, image generation-based methods often suffer from constraints in viewpoint selection and multi-view inconsistencies. In this work, we present Video Perception models for 3D Scene synthesis (VIPScene), a novel framework that exploits the encoded commonsense knowledge of the 3D physical world in video generation models to ensure coherent scene layouts and consistent object placements across views. VIPScene accepts both text and image prompts and seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object in a scene. This enables flexible scene synthesis with high realism and structural consistency. For more precise analysis, we further introduce First-Person View Score (FPVScore) for coherence and plausibility evaluation, utilizing continuous first-person perspective to capitalize on the reasoning ability of multimodal large language models. Extensive experiments show that VIPScene significantly outperforms existing methods and generalizes well across diverse scenarios. The code will be released.

Figures

Figures reproduced from arXiv: 2506.20601 by the authors.

Figure 1
Figure 1. We present VIPSCENE, a generative framework for synthesizing realistic and decomposable 3D scenes. Conditioned on text or image prompts, our method generates diverse indoor and outdoor environments by leveraging the commonsense priors of video generation models for scene layout and object placements. From the generated video, we reconstruct the 3D scene and extract individual objects. The final scene is synthesized … view at source ↗
Figure 2
Figure 2. Illustration of First-Person View Score. Rather than relying on a single top-down view, our metric uses a sequence of first-person view images for each generated scene (left). A multimodal language model (MLLM) then analyzes and ranks the sequences based on multiple evaluation criteria (right). 5 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Qualitative Results. We present top-down and close-up views of our VIPSCENE, comparing it against Holodeck [69] and Architect [60] (columns) across four room types (rows). Holodeck leaves large areas unused while over-cluttering others, whereas Architect produces implausible arrangements that are impractical and rarely seen in real environments. VIPSCENE generates room layouts that are overall more realistic and nat… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Image Prompting and Scene Realism. Left: Examples of image-based prompting: given an input image, VIPSCENE generates a video and reconstructs a full 3D scene. Note that based on the information from the first frame, the generated video can plausibly infer objects beyon…
Figure 5
Figure 5. Figure 5: Exemplary output of GPT-4o of a top-down view [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ablation Study. Win ratio of VIPSCENE versus variants measured by prompt adherence (PA), layout correctness (LC), and overall performance (OP). Variants include the original model without noise reduction (NR), without object pose refinement (OPR), and using a 3D instea…
Figure 7
Figure 7. Figure 7: Qualitative Results from Multimodal Inputs. The generated video respects the original field of view provided by the input image while leveraging the accompanying text to plausibly infer and complete scene elements beyond the visible area. 17 [PITH_FULL_IMAGE:figures/f…
Figure 8
Figure 8. Figure 8: Additional Qualitative Results. We present scenes generated by VIPSCENE, comparing it against Holodeck [69] and Architect [60] (columns) across four room types (rows). For better visibility, ceilings and walls are removed. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: User Study Instructions. This page was shown to participants at the beginning of the study to explain the task, interface, and evaluation criteria. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Example Page. Participants were shown a 360-degree video captured from the center of each scene, along with a top-down rendered image. This setup allowed them to evaluate both the global structure and fine details. Each scene was rated on a 3-point scale (1 = lowest, …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

84 extracted references · 54 canonical work pages

  1. [1]

    Augmented reality meets computer vision: Efficient data generation for urban driving scenes.International Journal on Computer Vision (IJCV), 2018

    Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, and Carsten Rother. Augmented reality meets computer vision: Efficient data generation for urban driving scenes.International Journal on Computer Vision (IJCV), 2018

    2018

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Open-universe indoor scene generation using llm program synthesis and uncurated object databases.arXiv preprint arXiv:2403.09675, 2024

    Rio Aguina-Kang, Maxim Gumin, Do Heon Han, Stewart Morris, Seung Jean Yoo, Aditya Ganeshan, R Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, and Daniel Ritchie. Open-universe indoor scene generation using llm program synthesis and uncurated object databases.arXiv preprint arXiv:2403.09675, 2024

    arXiv 2024

  4. [4]

    I-design: Personalized llm interior designer

    Ata Çelen, Guo Han, Konrad Schindler, Luc Van Gool, Iro Armeni, Anton Obukhov, and Xi Wang. I-design: Personalized llm interior designer. InEuropean Conference on Computer Vision (ECCV), 2025

    2025

  5. [5]

    Meshgen: Generating pbr textured mesh with render-enhanced auto-encoder and generative data augmentation

    Zilong Chen, Yikai Wang, Wenqiang Sun, Feng Wang, Yiwen Chen, and Huaping Liu. Meshgen: Generating pbr textured mesh with render-enhanced auto-encoder and generative data augmentation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  6. [6]

    Procthor: Large-scale embodied ai using procedural generation.International Conference on Neural Information Processing Systems (NeurIPS), 2022

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation.International Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  7. [7]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    work page 2023

  8. [8]

    Global-Local Tree Search in VLMs for 3D Indoor Scene Generation

    Wei Deng, Mengshi Qi, and Huadong Ma. Global-local tree search in vlms for 3d indoor scene generation. arXiv preprint arXiv:2503.18476, 2025

    work page Pith review arXiv 2025

Show all 84 references
  1. [9]

    Layoutgpt: Compositional visual planning and generation with large language models.International Conference on Neural Information Processing Systems (NeurIPS), 2023

    Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models.International Conference on Neural Information Processing Systems (...

    2023

  2. [10]

    Anyhome: Open-vocabulary generation of structured and textured 3d homes

    Rao Fu, Zehao Wen, Zichen Liu, and Srinath Sridhar. Anyhome: Open-vocabulary generation of structured and textured 3d homes. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    Materialmvp: Illumination-invariant material generation via multi-view pbr diffusion.arXiv preprint arXiv:2503.10289, 2025

    Zebin He, Mingxin Yang, Shuhui Yang, Yixuan Tang, Tao Wang, Kaihao Zhang, Guanying Chen, Yuhong Liu, Jie Jiang, Chunchao Guo, et al. Materialmvp: Illumination-invariant material generation via multi-view pbr diffusion.arXiv preprint arXiv:2503.10289, 2025

    2025 arXiv

  5. [13]

    Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  6. [14]

    Video diffusion models.International Conference on Neural Information Processing Systems (NeurIPS), 2022

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models.International Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  7. [15]

    Text2room: Extracting textured 3d meshes from 2d text-to-image models

    Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. InInternational Conference on Computer Vision (ICCV), 2023

    2023

  8. [16]

    Material anything: Generating materials for any 3d object via diffusion

    Xin Huang, Tengfei Wang, Ziwei Liu, and Qing Wang. Material anything: Generating materials for any 3d object via diffusion. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  9. [17]

    Midi: Multi-instance diffusion for single image to 3d scene generation

    Zehuan Huang, Yuan-Chen Guo, Xingqiao An, Yunhan Yang, Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu, Yan-Pei Cao, and Lu Sheng. Midi: Multi-instance diffusion for single image to 3d scene generation. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  10. [18]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 10

    2024 arXiv

  11. [19]

    How far is video generation from world model: A physical law perspective.arXiv preprint arXiv:2411.02385, 2024

    Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective.arXiv preprint arXiv:2411.02385, 2024

    2024 arXiv

  12. [20]

    Repurposing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  13. [21]

    A new measure of rank correlation.Biometrika, 1938

    Maurice G Kendall. A new measure of rank correlation.Biometrika, 1938

    1938

  14. [22]

    Kling, 2025.https://klingai.com/global/

    Kuaishou. Kling, 2025.https://klingai.com/global/

    2025

  15. [23]

    Scenecraft: automating interactive narrative scene generation in digital games with large language models

    Vikram Kumaran, Jonathan Rowe, Bradford Mott, and James Lester. Scenecraft: automating interactive narrative scene generation in digital games with large language models. InAssociation for the Advancement of Artificial Intelligence (AAAI), 2023

    2023

  16. [24]

    Grounding image matching in 3d with mast3r

    Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision (ECCV), 2024

    2024

  17. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning (ICML), 2023

    2023

  18. [26]

    Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    2024 arXiv

  19. [27]

    Instructscene: Instruction-driven 3d indoor scene synthesis with semantic graph prior

    Chenguo Lin and MU Yadong. Instructscene: Instruction-driven 3d indoor scene synthesis with semantic graph prior. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  20. [28]

    Towards language-guided interactive 3d generation: Llms as layout interpreter with generative feedback.arXiv preprint arXiv:2305.15808, 2023

    Yiqi Lin, Hao Wu, Ruichen Wang, Haonan Lu, Xiaodong Lin, Hui Xiong, and Lin Wang. Towards language-guided interactive 3d generation: Llms as layout interpreter with generative feedback.arXiv preprint arXiv:2305.15808, 2023

    2023 arXiv

  21. [29]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  22. [30]

    Scenethesis: A language and vision agentic framework for 3d scene generation.arXiv preprint arXiv:2505.02836, 2025

    Lu Ling, Chen-Hsuan Lin, Tsung-Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng, Yunhao Ge, Ming-Yu Liu, Aniket Bera, and Zhaoshuo Li. Scenethesis: A language and vision agentic framework for 3d scene generation.arXiv preprint arXiv:2505.02836, 2025

    2025 arXiv

  23. [31]

    Visual instruction tuning.International Conference on Neural Information Processing Systems (NeurIPS), 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.International Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  24. [32]

    Dream machine, 2024.https://lumalabs.ai/dream-machine

    Luma. Dream machine, 2024.https://lumalabs.ai/dream-machine

    2024

  25. [33]

    Mimicgen: A data generation system for scalable robot learning using human demonstrations

    Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. InConference on Robot Learning (CoRL), 2023

    2023

  26. [34]

    Cosmos, 2024.https://www.nvidia.com/en-us/ai/cosmos/

    Nvidia. Cosmos, 2024.https://www.nvidia.com/en-us/ai/cosmos/

    2024

  27. [35]

    Global structure-from-motion revisited

    Linfei Pan, Dániel Baráth, Marc Pollefeys, and Johannes L Schönberger. Global structure-from-motion revisited. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  28. [36]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InInternational Conference on Computer Vision (ICCV), 2023

    2023

  29. [37]

    Unidepth: Universal monocular metric depth estimation

    Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  30. [38]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  31. [39]

    Hsm: Hierarchical scene motifs for multi-scale indoor scene generation.arXiv preprint arXiv:2503.16848, 2025

    Hou In Derek Pun, Hou In Ivan Tam, Austin T Wang, Xiaoliang Huo, Angel X Chang, and Manolis Savva. Hsm: Hierarchical scene motifs for multi-scale indoor scene generation.arXiv preprint arXiv:2503.16848, 2025. 11

    2025

  32. [40]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning (ICML), 2021

    2021

  33. [41]

    Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159, 2024

    2024 arXiv

  34. [42]

    Gen 3, 2024.https://runwayml.com/research/introducing-gen-3-alpha

    Runway. Gen 3, 2024.https://runwayml.com/research/introducing-gen-3-alpha

    2024

  35. [43]

    Temporal generative adversarial nets with singular value clipping

    Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. InInternational Conference on Computer Vision (ICCV), 2017

    2017

  36. [44]

    Structure-from-motion revisited

    Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  37. [45]

    Pixelwise view selection for unstructured multi-view stereo

    Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. InEuropean Conference on Computer Vision (ECCV), 2016

    2016

  38. [46]

    Mask3d: Mask transformer for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. InInternational Conference on Robotics and Automation (ICRA), 2023

    2023

  39. [47]

    Controlroom3d: Room generation using semantic proxy rooms

    Jonas Schult, Sam Tsai, Lukas Höllein, Bichen Wu, Jialiang Wang, Chih-Yao Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, et al. Controlroom3d: Room generation using semantic proxy rooms. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  40. [48]

    Layoutvlm: Differentiable optimization of 3d layout via vision-language models.arXiv preprint arXiv:2412.02193, 2024

    Fan-Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, and Jiajun Wu. Layoutvlm: Differentiable optimization of 3d layout via vision-language models.arXiv preprint arXiv:2412.02193, 2024

    2024 arXiv

  41. [49]

    Neuralrecon: Real-time coherent 3d reconstruction from monocular video

    Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  42. [50]

    Retargetable ar: Context-aware aug- mented reality in indoor scenes based on 3d scene graph

    Tomu Tahara, Takashi Seno, Gaku Narita, and Tomoya Ishikawa. Retargetable ar: Context-aware aug- mented reality in indoor scenes based on 3d scene graph. In2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2020

    2020

  43. [51]

    Diffuscene: Denoising diffusion models for generative indoor scene synthesis

    Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Denoising diffusion models for generative indoor scene synthesis. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  44. [52]

    Mv-dust3r+: Single-stage scene reconstruction from sparse views in 2 seconds

    Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, and Zhicheng Yan. Mv-dust3r+: Single-stage scene reconstruction from sparse views in 2 seconds. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  45. [53]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  46. [54]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  47. [55]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  48. [56]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  49. [57]

    3d reconstruction with spatial memory.arXiv preprint arXiv:2408.16061, 2024

    Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory.arXiv preprint arXiv:2408.16061, 2024

    2024 arXiv

  50. [58]

    Vggt: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 12

    2025

  51. [59]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), pages 20697–20709, 2024

    2024

  52. [60]

    Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting.International Conference on Neural Information Processing Systems (NeurIPS), 2025

    Yian Wang, Xiaowen Qiu, Jiageng Liu, Zhehuan Chen, Jiting Cai, Yufei Wang, Tsun-Hsuan Johnson Wang, Zhou Xian, and Chuang Gan. Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting.International Conference on Neural Information Processing System...

    2025

  53. [61]

    Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation

    Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22227–22...

    2024

  54. [62]

    Amodal3r: Amodal 3d reconstruction from occluded 2d images.arXiv preprint arXiv:2503.13439, 2025

    Tianhao Wu, Chuanxia Zheng, Frank Guan, Andrea Vedaldi, and Tat-Jen Cham. Amodal3r: Amodal 3d reconstruction from occluded 2d images.arXiv preprint arXiv:2503.13439, 2025

    2025 arXiv

  55. [63]

    Structured 3d latents for scalable and versatile 3d generation.arXiv preprint arXiv:2412.01506, 2024

    Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation.arXiv preprint arXiv:2412.01506, 2024

    2024 arXiv

  56. [64]

    Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass

    Jianing Yang, Alexander Sax, Kevin J Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  57. [65]

    Diffusion probabilistic modeling for video genera- tion.Entropy, 2023

    Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video genera- tion.Entropy, 2023

    2023

  58. [66]

    Hunyuan3d 1.0: A unified framework for text-to-3d and image-to-3d generation.arXiv preprint arXiv:2411.02293, 2024

    Xianghui Yang, Huiwen Shi, Bowen Zhang, Fan Yang, Jiacheng Wang, Hongxu Zhao, Xinhai Liu, Xinzhou Wang, Qingxiang Lin, Jiaao Yu, et al. Hunyuan3d 1.0: A unified framework for text-to-3d and image-to-3d generation.arXiv preprint arXiv:2411.02293, 2024

    2024 arXiv

  59. [67]

    Physcene: Physically interactable 3d scene synthesis for embodied ai

    Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthesis for embodied ai. InInternational Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  60. [68]

    Llplace: The 3d indoor scene layout generation and editing via large language model.arXiv preprint arXiv:2406.03866, 2024

    Yixuan Yang, Junru Lu, Zixiang Zhao, Zhen Luo, James JQ Yu, Victor Sanchez, and Feng Zheng. Llplace: The 3d indoor scene layout generation and editing via large language model.arXiv preprint arXiv:2406.03866, 2024

    2024 arXiv

  61. [69]

    Holodeck: Language guided generation of 3d embodied ai environments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. InInternational Conference on Computer Vision and Pattern Recognition...

    2024

  62. [70]

    Mmgdreamer: Mixed-modality graph for geometry-controllable 3d indoor scene generation

    Zhifei Yang, Keyang Lu, Chao Zhang, Jiaxing Qi, Hanqi Jiang, Ruifei Ma, Shenglin Yin, Yifan Xu, Mingzhe Xing, Zhen Xiao, Jieyi Long, Xiangde Liu, and Guangyao Zhai. Mmgdreamer: Mixed-modality graph for geometry-controllable 3d indoor scene generation. InAssociation for the Adv...

    2025

  63. [71]

    Commonscenes: Generating commonsense 3d indoor scenes with scene graphs

    Guangyao Zhai, Evin Pinar Örnek, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Commonscenes: Generating commonsense 3d indoor scenes with scene graphs. In International Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  64. [72]

    Sg-bot: Object rearrangement via coarse-to-fine robotic imagination on scene graphs

    Guangyao Zhai, Xiaoni Cai, Dianye Huang, Yan Di, Fabian Manhardt, Federico Tombari, Nassir Navab, and Benjamin Busam. Sg-bot: Object rearrangement via coarse-to-fine robotic imagination on scene graphs. InInternational Conference on Robotics and Automation (ICRA), 2024

    2024

  65. [73]

    Echoscene: Indoor scene generation via information echo over scene graph diffusion

    Guangyao Zhai, Evin Pınar Örnek, Dave Zhenyu Chen, Ruotong Liao, Yan Di, Nassir Navab, Federico Tombari, and Benjamin Busam. Echoscene: Indoor scene generation via information echo over scene graph diffusion. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  66. [74]

    Fast and robust iterative closest point.Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2021

    Juyong Zhang, Yuxin Yao, and Bailin Deng. Fast and robust iterative closest point.Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2021

    2021

  67. [75]

    Dreammat: High-quality pbr material generation with geometry-and light-aware diffusion models.ACM Transactions on Graphics (TOG), 2024

    Yuqing Zhang, Yuan Liu, Zhiyu Xie, Lei Yang, Zhongyuan Liu, Mengzhou Yang, Runze Zhang, Qilong Kou, Cheng Lin, Wenping Wang, et al. Dreammat: High-quality pbr material generation with geometry-and light-aware diffusion models.ACM Transactions on Graphics (TOG), 2024

    2024

  68. [76]

    ground” for outdoor scenes or “floor

    Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang, Sheng Zhang, Xianghui Yang, et al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation.arXiv preprint arXiv:2501.12202, 2025. 13 Appendi...

    2025 arXiv

  69. [77]

    A bedroom with a large bed, two nightstands, a floor lamp, a wardrobe, and a big window

    Based on the typical function and layout of a {room type}, list the furniture, appliances, decorations, and other items commonly found in the space. 2. Prompts should describe the room’s contents clearly and in detail. Example: “A bedroom with a large bed, two nightstands, a f...

  70. [80]

    realistic

    Overall Preference Does the room layout look realistic and natural? Consider the visual coherence and harmony of the scene. Evaluation process: Carefully examine the multi-view images of all three 3D scenes. Focus on one criterion at a time and make independent judgments for e...

  71. [81]

    Semantic Correctness: The first one ...; The second one ...; The third one

  72. [82]

    Layout Correctness: The first one ...; The second one ...; The third one

  73. [83]

    Final answer: The first one: x x x The second one: x x x The third one: x x x (where x denotes ranks 1–3) (Please strictly follow the format above

    Overall Preference: The first one ...; The second one ...; The third one ... Final answer: The first one: x x x The second one: x x x The third one: x x x (where x denotes ranks 1–3) (Please strictly follow the format above. Do not include extra symbols like **, quotation mark...

  74. [84]

    Semantic Correctness Does the generated layout accurately reflect the text description? Check whether all described objects are present and correctly represented

  75. [85]

    Consider object positions, orientations, and user convenience

    Layout Correctness Is the room design physically plausible and functional? Evaluate if the layout supports practical use, space efficiency, and proper object functionality. Consider object positions, orientations, and user convenience

  76. [86]

    Overall Preference Does the room layout look realistic and natural? Consider the visual coherence and harmony of the scene. Provide only your final ranking of the three methods in the format below: Final answer: x x x (where x denotes ranks from 1 to 3) D User Study Details We...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.