Pith. sign in

REVIEW 4 major objections 7 minor 79 references

Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Scene Copilot claims that a two-stage LLM pipeline plus human editing in Blender lets users turn natural-language prompts into customized, photorealistic scene videos, including arbitrarily long renders.

desk verdict A credible integration of LLMs and Infinigen with a useful procedural asset dataset, but the editing loop is unmeasured and the stronger claims outrun the evidence. read the letter →

arxiv 2411.18644 v1 pith:HEDRKELB submitted 2024-11-26 cs.CV

Zhaofang Qian , Abolfazl Sharifi , Tucker Carroll , Ser-Nam Lim This is my paper · ORCID
classification cs.CV
keywords procedural3Dscenegenerationtext-to-videohuman-in-the-loopeditinglargelanguagemodelsretrieval-augmentedBlendervideoconsistencyassetdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a non-specialist can go from a typed sentence to a customized photorealistic video by pairing a large language model with Infinigen, a procedural 3D scene generator built on Blender, and adding a human editing step. The proposed system, Scene Copilot, uses one LLM stage to translate a prompt into a generator command and a second LLM stage to turn follow-up instructions into Blender editing code, with the user clicking objects in the Blender interface to indicate what should change. If this works, procedural scenes become accessible to people who do not know the underlying generation rules, and rendered videos inherit the scene's persistence: objects keep their identity and physical behavior holds no matter how long the video runs. The paper supports the claim with qualitative renderings, a comparison against four video-generation models on VBench's scenery category, and an ablation showing that retrieval context and few-shot examples raise executable command rate to 43.2 percent.

What carries the argument

The load-bearing mechanism is the two-stage LLM command pipeline. Scene Codex is a retrieval-augmented LLM that turns a prompt into an Infinigen run command. BlenderGPT is a chain-of-thought LLM that turns follow-up edits into Blender Python code, with the user selecting objects in the Blender GUI. Between the two sits a condensation step: each Blender scene is converted to a .usda textual scene description, every numerical value is replaced with the placeholder NUM, and the result is stored as a retrieval database so the editing LLM can see the scene's structure without drowning in numbers. The procedural asset dataset, 323 assets and materials converted into code, supplies the LLM with named, modifiable objects it can combine or repurpose.

What would settle it

Generate a set of scenes with known .usda files, issue targeted editing prompts such as "move the selected object 2 meters to the right" or "set the camera focal length to 35 mm" through BlenderGPT, and, without human code repair, compare the resulting scene's numeric values against the intended ones; if correct numeric edits occur near chance, the condensation claim collapses.

Watch

Extended reading notes

Core claim

The paper's claim is that LLM-generated commands combined with human-in-the-loop Blender editing make procedural scene generation controllable enough for ordinary users to act as directors of their own videos. Scene Codex maps text to an executable Infinigen command, Infinigen produces a coarse scene, BlenderGPT converts natural-language revisions into Blender Python code, and the user can also edit directly through the Blender GUI; the loop repeats once at a finer detail level before final rendering. The authors report that this workflow generated scenes and videos that follow prompts such as a graveyard at sunset, scored best among compared models on motion smoothness and dynamic degree in VBench's scenery category, and kept rendering success high at longer video lengths where plain Infinigen degrades. They further state that because the output is a render of a persistent 3D scene, videos of arbitrary length can be produced without concern for object integrity or the physical laws governing the world.

Load-bearing premise

The load-bearing premise is that a Blender scene, once converted to text with every number replaced by the placeholder NUM, still contains enough information for a language model to edit it correctly; if a requested change depends on an exact position, scale, or camera value, the model is guessing and the user must catch the error.

Editorial extensions

If this is right

  • Non-specialists can generate and revise photorealistic 3D scenes by typing requests and clicking objects, without reading Infinigen's parameter documentation.
  • Rendered videos keep object identities and physical behavior consistent across arbitrarily long durations, because the video is a render of a persistent 3D scene rather than a frame-by-frame prediction.
  • Adding RAG context and few-shot examples raises the fraction of executable generator commands from 0 percent to 43.2 percent, so documentation injection is a necessary part of the pipeline.
  • Users act as directors: they can force a camera to follow a chosen object or add assets, such as a church and gravestones, that the base generator would omit.
  • The 323 procedural assets in code form let the LLM combine and modify objects in ways a fixed asset library cannot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the NUM condensation is as information-preserving as the paper assumes, the same text-outline-of-a-scene trick could be applied to other procedural engines and CAD tools, letting LLMs edit any scene that can be serialized to text.
  • A safer design would keep exact numerical values available to the model through tool calls or selective retrieval, because the paper gives no measurement of how often BlenderGPT's edits are semantically reasonable but numerically wrong.
  • A cheap experimental test would replace NUM with exact numbers for a small set of edits and compare success rates; the gap would isolate how much precision the condensation step actually costs.
  • The dataset's automatic expansion scripts imply a route to continuously growing asset coverage, but they also raise a testable risk that older LLM-generated commands break as asset names and conversion conventions change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Scene Copilot, a framework that combines LLMs with the procedural 3D scene generator Infinigen and a Blender editing workflow. Given a text prompt, Scene Codex (an LLM with RAG over the Infinigen codebase and few-shot examples) generates an Infinigen command; after an initial coarse scene is produced, BlenderGPT converts the scene to a condensed textual .usda form and generates Blender Python code for user-requested edits, with the user able to select objects in Blender's GUI. The refined scene is fed back to Infinigen for a final render. The paper also contributes a dataset of over 300 procedural assets and materials in code form. Evaluation consists of qualitative key-frame examples, a VBench comparison on 100 scenery prompts, an executability-rate ablation for Scene Codex, and a long-video success-rate comparison with Infinigen.

Significance. If the controllability claims hold, the framework is a practical contribution: it lowers the barrier to using Infinigen, provides persistence of object identity across long renders, and is training-free. The paper's Scene Codex ablation (Fig. 11) gives a clear, quantitative demonstration that RAG and few-shot prompting each contribute to prompt-to-command executability (0%, 2.0%, 20.0%, 43.2% ER@1). The curated procedural asset dataset is also a useful resource for the community. The main significance hinges on whether BlenderGPT's editing behavior is reliable enough to make the human-in-the-loop loop practical; that point is not yet quantitatively established.

major comments (4)
  1. [§4.2, §5.3, Figure 11] The central claim that users can control the scene through text and GUI selection requires BlenderGPT to generate Blender Python code that both executes and implements the requested edit, but the paper reports no quantitative measurement of BlenderGPT's edit success rate. The only executability metric, ER@1 in Figure 11, evaluates Scene Codex's Infinigen commands, not BlenderGPT edits. Without such a measurement, the qualitative demonstrations in Figures 9 and 10 are potentially selected successes, and the contribution could reduce to 'experts can debug LLM-generated Blender code' rather than 'users can control the scene.'
  2. [§4.2 Preprocessing, Limitations] The NUM-masked .usda representation replaces all numerical values (e.g., matrix4d xformOp:transform = NUM, focal length, aperture) with placeholders before constructing the RAG database. For edits that depend on exact spatial or optical quantities—such as 'camera follow the selected object' in Figure 8 or repositioning geometry—the LLM cannot see the relevant numbers in context, and the paper does not explain how BlenderGPT recovers them at runtime. The Limitations section acknowledges that LLM hallucination is 'inevitable' and that users need 'advanced programming knowledge' to correct issues, which directly undercuts the paper's accessibility framing and leaves the reliability of the edit pipeline unquantified.
  3. [§5.2, Table 2] The VBench evaluation is restricted to 100 scenery prompts, includes no baseline from Infinigen alone, and reports no error bars or statistical significance. Without an Infinigen-only baseline, the scores in Table 2 do not isolate the contribution of Scene Copilot's editing components; without error bars, comparisons such as Motion Smoothness 99.36 vs. 97.58 are hard to interpret. The claim that Scene Copilot achieves the best scores in Motion Smoothness and Dynamic Degree should be supported with a confidence interval or multiple runs, or tempered accordingly.
  4. [§5.4, Figure 12] The long-video success rate comparison does not define what constitutes 'success' for a generated video, and the comparison between Infinigen and Scene Copilot is confounded by the human-in-the-loop component, which can manually set camera animations. The sentence in §5.4 claiming videos of 'arbitrary length without concern for the preservation of object integrity or the enforcement of the physical laws governing the generated world' overstates the evidence, which is based on 12 scenes per setting with no reported metric for object integrity or physical-law violations.
minor comments (7)
  1. [Title and Abstract] The title uses 'Scene Co-pilot' whereas the abstract, body, and system name use 'Scene Copilot'; please standardize the spelling.
  2. [§5.3] The text says 'Table 11 shows the results' but the ablation results appear in Figure 11, not a table; the cross-reference should be corrected.
  3. [§4.2] The phrase 'it defaults from a blank Blender project' should read 'it defaults to a blank Blender project'; also, 'such that' is used where 'so that' would be clearer.
  4. [§3.1] The 'probability function derived from a Gaussian distribution' that decides whether to use the procedural dataset or Shap-E is not specified; please provide the exact function or a reference so the behavior is reproducible.
  5. [§3.3 and Dataset release] The paper states the dataset is published under CC-BY 4.0 and 'includes procedural assets, the corresponding source code, direct links to the assets, and detailed metadata,' but no URL or repository link is given in the text; a link would be needed for the claim of accessibility to be verifiable.
  6. [Algorithm 1, line 8] The comment 'Format the improved code for readability' is followed by 'imp formatted code ← format(fine scene, LLMs)', which formats the scene, not the code; please rename for consistency.
  7. [Table 2 caption] The caption says 'The best result in each dimension is highlighted in bold, and the second-best result is highlighted in Italics,' but the table body does not visibly apply the italic formatting; please ensure the formatting is rendered.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are evaluated against external targets (Infinigen executability, VBench) and are not reducible to the paper's own inputs.

full rationale

Scene Copilot is a system paper with an engineering pipeline rather than a derivation, and no claimed prediction or first-principles result is equivalent to its inputs by construction. The main capability claims are supported by external evaluation: Scene Codex is measured by Executable Rate (ER@1), which checks whether generated Infinigen commands actually run in Infinigen, an external codebase; the video results are scored with VBench, an independent video-generation benchmark; and the long-video comparison uses render success at specified lengths. None of these metrics is defined in terms of the system's own outputs in a way that forces the reported numbers. The paper contains no load-bearing self-citations: Infinigen, BlenderGPT, VBench, Shap-E, and the LLM APIs are all external prior work, and none of the authors' own previous results is invoked to justify a central premise. The only self-involving element is that an LLM both writes Blender/Infinigen code and helps debug that code, but this is a system design loop, not a circular argument; the paper does not treat it as evidence for a derived claim. The Limitations section honestly concedes that LLM hallucination is inevitable and that users may need advanced programming knowledge to repair mistakes, which weakens the accessibility framing but is a correctness and usability limitation, not circularity. Overall, the paper's evaluation is self-contained against external benchmarks and code-execution checks, so no circular step rises to the level specified by the rubric.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No fitted parameters in the mathematical sense are present because the paper makes no quantitative derivation. The listed free parameters are hand-chosen settings that affect reproducibility. All axioms are domain assumptions inherited from Infinigen, Blender, and LLM behavior; no new physical or mathematical entities are introduced.

free parameters (2)
  • Shap-E selection probability function = not reported
    Sec. 3.1 states asset generation is guided by a probability function derived from a Gaussian distribution, but the mean, variance, or threshold are not reported, making the dataset construction not fully reproducible.
  • LLM decoding temperature = 0
    Sec. 5 sets decoding temperature to 0 for all generation. This is a deliberate anti-hallucination choice rather than a fitted parameter, but it is a hand-chosen setting that affects outputs.
assumptions (5)
  • domain assumption Infinigen generates photorealistic scenes from Python commands and can be extended with custom assets.
    Entire pipeline is built on Infinigen's capabilities; introduced in Sec. 1 and used throughout.
  • domain assumption Blender node graphs can be converted to executable, modifiable Python code via Node Transpiler and Node2Python.
    Sec. 3 states the dataset construction depends on this conversion; no failure analysis is provided.
  • domain assumption Condensed USD text with numerical values replaced by NUM retains enough scene information for LLM-based editing.
    Sec. 4.2 Preprocessing; this is the weakest link and is not validated quantitatively.
  • domain assumption Users can repair LLM-generated mistakes, so a 43.2 percent initial executable command rate suffices.
    Sec. 5.3 reports ER@1 of 43.2 percent; the Limitations section admits users need programming knowledge to correct issues.
  • domain assumption The chosen LLMs (Claude-3.5-Sonnet and GPT-4o) follow the provided system prompts and few-shot examples reliably.
    Core to both Scene Codex and BlenderGPT; the paper provides no formal guarantee and depends on proprietary model behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop." pith.science (2026). https://pith.science/paper/HEDRKELB

@misc{pith2026241118644,
  author       = {Pith},
  title        = {Pith review of: Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HEDRKELB}},
  note         = {Machine review of arXiv:2411.18644}
}
read the original abstract

Video generation has achieved impressive quality, but it still suffers from artifacts such as temporal inconsistency and violation of physical laws. Leveraging 3D scenes can fundamentally resolve these issues by providing precise control over scene entities. To facilitate the easy generation of diverse photorealistic scenes, we propose Scene Copilot, a framework combining large language models (LLMs) with a procedural 3D scene generator. Specifically, Scene Copilot consists of Scene Codex, BlenderGPT, and Human in the loop. Scene Codex is designed to translate textual user input into commands understandable by the 3D scene generator. BlenderGPT provides users with an intuitive and direct way to precisely control the generated 3D scene and the final output video. Furthermore, users can utilize Blender UI to receive instant visual feedback. Additionally, we have curated a procedural dataset of objects in code format to further enhance our system's capabilities. Each component works seamlessly together to support users in generating desired 3D scenes. Extensive experiments demonstrate the capability of our framework in customizing 3D scenes and video generation.

Figures

Figures reproduced from arXiv: 2411.18644 by the authors.

Figure 1
Figure 1. Dataset Samples: A subset of assets from our dataset, ac [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Materials in our dataset can serve as textures, and they [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Various types of materials are employed as textures for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Materials are combined with assets automatically, with [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Combining procedural assets and generative models to [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Overview of Scene Copilot with procedural dataset. Starting with a user textual input, Scene Codex combines an LLM with [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Code segment from converted .usda file. It shows [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Image demo file, which includes metadata ranging from the scene ter￾rain to object animation information. After acquiring the raw .usda file that describes the whole scene, we utilize a lightweight LLM (GPT-4o mini [47]) to further convert and clean the textual content…
Figure 9
Figure 9. Figure 9: Five samples of eight key frames from generated videos with the corresponding textual input prompt. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Left Three key frames from rendered videos of the original scene output from Infinigen. Right Three key frames from rendered videos of the edited scene using Scene Copilot. Associated text prompts from VBench [29] are used as the direct textual input. VideoCrafter1 [1…
Figure 11
Figure 11. Figure 11: The evaluation result of prompting RAG database Infinigen and few￾shot examples. Based on the result, Using both of them are valuable and vital for our Scene Codex. RAG Few-Shot ER@1 1 0.0% 2 ✓ 2.0% 3 ✓ 20.0% 4 ✓ ✓ 43.2% 1 2 5 10 30 0 20 40 60 80 100 Video length (min…
Figure 12
Figure 12. Figure 12: Video generation suc￾cess rate comparison between Infini￾gen and Ours across different video lengths. Infinigen’s performance de￾creases as video length increases. Codex, the ER@1 reaches 43.2%. This result demonstrates the value of including additional information wh…
Figure 13
Figure 13. Figure 13: Screenshots from 4 different outputs from Scene Copilot and their respective prompts. [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Thirty samples of videos generated by the framework are presented, showcasing the system’s capability to produce diverse and [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: The dataset has been updated with 41 new materials. This addition enhances the diversity and scope of the dataset, enabling it [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Four of the new assets added to the dataset from the internet. Each image was rendered inside of its original blender scene. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 44 canonical work pages

  1. [1]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024. 2, 6

    work page 2024

  2. [2]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 8

    arXiv 2023

  3. [3]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 8

    work page 2023

  4. [4]

    Nodetopython

    carls3d BrendanParmer. Nodetopython. https : //github.com/BrendanParmer/NodeToPython ,

    work page

  5. [5]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

  6. [6]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 2

    arXiv 2005

  7. [7]

    A naturalistic open source movie for opti- cal flow evaluation

    Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for opti- cal flow evaluation. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12 , pages 611–

    2012

  8. [8]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving, 2020. 1

    work page 2020

Show all 79 references
  1. [9]

    Coda: Col- laborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection

    Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Col- laborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. Advances in Neu- ral Information Processing Systems, 36, 2024. 2

    2024

  2. [10]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023. 7

    arXiv

  3. [11]

    Evalu- ating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hen- rique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 1, 5, 7

    2021 arXiv

  4. [12]

    Blender - a 3D modelling and rendering package

    Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. 1

    2018

  5. [13]

    Procthor: Large-scale embodied ai using procedural generation

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Ad- vances in Neural Information Processing Systems, 35:5982...

    2022

  6. [14]

    Pla: Language-driven open- vocabulary 3d scene understanding

    Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open- vocabulary 3d scene understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7010–7019, 2023. 2

    2023

  7. [15]

    The faiss library,

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazar´e, Maria Lomeli, Lucas Hosseini, and Herv´e J´egou. The faiss library,

  8. [16]

    Motsynth: How can synthetic data help pedestrian detection and tracking? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10849–10859, 2021

    Matteo Fabbri, Guillem Bras ´o, Gianluca Maugeri, Orcun Cetintas, Riccardo Gasparini, Aljo ˇsa Oˇsep, Simone Calder- ara, Laura Leal-Taix´e, and Rita Cucchiara. Motsynth: How can synthetic data help pedestrian detection and tracking? In Proceedings of the IEEE/CVF Internationa...

    2021

  9. [17]

    Large language models for code analysis: Do LLMs really do their job? In 33rd USENIX Security Symposium (USENIX Security 24) , pages 829–846, Philadelphia, PA, 2024

    Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Asmita, Ryan Tsang, Najmeh Nazari, Han Wang, and Houman Homayoun. Large language models for code analysis: Do LLMs really do their job? In 33rd USENIX Security Symposium (USENIX Security 24) , ...

    2024

  10. [18]

    Chat- edit-3d: Interactive 3d scene editing via text prompts

    Shuangkang Fang, Yufeng Wang, Yi-Hsuan Tsai, Yi Yang, Wenrui Ding, Shuchang Zhou, and Ming-Hsuan Yang. Chat- edit-3d: Interactive 3d scene editing via text prompts. In European Conference on Computer Vision, pages 199–216. Springer, 2025. 2

    2025

  11. [19]

    Scenescape: Text-driven consistent scene generation

    Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. Ad- vances in Neural Information Processing Systems, 36, 2024. 2

    2024

  12. [20]

    Scene-llm: Extending language model for 3d visual understanding and reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wen- han Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401, 2024. 2

    2024 arXiv

  13. [21]

    Retrieval-augmented generation for large language models: A survey, 2024

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024. 5

    2024

  14. [22]

    Procedural model- ing of plant ecosystems maximizing vegetation cover

    Cristina Gasch, Jos ´e Mart´ınez Sotoca, Miguel Chover, In- maculada Remolar, and Cristina Rebollo. Procedural model- ing of plant ecosystems maximizing vegetation cover. Mul- timedia Tools and Applications, 81(12):16195–16217, 2022. 2

    2022

  15. [23]

    Streamingt2v: Con- sistent, dynamic, and extendable long video generation from text

    Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Con- sistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403.14773, 2024. 1

    2024 arXiv

  16. [24]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 8

    2022

  17. [25]

    CogVideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. CogVideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 6

    2022 arXiv

  18. [26]

    An inverse procedural modeling pipeline for svbrdf maps

    Yiwei Hu, Chengan He, Valentin Deschaintre, Julie Dorsey, and Holly Rushmeier. An inverse procedural modeling pipeline for svbrdf maps. ACM Transactions on Graphics (TOG), 41(2):1–17, 2022. 2

    2022

  19. [27]

    Generating procedural materials from text or image prompts

    Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. Generating procedural materials from text or image prompts. In ACM SIGGRAPH 2023 Con- ference Proceedings, pages 1–11, 2023. 2

    2023

  20. [28]

    Scenecraft: An llm agent for synthesizing 3d scenes as blender code

    Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An llm agent for synthesizing 3d scenes as blender code. In Forty-first International Conference on Ma- chine Learning, 2024. 2

    2024

  21. [29]

    VBench: Com- prehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Com- prehensive benchmark suite for video generative models....

    2024

  22. [30]

    Putting nerf on a diet: Semantically consistent few-shot view synthesis

    Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5885–5894, 2021. 2

    2021

  23. [31]

    Zero-shot text-guided object genera- tion with dream fields

    Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object genera- tion with dream fields. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 867–876, 2022. 2

    2022

  24. [32]

    Shap-e: Generating condi- tional 3d implicit functions, 2023

    Heewoo Jun and Alex Nichol. Shap-e: Generating condi- tional 3d implicit functions, 2023. 3

    2023

  25. [33]

    Clip-mesh: Gen- erating textured meshes from text using pretrained image- text models

    N Khalid, T Xie, E Belilovsky, and T Popa. Clip-mesh: Gen- erating textured meshes from text using pretrained image- text models . PhD thesis, Concordia University Montr ´eal, Qu´ebec, Canada, 2023. 2

    2023

  26. [34]

    Kling AI

    Kuaishou. Kling AI. https://klingai.kuaishou. com, 2024. 1

    2024

  27. [35]

    Understanding pure clip guidance for voxel grid nerf models

    Han-Hung Lee and Angel X Chang. Understanding pure clip guidance for voxel grid nerf models. arXiv preprint arXiv:2209.15172, 2022. 2

    2022 arXiv

  28. [36]

    Retrieval-augmented genera- tion for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen tau Yih, Tim Rockt¨aschel, Sebas- tian Riedel, and Douwe Kiela. Retrieval-augmented genera- tion for knowledge-intensive nlp tasks, 2021. 5

    2021

  29. [37]

    Loogle: Can long-context language models under- stand long contexts?, 2024

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models under- stand long contexts?, 2024. 5

    2024

  30. [38]

    Uni3d-llm: Unifying point cloud perception, generation and editing with large language models

    Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang, Zhenfei Yin, Yongshun Gong, Peng Gao, and Wanli Ouyang. Uni3d-llm: Unifying point cloud perception, generation and editing with large language models. arXiv preprint arXiv:2402.03327, 2024. 2

    2024 arXiv

  31. [39]

    What makes good in-context examples for gpt- 3? arXiv preprint arXiv:2101.06804, 2021

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt- 3? arXiv preprint arXiv:2101.06804, 2021. 2

    2021 arXiv

  32. [40]

    Deep learning for procedural content generation

    Jialin Liu, Sam Snodgrass, Ahmed Khalifa, Sebastian Risi, Georgios N Yannakakis, and Julian Togelius. Deep learning for procedural content generation. Springer, 2021. 2

    2021

  33. [41]

    A multimodal gen- erative ai copilot for human pathology

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Melissa Zhao, Aaron K Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, et al. A multimodal gen- erative ai copilot for human pathology. Nature, pages 1–3,

  34. [42]

    Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data

    Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boular- ias, and Kostas Bekris. Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data. In Conference on Robot Learning, pages 1610–1620. PMLR, 2023. 2

    2023

  35. [43]

    When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models

    Xianzheng Ma, Yash Bhalgat, Brandon Smart, Shuai Chen, Xinghui Li, Jian Ding, Jindong Gu, Dave Zhenyu Chen, Songyou Peng, Jia-Wang Bian, et al. When llms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models. arXiv preprint arXiv:...

    2024

  36. [44]

    One million scenes for autonomous driving: Once dataset

    Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, et al. One million scenes for autonomous driving: Once dataset. arXiv preprint arXiv:2106.11037, 2021. 2

    2021 arXiv

  37. [45]

    Re- thinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Re- thinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022. 2

    2022 arXiv

  38. [46]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. OpenAI Blog, 2024. 2, 6

    2024

  39. [47]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. OpenAI Blog, 2024. 6

    2024

  40. [48]

    Infinite photore- alistic worlds using procedural generation

    Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, Alejandro Newell, Hei Law, Ankit Goyal, Kaiyu Yang, and Jia Deng. Infinite photore- alistic worlds using procedural generation. In Proceedings ...

    2023

  41. [49]

    Infinigen indoors: Photorealistic in- door scenes using procedural generation

    Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, Zeyu Ma, and Jia Deng. Infinigen indoors: Photorealistic in- door scenes using procedural generation. In Proceedings of the IE...

    2024

  42. [50]

    Increasing generality in machine learning through procedural content generation

    Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence, 2(8):428–436, 2020. 2

    2020

  43. [51]

    Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference o...

    2021

  44. [52]

    Code llama: Open foundation models for code, 2024

    Baptiste Rozi `ere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, J ´er´emy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan ...

    2024

  45. [53]

    Procedu- ral content generation in games

    Noor Shaker, Julian Togelius, and Mark J Nelson. Procedu- ral content generation in games. Springer, 2016. 2

    2016

  46. [54]

    Match: Differentiable material graphs for procedural mate- rial capture

    Liang Shi, Beichen Li, Milo ˇs Ha ˇsan, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. Match: Differentiable material graphs for procedural mate- rial capture. ACM Transactions on Graphics (TOG), 39(6): 1–15, 2020. 2

    2020

  47. [55]

    3d-gpt: Procedural 3d model- ing with large language models, 2024

    Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zis- han Qin, and Stephen Gould. 3d-gpt: Procedural 3d model- ing with large language models, 2024. 2

    2024

  48. [56]

    From sora what we can see: A survey of text-to-video gener- ation

    Rui Sun, Yumin Zhang, Tejal Shah, Jiahao Sun, Shuoying Zhang, Wenqi Li, Haoran Duan, Bo Wei, and Rajiv Ranjan. From sora what we can see: A survey of text-to-video gener- ation. arXiv preprint arXiv:2405.10674, 2024. 1

    2024 arXiv

  49. [57]

    Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors

    Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. In Pro- ceedings of the 32nd ACM International Conference on Mul- timedia, pages 6617–6626, 2024. 2

    2024

  50. [58]

    Plan2scene: Convert- ing floorplans to 3d scenes

    Madhawa Vidanapathirana, Qirui Wu, Yasutaka Furukawa, Angel X Chang, and Manolis Savva. Plan2scene: Convert- ing floorplans to 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10733–10742, 2021. 2

    2021

  51. [59]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023. 6

    2023 arXiv

  52. [60]

    Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai

    Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In IEEE Conference on Comput...

    2024

  53. [61]

    Lavie: High-quality video gener- ation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video gener- ation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023. 6

    2023 arXiv

  54. [62]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. 5

    2023

  55. [63]

    Using github copilot to solve simple programming problems

    Michel Wermelinger. Using github copilot to solve simple programming problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, pages 172–178, 2023. 1

    2023

  56. [64]

    Gos: A large-scale annotated outdoor scene synthetic dataset

    Mingye Xie, Ting Liu, and Yuzhuo Fu. Gos: A large-scale annotated outdoor scene synthetic dataset. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3244–3248. IEEE,

    2022

  57. [65]

    Sinnerf: Training neural radiance fields on complex scenes from a single image

    Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sinnerf: Training neural radiance fields on complex scenes from a single image. In European Conference on Computer Vision , pages 736–753. Springer,

  58. [66]

    Sketch2scene: Au- tomatic generation of interactive 3d game scenes from user’s casual sketches

    Yongzhi Xu, Yonhon Ng, Yifu Wang, Inkyu Sa, Yunfei Duan, Yang Li, Pan Ji, and Hongdong Li. Sketch2scene: Au- tomatic generation of interactive 3d game scenes from user’s casual sketches. arXiv preprint arXiv:2408.04567, 2024. 2

    2024

  59. [67]

    Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding

    Jihan Yang, Runyu Ding, Weipeng Deng, Zhe Wang, and Xi- aojuan Qi. Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19823–19832, 2024. 2

    2024

  60. [68]

    Physcene: Physically interactable 3d scene synthesis for embodied ai

    Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthesis for embodied ai. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  61. [69]

    pixelnerf: Neural radiance fields from one or few images

    Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4578–4587, 2021. 2

    2021

  62. [70]

    Procedural modeling of rivers from single image to- ward natural scene production

    Jian Zhang, Chang-bo Wang, Hong Qin, Yi Chen, and Yan Gao. Procedural modeling of rivers from single image to- ward natural scene production. The Visual Computer , 35: 223–237, 2019. 2

    2019

  63. [71]

    Text2nerf: Text-driven 3d scene generation with neu- ral radiance fields

    Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neu- ral radiance fields. IEEE Transactions on Visualization and Computer Graphics, 2024. 2

    2024

  64. [72]

    Scenex: Procedural control- lable large-scale scene generation via large-language models

    Mengqi Zhou, Yuxi Wang, Jun Hou, Chuanchen Luo, Zhaox- iang Zhang, and Junran Peng. Scenex: Procedural control- lable large-scale scene generation via large-language models. arXiv preprint arXiv:2403.15698, 2024. 7

    2024 arXiv

  65. [73]

    Scenex:procedural control- lable large-scale scene generation via large-language mod- els, 2024

    Mengqi Zhou, Yuxi Wang, Jun Hou, Chuanchen Luo, Zhaox- iang Zhang, and Junran Peng. Scenex:procedural control- lable large-scale scene generation via large-language mod- els, 2024. 2

    2024

  66. [74]

    Llafs: When large language models meet few-shot segmentation

    Lanyun Zhu, Tianrun Chen, Deyi Ji, Jieping Ye, and Jun Liu. Llafs: When large language models meet few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3065– 3075, 2024. 2

    2024

  67. [75]

    Wayvescenes101: A dataset and benchmark for novel view synthesis in autonomous driving, 2024

    Jannik Z ¨urn, Paul Gladkov, Sof´ıa Dudas, Fergal Cotter, Sofi Toteva, Jamie Shotton, Vasiliki Simaiaki, and Nikhil Mohan. Wayvescenes101: A dataset and benchmark for novel view synthesis in autonomous driving, 2024. 1 Scene Co-pilot: Procedural Text to Video Generation with H...

    2024

  68. [76]

    Think through the problem step by step within the 〈thinking〉tags

  69. [77]

    Reflect on your thinking to check for any errors or improvements within the 〈reflection〉tags

  70. [78]

    Make any necessary adjustments based on your reflection

  71. [79]

    Important: The 〈thinking〉and 〈reflection〉sections are for your internal reasoning process only

    Provide your final, concise answer within the 〈output〉tags. Important: The 〈thinking〉and 〈reflection〉sections are for your internal reasoning process only. Do not include any part of the final answer in these sections. The potential related object names should be gathered from...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.