{"id":"9cea1a48-b6a9-4f82-b56f-30e43f24f70a","arxiv_id":"2411.18644","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Scene Copilot is a training-free pipeline that combines an LLM with retrieval over Infinigen's codebase and human-in-the-loop Blender editing to generate customized 3D scenes and videos from text prompts.","lead":"Scene Copilot turns text prompts into controllable 3D scenes and videos by having a language model write commands for the open-source Blender generator Infinigen, with a human able to edit the results in Blender. It also contributes a code-format dataset of over 300 procedural assets and 200 materials to make LLM-driven scene editing work.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BlenderGPT's edit success is unmeasured, and the NUM-masked .usda representation in §4.2 may not carry the spatial/numerical information reliable edits need; the paper's own Limitations concede users require advanced programming knowledge to repair LLM mistakes.","rationale":"The paper's strongest claim is user control over Infinigen scenes and videos; the load-bearing link is BlenderGPT's edit correctness. The reader's weakest assumption points at the .usda NUM condensation, and I agree it is the right place to stress. The system's success path is: Scene Codex emits an Infinigen command (ER@1 43.2% in §5.3), Infinigen builds a coarse scene, BlenderGPT emits bpy edits, and Infinigen renders the final video. The first and last steps have some quantitative evidence (ER@1, VBench scores, success-rate figure), but the middle step, where user intent is translated into concrete scene changes, has no measured success rate. The NUM masking is a concrete reason to doubt it: exact camera transforms, focal lengths, object placements, and animation values are replaced by placeholders, so the LLM must infer or generate code without seeing the numbers. That can work for object-name-based requests, and BlenderGPT's system prompt says a scene.blend exists, so the generated code may query bpy at runtime. But the paper does not establish that this happens reliably, and the admitted need for advanced programming knowledge suggests it does not. This is not an internal inconsistency; it is a missing validation of the central premise. I credit the qualitative videos, the ER@1 ablation showing RAG and few-shot help, and the VBench numbers as evidence a working prototype exists for selected prompts. The long-video claim (§5.4) is also under-supported, but it is downstream of the edit-control claim, so I do not make it the primary attack. A focused edit benchmark with masked vs raw .usda input would settle whether the central premise holds. If it passes, the CONDITIONAL verdict can be upgraded; if it fails, the user-control claim collapses. The reader's verdict of CONDITIONAL remains the right call: acceptance should require this benchmark plus the already-noted release of code/data and softened overclaims.","tokens_in":17874,"tokens_out":9178,"duration_ms":84325,"concrete_test":"Build an edit benchmark of 50 prompts spanning the VBench scenery set plus object-centric edits (e.g., 'move the camera 2m left of the snake', 'rotate the gravestone 45 degrees', 'place the cup on the table'). For each prompt, run BlenderGPT on a fixed coarse scene, with and without NUM-masking of the .usda input, and record (a) the fraction of generated bpy scripts that execute without error, (b) the fraction of executed edits that visually satisfy the prompt as judged by independent raters, and (c) the median number of user repair interventions needed before success. Compare the masked vs raw conditions. If masking significantly lowers (a) or (b), the §4.2 condensation destroys information needed for reliable edits; if (b) is high even with masking and minimal repair, the central control claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that users can control Infinigen through Scene Codex plus BlenderGPT human-in-the-loop editing. That claim requires BlenderGPT to generate bpy code that both executes and implements the requested edit. The paper never measures this. §4.2 converts a Blender scene to .usda, then replaces every numerical value with NUM (e.g., matrix4d xformOp:transform = NUM), so the LLM cannot see exact transforms, focal lengths, or spatial relations when generating edits. For edits that depend on such values, code must either query bpy at runtime or guess; the condensed RAG description contains object names and structure but not the numbers. No edit-success rate, user study, or expert-baseline comparison is reported. The only quantitative code-generation metric in §5.3 is Scene Codex ER@1 = 43.2%, which concerns initial Infinigen commands, not BlenderGPT edits. The Limitations section explicitly states that LLM hallucination is inevitable and requires \"advanced programming knowledge from user to help correct issues and mistakes,\" which directly undercuts the accessibility framing. If edit success is low, the observed qualitative examples are successes on a selected subset, and the contribution reduces to \"experts can debug LLM-generated Blender code,\" not \"users can control the scene.\"","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Scene Copilot, a framework that combines LLMs with the procedural 3D scene generator Infinigen and a Blender editing workflow. Given a text prompt, Scene Codex (an LLM with RAG over the Infinigen codebase and few-shot examples) generates an Infinigen command; after an initial coarse scene is produced, BlenderGPT converts the scene to a condensed textual .usda form and generates Blender Python code for user-requested edits, with the user able to select objects in Blender's GUI. The refined scene is fed back to Infinigen for a final render. The paper also contributes a dataset of over 300 procedural assets and materials in code form. Evaluation consists of qualitative key-frame examples, a VBench comparison on 100 scenery prompts, an executability-rate ablation for Scene Codex, and a long-video success-rate comparison with Infinigen.","tokens_in":18119,"tokens_out":3056,"duration_ms":27484,"significance":"If the controllability claims hold, the framework is a practical contribution: it lowers the barrier to using Infinigen, provides persistence of object identity across long renders, and is training-free. The paper's Scene Codex ablation (Fig. 11) gives a clear, quantitative demonstration that RAG and few-shot prompting each contribute to prompt-to-command executability (0%, 2.0%, 20.0%, 43.2% ER@1). The curated procedural asset dataset is also a useful resource for the community. The main significance hinges on whether BlenderGPT's editing behavior is reliable enough to make the human-in-the-loop loop practical; that point is not yet quantitatively established.","major_comments":[{"comment":"The central claim that users can control the scene through text and GUI selection requires BlenderGPT to generate Blender Python code that both executes and implements the requested edit, but the paper reports no quantitative measurement of BlenderGPT's edit success rate. The only executability metric, ER@1 in Figure 11, evaluates Scene Codex's Infinigen commands, not BlenderGPT edits. Without such a measurement, the qualitative demonstrations in Figures 9 and 10 are potentially selected successes, and the contribution could reduce to 'experts can debug LLM-generated Blender code' rather than 'users can control the scene.'","section":"§4.2, §5.3, Figure 11"},{"comment":"The NUM-masked .usda representation replaces all numerical values (e.g., matrix4d xformOp:transform = NUM, focal length, aperture) with placeholders before constructing the RAG database. For edits that depend on exact spatial or optical quantities—such as 'camera follow the selected object' in Figure 8 or repositioning geometry—the LLM cannot see the relevant numbers in context, and the paper does not explain how BlenderGPT recovers them at runtime. The Limitations section acknowledges that LLM hallucination is 'inevitable' and that users need 'advanced programming knowledge' to correct issues, which directly undercuts the paper's accessibility framing and leaves the reliability of the edit pipeline unquantified.","section":"§4.2 Preprocessing, Limitations"},{"comment":"The VBench evaluation is restricted to 100 scenery prompts, includes no baseline from Infinigen alone, and reports no error bars or statistical significance. Without an Infinigen-only baseline, the scores in Table 2 do not isolate the contribution of Scene Copilot's editing components; without error bars, comparisons such as Motion Smoothness 99.36 vs. 97.58 are hard to interpret. The claim that Scene Copilot achieves the best scores in Motion Smoothness and Dynamic Degree should be supported with a confidence interval or multiple runs, or tempered accordingly.","section":"§5.2, Table 2"},{"comment":"The long-video success rate comparison does not define what constitutes 'success' for a generated video, and the comparison between Infinigen and Scene Copilot is confounded by the human-in-the-loop component, which can manually set camera animations. The sentence in §5.4 claiming videos of 'arbitrary length without concern for the preservation of object integrity or the enforcement of the physical laws governing the generated world' overstates the evidence, which is based on 12 scenes per setting with no reported metric for object integrity or physical-law violations.","section":"§5.4, Figure 12"}],"minor_comments":[{"comment":"The title uses 'Scene Co-pilot' whereas the abstract, body, and system name use 'Scene Copilot'; please standardize the spelling.","section":"Title and Abstract"},{"comment":"The text says 'Table 11 shows the results' but the ablation results appear in Figure 11, not a table; the cross-reference should be corrected.","section":"§5.3"},{"comment":"The phrase 'it defaults from a blank Blender project' should read 'it defaults to a blank Blender project'; also, 'such that' is used where 'so that' would be clearer.","section":"§4.2"},{"comment":"The 'probability function derived from a Gaussian distribution' that decides whether to use the procedural dataset or Shap-E is not specified; please provide the exact function or a reference so the behavior is reproducible.","section":"§3.1"},{"comment":"The paper states the dataset is published under CC-BY 4.0 and 'includes procedural assets, the corresponding source code, direct links to the assets, and detailed metadata,' but no URL or repository link is given in the text; a link would be needed for the claim of accessibility to be verifiable.","section":"§3.3 and Dataset release"},{"comment":"The comment 'Format the improved code for readability' is followed by 'imp formatted code ← format(fine scene, LLMs)', which formats the scene, not the code; please rename for consistency.","section":"Algorithm 1, line 8"},{"comment":"The caption says 'The best result in each dimension is highlighted in bold, and the second-best result is highlighted in Italics,' but the table body does not visibly apply the italic formatting; please ensure the formatting is rendered.","section":"Table 2 caption"}],"recommendation":"major_revision","confidential_remarks":"This is a systems paper whose main risk is the unmeasured reliability of the BlenderGPT editing component. The missing edit-success metric and the missing Infinigen-only baseline are the two load-bearing gaps; both are addressable within the manuscript's scope. If the authors add a quantitative evaluation of BlenderGPT edits (e.g., execution success and task success on a prompt suite with error bars) and an Infinigen-only comparison, the paper could be acceptable. The dataset release also needs a link and a clear statement of contents to back the accessibility claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Scene Copilot is a plausible integration of LLMs with Infinigen: Scene Codex turns text into Infinigen commands via RAG and few-shot examples, BlenderGPT helps edit the resulting scene in Blender, and a human can intervene through the GUI. The most concrete new artifact is the 323-asset procedural dataset in code format, which is genuinely reusable. The ER@1 ablation (0% -> 2% -> 20% -> 43.2%) is a clean, informative result: it shows both RAG and few-shots are necessary for executable Infinigen commands. The qualitative videos are consistent with the claim that the system can steer scene content and camera motion, and the architecture is clearly described.\n\nThe soft spot is that the central editing loop is unmeasured. The paper never reports how often BlenderGPT produces a correct edit, and the .usda preprocessing masks all numerical values with NUM placeholders, so the LLM sees object names and structure but not exact transforms, focal lengths, or spatial relations. The generated code could in principle query bpy at runtime to get those values, but the paper does not say that, and the Limitations section explicitly concedes that users need advanced programming knowledge to fix LLM mistakes—which undercuts the accessibility framing. The VBench evaluation is also narrow: 100 scenery prompts, no Infinigen-only baseline, no error bars, and the comparison is against diffusion-based T2V models rather than procedural-generation pipelines, so it says little about the value of the editing loop itself. The 'arbitrary length' claim in Sec. 5.4 is overstated; the data show Infinigen's success rate drops with length, and a human can manually extend a camera path, which is not the same as automatic arbitrary-length generation.\n\nStill, this is a working system with a useful dataset, and the ER@1 experiment is real evidence. The paper deserves a serious referee, but it needs major revision: measure edit success or run a small user study, add an Infinigen-only baseline, temper the overclaims, and release the code and dataset. I would send it to review with that expectation.","headline":"A credible integration of LLMs and Infinigen with a useful procedural asset dataset, but the editing loop is unmeasured and the stronger claims outrun the evidence.","tokens_in":18697,"tokens_out":2629,"would_cite":true,"duration_ms":24459,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Scene Copilot claims that a two-stage LLM pipeline plus human editing in Blender lets users turn natural-language prompts into customized, photorealistic scene videos, including arbitrarily long renders.","keywords":["procedural 3D scene generation","text-to-video generation","human-in-the-loop editing","large language models","retrieval-augmented generation","Blender scene editing","video consistency","procedural asset dataset"],"falsifier":"Generate a set of scenes with known .usda files, issue targeted editing prompts such as \"move the selected object 2 meters to the right\" or \"set the camera focal length to 35 mm\" through BlenderGPT, and, without human code repair, compare the resulting scene's numeric values against the intended ones; if correct numeric edits occur near chance, the condensation claim collapses.","tokens_in":17649,"feed_emoji":"🎬","tokens_out":6923,"duration_ms":62034,"temperature":0.7,"pith_summary":"The paper argues that a non-specialist can go from a typed sentence to a customized photorealistic video by pairing a large language model with Infinigen, a procedural 3D scene generator built on Blender, and adding a human editing step. The proposed system, Scene Copilot, uses one LLM stage to translate a prompt into a generator command and a second LLM stage to turn follow-up instructions into Blender editing code, with the user clicking objects in the Blender interface to indicate what should change. If this works, procedural scenes become accessible to people who do not know the underlying generation rules, and rendered videos inherit the scene's persistence: objects keep their identity and physical behavior holds no matter how long the video runs. The paper supports the claim with qualitative renderings, a comparison against four video-generation models on VBench's scenery category, and an ablation showing that retrieval context and few-shot examples raise executable command rate to 43.2 percent.","feed_headline":"Scene Copilot turns text prompts into editable 3D scene videos","feed_subtitle":"A two-stage LLM loop keeps scene objects persistent, so rendered videos stay physically consistent no matter how long they run.","key_machinery":"The load-bearing mechanism is the two-stage LLM command pipeline. Scene Codex is a retrieval-augmented LLM that turns a prompt into an Infinigen run command. BlenderGPT is a chain-of-thought LLM that turns follow-up edits into Blender Python code, with the user selecting objects in the Blender GUI. Between the two sits a condensation step: each Blender scene is converted to a .usda textual scene description, every numerical value is replaced with the placeholder NUM, and the result is stored as a retrieval database so the editing LLM can see the scene's structure without drowning in numbers. The procedural asset dataset, 323 assets and materials converted into code, supplies the LLM with named, modifiable objects it can combine or repurpose.","core_discovery":"The paper's claim is that LLM-generated commands combined with human-in-the-loop Blender editing make procedural scene generation controllable enough for ordinary users to act as directors of their own videos. Scene Codex maps text to an executable Infinigen command, Infinigen produces a coarse scene, BlenderGPT converts natural-language revisions into Blender Python code, and the user can also edit directly through the Blender GUI; the loop repeats once at a finer detail level before final rendering. The authors report that this workflow generated scenes and videos that follow prompts such as a graveyard at sunset, scored best among compared models on motion smoothness and dynamic degree in VBench's scenery category, and kept rendering success high at longer video lengths where plain Infinigen degrades. They further state that because the output is a render of a persistent 3D scene, videos of arbitrary length can be produced without concern for object integrity or the physical laws governing the world.","pith_inferences":["If the NUM condensation is as information-preserving as the paper assumes, the same text-outline-of-a-scene trick could be applied to other procedural engines and CAD tools, letting LLMs edit any scene that can be serialized to text.","A safer design would keep exact numerical values available to the model through tool calls or selective retrieval, because the paper gives no measurement of how often BlenderGPT's edits are semantically reasonable but numerically wrong.","A cheap experimental test would replace NUM with exact numbers for a small set of edits and compare success rates; the gap would isolate how much precision the condensation step actually costs.","The dataset's automatic expansion scripts imply a route to continuously growing asset coverage, but they also raise a testable risk that older LLM-generated commands break as asset names and conversion conventions change."],"forward_implications":["Non-specialists can generate and revise photorealistic 3D scenes by typing requests and clicking objects, without reading Infinigen's parameter documentation.","Rendered videos keep object identities and physical behavior consistent across arbitrarily long durations, because the video is a render of a persistent 3D scene rather than a frame-by-frame prediction.","Adding RAG context and few-shot examples raises the fraction of executable generator commands from 0 percent to 43.2 percent, so documentation injection is a necessary part of the pipeline.","Users act as directors: they can force a camera to follow a chosen object or add assets, such as a church and gravestones, that the base generator would omit.","The 323 procedural assets in code form let the LLM combine and modify objects in ways a fixed asset library cannot."],"supporting_citations":[{"why":"Supplies the procedural scene generator that Scene Codex must command and that ultimately renders the output video.","marker":"[48]"},{"why":"Provides the 3D environment and GUI that BlenderGPT edits and that Infinigen populates with scenes.","marker":"[12]"},{"why":"Defines the benchmark dimensions and comparison protocol used to evaluate the generated videos against transformer- and diffusion-based baselines.","marker":"[29]"},{"why":"Provides the retrieval-augmented generation approach used to inject Infinigen code context into Scene Codex.","marker":"[36]"},{"why":"Supplies the node-to-Python conversion that turns Blender geometry and material assets into editable code for the dataset.","marker":"[4]"},{"why":"Provides the text-to-3D fallback generator used when the procedural dataset lacks a requested object.","marker":"[32]"},{"why":"Demonstrates LLM code generation from natural language and motivates the Scene Codex design.","marker":"[11]"}],"fun_headline_variants":["LLM + Blender gives you director control over 3D scene videos","Scene Copilot: write scenes, edit renders, keep physics real","Human-in-the-loop LLM video generation with editable 3D scenes","From text to film: Scene Copilot keeps 3D physics intact","Direct your videos: Scene Copilot blends LLM and Blender"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a Blender scene, once converted to text with every number replaced by the placeholder NUM, still contains enough information for a language model to edit it correctly; if a requested change depends on an exact position, scale, or camera value, the model is guessing and the user must catch the error.","fun_headline_variants_meta":{"raw":{"variants":["LLM + Blender gives you director control over 3D scene videos","Scene Copilot: write scenes, edit renders, keep physics real","Human-in-the-loop LLM video generation with editable 3D scenes","From text to film: Scene Copilot keeps 3D physics intact","Direct your videos: Scene Copilot blends LLM and Blender"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000621,"raw_usage":{"total_tokens":2868,"prompt_tokens":926,"completion_tokens":1942,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1847}},"tokens_in":542,"tokens_out":1942,"duration_ms":11313,"temperature":1.0,"reasoning_tokens":1847,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:47:40.421739+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a set of scenes with known .usda files, issue targeted editing prompts such as \"move the selected object 2 meters to the right\" or \"set the camera focal length to 35 mm\" through BlenderGPT, and, without human code repair, compare the resulting scene's numeric values against the intended ones; if correct numeric edits occur near chance, the condensation claim collapses.","supporting_citations":[{"cited_title":"Infinite photore- alistic worlds using procedural generation","cited_arxiv_id":null,"evidence_quote":"Supplies the procedural scene generator that Scene Codex must command and that ultimately renders the output video."},{"cited_title":"VBench: Com- prehensive benchmark suite for video generative models","cited_arxiv_id":null,"evidence_quote":"Defines the benchmark dimensions and comparison protocol used to evaluate the generated videos against transformer- and diffusion-based baselines."},{"cited_title":"Retrieval-augmented genera- tion for knowledge-intensive nlp tasks, 2021","cited_arxiv_id":null,"evidence_quote":"Provides the retrieval-augmented generation approach used to inject Infinigen code context into Scene Codex."},{"cited_title":"Nodetopython","cited_arxiv_id":null,"evidence_quote":"Supplies the node-to-Python conversion that turns Blender geometry and material assets into editable code for the dataset."}],"review_version":1}