{"id":"aac6dc0e-ef03-4e46-b5e0-c01efc86a846","arxiv_id":"2505.13466","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AgentSGEN uses an internal LLM evaluator and editor loop to edit Holodeck 3D scenes so doors are blocked as requested, and human and LLM raters prefer the results.","lead":"An AI system with two large language model agents, one planning and one editing, changes computer-generated 3D room scenes to match safety goals such as blocked emergency exits. Work shows people and AI judges prefer these edited scenes over the original generator's scenes, but no downstream safety model was trained to prove the data helps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is that iterative LLM scene editing yields useful safety-critical training data, but the paper only measures preference on static renderings, not downstream task performance, leaving the data-utility claim unsupported.","rationale":"The reader's weakest assumption—that preference judgments on static renderings proxy for the usefulness of generated scenes as training data—is exactly the load-bearing concern. The paper's contributions are framed around synthetic data for safety-critical AI, yet Section 4 contains no downstream training or evaluation. The reported preference counts (38/53 vs. 6) and Likert scores only show that edited scenes look more aligned with the stated goal to human and LLM raters. The abstract explicitly says the framework 'can bridge this gap,' and Section 3.9 promises annotated outputs 'ready for training,' so the missing downstream validation is a gap in the central argument, not a peripheral issue. This is an externally-validity problem rather than an internal inconsistency, and it is addressable by a concrete training experiment. I also note the missing citation for the AABB collision checker in Section 3.5 ('[? ]'), which should be fixed, and the small annotator pool acknowledged in Section 5. These are secondary to the proxy-validity concern. Since the reader already marked the verdict CONDITIONAL, my assessment does not change that verdict; it reinforces the condition: the authors should demonstrate downstream utility before the contribution is fully accepted.","tokens_in":9003,"tokens_out":2351,"duration_ms":24940,"concrete_test":"Release the generated dataset and editing code, then train an object detector (e.g., YOLOv8) with identical hyperparameters on (i) AgentSGEN collision-aware synthetic scenes and (ii) unedited Holodeck scenes for the same room types. Evaluate both models on a held-out set of real-world construction-safety images containing blocked doors or exits. If the AgentSGEN-trained model does not achieve higher mAP or recall for the target hazard than the Holodeck-trained model, the preference results do not establish downstream utility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 evaluates only subjective preference—binary side-by-side choices and Likert ratings by humans and LLMs—on static renderings. It never trains or tests a downstream object detector, safety monitor, or other model on the generated scenes. The paper's stated purpose is to bridge the data-scarcity gap for safety-critical AI (Abstract, Section 5), and Section 3.9 claims the output is 'ready for training or evaluation of downstream computer vision or robotics models.' No evidence supports that annotator-preferred scenes improve downstream task performance over baseline scenes. The conclusion acknowledges a modest annotator count, but the deeper problem is that the evaluation proxy itself is unvalidated: human/LLM preference for 'doors blocked with large objects' may not correlate with, for example, a detector's ability to recognize blocked exits in real-world construction imagery. Without a downstream-task experiment, the central claim of practical utility is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AgentSGEN, a two-agent LLM system for editing 3D indoor scenes generated by Holodeck to satisfy safety goals such as blocking doors with large objects. An Evaluator Agent reasons over a scene graph and visual projections to produce an edit plan; an Editor Agent executes move, rotate, and delete operations; and an axis-aligned bounding box collision checker provides feedback. The evaluation covers 53 indoor scene types from the MIT Indoor Scenes dataset, comparing the edited scenes against the unedited Holodeck baseline through binary preference judgments and goal-oriented Likert ratings from human annotators and from GPT-4.1 and Gemini 2.5 Pro. The reported results show that the collision-aware version is preferred in 38 of 53 pairwise comparisons by both human judges and GPT-4.1, with moderate human inter-annotator agreement, and the paper concludes that the framework can generate synthetic safety-critical data.","tokens_in":9171,"tokens_out":3749,"duration_ms":39311,"significance":"If the reported results hold, the system offers a practical approach to goal-driven, fine-grained editing of synthetic 3D scenes, addressing a genuine bottleneck in safety-critical data generation. The separation of planning and execution is a sensible architectural choice, and the collision-aware variant's strong preference over the Holodeck baseline is a meaningful signal that interactive scene-graph editing can shift perceived semantic alignment. The paper also contributes a reproducible pipeline and reports encouraging human-LLM agreement in preference judgments. However, the significance of the contribution for downstream AI training is not yet established because the evaluation stops at perceptual preference on static renderings: no downstream object detector, safety monitor, or other vision model is trained or evaluated on the generated scenes. The abstract's framing as a solution to data scarcity therefore remains unsupported by the presented evidence.","major_comments":[{"comment":"The central utility claim is not supported by the evaluation. Section 3.9 states that the generated outputs are 'ready for training or evaluation of downstream computer vision or robotics models,' and the abstract frames the work as a way to bridge the data-scarcity gap. Yet Section 4 evaluates only binary preference and Likert ratings on static renderings; no downstream task is performed. It is therefore unknown whether annotator-preferred and LLM-preferred scenes lead to better performance in, for example, detecting blocked exits or recognizing safety violations. To support the stated contribution, the paper should either add a downstream experiment that trains or evaluates a model on the generated scenes, or substantially narrow the claims to perceptual scene quality rather than synthetic-data utility.","section":"Section 4 (Evaluation) and Section 3.9"},{"comment":"The claim that the system 'significantly improves task satisfaction in human perception' is statistically under-supported. The main text never reports the number of human annotators, and the Figure 8 caption says 'Each entry indicates how often a model was preferred by two annotators,' implying that only two annotators took part in the binary task. No confidence intervals, exact tests, or per-annotator breakdowns are provided for the 38-versus-6 preference count, and the Cohen's kappa values are reported as point estimates without uncertainty. Please report the annotator count, the per-annotator preference data, and at least a binomial confidence interval or exact test; otherwise the 'significantly improves' wording is not justified.","section":"Section 4.2 and Figure 8"},{"comment":"The LLM-based evaluation has a self-referentiality concern that is not fully addressed. The same class of model (LLM) is used inside the system as the Evaluator agent and externally as the automatic judge. Section 4.4 also reports that GPT-4.1 and Gemini preferred the Holodeck baseline over the method on the arrangement and scale criteria, with an attribution to visual priors from web-scale data. This is plausible but not verified. To make the automatic evaluation more convincing, the paper should report the exact prompts and temperatures used for the LLM judges, show the stability of the DCE results across repetitions, and provide a more detailed analysis of where human and LLM judgments diverge, rather than attributing the divergence to unmeasured visual priors.","section":"Section 4.4 and Figure 11"}],"minor_comments":[{"comment":"There is an unresolved citation placeholder in the sentence describing the 'AABB collision checking mechanism [?].' Please replace it with a proper reference or remove the placeholder.","section":"Section 3.5"},{"comment":"There are several typographical errors, including 'ann advancement' and 'lastests' in the conclusion; these should be corrected.","section":"Section 5"},{"comment":"The annotation protocol is described at a high level but does not state whether the side-by-side renderings were order-balanced, whether annotators saw all 53 room types, or whether the same annotators judged both the binary and Likert tasks. Please specify these details.","section":"Section 4.1"},{"comment":"Section 4.4 says LLM Likert-style ratings were unstable and were therefore deprioritized, yet Figure 11 is titled 'DCE results' and reports three goal-oriented scores from GPT-4.1 and Gemini. Please clarify how the scores in Figure 11 were obtained and how they relate to the statement about unstable ratings.","section":"Section 4.4 and Figure 11"}],"recommendation":"major_revision","confidential_remarks":"The paper is an interesting pipeline contribution, but the evaluation falls short of supporting the stated synthetic-data utility. The missing downstream validation and the very small apparent annotator pool are the main concerns. I recommend major revision rather than rejection because the central idea is defensible and the perceptual-preference evidence, while incomplete, is not contradicted by any internal inconsistency. The authors should be encouraged to add a downstream task experiment or—if that is infeasible—to reframe the paper's contribution as goal-directed scene editing and remove the data-scarcity bridging claim from the abstract and conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: if you need to generate synthetic indoor scenes where doors are blocked, this is the most straightforward recipe I've seen. The new bit is an Evaluator–Editor loop over a scene graph: one LLM plans (move/rotate/delete) with collision and safety constraints, another executes, and the loop iterates until the scene satisfies the goal. That is a real combination, not just a thin application of Holodeck. The paper is honest about Holodeck's limits, and the Dual Process Theory framing is mostly harmless scaffolding; the architecture is what matters.\n\nWhat the paper does well: the 53-scene comparison with humans and two LLM judges gives a clear preference signal. Collision-aware edits were preferred by a wide margin (38 vs 6 over the baseline in the binary task), and Cohen's kappa (0.406) is reported for human agreement. That supports the claim that iterative editing improves task satisfaction, at least on static renderings. I'd trust the central preference result; it is strong enough to survive the missing error bars.\n\nWhere it is soft:\n- The data-utility claim is not tested. Section 3.9 says the output is ready for downstream training, but no detector or monitoring model is trained. Whether human/LLM preference transfers to better safety-critical models is unmeasured, so the abstract's bridge-the-gap claim is supported only by implication.\n- Evaluation reporting is thin: no annotator count, no confidence intervals, one baseline, and no ablations except the collision on/off toggle. The toggle is useful; the rest is thin but fixable.\n- Section 3.5 has an unresolved \"[?]\" citation for the collision-checking mechanism. Minor but sloppy.\n- The internal Evaluator is an LLM and two of the external judges are LLMs; the human anchor keeps this from being circular, but LLM-vs-LLM agreement is weak evidence on its own. The paper mostly acknowledges this in Section 4.4.\n\nThe central claim—that an iterative LLM editing loop can produce scenes satisfying a specific safety goal better than one-shot generation—holds up. What does not hold up is the jump to \"synthetic data for safety-critical AI.\" That is a missing experiment, not a contradiction.\n\nWho it's for: anyone building scene-generation pipelines for safety or embodied AI, and anyone refereeing synthetic-data papers. It deserves a serious referee; with one downstream-task experiment (e.g., blocked-door detection or segmentation) and complete reporting it would be much stronger. I'd engage with it.","headline":"AgentSGEN is a practically useful multi-agent scene-editing pipeline whose core result—edited scenes beat the Holodeck baseline on blocking-door goals—holds, but the paper overclaims data-utility without a single downstream training experiment.","tokens_in":9689,"tokens_out":2412,"would_cite":false,"duration_ms":25422,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-agent LLM loop (Evaluator plans, Editor executes) edits 3D scenes to satisfy safety goals like blocked doors, beating the Holodeck baseline 38 to 6 in human preference.","keywords":["synthetic data generation","multi-agent LLM","scene graph editing","safety-critical scenes","3D scene generation","human evaluation","LLM-as-judge","construction safety"],"falsifier":"Compute, for all 53 rooms, the fraction of each exit's free area occluded by blocking objects in the final scene graphs, and check whether every scene that human raters and GPT-4.1 preferred shows a materially higher occlusion fraction than its Holodeck baseline; if many preferred scenes show no greater geometric obstruction, the perceptual preference claim is not evidence of actual safety alignment.","tokens_in":8785,"feed_emoji":"🚧","tokens_out":7576,"duration_ms":69908,"temperature":0.7,"pith_summary":"The paper tries to establish that a multi-agent loop of two large language models—an Evaluator that plans scene edits and an Editor that executes them—can turn generic 3D room scenes into safety-critical scenes, such as a bedroom where doors are blocked by large objects, more reliably than a one-shot scene generator. This matters because dangerous situations like blocked fire exits are ethically and practically difficult to photograph, and synthetic scenes that respect safety constraints could fill the training-data gap for safety-critical AI. The supporting evidence is a preference study across 53 indoor scene types: both human annotators and GPT-4.1 chose the collision-aware edited scene over the unedited Holodeck baseline in 38 of 53 comparisons, versus 6 for the baseline. If this holds, iterative LLM-guided scene editing becomes a viable, controllable route to synthetic safety data rather than relying on manual design or procedural rules.","feed_headline":"AI agent loop blocks doors in 3D scenes, beating baseline 38 to 6","feed_subtitle":"Human raters and GPT-4.1 both prefer the edited scenes, a step toward synthetic data for safety-critical AI.","key_machinery":"The central object is the scene graph—a symbolic encoding of objects, positions, orientations, and attributes—treated as an interactive environment rather than a static description. The central mechanism is the Evaluator–Editor loop: a reasoning-heavy LLM (System 2) plans atomic edits from the scene graph and 2D/3D renderings, a fast LLM (System 1) executes them, and an axis-aligned bounding-box collision checker reports validity back to the Editor, with the Evaluator re-entering at termination to validate constraints. SGRender supplies top-down 2D projections and overlay renderings as visual feedback, which is what makes the loop resource-efficient and keeps it free of domain-specific rules.","core_discovery":"On its own terms, AgentSGEN's central discovery is that separating planning from execution in an LLM-driven scene-editing loop produces scenes that satisfy a stated safety goal—'doors are blocked with large objects'—where the one-shot generator fails. The Evaluator agent reasons over a scene graph and multi-modal renderings to produce an action plan; the Editor agent executes atomic move, rotate, and delete operations; and a feedback loop with axis-aligned bounding-box collision checking validates each step. In 53 randomized pairwise comparisons, the collision-aware variant was preferred 38 times by both human annotators and GPT-4.1 (Cohen's kappa 0.406), while Holodeck was preferred only 6 times. The authors conclude that semantic and perceptual objectives are not inherently in conflict and that the pipeline yields reproducible, annotated synthetic data for downstream vision systems.","pith_inferences":["A direct test the paper leaves undone: train a downstream object detector or safety monitor on AgentSGEN renders versus Holodeck renders and measure performance on real images; preference judgments alone do not prove transfer, and this is the claim's weakest link.","Because planning and execution are decoupled, the Evaluator could be repurposed for other high-level goals (autonomous driving, warehouse safety) without changing the Editor's atomic operations; this modularity is our inference, not the authors' claim.","A cheaper and more objective evaluation could replace pairwise preference with a geometric obstruction metric: the fraction of a door's free area occluded by blocking objects; if that metric stops matching preferences, the perception-based evidence may overstate true safety alignment.","The authors' own observation that LLM Likert ratings are unstable suggests that future LLM-as-judge designs should stick to discrete-choice protocols, a practical lesson that extends beyond scene generation."],"forward_implications":["Safety-critical synthetic datasets—blocked-exit images with segmentation, depth, and object metadata—can be generated at scale without photographing real hazards.","The same two-agent loop should generalize to other safety constraints, such as unobstructed evacuation routes or equipment exclusion zones, by changing the goal prompt and constraint set.","LLM judges can serve as auxiliary evaluators for structured scene editing, since GPT-4.1 and Gemini preferences aligned with human preferences in pairwise choice.","Enabling collision checking is a key ingredient of the result: it raised human agreement (kappa 0.406 vs 0.151 without it) and produced the largest preference margin."],"supporting_citations":[{"why":"Supplies the one-shot baseline and the initial scene graphs that AgentSGEN edits; the main comparison is against it.","marker":"[24]"},{"why":"Provides the 53 indoor scene categories that define the goal prompts 'A {room_type}, where doors are blocked with large objects' used in evaluation.","marker":"[16]"},{"why":"Provides the cognitive model (System 2 planner vs System 1 executor) that the Evaluator–Editor architecture instantiates.","marker":"[1]"}],"fun_headline_variants":["Two-agent LLM loop wins 38–6 in safety scene generation","AgentSGEN's evaluator-editor duo beats one-shot scene gen","Collaborative AI agents generate safer synthetic 3D scenes","LLM pair improves synthetic data for safety-critical AI","Semantic collaboration loop yields better synthetic scenes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that human and LLM preferences between side-by-side renderings measure whether the scenes are actually useful as training data for safety-critical AI, but the evaluation never trains or tests a downstream detector on the generated scenes.","fun_headline_variants_meta":{"raw":{"variants":["Two-agent LLM loop wins 38–6 in safety scene generation","AgentSGEN's evaluator-editor duo beats one-shot scene gen","Collaborative AI agents generate safer synthetic 3D scenes","LLM pair improves synthetic data for safety-critical AI","Semantic collaboration loop yields better synthetic scenes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1666,"prompt_tokens":926,"completion_tokens":740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":657}},"tokens_in":542,"tokens_out":740,"duration_ms":7366,"temperature":1.0,"reasoning_tokens":657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:19:19.514143+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, for all 53 rooms, the fraction of each exit's free area occluded by blocking objects in the final scene graphs, and check whether every scene that human raters and GPT-4.1 preferred shows a materially higher occlusion fraction than its Holodeck baseline; if many preferred scenes show no greater geometric obstruction, the perceptual preference claim is not evidence of actual safety alignment.","supporting_citations":[{"cited_title":"Yang, F.-Y","cited_arxiv_id":null,"evidence_quote":"Supplies the one-shot baseline and the initial scene graphs that AgentSGEN edits; the main comparison is against it."},{"cited_title":"Quattoni and A","cited_arxiv_id":null,"evidence_quote":"Provides the 53 indoor scene categories that define the goal prompts 'A {room_type}, where doors are blocked with large objects' used in evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the cognitive model (System 2 planner vs System 1 executor) that the Evaluator–Editor architecture instantiates."}],"review_version":1}