{"id":"c3369f1b-17c2-45b7-85c1-efcf2e739af7","arxiv_id":"2504.15049","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ScanEdit uses hierarchical scene graphs and LLM-based planning, placement, and optimization to rearrange objects in real-world 3D scans from text instructions.","lead":"ScanEdit is a method that rearranges objects inside real-world 3D scans according to text instructions, using a hierarchy of scene graphs and language models. It targets scenes with hundreds of objects, which existing layout-generation methods cannot handle, and could make 3D scan editing practical for content creation and robotics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The final optimizer has no facing-preservation term, so the functional arrangements central to ScanEdit are not actually guaranteed after optimization.","rationale":"The reader's verdict is CONDITIONAL; I agree but for a sharper reason. The abstract and contributions claim edited scenes whose arrangements obey physics and common sense, and the optimization step is presented as the component that guarantees physical plausibility. Yet the only relation that encodes function ('facing') is excluded from L_Gs in §3.5: it is initialized in §3.4 and never enforced. Since θ is an optimization variable, all other losses can rotate objects and destroy the initialized facing relation. The claimed functional editing is therefore not actually guaranteed by the method as described. This is not a criticism of LLM/VLM quality; it is an internal consistency problem visible from the equations. The reader's weakest assumption (VLM/heuristic graph correctness) is plausible, but it is upstream and would require oracle labels to test; the facing omission can be tested directly from the paper's own pipeline. The paper does provide independent positives: hierarchical decomposition is reasonable, ablations isolate components, and the optimization losses are clearly specified. However, because the missing facing term/evaluation is easy to fix and directly affects the headline claim, the appropriate verdict is CONDITIONAL: the authors should add an explicit facing loss or freeze orientations, and report a facing/orientation fidelity metric. No code/data release further prevents us from checking this, but even with code the omission remains.","tokens_in":19305,"tokens_out":10295,"duration_ms":96561,"concrete_test":"Run ScanEdit on the subset of the 93 instructions requiring 'facing' (e.g., 'arrange chairs to face the TV'). For each moved object with a facing constraint, compute the signed angular error between its front normal F_i and the vector from its center to the intended reference object: after hierarchical placement, after optimization, and in a variant with θ frozen (or an explicit facing loss added). If the median absolute change in this angle exceeds 10° for more than 5% of constrained objects, or if final error is large while the frozen-orientation variant is fine, the optimization is breaking the functional facing relation. If angles change negligibly, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing gap is in §3.5. The placement LLM (§3.4) produces the initial pose T=(x,y,θ) and selects constraints including 'facing'; the final objective is L = L_Gs + α L_col + γ L_Gt. L_Gs contains only the on-top-of and against-wall losses, and the paper explicitly says the 'facing' constraint is 'resolved in the hierarchical placement step' rather than in optimization. However, §3.5 optimizes all three DOFs including θ. The collision, on-top-of, against-wall, and group losses can therefore rotate an object away from the facing direction that was initialized; the group loss preserves only center-center vectors, not object orientations. No loss or post-check measures deviation between the object's front normal and the vector to its intended parent/target. Consequently, a chair placed to face a TV or table may end up rotated out of facing alignment by the very optimization step designed to ensure physical plausibility. The evaluation in Table 1/2 contains no orientation or semantic-relation metric, so this failure mode is invisible. This is an internal gap: it does not depend on VLM annotation quality or LLM hallucination; even with a perfect hierarchical scene graph, the optimizer can break the functional relation that defines the edit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ScanEdit proposes a pipeline for instruction-driven editing of 3D scans. It constructs a hierarchical scene graph from an object-level decomposition, annotates node attributes via a VLM and edge relations via 3D heuristics, identifies a relevant subgraph with an LLM, generates localized per-object instructions with a planning LLM, places objects hierarchically via a placement LLM, and finally optimizes object transformations using convex differentiable losses that enforce on-top-of, against-wall, collision, and group-preservation constraints. The method is evaluated on ScanNet++ and Replica scenes with 69-306 objects, against LayoutGPT and LayoutVLM, using geometric metrics (NoFloat, InBound, ColVol, PIoU) and a perceptual user study.","tokens_in":19518,"tokens_out":4124,"duration_ms":32443,"significance":"The paper tackles a practically important and under-explored problem: editing real, cluttered 3D scans from natural language. The hierarchical decomposition is a sensible and scalable design, and the proposed pipeline integrates LLM/VLM reasoning with geometric optimization. The evaluation includes real-world scans, multiple scenes, and ablations. If the claims were robustly established, this would be a useful contribution to the community. However, as detailed in the major comments, the absence of orientation/semantic metrics, the self-referential geometric metrics, and the lack of statistical significance testing leave the central claims only partially supported at this stage.","major_comments":[{"comment":"The final objective L = LGs + αLcol + γLGt contains no term that preserves the 'facing' constraint introduced in Section 3.4. The text states that facing is resolved during hierarchical placement, but the optimization updates all three DOFs including θ. The group loss LGt preserves only center-to-center vectors, not object orientations. Consequently, the collision, on-top-of, against-wall, and group losses can rotate an object away from its initialized facing direction, breaking the functional relation that the edit is meant to create. The evaluation in Tables 1 and 2 contains no orientation or semantic-relation metric, so this failure mode is invisible. This is an internal gap: even with perfect VLM/LLM annotations, the optimizer can undo the facing alignment.","section":"Section 3.5"},{"comment":"The geometric metrics NoFloat, InBound, ColVol, and PIoU are essentially the same objectives that the optimization directly minimizes in Section 3.5 (support-surface loss, boundary/against-wall loss, collision loss). Reporting these as evidence of scene quality is partially self-referential. More importantly, the reported gains over LayoutGPT are extremely small: ColVol improves from 1.3440 to 1.3381 (Δ=0.006) and PIoU from 0.478 to 0.472 (Δ=0.006). No error bars, confidence intervals, or significance tests are reported across the 93 evaluation samples. Such tiny differences cannot support the claim of state-of-the-art performance.","section":"Tables 1 and 2"},{"comment":"The limitations paragraph states that the method 'cannot account for which possibilities would be the most common sense ones or the most aesthetically pleasing.' This directly contradicts the abstract's claim that the method generates arrangements that obey both physics and common sense. The perceptual user study is the only evidence for semantic plausibility, but the results are presented without confidence intervals or inter-rater agreement metrics. Either strengthen the perceptual evaluation with proper statistics or soften the claim.","section":"Section 5, Limitations"},{"comment":"The correctness of the entire pipeline depends on VLM-annotated node attributes and heuristic edge estimation ('on top of', 'against wall', 'facing'). No analysis of annotation accuracy or sensitivity to errors is provided. A false edge (e.g., an incorrect 'on top of' relation) or a wrong target surface cannot be recovered by the later optimization. The paper should include an error analysis or a sensitivity study to establish that the method is robust to the noise inherent in automatic graph construction.","section":"Supplement Section 9 and Section 3.1"}],"minor_comments":[{"comment":"There are typos such as 'objectswe' in the abstract and 'retieved' in Related Works that should be corrected.","section":"Abstract and Section 1"},{"comment":"In the definition of the against-wall loss, 'qre' should be 'are' in the description of the object center.","section":"Section 3.5"},{"comment":"The paper should report the number of runs (or seeds) for each configuration and provide standard deviations or confidence intervals for all metrics, given that the evaluation uses a finite set of 93 instructions.","section":"Tables 1 and 2"},{"comment":"The perceptual study results would be more informative if the figure legend described the rating scale and included confidence intervals or error bars.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a well-engineered system for a relevant problem, but the evaluation and one internal consistency issue currently prevent acceptance. The facing-preservation gap is a genuine algorithmic flaw that can be fixed with an orientation term and an orientation metric. The circularity of the geometric metrics and the negligible reported differences versus LayoutGPT require either a stronger evaluation protocol or a more modest claim. I recommend major revision rather than reject because the core pipeline is plausible and the issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the system: taking dense, real-world 3D scans (69–306 objects) and doing instruction-driven rearrangement via a hierarchical scene graph, LLM subgraph selection, localized planning, hierarchical placement, and a final convex optimization over physical constraints. Prior layout-generation methods target synthetic scenes with a handful of objects, so the scale and the explicit focus on real scans is a real step forward. The ablations are sensible and show each component earns its keep. I believe the central claim—that this is the first text-based editing approach for complex scans—holds up on a quick literature check.\n\nWhat the paper does well: the decomposition is thoughtful. The hierarchical graph makes the problem tractable, the localized planning reduces LLM hallucination risk, and the convex optimization with collision, surface, wall, and group losses is a clean way to clean up LLM proposals. The perceptual study, while not deeply detailed, at least tries to measure the actual outcome (instruction adherence and layout quality), which is more than many papers do.\n\nNow the soft spots, in proportion. The biggest internal gap is the one the stress-test note catches: the final optimizer optimizes x, y, and theta, but the 'facing' constraint is only handled in the initial placement. The group loss preserves center-center vectors, not orientations, and no loss or post-check enforces that a chair still faces the TV after optimization. So the optimizer can rotate an object away from the intended facing relation, and the paper's own metrics (NoFloat, InBound, ColVol, PIoU) do not measure orientation or semantic relation fidelity. That is a real hole in the claim that arrangements obey 'common sense.' It is not fatal—the initial placement might often survive optimization—but it is unaddressed.\n\nSecond, the geometric metrics partly mirror the optimized losses: NoFloat and InBound are essentially the support and against-wall losses, ColVol and PIoU are the collision loss. Reporting them as independent success is partially self-referential. The differences over LayoutGPT on ColVol and PIoU are tiny (0.006 and 0.006), with no error bars or significance tests. A stronger evaluation would include an instruction-fidelity metric, or at least a human study that isolates semantic correctness.\n\nThird, no code, data, or model versions are released. The method depends on several LLM/VLM calls and 3D heuristics; without code, reproducing or building on it is hard. The heuristic edge estimation in the supplement is plausible but brittle, as the reader notes.\n\nMinor: the writing has a few typos ('objectswe', 'retieved'), and the limitations paragraph honestly admits the method cannot handle 'common sense' or aesthetics—which slightly undercuts the abstract's phrasing. But the paper is candid about its limits.\n\nWho is this for? Anyone working on 3D scene editing, layout generation, or embodied AI goal-state visualization. It deserves a serious referee—the problem is real, the system is novel, and the gaps are addressable. I would like to see a revision that adds an orientation/semantic metric, error bars, and at least a promise of code.","headline":"A well-engineered hierarchical LLM+VLM system for editing real 3D scans, with a genuine novelty claim but a few measurement and reproducibility gaps that keep me from fully endorsing the results.","tokens_in":20087,"tokens_out":779,"would_cite":true,"duration_ms":8783,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ScanEdit claims the first text-instruction-driven method for editing complex real-world 3D scans, handling scenes with 69–306 objects by hierarchically decomposing the edit.","keywords":["3D scene editing","scene graphs","large language models","vision-language models","physical plausibility","indoor scenes","instruction-driven editing","3D understanding"],"falsifier":"Take a set of edited scene outputs, manually label the true support, facing, and wall relations for moved objects, and compare them with the scene-graph edges produced by the VLM/heuristic stage; if a substantial fraction of the load-bearing edges are wrong (e.g., more than 10%) while the editing still appears successful, or randomly corrupting a portion of the edges does not degrade output quality, then the claimed causal role of the hierarchical constraints is in doubt. Conversely, if manually correcting the edges substantially improves results, that supports the paper's mechanism.","tokens_in":19063,"feed_emoji":"🛋️","tokens_out":5811,"duration_ms":47190,"temperature":0.7,"pith_summary":"This paper introduces ScanEdit, a method that edits a real-world 3D scan by following a natural-language instruction such as 'rearrange the chairs to face the television.' It claims to be the first approach to handle instruction-driven rearrangement of complex scans containing hundreds of objects, far beyond the synthetic scenes that earlier text-to-3D layout systems handle. The key idea is to decompose the edit into a hierarchy: build a scene graph of objects with attributes and relations, select the subgraph relevant to the instruction, generate per-object instructions with an LLM planner, place objects hierarchically relative to their parents, then jointly optimize positions and orientations with convex losses for support, wall contact, collision avoidance, and group preservation. If correct, this makes interactive functional editing of captured indoor scenes practical.","feed_headline":"Text prompt rearranges 3D scans with up to 306 objects","feed_subtitle":"A hierarchical scene graph plus LLM guidance keeps rearranged objects physically plausible and instruction-faithful.","key_machinery":"The load-bearing object is the hierarchical scene graph: nodes are object instances annotated by a VLM with class, color, material, short description, front-facing direction, sampled surface points, and support surfaces; edges are directed relations ('on top of', 'facing', 'against wall') estimated with geometric heuristics. This representation lets the method decompose a high-level instruction into tractable local edits: an LLM subgraph selector prunes irrelevant context, an LLM planner and placement agent propose object-specific movements in parent-relative frames, and a set of convex differentiable losses (on-top-of surface loss, against-wall loss, collision loss with a stop condition, and group-structure loss) jointly optimize the final arrangement under physical constraints. The hierarchy is what makes scenes with 69–306 objects feasible within LLM context limits and what grounds the LLM's spatial guesses in geometry.","core_discovery":"ScanEdit's central claim is that functional text-based editing of complex, real-world 3D scans becomes tractable when the edit is structured hierarchically and grounded in explicit geometric constraints. Given a scan decomposed into object instances, the method constructs a hierarchical scene graph whose nodes carry VLM-estimated attributes (class, color, material, description, front normal, support surfaces) and whose directed edges encode 'on top of', 'facing', and 'against wall' relations estimated by 3D heuristics. An LLM agent prunes this graph to the instruction-relevant subgraph; a planner LLM converts the instruction into localized per-object instructions in parent-relative frames; a placement LLM proposes concrete positions, orientations, and constraints; and a final convex optimization resolves support, wall-contact, collision, and group-structure losses. The paper reports that this pipeline outperforms two state-of-the-art layout-generation baselines on geometric plausibility metrics (NoFloat, InBound, ColVol, PIoU) on 93 evaluation samples from ScanNet++ and Replica, and is strongly preferred by human raters.","pith_inferences":["The hierarchy idea could transfer to other LLM-based spatial tasks, such as robot task planning or multi-room scene synthesis, by using the same subgraph-selection and parent-relative placement pattern.","The reliance on heuristic edge estimation suggests a testable extension: learning edge proposers from data could reduce the method's dependence on VLM attribute quality and make it more robust to partial scans.","The optimization framework could be reused for interactive editing where a user iteratively modifies instructions, since each edit only needs to re-optimize the affected subgraph rather than the whole scene.","If the method is combined with mesh completion or inpainting to fill holes left by moved objects, it would become a complete scene-editing tool; the paper itself notes that holes can be visible after rearrangement."],"forward_implications":["Complex real-world scans with hundreds of objects become editable through natural language, enabling content creation, VR/AR staging, and robot goal-state visualization.","The hierarchical decomposition allows LLM context limits to be respected by sending only instruction-relevant subgraphs to the language models.","The convex optimization with physical losses fixes the LLM's spatial blind spots, producing non-floating, in-bounds, collision-free arrangements.","The method beats the two compared layout-generation baselines on geometric plausibility and human preference, suggesting that hierarchy plus physics-style constraints is a promising route for realistic scene editing.","Because the output scene is a rearrangement of the original object meshes, it preserves instance identity and original geometry, unlike methods that replace objects with retrieved assets."],"supporting_citations":[{"why":"The LayoutGPT baseline that ScanEdit compares against; represents LLM-based layout generation for synthetic scenes.","marker":"[10]"},{"why":"The LayoutVLM baseline, a VLM-based differentiable layout optimization method that ScanEdit outperforms.","marker":"[39]"},{"why":"Holodeck, which uses LLMs to generate spatial scene graphs for synthetic scene generation, a precursor to ScanEdit's graph-based approach.","marker":"[51]"},{"why":"DiffuScene, the source of the PIoU collision metric and an example of synthetic scene synthesis with limited object counts.","marker":"[40]"},{"why":"Mask3D, used to produce the instance segmentation masks that decompose scans into object instances.","marker":"[31]"},{"why":"The ScanNet++ dataset, one of the two evaluation sources for the edited scenes.","marker":"[53]"},{"why":"The Replica dataset, the other evaluation source for the edited scenes.","marker":"[37]"}],"fun_headline_variants":["Hierarchical scene graph powers text-driven 3D scan edits","LLM-guided editing keeps 3D rearrangements physically plausible","Text instructions reorder 3D scans with physical constraints","Scene graph + LLM brings functional editing to 3D scans","Scale 3D edit to 300+ objects via hierarchical guidance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes that the VLM-annotated object attributes and the geometric heuristics for 'on top of', 'facing', and 'against wall' edges are correct enough, and that the LLM's subgraph and placement proposals are plausible enough, that the convex optimizer can turn them into a physically and semantically valid scene; if any one of these upstream signals is badly wrong, the optimizer cannot recover the intended arrangement.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical scene graph powers text-driven 3D scan edits","LLM-guided editing keeps 3D rearrangements physically plausible","Text instructions reorder 3D scans with physical constraints","Scene graph + LLM brings functional editing to 3D scans","Scale 3D edit to 300+ objects via hierarchical guidance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2858,"prompt_tokens":945,"completion_tokens":1913,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":1826}},"tokens_in":561,"tokens_out":1913,"duration_ms":11946,"temperature":1.0,"reasoning_tokens":1826,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:33:55.991718+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of edited scene outputs, manually label the true support, facing, and wall relations for moved objects, and compare them with the scene-graph edges produced by the VLM/heuristic stage; if a substantial fraction of the load-bearing edges are wrong (e.g., more than 10%) while the editing still appears successful, or randomly corrupting a portion of the edges does not degrade output quality, then the claimed causal role of the hierarchical constraints is in doubt. Conversely, if manually correcting the edges substantially improves results, that supports the paper's mechanism.","supporting_citations":[{"cited_title":"Layoutgpt: Compositional visual plan- ning and generation with large language models","cited_arxiv_id":null,"evidence_quote":"The LayoutGPT baseline that ScanEdit compares against; represents LLM-based layout generation for synthetic scenes."},{"cited_title":"Holodeck: Language guided gen- eration of 3d embodied ai environments","cited_arxiv_id":null,"evidence_quote":"Holodeck, which uses LLMs to generate spatial scene graphs for synthetic scene generation, a precursor to ScanEdit's graph-based approach."},{"cited_title":"Diffuscene: Denoising diffu- sion models for generative indoor scene synthesis","cited_arxiv_id":null,"evidence_quote":"DiffuScene, the source of the PIoU collision metric and an example of synthetic scene synthesis with limited object counts."},{"cited_title":"Mask3d: Mask trans- former for 3d semantic instance segmentation","cited_arxiv_id":null,"evidence_quote":"Mask3D, used to produce the instance segmentation masks that decompose scans into object instances."},{"cited_title":"Scannet++: A high-fidelity dataset of 3d indoor scenes","cited_arxiv_id":null,"evidence_quote":"The ScanNet++ dataset, one of the two evaluation sources for the edited scenes."}],"review_version":1}