{"id":"77eaf274-728d-49ef-92c5-56cbffa71de6","arxiv_id":"2507.17585","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A pipeline that converts real-world 3D scans into application-specific USD formats, enabling LLM-based object insertion (80% success) and robotic drawer-opening policies (87% success).","lead":"This paper shows a way to convert detailed real-world 3D scans into a standard format (USD) that lets AI models edit scenes and robots learn manipulation tasks. The method reaches 80% success for AI-placed objects and 87% success for a robot opening a drawer, but a whole category of bathroom objects failed completely.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 87% drawer-opening result may reflect exploitation of processed scan geometry rather than robust manipulation; absent real-robot or cross-simulator validation, the 'Scan to Action' claim is unsupported.","rationale":"I agree with the reader's identification of the weakest assumption. The central 'Scan to Action' narrative needs the simulation to be predictive of real interaction; otherwise the paper demonstrates only that PPO can exploit a processed USD scene. The paper deserves credit for a concrete pipeline and for reporting the negative side-grasp finding in Table 1, but that same finding is internal evidence of the fidelity gap. Other weaknesses—no baselines, single seed, the 87% versus 88% inconsistency, and the bathroom failure in the user study—are real but secondary; they affect robustness of the numbers, not the core assumption. The 'first' claim should also be checked against reference [19]'s real-to-sim-to-real method, but that is a novelty question and less load-bearing than physical fidelity. Conditional acceptance remains appropriate: the pipeline is worth publishing as an engineering contribution, but the headline empirical claims should be framed as simulation-only until a transfer check is done.","tokens_in":6603,"tokens_out":7621,"duration_ms":83865,"concrete_test":"Zero-shot deploy the trained PPO drawer-opening policy from Sec. 3.2 on a physical replica of the same ScanNet++ cabinet, using the same Franka mount and drawer friction, and record whether the drawer opens >=0.2 m. If the policy relies on side-grasps that fail on the physical drawer, or if success drops by more than 20 points from the simulator, the processed USD did not preserve the required interaction properties and the central claim needs to be weakened to 'simulation-only'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the USD pipeline in Sec. 2.3—convex decomposition, mesh decimation, merging of static meshes, and ground-fixing of manipulated objects—preserves the physical interaction properties (friction, mass, contacts) of the real scans. The paper never validates this with a real robot, and its own Table 1 shows a fidelity gap: on the real-scene asset PPO reaches only 87% versus 98% on the synthetic asset, and Franka 'frequently grasps the drawer from the side, bypassing the handle'; manual grasping-point corrections did not resolve this. Since success is defined only as drawer opening >=0.2 m, a side-grasp of the decimated, ground-fixed mesh can earn success in IsaacSim while failing on a real cabinet. Thus the headline 87% and the implied 'Action' utility are not established; they may be artifacts of the processed simulation geometry rather than evidence of real-world manipulation capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a methodology for converting richly annotated real-world 3D scans (Articulate3D, built on ScanNet++) into USD representations with two application-specific flavors: a descriptive USD for LLM-based scene editing and a geometry-focused USD for robot simulation. The authors present an LLM-driven object-insertion pipeline evaluated by a user study at 80% success, and a simulation pipeline in IsaacLab/IsaacSim where a PPO policy achieves 87% success on a drawer-opening task and planner-based policies achieve 100% on the evaluated interactions. The central claim is that holistic real-world scene scans can support embodied scene understanding and manipulation policy learning at scale.","tokens_in":6729,"tokens_out":3463,"duration_ms":36993,"significance":"If the quantitative claims held, the paper would provide a useful bridge from real-world scan datasets to training environments for embodied agents, with practical recipes for dealing with incomplete meshes, collision geometry, and articulation annotations. The paper is honest about the challenges encountered and offers concrete mitigation strategies, and the two USD flavors are a sensible design idea that could be reused by the community. However, the evidence base is narrow: there is no real-robot validation, no comparison to prior methods, no baselines in the user study, and the RL result is a single run with an internal inconsistency. The headline numbers are therefore not yet established, and the paper should be read primarily as a system description whose quantitative claims require substantial additional support.","major_comments":[{"comment":"The PPO result is reported inconsistently and without statistical support. The text states an 87% success rate for drawer opening, while Table 1 lists 88% for the same 'Real [9]' reward configuration; no seeds, standard deviations, or repeated runs are reported. A single PPO training run, even with 1024 environments, is not enough to support the headline 87%; the authors should report mean and standard deviation over at least 3-5 seeds and reconcile the 87%/88% discrepancy.","section":"Sec. 3.2, Table 1"},{"comment":"The user study has no baseline condition, and the 80% overall number masks a complete failure in one category. The evaluation used 100 insertions (20 per category), but the bathroom category achieved 0/20, so the 80% result is entirely carried by the other four categories. Without a comparison condition (e.g., LLM operating on plain text scene graphs, random placement, or a prior insertion method), the claim that the descriptive USD representation enables the pipeline's performance is unsupported. The authors should provide a per-category breakdown with confidence intervals and at least one baseline.","section":"Sec. 3.1"},{"comment":"The physical fidelity of the simulation pipeline is load-bearing for the 'Scan to Action' claim, and the paper does not validate it. Convex decomposition, mesh decimation, ground-fixing of manipulated objects, and merging of static meshes can all alter contact, friction, mass distribution, and articulation dynamics. The paper itself reports that Franka frequently grasps the drawer from the side, bypassing the handle, and that manual grasping-point corrections did not resolve the issue; since success is defined only as drawer opening of at least 0.2 m, a side-grasp on decimated geometry can count as success in IsaacSim while failing on a real cabinet. The authors should provide real-robot validation or, at minimum, cross-simulator validation (e.g., deploying the same policy in a second physics engine such as MuJoCo or PyBullet) to show that the 87% result reflects robust manipulation rather than an artifact of the processed simulation geometry.","section":"Sec. 2.3 and Sec. 3.2"},{"comment":"The claim of being 'the first to enable large-scale training for manipulation tasks using real-world scene scans within robotics simulation' is not supported by any comparison. The closest prior work, ReAL [19], is cited as the basis for the asset structure and ground-fixing approach, but there is no quantitative or even structural comparison against it or against any other manipulation-simulation pipeline. The authors should either provide a direct comparison or carefully qualify the novelty claim relative to [19] and related work.","section":"Sec. 1 and Sec. 3.2"},{"comment":"The statement that 'planner-based policies achieved 100% success rate on all objects interacted with, both rigid and deformable' is not backed by a task list, success criteria, or failure analysis. The experiments described are one drawer-opening task and one pick-and-place task with a teddy bear; no definition of success for the pick-and-place task is given. The 100% claim should be scoped to the specific evaluated tasks or removed.","section":"Sec. 3.2, Results"}],"minor_comments":[{"comment":"There is a typo in the contributions list: 'aggregation appraoch' should be 'aggregation approach'.","section":"Sec. 1"},{"comment":"The abbreviations 'USD-Core' and 'RANSAC' are used without definition or citation; they should be introduced at first use.","section":"Sec. 2.2"},{"comment":"The security mechanism is described only as an allowlist of USD-Core operations with import-statement checks; more detail on how the allowlist is constructed and enforced would make the safety claim credible.","section":"Sec. 2.2"},{"comment":"The sentence 'The decimation strategy is solely applied for policy learning to enable faster learning' is repetitive and would benefit from rewording.","section":"Sec. 2.3"},{"comment":"The reward-component columns 'Opening' and 'Finger' are ambiguous; the reward terms should be explicitly defined in the text or caption, and the table should be reformatted for readability.","section":"Table 1"},{"comment":"Reference [7] is malformed: 'Kawana Y . et al.' is missing the full author list and should be corrected.","section":"References"},{"comment":"The qualitative insertion results do not indicate which LLM (GPT-4o or GPT-4o mini) generated each example; labeling each row or column would help the reader interpret the comparison.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies heavily on the authors' own Articulate3D dataset and on a self-citation for the primary input representation; this is a dependency rather than circular reasoning, but it increases the need for independent validation. The editors may wish to emphasize that the 80% user-study and 87% RL numbers are likely to be over-interpreted by readers unless the paper adds baselines, error bars, and a real-robot or cross-simulator check. The current evidence is not sufficient to support the 'Scan to Action' framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this is a competent engineering paper, not a breakthrough. It shows how to take Articulate3D's full-scene scans, convert them into two USD flavors (descriptive for LLMs, geometry-focused for physics sim), and use them for object insertion and drawer-opening. That integration is genuinely new — prior work used single-object scans or fully synthetic scenes. The USD flavor split is a sensible design choice, and the reward-engineering ablation in Table 1 is honest about the sim-to-real gap on handle grasping.\n\nThe soft spots are real but addressable. The 80% user-study success is technically true but hides that the bathroom category scored 0/20; saying \"failures limited to bathroom objects\" in the conclusion is an odd way to frame a category that completely failed. The PPO success rate is reported as 87% in the text and abstract but 88% in Table 1 for the same asset and reward setting; minor, but sloppy. No baselines, no error bars, and no real-robot validation. The load-bearing assumption is that ground-fixing, decimation, and convex decomposition preserve the physical properties that matter for manipulation. The paper's own Table 1 shows the real asset underperforms the synthetic one and that the Franka often grabs the side of the drawer instead of the handle — exactly the kind of geometry-exploiting failure that can look like success in IsaacSim but not on a real cabinet. The stress-test note that the 87% may be an artifact of the processed geometry is fair, though I'd phrase it as \"plausible but unproven\" rather than \"unsupported.\" The paper never claims real-robot transfer, so it's not a sin, but the title says \"to Action\" and the framing invites that reading.\n\nAll this is fixable. The pipeline is described clearly enough to reproduce, the code is not shipped but could be, and the authors are transparent about the grasping-point issue. The \"first to enable large-scale training\" claim is unverifiable without a broader comparison, but it's a minor overclaim, not a fraud.\n\nWho this is for: researchers working on sim-to-real pipelines, USD-based scene representations, or LLM-driven 3D editing. They'll get a useful recipe and a clear list of pitfalls. It deserves a serious referee — the methodology is sound enough to warrant scrutiny, and the evaluation gaps can be closed in revision. I'd send it to peer review with a request for baselines, multiple seeds, consistent reporting, and either a cross-simulator check or a frank discussion of why the fidelity assumption is acceptable. I wouldn't cite it yet, but I'd keep it in the stack.","headline":"A clearly-written USD pipeline for turning real scene scans into usable simulation assets, but the headline numbers hide a 0% bathroom category and an unreplicated PPO run.","tokens_in":7304,"tokens_out":2016,"would_cite":false,"duration_ms":23082,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Real-world 3D scene scans, converted into two flavors of USD, can replace synthetic assets for both LLM-driven scene editing and robot manipulation training, with 80% and 87% success rates respectively.","keywords":["USD scene representation","real-world 3D scans","LLM scene editing","robotic manipulation simulation","articulated objects","sim-to-real transfer","reinforcement learning","scene understanding"],"falsifier":"Deploy the PPO drawer-opening policy trained on the real-scan office scene on a physical robot facing the same cabinet from the same starting pose, and run 20 episodes; if the success rate falls well below the simulated 87% or the robot cannot grasp the handle, then the processed scans do not preserve the contact and mass properties the simulation assumed.","tokens_in":6374,"feed_emoji":"🤖","tokens_out":7599,"duration_ms":83244,"temperature":0.7,"pith_summary":"The paper argues that richly annotated real-world 3D scene scans, usually too fragmented, holey, and heavy to use, can be turned into working assets for embodied AI. Its method converts the scans and their object, part, and articulation annotations into USD, a hierarchical 3D format, and then splits the result into two flavors: a geometry-free descriptive version that large language models can read, and a geometry-focused version for physics simulation. On this pipeline, an LLM-driven insertion system places objects into scenes with 80% user-rated success, and a PPO-trained policy opens a cabinet drawer with 87% success in simulation, while planner-based policies reach 100%. If these numbers hold outside the simulator, real scans would replace synthetic assets for training manipulation policies, giving robots contact with real-world geometry and layout from the start.","feed_headline":"87% success: real scans train robot policies","feed_subtitle":"Two USD flavors turn messy scene scans into simulation assets for robot learning and LLM-readable scene graphs.","key_machinery":"The load-bearing object is the dual-flavor USD representation. Universal Scene Description (USD) is a hierarchical 3D scene format in which objects, parts, and attributes can be nested and annotated. The descriptive flavor omits geometry and keeps labels, bounding boxes, hierarchies, and articulation annotations, giving an LLM a compact symbolic scene to reason over. The geometry-focused flavor keeps mesh data but flattens part hierarchies into separate objects, using articulation annotations to encode relationships, and applies convex decomposition for collision detection, ground-fixing for stabilization, mesh decimation (10% of faces for walls, floors, and ceilings, 30% for other static objects, and full resolution for the manipulated object), and mesh merging for static objects. This split is what lets one scan dataset serve both an LLM planner and a physics simulator.","core_discovery":"The central claim is that application-specific USD flavors make real-world full-scene scans usable for both semantic and physical downstream tasks, and that this is the first large-scale training of manipulation policies on real-world scene scans inside a robotics simulator. Concretely, the paper reports that a descriptive USD containing labels, bounding boxes, hierarchies, and articulation data lets two recent LLMs decide where and how to place an object, yielding 80% successful insertions over 100 user-evaluated cases, with failures concentrated in bathroom scenes and rare labels. It further reports that a geometry-focused USD, built by treating parts as separate objects, merging static meshes, and applying convex decomposition and decimation, supports PPO drawer-opening at 87% success and planner-based pick-and-place at 100% success.","pith_inferences":["The absence of real-robot trials means the 87% figure is a simulation metric; the paper's own difficulty fixing grasping points on the real cabinet hints that the sim-to-real gap could be large, so the strongest supported reading is that scans are usable for simulation, not yet that policies transfer to reality.","The same USD-conversion recipe could be applied to other scan datasets with object- and part-level annotations, turning annotation unification into a general bridge rather than a single-dataset trick.","The descriptive USD could power other LLM 3D tasks such as spatial question answering or rearrangement planning, because it exposes a compact symbolic scene graph without mesh noise.","A testable extension is to supply the LLM with detected planar support surfaces or affordance priors for hard geometries like sinks; the bathroom category's zero successes suggest the bottleneck is geometric affordance detection rather than semantic reasoning."],"forward_implications":["If the approach holds, any richly annotated real-scan dataset can be converted into simulation-ready USD assets, removing a key bottleneck in using real-world data for robot learning.","LLM-driven scene editing becomes a practical way to extend real-scan scenes with new objects, since 80% of insertions were judged successful and failures were concentrated in one object category and rare labels.","Planner-based policies on full real scenes can collect diverse trajectories and realistic images, supporting large-scale data collection for downstream imitation or evaluation.","The reward-ablation result implies that real-scan geometry changes learned behavior, so sim-to-real tuning must account for geometric differences, not just appearance.","The dual USD flavors establish a shared representation for semantic reasoning and physics, so the same underlying scene data can be reused across disparate embodied tasks."],"supporting_citations":[{"why":"Supplies the object- and part-level annotations, articulation parameters, and grasping regions on real scene scans that the whole pipeline consumes.","marker":"[9]"},{"why":"Provides the simulation asset structure and ground-fixing approach that the authors follow for the geometry-focused USD.","marker":"[19]"},{"why":"Source of the real-world indoor scene scans on which the annotations are built.","marker":"[21]"},{"why":"Provides the 3D object models used in the LLM insertion user study.","marker":"[4]"},{"why":"The simulation and reward framework used for both PPO and planner-based experiments.","marker":"[12]"},{"why":"The PPO algorithm used to train the drawer-opening policy.","marker":"[18]"},{"why":"One of the two LLMs tested in the object-insertion pipeline.","marker":"[15]"},{"why":"The second LLM tested, showing that insertion success holds across models.","marker":"[16]"}],"fun_headline_variants":["Real scans boost robot policies to 87% success","USD flavors turn scans into robot and LLM assets","Real-world scans enable 87% robot success rate","From messy scans to 87% robot policy success","Scene scans train robots: 87% success in simulation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulation pipeline assumes that convex decomposition, mesh decimation, ground-fixing, and mesh merging preserve the real objects' physical behavior, such as contacts, friction, and mass distribution, so that a policy trained in these processed USD scenes would transfer to a real robot; the paper validates only inside simulation and notes that even adjusted grasping points failed on the real cabinet.","fun_headline_variants_meta":{"raw":{"variants":["Real scans boost robot policies to 87% success","USD flavors turn scans into robot and LLM assets","Real-world scans enable 87% robot success rate","From messy scans to 87% robot policy success","Scene scans train robots: 87% success in simulation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1846,"prompt_tokens":803,"completion_tokens":1043,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":966}},"tokens_in":419,"tokens_out":1043,"duration_ms":8387,"temperature":1.0,"reasoning_tokens":966,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:44:40.559539+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Deploy the PPO drawer-opening policy trained on the real-scan office scene on a physical robot facing the same cabinet from the same starting pose, and run 20 episodes; if the success rate falls well below the simulated 87% or the robot cannot grasp the handle, then the processed scans do not preserve the contact and mass properties the simulation assumed.","supporting_citations":[{"cited_title":"Rec- onciling Reality through Simulation: A Real-To-Sim-to-Real Approach for Robust Manipulation","cited_arxiv_id":null,"evidence_quote":"Provides the simulation asset structure and ground-fixing approach that the authors follow for the geometry-focused USD."},{"cited_title":"Scannet++: A high-fidelity dataset of 3d indoor scenes","cited_arxiv_id":null,"evidence_quote":"Source of the real-world indoor scene scans on which the annotations are built."},{"cited_title":"Objaverse-xl: A universe of 10m+ 3d objects","cited_arxiv_id":null,"evidence_quote":"Provides the 3D object models used in the LLM insertion user study."},{"cited_title":"Orbit: A unified simulation framework for interactive robot learning environments","cited_arxiv_id":null,"evidence_quote":"The simulation and reward framework used for both PPO and planner-based experiments."},{"cited_title":"GPT-4o mini: advancing cost-efficient intelligence","cited_arxiv_id":null,"evidence_quote":"One of the two LLMs tested in the object-insertion pipeline."},{"cited_title":"Hello GPT-4o","cited_arxiv_id":null,"evidence_quote":"The second LLM tested, showing that insertion success holds across models."}],"review_version":1}