{"id":"ac11655c-044b-42cb-817c-2e35e2a747ef","arxiv_id":"2505.04831","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A single diffusion scene prior over SE(3) object sets can be steered by RL post training, text conditioning, and MCTS search, backed by a new 44 million scene dataset.","lead":"This paper trains a diffusion model to generate 3D indoor scenes made of objects with full positions and rotations, then steers that model with reinforcement learning, language conditioning, and search to hit task-specific goals. The result is a released dataset of 44 million scenes and evidence that a single scene prior can be adapted past its training distribution.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feasibility 'guarantee' is conditional on SNOPT convergence and Drake thresholds; B.4's conceded failures undermine the MCTS reward and Section 1 claim.","rationale":"The reader identified SNOPT convergence and Drake thresholds as the weakest assumption; I agree and sharpen it with the internal contradiction between Section 1, Appendix B.4, and Appendix C. The concern is load-bearing because the paper's advertised guarantee and the MCTS feasibility objective both depend on the projection solver succeeding and on the simulator thresholds being meaningful. If SNOPT fails on a nontrivial fraction of Restaurant scenes, the 'guaranteed zero MTP after projection' statement is false, and the MCTS reward counts objects that may not be physically feasible under any more conservative check. I do not think this invalidates the broader steering claim: RL post training with an object-count reward and conditional generation are demonstrated independently and show consistent advantages over the baselines, and the release of code/data/weights makes the failure rate directly checkable. The paper also contains appropriate limitations on procedural realism, overoptimization, and asset-library generality. However, the physical-feasibility guarantee should be either downgraded to a best-effort post-processing statement with quantified SNOPT convergence and fallback behavior, or verified by an independent simulation protocol. This is exactly the kind of condition that keeps the paper at CONDITIONAL rather than ACCEPT, so I recommend UNCHANGED relative to the reader's verdict.","tokens_in":24271,"tokens_out":7859,"duration_ms":80542,"concrete_test":"Run the released post-processing pipeline on 1,000 held-out generated Restaurant (High-Clutter) scenes, logging SNOPT exit status for every Eq. (1) solve and computing post-projection signed distances for converged and non-converged solves separately. If any non-converged solve is returned with positive penetration, or if non-converged scenes are silently discarded without disclosure, the Section 1 guarantee is false as stated and the MCTS reward is not a verified physical ground truth.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that generated scenes are physically feasible and simulation-ready rests on Section 1's assertion that projection plus simulation 'guarantee physical correctness.' Appendix B.4 explicitly concedes that SNOPT, used for the non-penetration projection in Eq. (1), 'may fail to converge' in complex Restaurant scenes with many nonlinear non-convex constraints, yet Appendix C states that after projection MTP is 'guaranteed to be zero.' The paper provides no fallback for failed solves, no convergence statistics, and no post-projection penetration measurements. The same Drake signed-distance checker and 0.1 s settle simulation define both the MCTS mask generator (B.7.1) and the MCTS reward (B.7.2), so a solver failure or a threshold misclassification directly inflates the 'number of physically feasible objects' and the headline MCTS result in Figure 7. Because the claimed guarantee fails exactly in the high-clutter regime the method targets, the steering claim for physical feasibility is not yet established for the hardest scenes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for SE(3) scene generation in which a diffusion model over object sets (asset ID, translation, and 9D rotation) is trained on procedurally generated scenes and then steered toward downstream objectives by three mechanisms: DDPO-based RL post training, text-conditioned generation and inpainting, and an MCTS procedure that iteratively inpaints masked invalid objects. Physical feasibility is enforced by a nonlinear projection step solved with SNOPT, followed by Drake simulation. The authors report unconditional and conditional generation results on five procedurally generated scene types, compare with DiffuScene and MiDiffusion, demonstrate clutter-maximizing RL and MCTS steering, and release code, model weights, and a dataset of 44.79 million scenes, including a teleoperation demo.","tokens_in":24528,"tokens_out":4861,"duration_ms":48456,"significance":"If the claims hold, the paper makes a meaningful contribution: a large-scale SE(3) scene dataset, a unified scene prior supporting three distinct steering paradigms, and an MCTS inference-time search strategy for diffusion-based scene synthesis. The release of code, data, and weights is a concrete strength that should make the results reproducible. The significance is tempered by two issues: the feasibility guarantee is conditional on a solver convergence caveat that is conceded in Appendix B.4, and the evaluation of physical feasibility and steering quality is largely internal to the same procedural and simulation pipeline used to define and optimize the objectives.","major_comments":[{"comment":"The Section 1 claim that projection and simulation 'guarantee physical correctness' is not supported for the hardest scenes. Appendix B.4 states that in complex Restaurant scenes 'SNOPT may fail to converge due to the presence of many complicated nonlinear non-convex constraints,' yet Appendix C asserts that 'after projection, MTP is guaranteed to be zero.' The manuscript provides no convergence statistics, no fallback procedure for failed solves, and no post-projection measurements of residual penetration. Because Section 3.5 and Figures 29–31 use the number of physically feasible objects as the MCTS reward, and this number is determined by the same Drake feasibility checks, a solver failure in the high-clutter regime directly affects the headline inference-time search result. Please report SNOPT convergence rates per dataset and scene complexity, specify the fallback behavior, and verify post-projection penetration with measurements rather than an unconditional guarantee.","section":"§1, Appendix B.4, Appendix C"},{"comment":"The MCTS evaluation is circular with respect to the feasibility definition. The mask generator and reward function (Appendices B.7.1 and B.7.2) use Drake's signed-distance checker and a 0.1 s settle simulation with 'predefined thresholds' that are never specified, and the same criteria are the only evidence that the final scenes are physically feasible. Please state the translation/rotation thresholds, report the sensitivity of the MCTS reward to those thresholds, and validate the feasibility determinations against an independent source (e.g., a different simulator, longer simulation horizons, or real-world checks) so that the steering objective is not merely conformity to one simulator's settings.","section":"§3.5, Appendix B.7"},{"comment":"The RL post-training section is presented mostly qualitatively. Figure 4 and Figure 19 show sample scenes and a reward curve, but there are no quantitative comparisons of CA, FID, MTP, or the object-count distribution before versus after post training, and no comparison against the other two steering mechanisms on the same clutter objective. Because the central contribution is that post training steers the pretrained model beyond its training distribution, the paper should provide quantitative evidence that the selected checkpoint preserves scene quality while increasing object count, and ideally a comparison with conditional generation and MCTS under a common budget.","section":"§3.3, Appendix D.2"},{"comment":"The MCTS contribution lacks a comparison against simpler inference-time baselines. Figure 7 compares MCTS only with the initial inpainting sample and the best of B=3 unconditional samples, and Section 2.4.3 states that the method reduces to Random Search when B→∞, but no random-search or best-of-k baseline is evaluated under the same number of forward passes. Because Dimsum Table scenes have a small maximum object count (34), the claim that MCTS 'reaches the maximum possible 34 objects' would be more persuasive if compared with, for example, best-of-N independent samples or a greedy inpainting baseline using the same reward. Please add such a comparison, including compute cost.","section":"§3.5, §2.4.3"}],"minor_comments":[{"comment":"Please specify the exact stability thresholds used in the 0.1 s Drake simulation; the current text only says 'predefined thresholds.'","section":"Appendix B.7.1"},{"comment":"Please clarify whether 'MTP' is reported as the median over scenes of total penetration or as a median per-object quantity; the caption of Table 1 calls it 'median total penetration,' which is ambiguous.","section":"§3.1"},{"comment":"Please report the hyperparameters used for MCTS (exploration constant c, branching factor B, number of iterations, and noise initialization strategy) in the main text or a table, since Figure 7 reports only B=3 and 313 iterations.","section":"§2.4.3"},{"comment":"The novelty analysis with Sinkhorn-Knopp distance is a useful addition, but it reports only qualitative nearest-neighbor examples; a quantitative threshold or distribution of distances would strengthen the claim that the model does not memorize training scenes.","section":"Appendix D.5"},{"comment":"Please define how the empty-object token interacts with the discrete diffusion and with the upper bound N on scene size; this affects how variable-size scenes are sampled and how the MCTS mask treats empty slots.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a robotics venue, and the released dataset, code, and model weights are valuable assets. The main correctness risk is the feasibility guarantee: it depends on SNOPT convergence and Drake's thresholds, and the manuscript itself concedes convergence failures in the high-clutter regime. This is fixable with convergence statistics, a documented fallback, and post-projection measurements. Please also ensure that the RL and MCTS steering claims are backed by quantitative comparisons against simpler baselines before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick read of arXiv:2505.04831. Worth your time. It builds an SE(3) scene diffusion model on 44 million procedurally generated scenes, then shows three steering methods: DDPO RL post-training, text conditioning/inpainting, and MCTS inference search. What is genuinely new isn't any single component — mixed diffusion, CFG, DDPO, UCT all exist — it's that they are combined in a working scene-generation system with full 6-DOF poses, a fixed asset library, and a released dataset/code. The dataset alone is a resource for anyone doing simulation-based manipulation training.\n\nThe quantitative story is mostly solid. Against DiffuScene and MiDiffusion adapted to SE(3), the model wins or ties on CA/FID/APF across the five scene types, with consistently lower MTP. The MCTS demo — reaching the maximum 34 physically feasible dimsum objects from 21 — is nice evidence that inference-time search can push beyond the training distribution. The teleop video supports simulation-readiness. I'd trust the central claim that one pretrained prior can be steered toward clutter or feasibility objectives.\n\nSoft spots, in order of seriousness. First, the \"guarantee physical correctness\" language in Section 1 and \"MTP is guaranteed zero after projection\" in Appendix C are overstatements. Appendix B.4 concedes SNOPT may fail to converge on complex Restaurant scenes. There are no convergence statistics, no fallback, and no post-projection penetration measurements. Since the same simulation thresholds define the MCTS mask and reward, a solver failure or misclassified stability directly inflates the headline search result. This doesn't sink the paper, but the authors should replace \"guarantee\" with a reported success rate, or add failure handling. Second, evaluation is internal to the procedural distribution: models train on synthetic scenes, metrics compare against those same synthetic scenes, and feasibility is judged by the same Drake pipeline used as reward. Some external validation — real scans, a different simulator, human ratings — would strengthen it. Third, several headline metrics lack error bars, and RL post-training is only shown on the continuous variant with one reward. Fourth, the novel-scenes analysis via Sinkhorn nearest neighbors is qualitative but at least addresses memorization.\n\nNone of these are fatal; they're fixable in revision. The paper is a serious engineering contribution and deserves referee time. If I were handling it, I'd send it out, with a request for SNOPT convergence numbers and a softened feasibility claim.","headline":"A solid systems paper with a genuinely useful released dataset; the feasibility 'guarantee' is softer than advertised, but the core steering story survives.","tokens_in":742,"tokens_out":749,"would_cite":true,"duration_ms":32007,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A scene diffusion model trained on 44 million procedurally generated scenes can be steered toward objectives such as higher clutter through RL post training, conditional generation, or inference-time tree search, without retraining from…","keywords":["SE(3) scene generation","scene diffusion models","Monte Carlo tree search","reinforcement learning post-training","procedural scene generation","conditional generation","physical feasibility","robot simulation"],"falsifier":"Generate 1,000 scenes from the released Restaurant model, run the stated post-processing, and record how many fail the SNOPT projection step outright, then re-simulate the surviving 'stable' scenes for 10 seconds to see whether objects judged at rest after 0.1 seconds stay at rest. A substantial fraction of failures on either check would show that the feasibility guarantee, and the MCTS reward built on it, do not hold as stated.","tokens_in":24101,"feed_emoji":"🤖","tokens_out":16353,"duration_ms":136152,"temperature":0.7,"pith_summary":"This paper argues that a single pretrained generative model over full six-degree-of-freedom scenes can be adapted to downstream objectives without being retrained from scratch. The authors train a diffusion model on more than 44 million procedurally generated scenes spanning five environments, where a scene is an unordered set of objects drawn from a fixed asset library, each placed at a full six-degree-of-freedom $\\mathrm{SE}(3)$ pose. They then demonstrate three steering routes: reinforcement-learning post training, text-conditioned or inpainting-based conditional generation, and an inference-time Monte Carlo tree search that incrementally repairs and completes scenes while optimizing a task reward. If the claim holds, robots can obtain task-specific training scenes — high clutter, stable arrangements, prompt-specified layouts — from one flexible prior, with physical feasibility enforced afterwards by collision-free projection followed by gravity settling in simulation.","feed_headline":"Steer one scene model with RL, prompts, or tree search","feed_subtitle":"A 44M-scene diffusion prior adapts to clutter and simulation-ready layouts without retraining.","key_machinery":"The central object is the mixed discrete-continuous $\\mathrm{SE}(3)$ scene diffusion model: a transformer-based denoiser, trained to be permutation-equivariant over unordered object sets, in which each object is a one-hot asset category diffused discretely together with a translation vector and a nine-dimensional rotation matrix diffused continuously and projected onto $\\mathrm{SO}(3)$ at sampling time. It carries the argument because it is the single pretrained prior that all three steering strategies act on, and it makes steering possible by being differentiable, conditionable, and cheap to sample. The argument is completed by two feasibility mechanisms: a signed-distance projection solved with SNOPT that removes inter-object penetration while keeping orientations fixed, and a Drake simulation step that lets unstable objects settle under gravity for 0.1 seconds; together these define the 'physically feasible' label that serves as the MCTS reward, the inpainting mask criterion, and the paper's claim that outputs are simulation-ready.","core_discovery":"The paper's central claim is that a scene generative model trained on broad procedural data can be steered toward task-specific objectives, such as increasing clutter, and that this steering works through three complementary mechanisms: reinforcement-learning post training with a task reward, conditional generation via text prompts and structured inpainting, and a Monte Carlo tree search that builds scenes incrementally at inference time. The pretrained model predicts which objects to place from a fixed asset library together with their continuous $\\mathrm{SE}(3)$ poses, using a mixed discrete-continuous diffusion objective over permutation-equivariant object sets; because the model is differentiable and fast to sample from, its output distribution can be shifted after the fact. The authors treat physical feasibility as a separate guarantee rather than leaving it to the diffusion model: every generated scene is projected to a collision-free configuration and then settled under gravity in simulation, so outputs are claimed to be non-penetrating, statically stable, and directly usable for robot data collection. The reported experiments show scenes matching dataset statistics on all five scene types, inference-time search reaching the 34-object maximum of a Dimsum dataset from a 17.1-object average, and RL post training producing shelf scenes denser than anything in the pretraining distribution.","pith_inferences":["If the steering claim generalizes, scene generation inherits an alignment workflow: pick the cheapest of RL, conditioning, or search for each new objective, and reuse the same broad prior. The paper's separation of the MCTS mask generator from the reward function is a natural interface for arbitrary non-differentiable goals such as grasp success or human preference.","A testable extension the paper does not run is to keep the MCTS machinery but replace the feasibility-count reward with a semantic one, such as the number of requested object categories. The framework's modular design predicts the search should still improve that reward, which would show the steering result is not merely an artifact of the physics post-processor.","The element to stress-test is the feasibility oracle itself: the paper reports that SNOPT can fail to converge in complex Restaurant scenes (Appendix B.4), and stability is judged from 0.1 seconds of simulation, so 'physically feasible' is only as strong as those two checks. A longer simulation horizon or an analytic equilibrium certificate would be the natural safeguard."],"forward_implications":["A single pretrained scene model can be redirected toward an objective its training data never emphasized: the paper shows object-count rewards driving denser shelves after RL post training, and MCTS pushing a Dimsum scene to the 34-object maximum from a training average of 17.1 objects.","Generated scenes can be handed to a robot simulator without manual correction, since the projection-plus-simulation pipeline is designed to eliminate penetrations and unstable equilibria; the authors demonstrate this by teleoperating a mobile manipulator through pick-and-place interactions in Drake.","The three steering routes compose: a model refined by RL can still be prompted by text or searched at inference time, so scene generation gains the same kind of post-hoc alignment workflow used for large language models.","The released resource, over 44 million full six-degree-of-freedom scenes across five environments, gives future work a benchmark beyond the 18,968 floor-plan layouts of 3D-FRONT, with manipulable small objects rather than static furniture.","Cotraining across scene types yields a single model that can interpolate between environments, generating scenes that mix object identities from the Living Room Shelf and Pantry Shelf datasets when prompted with both."],"supporting_citations":[{"why":"Supplies the procedural scene grammars that generated the 44-million-scene training data.","marker":"[16]"},{"why":"Contributes the unordered object-set scene representation with an empty-object slot for variable scene sizes, plus the DiffuScene baseline.","marker":"[30]"},{"why":"Provides the mixed discrete-continuous diffusion training objective that the model and the conditioning-between-modalities mechanism are built on.","marker":"[31]"},{"why":"The DDPO algorithm that carries the reinforcement-learning post-training steering route.","marker":"[34]"},{"why":"The R9-plus-SVD rotation representation that lets the model learn and sample full $\\mathrm{SO}(3)$ poses.","marker":"[37]"},{"why":"The Drake simulator that defines physical feasibility: it settles scenes under gravity and evaluates penetration and static equilibrium for the MCTS mask and reward.","marker":"[44]"},{"why":"The frozen BERT encoder that turns text prompts into conditioning for the conditional-generation steering route.","marker":"[45]"},{"why":"Classifier-free guidance, the mechanism that lets one model do both conditional and unconditional generation and be steered by prompts.","marker":"[46]"},{"why":"The UCT bandit rule that selects which partial scene to expand during the inference-time MCTS search.","marker":"[49]"},{"why":"The SNOPT solver used to project object translations to a collision-free configuration before simulation.","marker":"[54]"}],"fun_headline_variants":["Diffusion scene prior steered by RL, prompts, or MCTS","Post-train or search: steer a scene diffusion model","RL, prompts, tree search: one scene model adapts","From 17 to 34 objects: inference-time scene search","44M-scene diffusion prior with three steering methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that physical feasibility is fully captured by two mechanical checks — a signed-distance projection solved with SNOPT and a 0.1-second Drake settling simulation with its stability thresholds — so the paper's guarantee that outputs are physically correct (Section 1), and the MCTS reward that counts objects passing those checks, both inherit all their trust from an optimizer that the appendix itself concedes may fail to converge on complex Restaurant scenes.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion scene prior steered by RL, prompts, or MCTS","Post-train or search: steer a scene diffusion model","RL, prompts, tree search: one scene model adapts","From 17 to 34 objects: inference-time scene search","44M-scene diffusion prior with three steering methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001006,"raw_usage":{"total_tokens":4277,"prompt_tokens":990,"completion_tokens":3287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":3202}},"tokens_in":606,"tokens_out":3287,"duration_ms":20994,"temperature":1.0,"reasoning_tokens":3202,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:20:12.467327+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate 1,000 scenes from the released Restaurant model, run the stated post-processing, and record how many fail the SNOPT projection step outright, then re-simulate the surviving 'stable' scenes for 10 seconds to see whether objects judged at rest after 0.1 seconds stay at rest. A substantial fraction of failures on either check would show that the feasibility guarantee, and the MCTS reward built on it, do not hold as stated.","supporting_citations":[{"cited_title":"Izatt and R","cited_arxiv_id":null,"evidence_quote":"Supplies the procedural scene grammars that generated the 44-million-scene training data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the unordered object-set scene representation with an empty-object slot for variable scene sizes, plus the DiffuScene baseline."},{"cited_title":"Black, M","cited_arxiv_id":null,"evidence_quote":"The DDPO algorithm that carries the reinforcement-learning post-training steering route."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The R9-plus-SVD rotation representation that lets the model learn and sample full $\\mathrm{SO}(3)$ poses."},{"cited_title":"Tedrake and the Drake Development Team","cited_arxiv_id":null,"evidence_quote":"The Drake simulator that defines physical feasibility: it settles scenes under gravity and evaluates penetration and static equilibrium for the MCTS mask and reward."},{"cited_title":"Kocsis and C","cited_arxiv_id":null,"evidence_quote":"The UCT bandit rule that selects which partial scene to expand during the inference-time MCTS search."}],"review_version":1}