{"id":"051e5193-18f4-459d-b2e3-db1e9c032ceb","arxiv_id":"2507.06710","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A diffusion-based visuomotor policy gains 3D and 4D scene awareness from a dynamic Gaussian world model, improving simulated and real robot manipulation success rates.","lead":"DP4 is a robot imitation-learning method that uses a dynamic 3D Gaussian world model to reconstruct the current scene and predict the next scene, feeding this spatial-temporal awareness into a diffusion-based action policy. The authors report higher task success rates than prior visuomotor policies on 17 simulation tasks and 3 real-robot manipulation tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that DP4's gains come from instilled 3D/4D structure is not established: L3D is same-view reconstruction, L4D is one-step prediction, and the ablation/evaluation evidence is internally inconsistent.","rationale":"The reader's weakest assumption is exactly the one I consider load-bearing: the auxiliary Gaussian losses must actually create spatial-temporal structure for the paper's mechanistic claim to hold. I agree that the current evidence does not establish this, and I do not see a more fundamental flaw in the method itself. The concern is strengthened by the internal inconsistency between the ablation text and Table 4, by the top-five checkpoint selection for Adroit/DexArt, and by the omission of DP3 from the RLBench comparison. These problems do not disprove the method, but they prevent accepting the central claim as stated. The reader's CONDITIONAL verdict is the right calibration, and the concrete test I propose would directly probe both the mechanism and the robustness of the reported empirical gains.","tokens_in":17362,"tokens_out":13019,"duration_ms":160834,"concrete_test":"On a held-out split of an Adroit/RLBench task, take a trained DP4 and evaluate the auxiliary heads directly: render the reconstructed current scene from a novel camera viewpoint not used in training and compare RGB/depth to ground truth; then, with the expert action a(t), render the predicted future scene at t+1 and compare to the true next frame. Also compute final-checkpoint success rates (not top-five). If novel-view/future-frame errors are near a DP3-without-Gaussian-loss baseline, or if final-checkpoint success no longer beats DP3 on RLBench, the reported mechanism and headline gains are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition is that L3D (Eq. 2) and L4D (Eq. 4) make the policy spatially and temporally aware rather than merely acting as regularizers. That condition is not demonstrated. L3D compares a Gaussian rendering to the same single view that generated the point cloud; with 3D coordinates already given by depth, the loss can be satisfied by per-point appearance overfitting, with no multi-view consistency. No novel-view evaluation is reported; Sec. 4.3 even concedes renderings 'lack fine detail.' L4D is one-step future-frame reconstruction using actions in Eq. (3); at inference the Gaussian world model is discarded (Sec. 4.4), so any benefit must transfer through the representation, yet no probing or control ablation shows temporal structure is what matters. The component ablation (Table 4) is internally inconsistent with the text (Door base is 64 in the table but 94 in the text; final DP4 is 80 in the table but 100 in the text), so component attribution is unreliable. Adroit/DexArt numbers are means of top-five checkpoints, which can inflate differences, and RLBench omits DP3, the primary baseline. If the auxiliary losses are only regularizers, or if the evaluation is optimistic, the central mechanism claim is unsupported even if headline success rates survive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 4D Diffusion Policy (DP4), a visuomotor diffusion policy augmented with two auxiliary scene-level losses: a 3D loss that reconstructs the current RGB-D observation via Gaussian splatting from a single view, and a 4D loss that predicts the next RGB-D frame from the current observation and action through a deformable Gaussian world model. The authors report consistent improvement over existing visuomotor policies across Adroit, DexArt, and RLBench simulation benchmarks (aggregate gains of +16.4%, +14.0%, and +6.45%) and across three real-robot tasks (+8.6% on average). The Gaussian world model is used only during training; at inference the policy relies on the multi-level 3D representation. The central claim is that these losses instill genuine 3D spatial and 4D spatiotemporal awareness in the representation, which in turn improves trajectory generation.","tokens_in":17628,"tokens_out":4329,"duration_ms":48601,"significance":"If the mechanism claim is correct, DP4 is a meaningful step toward explicit scene-level spatial-temporal supervision in visuomotor policy learning: it is one of relatively few approaches that couple a diffusion policy with a dynamic Gaussian world model, and the experimental scope (17 simulated tasks plus real-robot tasks) is appropriate for a policy-learning paper. The paper also provides architecture details and PyTorch snippets for the main components, which aids reproducibility. However, the significance is currently conditional: the paper does not show that the auxiliary losses actually produce geometric or temporal understanding rather than serving as generic regularizers, and the evaluation and ablation contain inconsistencies that weaken the quantitative claims. These issues are addressable, but they need to be fixed before the paper can be accepted.","major_comments":[{"comment":"The 3D spatial supervision is a same-view reconstruction loss: L3D compares a rendered RGB-D image to the exact input view from which the point cloud was created. Because the depth image already provides 3D coordinates, this loss can in principle be minimized by per-point appearance fitting without any multi-view or cross-geometry consistency. The paper calls this '3D spatial awareness' but provides no novel-view rendering evaluation or any other evidence that this loss enforces consistent 3D structure. The authors even concede in §4.3 that the renderings 'lack fine detail.' I recommend adding a quantitative novel-view or held-out-view rendering evaluation, or an ablation that removes depth supervision to separate geometry from appearance, to support the mechanism claim.","section":"§3.2, Eq. (2)"},{"comment":"The 4D loss is a one-step future-frame reconstruction computed only during training, and §4.4 states that the Gaussian world model is not constructed at inference. Therefore any benefit of L4D must transfer to the shared multi-level 3D representation. The paper does not provide evidence that L4D contributes temporal structure rather than acting as an extra regularizer. Concretely, I would like to see a control experiment where the future target in Eq. (4) is replaced by the current frame or by an action-independent random future frame, or a representation-probing experiment that checks whether the learned features encode future state information. Without such a control, the central '4D spatiotemporal awareness' interpretation is unsupported.","section":"§3.3, Eq. (4) and §4.4"},{"comment":"The component ablation contains explicit numerical inconsistencies between the table and the surrounding text. For the Door task, the text reports the base model at 94.0% and DP4 at 100.0%, whereas Table 4 reports 64.0% and 80.0%, respectively. For the Hammer task, the text says DP4 reaches 98.0% while the table shows 100.0%. Since this table is the primary evidence for component attribution (3D loss, 4D loss, and their combination), these inconsistencies make the attribution unreliable. The authors should correct the numbers and state clearly which result is final.","section":"§4.4, Table 4 and text"},{"comment":"The evaluation protocol for Adroit and DexArt uses the mean of the top five success rates across checkpoints, and the paper reports no significance tests for any benchmark. Selecting the best five checkpoints out of the evaluation history can systematically inflate reported performance and can inflate the measured gap between methods, especially with only three seeds. In addition, Table 3 (RLBench) omits DP3, which is identified in §4.1 as the primary baseline of the paper. Since DP3 is the closest point-cloud-based comparison and the one most likely to isolate the effect of the proposed auxiliary losses, its absence is a significant gap. I recommend reporting fixed-checkpoint performance with confidence intervals and adding DP3 to the RLBench comparison.","section":"§4.1 evaluation metric and Table 3"}],"minor_comments":[{"comment":"The abstract states '17 simulation tasks with 173 variants' and the introduction mentions 'four benchmarks,' but Section 4.1 only lists Adroit, DexArt, and RLBench and says 'we gathered 22 tasks.' Please reconcile these counts.","section":"Abstract and §4.1"},{"comment":"The text states that 'w denotes the camera pose used to project the Gaussian primitives,' but the camera pose does not appear in Eq. (3); the rendering is written as R(θ(t+1), p). Either include w in the notation or remove the undefined symbol.","section":"§3.3, Eq. (3)"},{"comment":"The caption and axis labels are garbled ('T+3 T+6 T+9T'). Please redraw the figure with clear ordering of the predicted future frames.","section":"Figure 5"},{"comment":"The real-robot evaluation would benefit from a statement of the number of trials and whether success rates are averaged over the same initial conditions; currently only percentages are reported without error bars.","section":"§4.5 and Table 5"},{"comment":"Typo: 'KINOV A GEN2' should be 'Kinova Gen2'.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the experimental scope is broad, but the paper's central interpretation ('spatial-temporal awareness') is not yet empirically established, and the ablation inconsistency plus the checkpoint selection protocol undermine the quantitative claims. The requested changes (novel-view/probing evidence for 3D/4D awareness, corrected ablations, standard evaluation, and inclusion of DP3 on RLBench) are well within the scope of a revision. I do not see evidence of bad faith, only under-supported interpretation and reporting issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a ManiGaussian variant with different encoders and a broader benchmark spread. The consistent success-rate gains across Adroit, DexArt, RLBench, and three real-robot tasks suggest the extra supervision does some good, but the paper doesn't show it instills genuine 3D/4D awareness, and the evaluation has problems that could inflate the numbers.\n\nTo its credit, the paper is not sloppy in the usual ways. It evaluates on a large set of tasks (17 sim tasks, 173 variants), includes real-robot results, and ships implementation details plus PyTorch snippets in the appendix. The multi-level local/global encoder is a reasonable addition over DP3. If the gains are real, this is a modest step forward for visuomotor imitation learning.\n\nSoft spots, in order of seriousness. First, novelty. ManiGaussian already uses a dynamic Gaussian world model to provide future-scene supervision for a manipulation policy. DP4 cites it only as a baseline and never clearly says what is genuinely new beyond the encoder architecture and benchmark set. A referee needs that delineation.\n\nSecond, the ablation is internally inconsistent. The text says the base Door success is 94% and final DP4 is 100%, but Table 4 shows 64% and 80%. The Hammer numbers also conflict (98 vs 100). This isn't a typo; it undermines the attribution of gains to the 3D/4D losses.\n\nThird, the evaluation protocol is optimistic. Adroit/DexArt report the mean of the top five checkpoints across training, which can exaggerate differences. Table 3 on RLBench omits DP3, the primary baseline, while including weaker 2D methods, and no significance tests are provided.\n\nFourth, the mechanism claim is not established. L3D compares a rendered image to the same single view that generated the point cloud, and L4D is a one-step future-frame prediction. There's no multi-view consistency test, no probing of the learned representation, and the supplementary admits renderings \"lack fine detail.\" The losses may simply act as regularizers. The headline improvements could survive either way, but the stated spatial-temporal-awareness story is unsupported.\n\nThe paper is for people working on diffusion policies for manipulation who want to know whether world-model supervision helps. I'd send it to peer review, but with major revision required: fix the ablation numbers, add DP3 to the RLBench comparison, clarify novelty relative to ManiGaussian, and either demonstrate that the world model imparts 3D/4D structure or soften the claim. As it stands, I wouldn't cite it as evidence for the mechanism.","headline":"A plausible but incremental extension of ManiGaussian: consistent gains across many benchmarks, but the spatial-temporal awareness story is not supported, the ablation text conflicts with Table 4, and a key baseline is missing.","tokens_in":18164,"tokens_out":3813,"would_cite":false,"duration_ms":92976,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training a diffusion visuomotor policy to reconstruct the current scene and predict the next scene through a dynamic Gaussian world model raises success rates by 16.4 percentage points on Adroit, 14.0 on DexArt, 6.45…","keywords":["visual imitation learning","diffusion policy","Gaussian splatting","world model","robot manipulation","3D spatial awareness","4D spatiotemporal dynamics","RGB-D observation"],"falsifier":"Train DP4 with the depth channel stripped from $L_{3D}$ while keeping the RGB reconstruction loss; if success rates on Adroit stay near 84.7%, then the 3D spatial mechanism is not what carries the gain. Alternatively, replace the ground-truth future frame in $L_{4D}$ with a randomly chosen future frame from another episode; if performance does not drop, the temporal consistency term is not the cause.","tokens_in":17169,"feed_emoji":"🤖","tokens_out":9336,"duration_ms":92949,"temperature":0.7,"pith_summary":"The paper sets out to show that imitation learning from recorded trajectories misses the 3D structure of scenes and the 4D dynamics of interaction, and that this gap can be closed with auxiliary supervision rather than new hardware. To that end it introduces DP4, a diffusion policy whose representation is trained by reconstructing the current single-view RGB-D scene as 3D Gaussians and by deforming those Gaussians under the candidate action to render the predicted next scene. The two extra losses—$L_{3D}$ for current-scene reconstruction and $L_{4D}$ for future-scene prediction—are added to the action-diffusion objective. Across 17 simulated tasks with 173 variants and 3 real-robot tasks, the paper reports higher average success than 2D and 3D diffusion baselines, including +16.4% on Adroit, +14.0% on DexArt, +6.45% on RLBench, and +8.6% on real robots. If true, this is a practical route to spatial-temporal awareness in visuomotor policies because the world model is used only during training and adds no inference cost.","feed_headline":"A scene-predicting robot policy beats baselines by 16.4%","feed_subtitle":"Extra 3D and 4D scene supervision lifts success on 17 simulated and 3 real robot tasks at no extra inference cost.","key_machinery":"The engine of the method is the dynamic Gaussian world model: a scene is represented as a set of 3D Gaussian primitives (positions, colors, rotations, scales, opacities) rendered into images by alpha-blending, and a deformable MLP predicts how those primitives change under an action to render the next frame. It does three jobs at once: it supervises the 3D global representation through rendered RGB and depth, it supervises temporal dynamics through the future-frame rendering loss, and it conditions the diffusion policy's trajectory generation on the resulting multi-level 3D features. Because the world model is discarded at inference, the claimed benefit is representation quality rather than extra computation.","core_discovery":"The central discovery is that a visuomotor diffusion policy can be made spatially and temporally aware by supervising a dynamic Gaussian world model learned alongside the policy, without changing how the policy acts at test time. From one RGB-D image the model builds a voxel scene representation, a generalizable Gaussian regressor turns it into 3D Gaussian primitives, and a rendering loss enforces agreement with the observed color and depth. A deformable mapper then shifts the Gaussian parameters using the current action, renders a predicted next RGB-D frame, and a second loss enforces agreement with the real future frame. The full objective is $L_{DP4} = L_{action} + \\lambda_{3D} L_{3D} + \\lambda_{4D} L_{4D}$, and the paper reports that the added terms improve success rates on all evaluated benchmarks while leaving inference time essentially unchanged.","pith_inferences":["A testable extension, not made in the paper, is to attach the same two rendering losses to non-diffusion visuomotor policies, since the loss structure is not diffusion-specific.","The paper evaluates only one-step future prediction, so a natural next test is a multi-step rollout of the deformable mapper; long-horizon tasks such as pouring would reveal whether the temporal model generalizes beyond a single step.","The reported gains could in principle come from regularization rather than geometric understanding; ablating depth from $L_{3D}$ or shuffling future frames in $L_{4D}$ would separate those explanations.","Because the world model is trained on the same view used by the policy, a further test of genuine 3D structure would be to render from a held-out camera pose and measure reconstruction error there."],"forward_implications":["Point-cloud diffusion policies gain consistent success improvements from 3D and 4D rendering losses across Adroit, DexArt, and RLBench, with the largest relative gains on dexterous and articulated manipulation.","In the ablations, the 4D future-scene term is the largest single lever, lifting Pen success from 47% to 75%.","A single RGB-D camera with known intrinsics and extrinsics suffices for both training supervision and deployment, so the method does not require a multi-camera rig.","Because the world model is training-only, the policy's inference cost stays close to the point-cloud diffusion baseline, making the added supervision a low-cost upgrade for existing visuomotor policies."],"supporting_citations":[{"why":"Supplies Gaussian splatting, the differentiable renderer and primitive parametrization used for 3D and 4D supervision.","marker":"[19]"},{"why":"The 3D point-cloud diffusion policy that DP4 builds its encoder and diffusion backbone on and reports gains against.","marker":"[45]"},{"why":"The 2D diffusion policy baseline that DP4 is compared with and whose action-diffusion framework it extends.","marker":"[5]"},{"why":"Provides the strongest dynamic-Gaussian world-model baseline on RLBench that DP4 must outperform.","marker":"[23]"},{"why":"Supplies the dynamic 4D Gaussian splatting idea used to deform the current scene into a predicted future scene.","marker":"[39]"},{"why":"Provides the DexArt articulated-object benchmark and task definitions used for evaluation.","marker":"[2]"},{"why":"Provides the RLBench benchmark and the 10 manipulation tasks with 166 variations used for evaluation.","marker":"[17]"},{"why":"Provides the Adroit dexterous manipulation benchmark and the expert demonstrations used for training.","marker":"[28]"}],"fun_headline_variants":["DP4: diffusion policy with 4D scene understanding, +16%","Gaussian world model lifts robot imitation success by 16.4%","Spatiotemporally aware diff policy improves robot tasks","4D scene prediction boosts robot success, no extra cost","Robot policy gains 4D awareness, beats baselines by 16%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single-view rendering losses teach genuine 3D geometry and temporal dynamics, since the 3D loss compares a rendered image with the same view that produced it and the 4D loss checks only one step ahead, so the model could in principle memorize appearance without building consistent structure.","fun_headline_variants_meta":{"raw":{"variants":["DP4: diffusion policy with 4D scene understanding, +16%","Gaussian world model lifts robot imitation success by 16.4%","Spatiotemporally aware diff policy improves robot tasks","4D scene prediction boosts robot success, no extra cost","Robot policy gains 4D awareness, beats baselines by 16%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000848,"raw_usage":{"total_tokens":3695,"prompt_tokens":955,"completion_tokens":2740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2648}},"tokens_in":571,"tokens_out":2740,"duration_ms":20609,"temperature":1.0,"reasoning_tokens":2648,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:57:24.119738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train DP4 with the depth channel stripped from $L_{3D}$ while keeping the RGB reconstruction loss; if success rates on Adroit stay near 84.7%, then the 3D spatial mechanism is not what carries the gain. Alternatively, replace the ground-truth future frame in $L_{4D}$ with a randomly chosen future frame from another episode; if performance does not drop, the temporal consistency term is not the cause.","supporting_citations":[{"cited_title":"3d diffusion policy: Gen- eralizable visuomotor policy learning via simple 3d repre- sentations","cited_arxiv_id":null,"evidence_quote":"The 3D point-cloud diffusion policy that DP4 builds its encoder and diffusion backbone on and reports gains against."},{"cited_title":"Diffusion policy: Visuomotor policy learning via action dif- fusion","cited_arxiv_id":null,"evidence_quote":"The 2D diffusion policy baseline that DP4 is compared with and whose action-diffusion framework it extends."},{"cited_title":"Manigaussian: Dynamic gaus- sian splatting for multi-task robotic manipulation","cited_arxiv_id":null,"evidence_quote":"Provides the strongest dynamic-Gaussian world-model baseline on RLBench that DP4 must outperform."},{"cited_title":"Dexart: Benchmarking generalizable dexterous manipu- lation with articulated objects","cited_arxiv_id":null,"evidence_quote":"Provides the DexArt articulated-object benchmark and task definitions used for evaluation."},{"cited_title":"Rlbench: The robot learning benchmark & learning environment","cited_arxiv_id":null,"evidence_quote":"Provides the RLBench benchmark and the 10 manipulation tasks with 166 variations used for evaluation."}],"review_version":1}