{"id":"cfd3a240-ad43-4d66-9aff-18023b2c5c3e","arxiv_id":"2501.00602","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A feed-forward Transformer turns sparse multi-view video frames into 3D Gaussians with velocities, reconstructing dynamic driving scenes in 0.2 seconds and estimating scene flow without motion labels.","lead":"STORM is a neural network that reconstructs moving outdoor scenes from a few posed camera frames in one forward pass, taking about 0.2 seconds per two-second clip. It also estimates object motion and moving-object masks without explicit motion labels, which matters for self-driving simulators and real-time 3D editing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'self-supervised' claim is internally contradicted: §A.2 tests on externally provided depth maps D and pre-trained sky masks M, so 'using only reconstruction losses' is not established.","rationale":"The reader's CONDITIONAL verdict is appropriate, and I agree that Eq. (1)'s constant-velocity assumption is a real limitation; the absence of an acceleration ablation is worth checking. But I think the more load-bearing, immediately verifiable defect is the contradiction between the paper's 'self-supervised' and 'using only reconstruction losses' claims and the loss defined in §A.2. The strongest claim includes 'self-supervised scene flows' and 'using only reconstruction losses'; if external depth and sky-mask supervision are required, that claim is false, and the 'emergent' motion segmentation and flow estimation lose much of their claimed novelty. This is not a semantic quibble: Eq. A2/A3 literally feed D and M into the training loss. The decisive fix is a single ablation that removes both external signals. I therefore keep the reader's CONDITIONAL verdict unchanged: the concern strengthens the conditions (disclose all supervision, add the ablation) but does not warrant rejecting the reconstruction results outright.","tokens_in":21498,"tokens_out":7198,"duration_ms":70942,"concrete_test":"Retrain STORM with Eq. 7 restricted to RGB and LPIPS reconstruction only: remove the depth term from Eq. A2 and set λsky=0 in Eq. A3, keeping λreg. Train for the same 100k iterations on the same Waymo training split and evaluate dynamic PSNR/SSIM and EPE3D on the same validation clips. If dynamic PSNR drops by more than roughly 1 dB or training collapses, the headline results depend on external depth/sky supervision and the self-supervision claim fails; if not, this ablation should be reported and the wording changed accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that STORM learns dynamic 3D reconstruction and scene flow 'using only reconstruction losses' and 'without explicit motion supervision.' The paper's own training loss says otherwise. Eq. A2 includes a depth term ||(Dhat-D)/max(D)||_1 against 'observed depth maps D,' and Eq. A3 includes Lsky = ||Ohat-(1-M)||_1 where M is 'predicted by a pre-trained segmentation model.' These are external geometric and semantic supervision signals, not reconstruction losses. A self-supervised system that derives dynamics purely from RGB reconstruction would require an ablation removing D and M and showing the model still reaches its reported dynamic PSNR and EPE3D. No such ablation appears: the ablations in Fig. B.1 vary only λreg and M, and §A.3 states all models are trained with 'the same color, depth, perceptual and sky supervision.' Thus the novelty and the 'emergent' motion-mask and scene-flow claims are confounded: the predictor may be learning to regress externally supplied geometry and sky semantics rather than acquiring dynamics from reconstruction alone. This is an internal inconsistency, not a disagreement with community consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents STORM, a feed-forward Transformer that takes sparse, posed, multi-view images at several timesteps and outputs per-frame 3D Gaussians plus velocity bases. Predicted Gaussians are transported to a target timestep using piecewise-constant velocities and aggregated into an amodal representation; training combines color, depth, perceptual, sky, and velocity-regularization losses. Experiments on Waymo Open, NuScenes, and Argoverse2 report higher dynamic-region PSNR/SSIM than per-scene optimization and feed-forward baselines, lower depth RMSE, and improved scene-flow EPE3D/Acc5 at 0.18s inference. The paper also claims self-supervision and emergent motion masks using only reconstruction losses.","tokens_in":21743,"tokens_out":6706,"duration_ms":65718,"significance":"If the results withstand scrutiny, STORM is a meaningful advance: it is the first feed-forward model to reconstruct dynamic outdoor scenes and estimate camera-only scene flow in a single pass, with strong cross-dataset generalization and real-time speed. The use of public benchmarks, the ablations for velocity regularization, motion-token count, and input timesteps, and the conservative comparison against LiDAR-based flow baselines are notable strengths. The central 'self-supervised / only reconstruction losses' claim, however, is contradicted by the actual loss function, which includes externally provided depth maps and pre-trained sky masks. The significance of the emergent segmentation and scene-flow results therefore depends on additional ablations that separate those external cues from pure reconstruction.","major_comments":[{"comment":"The paper repeatedly claims self-supervision 'using only reconstruction losses' (abstract and §3.1), but Eq. (A2) includes a depth term ||(Dhat-D)/max(D)||_1 against observed depth maps D, and Eq. (A3) includes Lsky = ||Ohat-(1-M)||_1 with M supplied by a pre-trained segmentation model. These are external geometric and semantic supervision signals, not reconstruction losses. Since Appendix A.3 states that all models are trained with 'the same color, depth, perceptual and sky supervision,' the reported gains and the 'emergent' motion-mask and scene-flow claims are confounded with supervised depth and sky cues. Please add an ablation that removes the depth term and the sky term (or replaces them with self-supervised monocular estimates) and reports dynamic-region PSNR/SSIM and EPE3D/Acc5. Without such an ablation, the central self-supervision claim is not established.","section":"Abstract, §3.3 Eq. (7), Appendix A.2 Eqs. (A2)-(A3), Appendix B.2"},{"comment":"The motion model assumes piecewise constant velocity over the clip, as stated in Eq. (1): mu_{t->t'} = mu_t +/- (t'-t)v_t. This assumption is described as an empirical finding but is not ablated; no experiment considers accelerated motion, turning, or non-rigid deformation. Because the amodal aggregation in Eq. (2) and the scene-flow evaluation in §4.2 both build on Eq. (1), the reported dynamic-region PSNR and EPE3D gains may partly reflect a favorable test distribution (gentle longitudinal motion over 0.5s intervals) rather than a general motion model. Please add an evaluation on high-acceleration or turning segments, or compare against a quadratic or rigid-body motion variant, to quantify the impact of this assumption. At minimum, the limitation should be stated explicitly in the main text rather than only in passing.","section":"§3.1 Eq. (1), §4.2"}],"minor_comments":[{"comment":"Figures B.1 and B.2 each appear twice in the compiled PDF with identical content; please remove the duplicates.","section":"Appendix B.2, Figures B.1 and B.2"},{"comment":"The definition of 'dynamic regions' is not specified; state whether the evaluation mask comes from ground-truth bounding boxes, from the motion-token assignments, or from an external detector.","section":"§4.1"},{"comment":"For the scene-flow evaluation, clarify whether EPE3D/Acc5 are computed at 0.5s intervals and how the ground-truth flows are interpolated to the 1st, 5th, 10th, and 15th context frames; also state which STORM variant (STORM or Latent-STORM) produces the 'Ours' row in Table 3.","section":"§4.2"},{"comment":"The denominator max(D) in the depth loss is ambiguous: is it a per-image maximum, a per-batch maximum, or a global constant? Please clarify and add a small epsilon for numerical stability.","section":"Appendix A.2, Eq. (A2)"},{"comment":"The sign convention in Eq. (1) is stated as '+/-'; consider writing the backward and forward cases with explicit v^- and v^+ to avoid confusion with the two-velocity representation introduced earlier.","section":"Eq. (1)"},{"comment":"The sky token, affine token, and latent decoder are evaluated only qualitatively; a small quantitative table (e.g., PSNR/SSIM with and without each component) would strengthen the claims in §3.2.","section":"§3.2, Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be a strong contribution once the self-supervision overclaim is resolved. The gap between the abstract's 'only reconstruction losses' phrasing and the actual loss in Appendix A.2 is conspicuous and should be addressed before publication. The core method and evaluations appear sound, so I would not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read STORM (arXiv:2501.00602). The headline: this is a real step forward—a feed-forward, scene-level dynamic 3DGS model that reconstructs 2-second outdoor clips in 0.2s and beats per-scene optimizers on dynamic-region PSNR by 4-6 dB. If the numbers hold, it changes how people think about dynamic urban reconstruction as a learnable prior rather than per-scene fitting. The motion-token design (shared velocity bases with SAM-style assignment) is clean, and the cross-dataset results on NuScenes and Argoverse2 plus the camera-only scene flow numbers give the central claim teeth.\n\nThe paper is not, however, 'self-supervised using only reconstruction losses' as the abstract says. Appendix A.2 includes depth loss against observed depth maps D and sky loss against masks M from a pretrained segmentation model, and the baselines are trained with the same supervision. That is external geometric and semantic signal; 'self-supervised' overstates it. The reader flags this, and the stress-test note is right. An ablation dropping depth and sky supervision would settle whether the dynamic PSNR and flow gains come from reconstruction alone or from regression to external geometry. No such ablation is present; the ablations vary λreg, M, and timesteps. So the novelty claim is confounded at the margin, though not fatally—the core architectural contribution stands regardless.\n\nOther soft spots: no code, no error bars, and the per-scene baselines are run with reduced iterations and no LiDAR, which the authors explain but which could hide a gap. L4GM, the closest object-level feed-forward dynamic baseline, is cited but not compared; a same-protocol comparison would sharpen the claim. The constant-velocity assumption (Eq. 1) is stated as an empirical choice and not ablated; for two-second clips with braking/turning cars it's a real limitation, but the flow numbers suggest the model copes well in practice.\n\nThe citation pattern is fine—the paper builds on GS-LRM, EmerNeRF, DynMF/MoSca, SAM, and gives credit. The internal tables are consistent and the ablations are honest about the regularizer being necessary.\n\nWho is this for? Anyone working on autonomous driving simulation, 4D editing, or generalizable dynamic reconstruction. A serious referee should engage with it. My recommendation: send it to review, but demand that the authors either revise the self-supervision language, add the depth/sky ablation, or provide code so the ambiguity can be resolved. Conditional accept territory, not a reject.","headline":"Real feed-forward dynamic 3DGS result with strong numbers, but the 'self-supervised' claim doesn't survive the appendix: external depth and sky masks are in the training loss.","tokens_in":22349,"tokens_out":1859,"would_cite":true,"duration_ms":17384,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One feed-forward pass can rebuild a moving street scene from sparse camera frames in 0.2 seconds, with motion estimated from reconstruction losses alone.","keywords":["dynamic scene reconstruction","3D Gaussian splatting","feed-forward reconstruction","scene flow estimation","self-supervised learning","motion segmentation","autonomous driving","novel view synthesis"],"falsifier":"On clips with hard braking or sharp turns, measure object acceleration from LiDAR ground truth and compare STORM's camera-only scene flow against that ground truth separately for high-acceleration and low-acceleration objects; if constant velocity is load-bearing, EPE3D should rise with acceleration and dynamic-region PSNR should drop on the accelerating frames.","tokens_in":21284,"feed_emoji":"🚗","tokens_out":12802,"duration_ms":117272,"temperature":0.7,"pith_summary":"Normally, reconstructing a dynamic outdoor scene from sparse cameras means fitting every scene separately for minutes to hours, often guided by masks, optical flow, or LiDAR. STORM attempts to replace that with one feed-forward pass: given four posed frames from a two-second clip, a Transformer emits a 3D Gaussian cloud for each frame and a velocity for every Gaussian. Those velocities carry the clouds to a common time, where they merge into an amodal scene (one that includes regions no single frame saw), and the only training signal is the error between rendered and observed images. If the paper's results hold, dynamic street scenes can be reconstructed in about 0.2 seconds per clip, their motion estimated without LiDAR or pseudo-labels, and the learned motion tokens can even segment moving objects for editing.","feed_headline":"Sparse camera frames become moving 3D scenes in 0.2 seconds","feed_subtitle":"STORM infers geometry, velocity, and object masks together, beating per-scene optimizers without motion labels.","key_machinery":"The load-bearing object is the per-frame, pixel-aligned 3D Gaussian Splat: a cloud of anisotropic 3D Gaussians, one per image patch, each carrying center, rotation, scale, opacity, color, and forward/backward velocity vectors. Motion is transported by Eq. (1), a constant-velocity translation, and Eq. (2) takes the union of all transported clouds at the target time, yielding the amodal representation that is rendered and supervised. To keep motion low-dimensional, $M=16$ learnable motion tokens are prepended to the Transformer; a mask decoder computes softmax weights over $M$ velocity bases, so every Gaussian's velocity is a convex combination of shared motion primitives (Eqs. (3)--(4)). The training loss is reconstruction, sky-opacity, and velocity-regularization only; no optical flow, masks, trajectories, or LiDAR enter the loss.","core_discovery":"The central claim is that dynamic reconstruction and scene-flow estimation are one self-supervised problem: if the per-frame 3D Gaussians are carried to a target timestep by the correct velocities, the union of all transported clouds must re-render the observed frames, so reconstruction error alone can supervise motion. On the Waymo Open Dataset, STORM reports dynamic-region PSNR of 22.10, surpassing per-scene optimization methods by 4.3 to 6.6 dB and feed-forward baselines by 2.1 to 4.7 dB, with 0.18 s inference per clip; it also reports camera-only scene flow with 0.276 m EPE3D and 81.12% Acc5, above LiDAR-based baselines on the same metrics. An emergent property, the weights that combine shared motion bases form instance-like masks with no segmentation labels, and the latent variant reconstructs fine human motion and edits scenes by adding or removing Gaussians.","pith_inferences":["Because Eq. (1) fixes velocity for the whole clip, the approach is likely to degrade on hard braking, sharp turns, or articulated motion; a variant that predicts acceleration or piecewise trajectories would be the natural stress test.","With only $M=16$ shared motion bases, we would expect STORM to merge or drop distinct movers when a single clip contains more than about sixteen independently moving groups.","If this model scales like other feed-forward reconstruction Transformers, dynamic-region PSNR should improve monotonically with training-set size; a training-fraction sweep would settle that, and the paper does not run it."],"forward_implications":["Dynamic outdoor scenes can be reconstructed in about 0.2 s per two-second clip on a single A100 GPU, versus tens of minutes for per-scene fitting, making onboard or interactive use possible.","Scene flow is obtainable from cameras alone and is reported to beat LiDAR-fed flow methods in EPE3D and Acc5, so depth sensors would not be needed at test time.","Motion-token assignments give unsupervised motion masks, so objects can be removed, cloned, replaced, or added directly in the recovered Gaussian scene.","Because the model is a sequence-to-sequence Transformer, it transfers to fewer or more input timesteps without retraining and can be run iteratively to cover 20-second videos in under a second."],"supporting_citations":[{"why":"Supplies the 3D Gaussian Splatting representation whose per-frame clouds and velocities are the model's output.","marker":"Kerbl et al. (2023)"},{"why":"Supplies the self-attention architecture used to process the image, ray, and motion token sequence.","marker":"Vaswani et al. (2017)"},{"why":"Supplies the vision-transformer patch and token scheme used to embed images, ray maps, and time.","marker":"Dosovitskiy (2020)"},{"why":"Provides the GS-LRM baseline and the pixel-to-3DGS output parameterization that STORM follows.","marker":"Zhang et al. (2024)"},{"why":"Provides the mask-decoder design and query-key similarity operation reused to compute motion-base weights.","marker":"Kirillov et al. (2023)"},{"why":"Supplies the Waymo Open Dataset and evaluation protocol used for all main reconstruction and flow results.","marker":"Sun et al. (2020)"},{"why":"Defines the EPE3D, Acc5, and Acc10 metrics and the LiDAR-based NSFP baseline for scene-flow comparison.","marker":"Li et al. (2021a)"},{"why":"Motivates the amodal aggregation idea: accumulated observations over time form a complete scene representation.","marker":"Huang et al. (2022)"},{"why":"Provides the open-source Gaussian rasterizer used to render clouds into images for supervision and visualization.","marker":"Ye et al. (2024)"}],"fun_headline_variants":["STORM rebuilds dynamic 3D scenes from sparse views in 0.2s","Self-supervised STORM beats per-scene optimizers without motion labels","One forward pass: STORM outputs 3D Gaussians, velocities, and masks","Emergent masks and scene flow from reconstruction-only training","Sparse camera input to full dynamic 3D, STORM takes 200ms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes each 3D point glides at constant velocity for the roughly two-second clip (Eq. (1)), so vehicles that brake or swerve and humans who swing limbs are treated as piecewise-constant translations, and the reported reconstruction and flow gains inherit that assumption.","fun_headline_variants_meta":{"raw":{"variants":["STORM rebuilds dynamic 3D scenes from sparse views in 0.2s","Self-supervised STORM beats per-scene optimizers without motion labels","One forward pass: STORM outputs 3D Gaussians, velocities, and masks","Emergent masks and scene flow from reconstruction-only training","Sparse camera input to full dynamic 3D, STORM takes 200ms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1566,"prompt_tokens":1044,"completion_tokens":522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":420}},"tokens_in":660,"tokens_out":522,"duration_ms":4913,"temperature":1.0,"reasoning_tokens":420,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:46:57.472130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On clips with hard braking or sharp turns, measure object acceleration from LiDAR ground truth and compare STORM's camera-only scene flow against that ground truth separately for high-acceleration and low-acceleration objects; if constant velocity is load-bearing, EPE3D should rise with acceleration and dynamic-region PSNR should drop on the accelerating frames.","supporting_citations":[],"review_version":1}