{"id":"dfebf686-6c8a-483c-8c75-3f938b4927ea","arxiv_id":"2506.18890","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"4D-LRM is a transformer that maps sparse posed frames scattered across time to a cloud of 4D Gaussians and renders any query view at any query time in under 1.5 seconds.","lead":"4D-LRM learns a continuous 4D representation of an object from sparse photos taken at different times and angles, then renders the object from new angles at new moments in a single pass. The authors report large gains over per-frame 3D reconstruction baselines and show the approach can be fine-tuned for 4D asset generation from a single video.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"4DGS representation's affine-in-time conditional Gaussians (Eq. 11) bound the central any-view/any-time claim to locally linear motion; Fig. 11 admits failures on non-linear and occluded trajectories, so the generalization claim is unscoped.","rationale":"I read the paper as making a deliberately broad claim: a single feed-forward transformer can take arbitrarily posed, timestamped images and output a representation that renders any novel view-time combination. For this to hold, the output representation must cover the dynamics in the test distribution. The adopted 4DGS representation is the point where this is least secure: Eq. (11) makes each primitive's conditional mean affine in t and its shape constant, so a curved or accelerating trajectory cannot be carried by one primitive. The authors themselves flag this in Fig. 11 and App. B.2, reporting ghosting and motion trails for non-linear motion and self-occlusion. No evaluation in Tables 1-3 stratifies by motion complexity or includes real-world dynamic video, so the extent of the limitation is unquantified. I considered other potential objections, including synthetic-only evaluation, dense-canonical underperformance versus GS-LRM in Table 3, and lack of released code and checkpoints, but those are validation or scope gaps rather than the constraint that most directly affects the headline wording. The representation-capacity limit is not an internal contradiction, and the paper presents substantial held-out synthetic evidence and an honestly reported failure case, so the appropriate verdict remains CONDITIONAL as the reader stated. The condition should explicitly require demonstrating the model on non-linear and occluded motions, ideally on real dynamic captures, before the broad 'any view at any time' claim is taken at face value.","tokens_in":25400,"tokens_out":10204,"duration_ms":112338,"concrete_test":"Construct a held-out benchmark of 60 Objaverse4D animations, 30 with high-curvature or accelerating motion (e.g., pendulum, bouncing, articulated hinges with self-occlusion) and 30 with matched linear or rotational motion, rendered under the four camera setups of Fig. 4. Run 4D-LRM-Large on both subsets and compare PSNR, SSIM, and LPIPS, plus a temporal-ghosting metric such as the fraction of pixels whose rendering contradicts ground-truth optical flow. If the non-linear or occluded subset degrades markedly (e.g., more than 2 dB PSNR) relative to the linear subset, Eq. 11's affine conditional mean is the binding limitation and the 'any view at any time' claim must be scoped to locally linear dynamics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support 'arbitrary novel view-time combinations', the decoded 4D Gaussian primitives must be able to represent the motions in the test objects. In the adopted 4DGS formulation, each primitive's conditional spatial mean is affine in time (Eq. 11: mu_xyz|t = mu_1:3 + Sigma_1:3,4 Sigma_4,4^-1 (t - mu_4)), and its conditional covariance is time-independent. Thus a single primitive can only move along a straight line with constant shape; curved, accelerating, or occlusion-breaking motion can only be approximated by many primitives. The paper's own failure analysis (Fig. 11, App. B.2) documents exactly this: non-linear trajectories and self-occlusion produce temporal ghosting and motion trails. This is a load-bearing boundary on the central 'any view at any time' claim, not a cosmetic artifact. Moreover, none of the Table 1-2 camera setups is stratified by motion complexity, and all evaluations are synthetic re-renders, so the regime in which the central claim holds is not measured. Without a test that isolates non-linear and occluded dynamics, the headline over-generalizes from smooth, well-covered motions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 4D-LRM, a transformer-based feed-forward model that takes sparse posed images with timestamps and predicts per-pixel 4D Gaussian primitives, adapting the 4DGS representation of Yang et al. The model is trained on a curated subset of Objaverse (animated objects from Diffusion4D, plus static objects with small linear displacements) and evaluated by re-rendering Consistent4D and a held-out Objaverse4D split. The paper reports PSNR above 30 on several structured camera setups, strong performance relative to GS-LRM and SoM, 24-frame reconstruction in less than 1.5 seconds on a single A100, scaling analyses, and an application to 4D generation by chaining with SV3D. The authors claim this is the first large-scale 4D reconstruction model that accepts unconstrained view-time inputs and renders arbitrary novel view-time combinations.","tokens_in":25663,"tokens_out":5801,"duration_ms":62315,"significance":"If the empirical claims hold, this is a meaningful step for feed-forward 4D reconstruction: a single model that jointly reasons over space and time, generalizes to held-out objects, interpolates across time, and handles diverse camera configurations. The strengths of the paper are its clear formulation of the unified space-time representation, the breadth of camera setups tested, the held-out evaluation protocol, the inclusion of external baselines, and a candid failure analysis. At the same time, the central claim is currently scoped too broadly: the representation itself can only express affine-in-time, constant-shape primitives, and all dynamic evaluations are synthetic re-renders, so the regime where the headline claim holds is not yet measured. The manuscript also lacks error bars on all main quantitative tables, which is important because several reported differences are small or non-monotonic.","major_comments":[{"comment":"The central claim of arbitrary novel view-time combinations is bounded by the 4DGS representation used. In Eq. (11), the conditional mean μ_{xyz|t} is affine in t and the conditional covariance is independent of t, so each primitive can only follow a straight-line, constant-shape trajectory. The paper's own failure analysis in App. B.2 and Fig. 11 confirms that non-linear trajectories and self-occlusion produce temporal ghosting and motion trails. Since all Table 1–2 evaluations are synthetic re-renders and no setup is stratified by motion complexity or occlusion, the regime in which 'any view at any time' holds is not measured. Please either add a test split that varies motion curvature and occlusion (or a real dynamic video benchmark) and report performance there, or restrict the headline claim to smooth, well-covered motions.","section":"§2.1, Eq. (11); App. B.2, Fig. 11"},{"comment":"All quantitative claims rest on single-run point estimates without error bars, repeated seeds, or significance tests. Several reported comparisons are close or non-monotonic, e.g., Table 3 shows 4D-LRM at 28.343 dB for 1 random view per frame versus 27.940 dB for 4 random views per frame, and Table 1 shows 128-resolution results that are sometimes higher than 256-resolution results for the same model. Without variance estimates it is hard to know whether the reported ordering is robust. Please report means with standard deviations over at least three seeds or a bootstrap over test objects for the main tables.","section":"Tables 1–3"},{"comment":"The evaluation contains no real-world dynamic scene. All 4D benchmarks are re-renders of synthetic Objaverse and Consistent4D assets, and the only real-image evaluation is the static GSO dataset in Table 5. Because the contribution is a learned general 4D prior, the paper needs at least a qualitative or quantitative evaluation on real dynamic videos (with the resulting limitations on segmentation, masks, and non-synthetic motion) to support the claim of general 4D reconstruction. Without this, the central result is only demonstrated for synthetic object videos.","section":"§3.2–3.3; §B.1"},{"comment":"The training data relies on the assumption that static Objaverse objects with minor frame-by-frame displacements along a single random direction are useful dynamic training data, and this assumption is not isolated in any ablation. The mixing ratio HQ4D:4D:static is fixed at 200:50:1, so the model's ability to handle arbitrary dynamics is only shown for the limited motion distribution of the training set. Please add an ablation or a test set with dynamics substantially outside the training distribution (e.g., articulated or occluded motions, non-linear paths) to support the scaling and generalization claims.","section":"§3.1, App. A.3"}],"minor_comments":[{"comment":"The text says the evaluation re-renders the first 48 (2×24) frames of Consistent4D, but Tables 1–2 and Figure 4 discuss 24-frame sequences; please clarify the exact frame range used in each table and figure.","section":"§3.1"},{"comment":"The architecture diagram includes Free Gaussian Tokens with dimension N, but the default pretraining uses no free Gaussians; please mark this component as optional and state N=0 for the main reconstruction experiments.","section":"Figure 3"},{"comment":"The perceptual loss weight λ is set to 0.5 and the appendix says the relevant hyperparameters are 'relatively insensitive,' but no sensitivity study is shown; a short plot or table would make this claim verifiable.","section":"Eq. (3) and App. A.1"},{"comment":"For the 'Canon. 4' row, please state the total number of input images (4 views per frame × 24 frames = 96) and clarify whether the random-view rows use the same total input budget or the same number of views per frame.","section":"Table 3"},{"comment":"The notation for the decoded Gaussian parameters is split across Section 2.2, Algorithm 1, and App. A.1; unifying the names (e.g., g_scale vs. scale_xyz) and stating the depth clipping range [−1,1]^3 in the main text would improve readability.","section":"App. A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong systems contribution with a clear benchmark protocol and honest failure analysis. My main concern is scoping: the representation and the synthetic-only evaluation do not yet support the literal 'any view at any time' claim, and the missing error bars make several comparative statements fragile. These issues are addressable with additional experiments or a rescoped claim, so I recommend major revision rather than rejection. I do not see citation or novelty concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The paper delivers the first feed-forward model that takes sparse posed images scattered across time and renders arbitrary novel view-time combinations. That task formulation is genuinely new, and the results back it up on held-out synthetic objects: 4D-LRM beats GS-LRM and SoM across several camera configurations, interpolates frames convincingly, and runs in about 1.5 seconds for 24 frames. The scaling analyses and the ablation on temporal alignment are also useful. The authors deserve credit for putting a typical failure case and a limitations discussion in the main text, not hiding them in a footnote.\n\nThe main caveat is real but not fatal: the 4DGS representation bounds each primitive to affine motion in time (Eq. 11), so curved or occlusion-breaking trajectories need many primitives, and Figure 11 shows exactly the resulting ghosting. The paper claims \"any view at any time\" but does not stratify evaluation by motion complexity, and all headline numbers come from synthetic re-renders. No real dynamic video, no error bars, no code or checkpoints. Those are addressable gaps, not load-bearing flaws, but they do cap how much I trust the generalization claim.\n\nThe stress-test note about the affine-in-time constraint is correct and worth taking seriously as a scoping condition. I would not call it a refutation, because the representation is still expressive enough for many object motions and the authors explicitly acknowledge the failure mode. The larger concern is empirical: without real-world dynamic evaluation or a motion-complexity split, the reader cannot tell how wide the usable envelope is. The absence of code and data also makes the reconstruction claim harder to verify, though the architecture description is clear enough to reproduce with effort.\n\nWho gets value: anyone working on 4D reconstruction or generation, especially those tired of per-scene optimization. This is a solid base for follow-up work on longer contexts, higher resolution, and scene-level dynamics. I would send it to serious peer review, with the expectation of a revise-and-resubmit that adds a real-video test, a motion-complexity breakdown, and at least one error-bar column. I would cite it myself.\n\nRecommendation: accept for review, conditional on those additions.","headline":"A real step forward for feed-forward 4D reconstruction, with an honest limitations section; the \"any view at any time\" headline overstates what the affine-in-time Gaussian representation and synthetic-only evaluation can support.","tokens_in":26242,"tokens_out":1690,"would_cite":true,"duration_ms":21516,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"4D-LRM claims that a single feed-forward transformer can reconstruct a dynamic object from a few posed images at scattered times and render any view at any time, in under 1.5 seconds per 24-frame sequence.","keywords":["4D reconstruction","large reconstruction model","4D Gaussian splatting","novel view-time synthesis","spatiotemporal pretraining","dynamic object reconstruction","feed-forward transformer","frame interpolation"],"falsifier":"Take a held-out dynamic object whose motion is deliberately non-linear or includes occlusion (e.g., a swinging cloth or a closing hand), render sparse posed views around it, and run the released 4D-LRM: if PSNR against ground truth drops substantially and ghosting or motion trails appear, the central claim is false for such motions; if the render stays sharp, the representation generalizes beyond smooth trajectories.","tokens_in":25180,"feed_emoji":"🎞️","tokens_out":5812,"duration_ms":54462,"temperature":0.7,"pith_summary":"The paper argues that 4D reconstruction—rebuilding a moving object from a few images—can be treated as a large-scale pretraining problem rather than per-scene optimization. Its model, 4D-LRM, takes any number of posed images at arbitrary timestamps and, in one transformer forward pass, predicts a cloud of 4D Gaussians that can render any novel view at any novel time. The authors claim this is the first large-scale 4D reconstruction model with unconstrained spatiotemporal inputs, and report that a 24-frame sequence is reconstructed in under 1.5 seconds on a single A100 GPU, with PSNR above 30 on structured camera setups. If true, this would make dynamic-object reconstruction fast enough for practical 4D asset generation, AR/VR, and robotics pipelines, replacing slow optimization or unreliable generative synthesis.","feed_headline":"One forward pass rebuilds 24-frame dynamic objects in under 1.5s","feed_subtitle":"Sparse posed images at scattered times become novel views at any time, trained on synthetic object videos.","key_machinery":"The load-bearing object is the 4D Gaussian primitive: a 20-dimensional parameter vector with RGB color, four scales, two quaternions for 4D rotation, opacity, and a space-time center (x, y, z, t). Its covariance is a 4D ellipsoid, and Eq. (11) gives the conditional 3D Gaussian at any time $t$ by a closed-form marginalization and conditioning of the multivariate normal; the tile-based Gaussian rasterizer then splats these conditional Gaussians. A transformer with Plücker-ray and timestamp tokenization decodes one such primitive per pixel, optionally joined by learnable free Gaussian tokens for sparse-view or generative setups. This machinery is what lets the model share information across both views and time in a single forward pass and render at effectively continuous frame rates.","core_discovery":"The central claim is that a unified space-time representation, in which every primitive is a 4D Gaussian with a spatial and temporal center, lets a feed-forward transformer learn a general prior over dynamic objects. Given posed image tokens augmented with ray geometry and a timestamp, the model directly regresses per-pixel 4D Gaussian parameters; at render time each 4D Gaussian is marginalized in time and conditioned to a 3D Gaussian at the queried instant via Eq. (11). The paper reports that this representation, trained on rendered synthetic objects, generalizes to novel objects, interpolates missing timestamps, and handles diverse camera setups, matching or beating per-frame 3D reconstruction baselines when views are sparse, and serving as a fast, faithful backbone for 4D generation.","pith_inferences":["A reader might infer that extending the same architecture with long-context handling and test-time training—both named by the paper as future work—could push toward hundreds of high-resolution inputs without changing the core representation.","A testable extension is to train or fine-tune 4D-LRM on real in-the-wild dynamic video with estimated poses; the paper's smooth-ellipsoid motion assumption predicts visible ghosting on fast, non-rigid, or occluded motion, so real-world evaluation would show whether the synthetic-data prior transfers.","The 4D Gaussian representation ties reconstruction quality to trajectory smoothness; for articulated or branching motion, one could hybridize with deformation fields or trajectory splines, though the paper does not explore this.","Since static 3D reconstruction with time collapsed to zero still gives usable quality, the same model could serve as a single backbone for both 3D and 4D tasks."],"forward_implications":["A 24-frame dynamic object can be reconstructed in one forward pass in under 1.5 seconds on one A100 GPU, and rendering can be done at arbitrary intermediate timestamps, in principle at infinite frame rate.","The model generalizes to novel objects and to unconstrained camera setups, including random views per frame and missing half the timestamps, with PSNR above 30 in the easier canonical setups.","Scaling model size and training data improves reconstruction quality, and initializing from a static-scene LRM accelerates convergence.","With free Gaussian tokens and a generative prior for the first frame, the same model exceeds generation-based 4D methods in faithfulness and speed on held-out 4D benchmarks.","Performance improves with more input views up to about 48 views, then declines slightly because of overcrowded Gaussians and long input sequences."],"supporting_citations":[{"why":"supplies the 4D Gaussian representation with conditional 3D sampling at time t.","marker":"[82]"},{"why":"provides the pixel-aligned Gaussian formulation, Plücker ray conditioning, and pretrained initialization.","marker":"[92]"},{"why":"supplies the tile-based Gaussian rasterizer used for differentiable rendering.","marker":"[31]"},{"why":"provides the large pool of 3D object assets used to build the training dataset.","marker":"[13]"},{"why":"curates the filtered animated-object subset that forms the core 4D training data.","marker":"[38]"},{"why":"provides the dynamic-object benchmark used for evaluation.","marker":"[28]"},{"why":"defines the generation-based 4D baseline and the free-Gaussian fine-tuning setup.","marker":"[53]"},{"why":"provides a second generation-based 4D baseline for comparison.","marker":"[78]"},{"why":"supplies an optimization-based 4D baseline that struggles with unconstrained inputs.","marker":"[69]"}],"fun_headline_variants":["4D-LRM: any view, any time, one forward pass","Sparse views to full 4D in under 1.5s","4D pretraining learns general dynamic scene reconstruction","From few views at some times to any view at any time","24-frame dynamic object, one shot, on a single A100"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a moving object can be divided into smooth, ellipsoidal, locally linear pieces, so that each 4D Gaussian with a single time-variance captures one chunk of motion; the model's own failure case shows this premise breaks for non-linear trajectories and occlusion, producing ghosting artifacts.","fun_headline_variants_meta":{"raw":{"variants":["4D-LRM: any view, any time, one forward pass","Sparse views to full 4D in under 1.5s","4D pretraining learns general dynamic scene reconstruction","From few views at some times to any view at any time","24-frame dynamic object, one shot, on a single A100"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000447,"raw_usage":{"total_tokens":2231,"prompt_tokens":892,"completion_tokens":1339,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1250}},"tokens_in":508,"tokens_out":1339,"duration_ms":11938,"temperature":1.0,"reasoning_tokens":1250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:41:13.167091+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out dynamic object whose motion is deliberately non-linear or includes occlusion (e.g., a swinging cloth or a closing hand), render sparse posed views around it, and run the released 4D-LRM: if PSNR against ground truth drops substantially and ghosting or motion trails appear, the central claim is false for such motions; if the render stays sharp, the representation generalizes beyond smooth trajectories.","supporting_citations":[{"cited_title":"Real-time photorealistic dynamic scene representationandrenderingwith4dgaussiansplatting","cited_arxiv_id":null,"evidence_quote":"supplies the 4D Gaussian representation with conditional 3D sampling at time t."},{"cited_title":"Gs-lrm: Large reconstruction model for 3d gaussian splatting","cited_arxiv_id":null,"evidence_quote":"provides the pixel-aligned Gaussian formulation, Plücker ray conditioning, and pretrained initialization."},{"cited_title":"Diffusion4d: Fast spatial-temporal consistent 4d generationviavideodiffusionmodels","cited_arxiv_id":null,"evidence_quote":"curates the filtered animated-object subset that forms the core 4D training data."},{"cited_title":"L4gm: Large 4d gaussian reconstruction model","cited_arxiv_id":null,"evidence_quote":"defines the generation-based 4D baseline and the free-Gaussian fine-tuning setup."}],"review_version":1}