{"id":"4b7f47ee-cd47-4557-baae-81ddff3ec3f1","arxiv_id":"2505.13440","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-stage self-supervised pipeline, LVSM-style implicit pretraining plus explicit Gaussian Splatting and depth alignment, learns novel view synthesis and camera poses from uncalibrated videos.","lead":"This paper trains a neural view-synthesis model from raw, uncalibrated video frames, with no camera poses or depth labels. Its two-stage recipe, implicit latent pretraining followed by explicit 3D Gaussian alignment, reports strong new-view quality on RealEstate10K and DL3DV-10K without needing calibration.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline NVS numbers use a target-aware protocol the paper itself says is not novel view synthesis; the true-NVS margin is smaller and partly relies on test-view pose optimization.","rationale":"Good-faith reading: the two-stage idea is coherent, the Stage 2 ablations show large pose gains, and the target-aligned non-optimized results do beat PF3plat on both datasets. The conditional verdict is therefore appropriate. However, the strongest reported NVS numbers are produced by a protocol the paper itself says is not true NVS, which is a concrete, manuscript-attested flaw in the support for the central claim rather than a speculative ambiguity. The reader's gauge concern is real but secondary: relative pose metrics are invariant to the global similarity ambiguity, and the target-aligned protocol already aligns test poses into the model's gauge. The target-aware issue, by contrast, directly affects the headline NVS margins. The proposed check would determine whether the reported advantage survives without target-image leakage. Pending that check and the reader's other reproducibility conditions, the verdict remains conditional.","tokens_in":13910,"tokens_out":15805,"duration_ms":163836,"concrete_test":"Re-run the RealEstate10K and DL3DV-140 evaluations with the target image withheld from the Camera Transformer input in the target-aware protocol (i.e., predict the test camera from context frames only) and compare PSNR/SSIM/LPIPS to the reported target-aware numbers. If the PSNR drops by more than 2 dB or falls to the target-aligned level, the target-aware protocol was encoding the target image, and the abstract's high-quality NVS claim must be re-baselined on target-aligned non-optimized results across at least three seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's own Sec. 4.3 says of the target-aware evaluation: 'this actually encodes and decodes the target view, rather than performing true novel view synthesis.' Nevertheless, Tables 1 and 2 lead with target-aware results, and Sec. 4.4 concludes from them that our method 'achieves the best novel view synthesis quality.' Because the target view's image is fed to the camera network, the predicted test camera can carry target-specific appearance, so the large PSNR margins (26.53 vs. 22.84 on RealEstate10K) are not measurements of NVS. The only valid NVS evidence is the target-aligned block: without test-view pose optimization, Ours reaches 22.20 PSNR (vs. PF3plat 20.04); with 40 iterations of test-pose optimization, Ours* reaches 23.96. Thus the central NVS claim rests on a weaker margin than the abstract suggests, and the stronger margin comes from a protocol the authors themselves identify as encoding the target. This does not refute the method, but it is the most load-bearing weakness in the empirical case for the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage training strategy for novel view synthesis (NVS) models trained from uncalibrated video frames or multi-view images, without camera parameters, depth, or other geometric priors. In Stage 1, the model performs implicit latent reconstruction: it predicts per-frame latent cameras and context features and uses an LVSM-style view synthesis transformer to re-render input frames, supervised by MSE and LPIPS losses. In Stage 2, the pretrained model additionally predicts pixel-aligned 2D Gaussian primitives and is trained with a 3D Gaussian Splatting rendering loss and depth projection/smoothness losses to align the latent representation with physical 3D geometry. The paper also proposes an inference-time interpolated-frame scheme for the two-input-frame case. Experiments on RealEstate10K and DL3DV-140 report state-of-the-art NVS quality and competitive camera pose estimation compared with methods that use calibration, pose, or depth supervision. Ablations show that Stage 1 pretraining is necessary for convergence and that the two stages are complementary.","tokens_in":14089,"tokens_out":3398,"duration_ms":33630,"significance":"If the claims hold, the contribution is significant: it demonstrates a practical path toward training generalizable NVS and pose-estimation models on large-scale raw video without SfM preprocessing, and the two-stage design is a reasonable response to the optimization difficulties of fully self-supervised explicit 3D reconstruction. The paper ships code, and the ablation evidence that Stage 1 is necessary is concrete and internally consistent. However, the empirical support for the headline claim is weakened by the evaluation protocol: the leading tables use a target-aware protocol that the authors themselves acknowledge is not true novel view synthesis, and the pose metrics are partly insensitive on the chosen datasets. The central idea is defensible, but the paper needs a substantially restructured evaluation before the claims can be accepted as stated.","major_comments":[{"comment":"The headline NVS claim is based on the target-aware evaluation, which the paper itself states 'actually encodes and decodes the target view, rather than performing true novel view synthesis' (Sec. 4.3). Nevertheless, Tables 1 and 2 lead with target-aware numbers and Sec. 4.4 concludes from them that the method 'achieves the best novel view synthesis quality.' The valid target-aligned evidence is much weaker: on RealEstate10K, Ours reaches 22.20 PSNR versus PF3plat's 20.04, and the improved 23.96 PSNR (Ours*) requires 40 iterations of test-view pose optimization with the Gaussian Splatting rendering loss. The abstract and conclusion should be rephrased so that the true-NVS claim rests on the target-aligned results, and the target-aware protocol should be presented as a diagnostic or as a separate 'target-encoding' setting, not as the primary NVS comparison.","section":"Sec. 4.3, Tables 1-2, Sec. 4.4, Abstract"},{"comment":"The pose accuracy evidence is partly undermined by metric insensitivity. The paper notes that RealEstate10K contains many scenes with minimal camera rotation and that the untrained model already achieves RRA@5 = 36.6, the same value as the failed 'w/o Stage 1' model. Given this, RRA@5 alone cannot support the claim of 'accurate camera pose estimation.' The authors should report additional pose metrics such as median RRA/RTA, RRA@1, or results on a subset of scenes with large rotation and translation, and they should provide error bars or multiple-seed results, since all tables appear to report a single run.","section":"Sec. 4.5, Table 4"},{"comment":"The self-supervised Stage 2 losses do not observe metric scale or absolute orientation, so the model could in principle converge to a self-consistent but physically incorrect solution. The paper does not explicitly test this gauge ambiguity, and the datasets used (RealEstate10K and DL3DV) are dominated by relatively constrained camera motions. To substantiate the pose-estimation claim, the authors should evaluate on sequences with arbitrary or wide-baseline motions, or otherwise demonstrate that the predicted cameras align with ground truth beyond the near-identity-rotation regime where RRA@5 is already inflated.","section":"Sec. 3.4, Eqs. (6)-(7), Sec. 4.1"},{"comment":"The contribution of Stage 2 to true novel view synthesis quality is not isolated under the valid protocol. Table 3 (target-aligned) includes 'w/o Stage 1' and 'w/o IF' but no 'w/o Stage 2'; Table 4 includes 'w/o Stage 2' only under the target-aware protocol, which is not a measure of NVS. Since the paper argues that the two stages are complementary and that Stage 2 'enforces 3D consistency,' the target-aligned effect of removing Stage 2 should be reported to support that specific claim.","section":"Sec. 4.5, Tables 3-4"}],"minor_comments":[{"comment":"There is a typo in 'this acutally encodes and decodes the target view'—should be 'actually.'","section":"Sec. 4.3"},{"comment":"The captions read 'Qualitative Comparison,' but the tables contain quantitative metrics; the captions should say 'Quantitative Comparison.'","section":"Tables 1-2"},{"comment":"The projection loss is described for an i-th frame projected onto a j-th frame, but the notation does not specify how the loss is aggregated over i and j; please clarify the summation and the random sampling of j.","section":"Sec. 3.4, Eq. (6)"},{"comment":"The interpolated-frame scheme is introduced as an inference-time strategy, but it is unclear whether it is applied for all two-frame test cases and whether the comparison methods receive any analogous treatment; please state this explicitly.","section":"Sec. 3.5"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and potentially publishable, but the evaluation must be restructured so that the true-NVS claims rest on the target-aligned protocol, and the pose metrics need to be made more informative. I would not reject on the basis of the current evidence, but I would not accept without these changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the two-stage scheme is the real contribution, and I believe it mostly works. The genuinely new bit is using LVSM-style implicit reconstruction as a pretraining warm-up before explicit 3D Gaussian and depth alignment, all without pose supervision. The ablations support that Stage 1 is load-bearing: without it, the model collapses to 12.97 PSNR on the target-aligned protocol, and the learned Gaussians are meaningless. That is credible evidence that the implicit phase solves an optimization problem that explicit-only methods such as SelfSplat hit. DL3DV results also consistently favor the method in target-aligned metrics, which is a good sign.\n\nThe soft spots are real, though. The target-aware evaluation is not novel view synthesis. The paper itself says that protocol \"actually encodes and decodes the target view, rather than performing true novel view synthesis.\" Yet Tables 1 and 2 lead with target-aware numbers, and Section 4.4 concludes \"best novel view synthesis quality\" from them. In the target-aligned protocol, the margin over PF3plat is 2.2 dB, or 3.9 dB after 40 iterations of test-view pose optimization. That is still a decent result, but it is a weaker claim than the tables imply. The test-pose optimization also means part of the reported gain is not from the feed-forward model.\n\nOther issues are smaller. There are no error bars or multiple seeds anywhere. The RRA@5 pose metric on RealEstate10K is partly trivial because many scenes have near-identity rotation; an untrained model already scores 36.6, so the pose numbers are less informative than they look. The gauge ambiguity of self-supervised camera/geometry alignment is acknowledged only implicitly, and the paper never tests on scenes with arbitrary large rotations. None of this sinks the central argument, but it should be addressed in revision.\n\nCitation pattern looks reasonable, and the code link is a plus even without a commit hash or full artifact. The paper is aimed at researchers working on camera-free NVS and self-supervised 3D; it deserves serious refereeing and would make a good reading-group discussion. I would recommend major revision: make the target-aligned evaluation primary, report variance, release reproducible code, and explicitly discuss the similarity-transform ambiguity.","headline":"The two-stage implicit-then-explicit training idea is genuinely new and mostly works, but the headline NVS numbers are inflated by a target-aware protocol that the paper itself admits is not true novel view synthesis.","tokens_in":14649,"tokens_out":2639,"would_cite":true,"duration_ms":27012,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-stage, fully self-supervised training can learn novel view synthesis and camera pose estimation from raw uncalibrated video frames alone, and beat methods that use camera calibration, pose, or depth supervision.","keywords":["novel view synthesis","uncalibrated video","camera pose estimation","self-supervised learning","3D Gaussian splatting","implicit scene representation","latent camera","video-based 3D reconstruction"],"falsifier":"Train the same pipeline on a set of videos with large, varied rotations and translations, and compare the predicted relative poses against a reliable structure-from-motion reconstruction on every scene; if the photometric losses stay low while pose errors grow large on the high-rotation scenes, the Stage 2 losses have found a self-consistent but geometrically wrong solution, and the pose accuracy reported on RealEstate10K would be inflated by its many videos with only small rotations.","tokens_in":13674,"feed_emoji":"🎥","tokens_out":10713,"duration_ms":90191,"temperature":0.7,"pith_summary":"Almost all state-of-the-art novel view synthesis models need calibrated cameras or geometric priors during training, which shuts them out of the massive supply of raw, uncalibrated video. This paper claims that a two-stage training scheme removes that requirement. The first stage pretrains the network as an implicit scene reconstructor: it predicts per-frame latent cameras and context features and uses a view synthesis network as a proxy renderer, so the model learns cross-view correspondence without any explicit 3D representation. The second stage then predicts explicit 3D Gaussian primitives and applies Gaussian splatting rendering, depth reprojection, and smoothness losses to pull the latent representation into alignment with real 3D geometry. If the claim holds, view synthesis and pose estimation can be trained directly from raw video at scale, without preprocessing pipelines such as structure-from-motion.","feed_headline":"Two-stage training lets view synthesis learn without camera poses","feed_subtitle":"A latent pretraining stage makes convergence possible; the alignment stage pins the model to real 3D geometry.","key_machinery":"The load-bearing mechanism is the pairing of two training objectives that share the same predicted cameras. In Stage 1, a view synthesis transformer with Plücker-coordinate camera embeddings—each ray encoded by its direction and moment—acts as the only renderer, making every target pixel differentiable to all input pixels while the camera parameters serve as an intermediate latent code; the strict-subset context rule prevents the code from degenerating into a frame index lookup. In Stage 2, the same cameras are used to back-project predicted depth into 3D Gaussian primitives, and the Gaussian splatting rendering loss plus depth projection and smoothness losses force the latent code to respect real 3D geometry. The essential identity is that the cameras must satisfy both the implicit rendering objective and the explicit geometric rendering objective simultaneously, which is what transfers the latent representation into a physically consistent world frame.","core_discovery":"On the paper's own terms, the discovery is that implicit latent reconstruction pretraining and explicit geometric alignment are complementary, and together they let a single network produce both novel views and camera poses from uncalibrated video. In Stage 1 the model treats the predicted per-frame camera as a latent code with the right number of degrees of freedom to encode the target image; the context frames are always a strict subset of the input frames so the network cannot trivially copy the target. In Stage 2 the model predicts pixel-aligned 2D Gaussian primitives, with centers obtained by back-projecting predicted depth through the predicted cameras, renders them with Gaussian splatting, and adds depth projection and edge-aware smoothness losses alongside the Stage 1 rendering loss. The authors report that on RealEstate10K this self-supervised model reaches 26.53 dB PSNR for target-aware novel view synthesis, surpassing baselines trained with intrinsics, poses, depth, or matching, and that on DL3DV-140 it also leads in pose accuracy; ablations show that without Stage 1 training fails to converge, and without Stage 2 the latent cameras are not anchored to real space.","pith_inferences":["If the result transfers to longer, more diverse videos, it suggests the field's reliance on precomputed calibration is partly a habit rather than a hard requirement; uncalibrated training could become the default for large-scale 3D foundation models.","The static-scene limitation points to a concrete extension: replace the rigid world assumption with per-frame latent motion or scene flow in Stage 1, then align a dynamic Gaussian representation in Stage 2.","A reader could probe how much geometry the implicit stage already learns by inspecting the latent cameras before Stage 2; the ablations show rotations and translations already trend toward ground truth, which is more geometric content than the paper's framing claims.","The depth reprojection loss in Stage 2 samples a single neighbor frame per projection; using multiple concurrent projection targets might lock down poses more tightly on long videos and would be a direct testable modification of the loss."],"forward_implications":["Training view synthesis models directly on raw video removes the structure-from-motion or SLAM preprocessing step, which is the main bottleneck for scaling 3D vision models to diverse, unconstrained web video.","Because pose estimates emerge from the same network without pose supervision, the method offers a way to obtain relative camera trajectories from video alone, useful for downstream reconstruction, navigation, and video editing.","The ablations imply that the implicit pretraining stage is not an optional warm-up: removing it makes the explicit Gaussian alignment fail to converge, so latent correspondence learning is a necessary precondition for self-supervised geometric alignment.","The interpolated-frame inference scheme shows that two-view inputs are a degeneracy point for the alignment; robustness improves whenever context overlap is increased, suggesting a direct trade-off between input frame spacing and reconstruction quality.","Enforcing explicit 3D consistency in Stage 2 lowers image quality slightly relative to pure latent reconstruction, so applications that prioritize appearance over geometry would make a different trade-off than applications that need geometrically consistent scenes."],"supporting_citations":[{"why":"Supplies the implicit view synthesis renderer used in Stage 1 pretraining, whose full-image differentiability avoids the local-neighborhood gradient limits of explicit 3D representations.","marker":"[19]"},{"why":"Shows the explicit Gaussian-splatting plus depth-projection self-supervised recipe that Stage 2 adapts, and serves as the closest self-supervised baseline.","marker":"[20]"},{"why":"Provides the 3D Gaussian splatting rasterizer used for the Stage 2 rendering loss.","marker":"[21]"},{"why":"Provides the geometrically accurate 2D Gaussian primitive formulation the network predicts.","marker":"[15]"},{"why":"Supplies the depth reprojection loss that aligns predicted depth with the target view across frames.","marker":"[59]"},{"why":"Supplies the edge-aware depth smoothness loss used to regularize predicted and rendered depth.","marker":"[10]"},{"why":"Provides the large-scale video dataset used to train and evaluate the method.","marker":"[60]"},{"why":"Defines the RealEstate10K train-test split and the two-view context/test evaluation protocol used for comparisons.","marker":"[4]"},{"why":"A comparison baseline trained with intrinsics, depth, and a pretrained matching network that the method outperforms on DL3DV.","marker":"[13]"},{"why":"A comparison baseline that uses pose supervision and defines the target-aligned evaluation protocol.","marker":"[14]"}],"fun_headline_variants":["No camera poses, no priors: view synthesis from raw video","Two-stage training learns 3D from uncalibrated video","Latent pretraining + explicit 3D alignment: pose-free view synthesis","Self-supervised view synthesis: two complementary stages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the self-supervised photometric losses in Stage 2—Gaussian splatting rendering plus depth reprojection and smoothness—are strong enough to force the predicted cameras and geometry to agree with the true shape of the scene, even though no real-world scale, orientation, or camera calibration is ever observed.","fun_headline_variants_meta":{"raw":{"variants":["No camera poses, no priors: view synthesis from raw video","Two-stage training learns 3D from uncalibrated video","Latent pretraining + explicit 3D alignment: pose-free view synthesis","Self-supervised view synthesis: two complementary stages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000407,"raw_usage":{"total_tokens":2182,"prompt_tokens":1083,"completion_tokens":1099,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":1026}},"tokens_in":699,"tokens_out":1099,"duration_ms":10768,"temperature":1.0,"reasoning_tokens":1026,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:13:04.026414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same pipeline on a set of videos with large, varied rotations and translations, and compare the predicted relative poses against a reliable structure-from-motion reconstruction on every scene; if the photometric losses stay low while pose errors grow large on the high-rotation scenes, the Stage 2 losses have found a self-consistent but geometrically wrong solution, and the pose accuracy reported on RealEstate10K would be inflated by its many videos with only small rotations.","supporting_citations":[{"cited_title":"Unsupervised learning of depth and ego-motion from video","cited_arxiv_id":null,"evidence_quote":"Supplies the depth reprojection loss that aligns predicted depth with the target view across frames."},{"cited_title":"Unsupervised monocular depth estimation with left-right consistency","cited_arxiv_id":null,"evidence_quote":"Supplies the edge-aware depth smoothness loss used to regularize predicted and rendered depth."},{"cited_title":"Stereo magnification: Learning view synthesis using multiplane images","cited_arxiv_id":null,"evidence_quote":"Provides the large-scale video dataset used to train and evaluate the method."},{"cited_title":"pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction","cited_arxiv_id":null,"evidence_quote":"Defines the RealEstate10K train-test split and the two-view context/test evaluation protocol used for comparisons."},{"cited_title":"Unifying correspondence pose and nerf for generalized pose-free novel view synthesis","cited_arxiv_id":null,"evidence_quote":"A comparison baseline that uses pose supervision and defines the target-aligned evaluation protocol."}],"review_version":1}