{"id":"0359b2a6-999b-4112-8072-289cb30b46d1","arxiv_id":"2411.17190","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SelfSplat jointly predicts depth, camera poses and 3D Gaussians from unposed image triplets, and outperforms prior pose-free baselines on RealEstate10K, ACID and DL3DV.","lead":"SelfSplat is a neural network that builds 3D models from ordinary videos without needing camera positions or 3D training data. It pairs self-supervised depth and pose learning with 3D Gaussian Splatting, and reports better image quality than earlier pose-free methods on three large real-world datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'superior over previous state-of-the-art' claim is not yet evidenced because the closest pose-free 3D-GS baseline, PF3plat, is absent from the main comparisons, leaving the reported margins against older NeRF-based methods only.","rationale":"The reader's weakest_assumption (known camera intrinsics) is a disclosed and conventional assumption, and the static-scene caveat is acknowledged in the Limitations. In contrast, the absence of PF3plat and GGRT from the main comparison directly undermines the paper's central comparative claim: the reader flagged this in the rationale but did not make it the primary load-bearing point. I focus on the baseline gap because a single comparison against the closest pose-free 3D-GS method would settle whether 'superior over previous state-of-the-art' is true. The intrinsics assumption would remain relevant only if the method were claimed to be fully uncalibrated, which the paper does not claim. The verdict stays conditional: the empirical claim may be correct, but the evidence as presented is incomplete.","tokens_in":19226,"tokens_out":13606,"duration_ms":136200,"concrete_test":"Run the released PF3plat (and GGRT, if its code and protocol permit) on the RE10k, ACID, and DL3DV test splits used in Sections 5.1-5.2, using exactly the same context/target pairs, overlap categories, and evaluation protocol (including the use of the target image for pose estimation in all methods). Compare PSNR, SSIM, LPIPS, and pose errors against Tables 2, 3, and 6. If SelfSplat retains its margins, the state-of-the-art claim is supported; if PF3plat matches or exceeds SelfSplat, the central empirical claim is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: SelfSplat 'achieves superior results over previous state-of-the-art methods in both appearance and geometry quality.' The experiments in Section 5.2 compare against VAE, DBARF, FlowCAM, and CoPoNeRF on RE10k and ACID (Tables 2-5), and only FlowCAM on DL3DV (Table 6). PF3plat, cited in Section 2 as a pose-free feed-forward 3D Gaussian Splatting method, is a direct competitor for exactly this setting and is never evaluated on these benchmarks. GGRT, also cited there, is likewise absent. The supplementary Splatt3R comparison (Table 12) is on DTU only, uses a model trained on ScanNet++, and reports only appearance metrics, so it does not settle either the state-of-the-art ranking or the geometry-quality claim. Because the headline claim is a comparative statement, omitting the closest recent methods makes the numerical margins unverified: if PF3plat were run under the same evaluation protocol, the ranking could change. This is an evidential gap rather than an internal contradiction; nonetheless it is load-bearing for the paper's central assertion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SelfSplat, a feed-forward 3D Gaussian Splatting network that, given unposed image triplets, jointly predicts per-pixel Gaussian attributes, depth maps, and relative camera poses. It is trained end-to-end with a photometric reprojection loss and a rendering loss, without ground-truth poses or depth and without per-scene fine-tuning. The method combines a CroCo-initialized monocular encoder, a multi-view Swin Transformer encoder, a DPT-style fusion head, a matching-aware pose network, and a pose-aware depth refinement module. It is evaluated for novel view synthesis and relative pose estimation on RealEstate10K, ACID, and DL3DV, with additional cross-dataset generalization experiments, multi-context-view tests, and ablations. The paper reports strong appearance metrics against VAE, DBARF, FlowCAM, and CoPoNeRF, and the abstract claims 'superior results over previous state-of-the-art methods in both appearance and geometry quality.'","tokens_in":19456,"tokens_out":7903,"duration_ms":62344,"significance":"If the results hold, SelfSplat is a useful contribution: a single-forward-pass, optimization-free model that removes pose supervision and per-scene fine-tuning for 3D-GS reconstruction. The ablations are informative and support the design choices: the reprojection loss and rendering loss cooperate, and the matching-aware pose network and depth refinement module each contribute measurable improvements. The cross-dataset generalization results (Table 7) and the extension to more than two context views (Table 11) strengthen the practical value. However, the state-of-the-art claim is not yet fully verified because the closest pose-free 3D-GS baselines cited in the paper, PF3plat and GGRT, are absent from the main comparisons, and the pose-estimation prose is contradicted by the small-overlap entries in Tables 4 and 5. The claim of superior 'geometry quality' is also not backed by any direct geometric metric. These issues are addressable within the manuscript's scope.","major_comments":[{"comment":"The abstract's claim that SelfSplat 'achieves superior results over previous state-of-the-art methods' is a comparative statement, but the two most relevant pose-free feed-forward 3D-GS methods cited in the paper, PF3plat [23] and GGRT [32], never appear in the main benchmark tables. This is especially conspicuous for DL3DV, where the data split is described as 'following PF3plat' (Section 5.1) but Table 6 compares only FlowCAM. The supplementary Splatt3R comparison (Table 12) is on DTU, uses a model trained on ScanNet++, and reports only appearance metrics, so it does not settle the ranking among pose-free 3D-GS models. I request that the authors either include PF3plat and GGRT under the same evaluation protocol or explicitly limit the superiority claim to the baselines actually compared.","section":"§2.1, §5.1, §5.2, Tables 2–6"},{"comment":"The sentence 'Our approach consistently achieves lower errors in both average and median deviations' is not consistent with the tabulated results for the small-overlap category. On RE10k (Table 4), CoPoNeRF has lower average rotation error (1.281° vs. 1.523°) and lower average translation error (10.173° vs. 14.954°) than SelfSplat; on ACID (Table 5), CoPoNeRF again has lower small-overlap rotation error (1.731° vs. 1.787°) and translation error (20.572° vs. 21.631°). The sentence should be corrected, and a brief discussion of the small-overlap failure mode should be added, since this is the regime where the matching-aware pose network was expected to help most.","section":"§5.2, Tables 4 and 5"},{"comment":"The claim of superior 'geometry quality' is not supported by any direct geometric evaluation. The paper reports appearance metrics (PSNR/SSIM/LPIPS) and relative pose errors, but no depth-error or surface-accuracy metrics; the depth discussion in Fig. 10 is qualitative. Furthermore, in the only direct comparison against a 3D-GS method on a shared benchmark, Table 12, the pose-required MVSplat outperforms SelfSplat on PSNR (13.94 vs. 13.14), SSIM, and LPIPS on DTU, which is not mentioned in the text. I recommend either adding quantitative geometry metrics or restricting the 'geometry quality' claim to the criteria actually evaluated.","section":"Abstract, §5.2, Table 12"}],"minor_comments":[{"comment":"The caption contains the typo 'pose estimaion'; it should read 'pose estimation'.","section":"Table 6 caption"},{"comment":"The spellings 'CoPoNeRF' and 'CopoNeRF' are used inconsistently across the paper; please unify to a single convention.","section":"Tables 1, 2–6 and text"},{"comment":"The label 'V AE' appears with a space that looks like a LaTeX artifact; please typeset it consistently as 'VAE'.","section":"Tables 2–6"},{"comment":"The Plücker ray embedding E_ext and the intrinsic ray embedding E_int are described only in prose; explicit formulas for their construction would improve reproducibility.","section":"§4.2"},{"comment":"The sentence 'The main reason Splatt3R cannot estimate a consistent scene scale is its reliance on a fixed pretrained MASt3R model, which is trained using metric camera poses, and difference between estimated intrinsic parameters and ground truth intrinsic parameters' is grammatically incomplete and should be rewritten.","section":"Supplementary B.3"},{"comment":"In the 'No Rendering Loss' row, the PSNR/SSIM/LPIPS entries are listed as '-'; please add a footnote explaining that these values are omitted because training without the rendering loss did not produce usable renders.","section":"§5.3, Table 8"},{"comment":"The paper states that all baselines were trained with the same curriculum; please clarify whether the published baseline numbers were reproduced by the authors or whether the models were retrained with their own scheduler, and whether this affects comparability with numbers reported in the original papers.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the missing comparison to PF3plat, which is a direct competitor for the exact pose-free 3D-GS setting and is even the source of the DL3DV split. If the authors cannot add this baseline, the state-of-the-art claim should be significantly softened. The small-overlap pose discrepancy between the text and Tables 4/5 is the kind of internal inconsistency that careful readers will notice immediately, so it must be fixed. The 'geometry quality' phrasing should also be aligned with what is actually measured."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something real: it trains a feed-forward 3D-GS network from unposed triplets using only photometric and rendering losses, with no pose supervision, no depth supervision, and no 3D-pretrained encoder (just self-supervised CroCo). That combination is new as far as I can tell. The matching-aware pose network and pose-conditioned depth refinement are sensible additions, and the ablations show each component earns its keep. The efficiency numbers are also genuinely good: about 2 ms per 256x256 view and 1.8 GB peak memory. That is a practical capability, not a marginal one.\n\nThe NVS results on RE10k, ACID, and DL3DV are solid against the baselines they did include (VAE, DBARF, FlowCAM, CoPoNeRF), and the cross-dataset generalization table looks believable. The experimental setup is described carefully enough to reproduce.\n\nBut the stress-test note has it right: the abstract says \"superior over previous state-of-the-art,\" and the closest pose-free feed-forward 3D-GS baselines, PF3plat and GGRT, are cited in Related Work but absent from the main comparisons. The supplemental Splatt3R comparison is on DTU, uses a model trained on ScanNet++, and only reports appearance metrics. So the headline comparative claim is not actually verified against the methods that would most likely threaten it. That is an evidential gap, not an internal contradiction.\n\nTwo smaller issues. First, the pose section says the method \"consistently achieves lower errors,\" but Tables 4 and 5 show CoPoNeRF beats SelfSplat on small-overlap rotation and translation on both RE10k and ACID. That is an overstatement in the text, though the overall averages do favor SelfSplat. Second, the \"geometry quality\" claim is supported only by qualitative depth visualizations; there are no quantitative depth or reconstruction metrics. If they want to claim geometry superiority, they should measure it.\n\nThe intrinsics-from-metadata assumption is standard for this line of work and is stated clearly; the dynamic-scenes limitation is also acknowledged in the Limitations section, which is honest.\n\nAll in, this is a well-engineered paper with a genuinely new training formulation and useful efficiency results. The flaws are addressable and not fatal. A serious referee should ask for PF3plat/GGRT comparisons, a fix to the pose phrasing, and ideally some quantitative geometry evaluation. I would send it to peer review.\n\nBring it to reading group: the architecture discussion is worth it, and the missing-baseline issue is a good case study in how SOTA claims outrun experiments.","headline":"A genuinely self-supervised feed-forward 3D-GS pipeline with strong NVS results, but the top-line SOTA claim is undercut by missing the closest baselines and by overstated pose/geometry claims.","tokens_in":20051,"tokens_out":2061,"would_cite":true,"duration_ms":20833,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SelfSplat jointly estimates depth, poses, and 3D Gaussians from unposed image triplets using only photometric and rendering losses, beating prior pose-free methods on three real-world datasets with no per-scene fitting.","keywords":["3D Gaussian Splatting","pose-free reconstruction","self-supervised depth estimation","generalizable novel view synthesis","monocular video","relative camera pose estimation","cross-dataset generalization","unposed multi-view images"],"falsifier":"Take a trained SelfSplat model and feed it held-out image triplets whose camera intrinsics are deliberately mis-specified, for example a focal length scaled by 20 percent, without changing the images. If pose and novel-view metrics stay about the same, the known-intrinsics premise is not what carries the method; if they degrade sharply, the premise is confirmed.","tokens_in":19023,"feed_emoji":"🎥","tokens_out":12029,"duration_ms":88542,"temperature":0.7,"pith_summary":"SelfSplat sets out to show that a single feed-forward network can reconstruct a 3D scene and synthesize novel views from a triplet of images whose camera poses are unknown, without any pretrained 3D priors or per-scene optimization. It does this by coupling self-supervised depth and pose estimation, trained with photometric reprojection losses, to a pixel-aligned 3D Gaussian Splatting pipeline trained with a rendering loss. The two losses reinforce each other: the reprojection signal positions Gaussians consistently, and the rendering signal sharpens pose and depth estimates. If the claim holds, reconstruction pipelines no longer need a separate structure-from-motion or pose-estimation stage, and models can be trained directly on large unlabeled video collections. The paper reports higher appearance and geometry scores than prior pose-free generalizable baselines on RealEstate10K, ACID, and DL3DV, along with stronger cross-dataset generalization.","feed_headline":"One pass yields depth, poses, and 3D Gaussians from unposed frames","feed_subtitle":"SelfSplat trains on raw video only, then reconstructs new scenes and synthesizes novel views without per-scene fitting","key_machinery":"The load-bearing object is the pixel-aligned 3D Gaussian representation fused with a self-supervised depth and pose loop. For each context image the network predicts per-pixel Gaussian attributes and a depth map; the depth is refined by a small U-Net that takes the predicted poses as Plücker ray embeddings, and the poses are estimated by a matching-aware network that concatenates cross-view features and camera-intrinsic ray embeddings. All Gaussians are transformed into the target view's coordinate frame by the predicted SE(3) poses and rendered with a differentiable splatting rasterizer. The training signal is the sum of a photometric reprojection loss and a rendering loss, with no ground-truth depth, poses, or 3D supervision, so the Gaussians act as the geometric substrate that makes the two loss terms share a consistent 3D scale.","core_discovery":"On its own terms, the central discovery is that an explicit 3D representation and a self-supervised geometric learning objective can be trained jointly from unposed videos with neither ground-truth poses nor pretrained 3D models, and that the two sides improve each other. SelfSplat predicts per-pixel 3D Gaussian attributes and depth for the two context views, predicts the relative SE(3) poses connecting context to target, unprojects the Gaussians into a common target-centered coordinate frame, and renders the target view through differentiable splatting. Training minimizes a photometric reprojection loss between actual and warped views together with a rendering loss between actual and splatted views. The ablation shows that removing either loss degrades pose or rendering accuracy: removing the rendering loss jumps rotation error from about 1.0 to 8.6 degrees and translation error from about 13.7 to 64.4 degrees on RealEstate10K at 50,000 iterations. The result is a single forward pass that yields depth, poses, and a 3D scene representation for novel-view synthesis, with no per-scene fine-tuning.","pith_inferences":["Beyond the paper: if the reciprocal pose-rendering loop transfers, adding a differentiable explicit renderer to other self-supervised depth and pose systems could reduce drift on long trajectories.","Beyond the paper: a natural extension the paper does not pursue is estimating camera intrinsics jointly with poses and depths, which would remove the known-calibration assumption from the pipeline.","Beyond the paper: because the method needs only unposed video, it could plausibly be pretrained on much larger and more diverse video collections than the three datasets reported, which would test whether the cross-dataset generalization continues to scale.","Beyond the paper: the supplementary experiments with three and four context views suggest the model already handles longer camera trajectories, but the paper stops short of claiming a full simultaneous localization and mapping system."],"forward_implications":["A single feed-forward pass on an unposed image triplet yields depth, relative camera poses, and a 3D Gaussian scene, so reconstruction pipelines can skip a separate structure-from-motion or pose-estimation preprocessing stage.","Because training uses only photometric and rendering losses, the model can be trained on large collections of raw monocular video without ground-truth poses, depths, or 3D scans.","The rendering loss feeds gradients back into pose estimation with an explicit functional form that the paper derives for rotation and translation, so the 3D representation actively regularizes pose learning rather than merely consuming poses.","Cross-dataset experiments, training on one dataset and testing on another without fine-tuning, show that the learned geometry transfers to new scene distributions."],"supporting_citations":[{"why":"Introduces the 3D Gaussian Splatting scene representation and differentiable rasterizer that the whole pipeline builds on.","marker":"[27]"},{"why":"Defines the pixel-aligned feed-forward Gaussian estimation framework that SelfSplat adapts to predict per-pixel Gaussian attributes from context views.","marker":"[3]"},{"why":"Provides the pixel-aligned Gaussian prediction and spherical-harmonic warping scheme used to move Gaussians into the target coordinate frame.","marker":"[50]"},{"why":"Supplies the pretrained self-supervised monocular vision transformer encoder that provides monocular features without any 3D priors.","marker":"[59]"},{"why":"Contributes the multi-view feature matching encoder design and serves as a direct prior work for the multi-view features in the model.","marker":"[7]"},{"why":"Is the pose-free generalizable baseline that relies on a pretrained flow model for pose estimation, the main comparison target for the pose-free setting.","marker":"[46]"},{"why":"Is a pose-free NeRF baseline that requires pose supervision during training, the comparison target for the claim of training with no pose labels.","marker":"[22]"},{"why":"Gives the photometric reprojection loss formulation used for self-supervised depth and pose training.","marker":"[18]"},{"why":"Provides the differentiable rasterizer used to render the predicted Gaussians.","marker":"[67]"}],"fun_headline_variants":["No poses, no priors, just 3D from raw video","SelfSplat: 3D from unposed images without finetuning","Single pass transforms unposed frames into 3D scene","Jointly learn depth, pose, and 3D Gaussians from scratch","Pose-free Gaussian splatting with self-supervised geometry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the camera's internal calibration is known and correct for every input; if those numbers are wrong, the ray embeddings and the unprojection that place Gaussians in 3D will be wrong in a way the photometric and rendering losses cannot fix, because both losses compare images in 2D.","fun_headline_variants_meta":{"raw":{"variants":["No poses, no priors, just 3D from raw video","SelfSplat: 3D from unposed images without finetuning","Single pass transforms unposed frames into 3D scene","Jointly learn depth, pose, and 3D Gaussians from scratch","Pose-free Gaussian splatting with self-supervised geometry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1689,"prompt_tokens":1014,"completion_tokens":675,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":582}},"tokens_in":630,"tokens_out":675,"duration_ms":6622,"temperature":1.0,"reasoning_tokens":582,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:26:12.785960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained SelfSplat model and feed it held-out image triplets whose camera intrinsics are deliberately mis-specified, for example a focal length scaled by 20 percent, without changing the images. If pose and novel-view metrics stay about the same, the known-intrinsics premise is not what carries the method; if they degrade sharply, the premise is confirmed.","supporting_citations":[{"cited_title":"pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction","cited_arxiv_id":null,"evidence_quote":"Defines the pixel-aligned feed-forward Gaussian estimation framework that SelfSplat adapts to predict per-pixel Gaussian attributes from context views."},{"cited_title":"Splatter image: Ultra-fast single-view 3d recon- struction","cited_arxiv_id":null,"evidence_quote":"Provides the pixel-aligned Gaussian prediction and spherical-harmonic warping scheme used to move Gaussians into the target coordinate frame."},{"cited_title":"Croco v2: Improved cross-view completion pre- training for stereo matching and optical flow","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained self-supervised monocular vision transformer encoder that provides monocular features without any 3D priors."}],"review_version":1}