{"id":"981be9f7-3f67-41cc-83a2-7c01ebfa4ed5","arxiv_id":"2506.08777","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Self-supervised pre-training with 3D Gaussian Splatting supervision improves downstream 3D object detection over a masked-autoencoder baseline on SUN RGB-D and ScanNetV2.","lead":"Gaussian2Scene pre-trains 3D point-cloud models by reconstructing masked 2D images and 3D points, then refining with 3D Gaussian Splatting renderings. It reports detection improvements over some earlier self-supervised baselines, mainly at the stricter AP50 threshold.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The second-stage gradient path from L_GS-image to the encoder is unspecified; if Scaffold-GS parameters are leaf tensors, the rendered-image loss never updates the backbone and the claimed joint 2D+3D supervision is not implemented.","rationale":"The reader's weakest assumption is precisely that the 3DGS branch is jointly optimizable with the backbone, and this is the condition on which the central claim depends. If the rendered-image loss cannot update the encoder, then the paper's headline contribution reduces to a one-way Chamfer loss from fitted Gaussian positions to reconstructed points, which is a different and weaker mechanism than the claimed end-to-end multimodal supervision. The paper's own wording in Section 3.2.2 ('after obtaining optimized 3DGS parameters') makes the ambiguity real rather than hypothetical, because it suggests a two-phase procedure in which Gaussians are fitted before the joint loss is applied. The ablation in Table 4 is not sufficient to resolve this, since the GS_IMG row could benefit from better 3DGS fitting or from random seed effects, and no error bars or code are provided. A gradient probe is a cheap, decisive check: it directly measures whether the purported gradient path exists. I do not see a stronger candidate concern. The scaling and detection results are secondary; if the gradient path is absent, the interpretation of all downstream gains changes. If the gradient path is present, the conditional verdict can be upgraded once code or gradient traces are released. Therefore the reader's CONDITIONAL verdict should remain unchanged pending this clarification.","tokens_in":11551,"tokens_out":5055,"duration_ms":61264,"concrete_test":"Run a gradient probe in the second-stage training loop: for a fixed batch, compute the norms of dL_GS-image/dtheta_enc and dL_GS-point/dtheta_enc, and print whether the Scaffold-GS anchor tensor is a leaf tensor or a differentiable function of the point-cloud decoder output. If dL_GS-image/dtheta_enc is exactly zero while anchors are leaf tensors, the 2D rendering loss never reaches the backbone, so the claimed joint optimization in Eq. 8 is not what produces the Table 4 GS_IMG gain. A complementary rerun with stop_gradient applied to PGS in Eq. 5 would show how much of the AP50 gain actually flows through Prec.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that L_GS-image and L_GS-point (Eqs. 5-6, combined in Eq. 7 and added to L_stage1 in Eq. 8) back-propagate into the encoder-decoder. Section 3.2.2 says Gaussians are 'seeded' from reconstructed point clouds, that Gaussian properties are 'optimized directly through back-propagation,' and that the joint loss is applied 'after obtaining optimized 3DGS parameters.' In the standard Scaffold-GS implementation, anchor positions are leaf parameters initialized from the input points, not differentiable functions of the decoder output; once fitted, rendered images depend on the fitted Gaussian parameters, not on the encoder. Therefore L_GS-image has no path to the backbone unless the code re-parameterizes Gaussian positions through the decoder or back-propagates through the initialization during fitting. L_GS-point can reach the encoder through Chamfer distance to Prec, but that is a one-way geometric loss and does not by itself provide 2D rendering supervision. Without specifying stop-gradient scopes or parameter partitions, the contribution 'end-to-end multimodal optimization' is not established, and Table 4's GS_IMG row could reflect 3DGS fitting behavior rather than backbone pre-training.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Gaussian2Scene, a two-stage self-supervised pre-training method for 3D scene understanding. Stage one trains a dual-branch masked autoencoder on point clouds and RGB images with complementary masking and cross-modal reconstruction. Stage two seeds Scaffold-GS Gaussians from the point branch's reconstructed points and adds losses on rendered images (L_GS-image) and Gaussian positions (L_GS-point). The pre-trained encoder is transferred to 3DETR for object detection on SUN RGB-D and ScanNetV2, reporting AP improvements. The paper claims that explicit 3DGS supervision is computationally lighter and geometrically stronger than implicit volume rendering.","tokens_in":11834,"tokens_out":6569,"duration_ms":65211,"significance":"If the central mechanism works as described, the idea of using explicit 3DGS supervision in SSL is timely and potentially valuable, offering a way to obtain geometric supervision beyond 2D rendering. The two-stage progressive design is clear, and the downstream experiments cover two benchmarks and several ablations. However, the paper does not provide code, error bars, or quantitative comparisons with volume-rendering SSL methods; the significance hinges on the joint gradient path, which is not specified.","major_comments":[{"comment":"The central claim of end-to-end multimodal optimization requires that L_GS-image and L_GS-point back-propagate into the encoder-decoder. The text states that Gaussians are seeded from reconstructed point clouds, optimized directly through back-propagation, and that the joint loss is applied 'after obtaining optimized 3DGS parameters.' In the standard Scaffold-GS implementation, Gaussian anchor positions are leaf parameters initialized from input points, not differentiable functions of the decoder output. Once fitted, rendered images depend on the fitted Gaussian parameters, not on the encoder. L_GS-point can reach the decoder through the reconstructed points Prec in Eq. (5), but L_GS-image has no path to the backbone unless the code re-parameterizes Gaussian positions through the decoder or back-propagates through initialization. The paper should specify stop-gradient scopes, parameter partitions, or an explicit backprop-through-initialization scheme. Without this, Table 4's GS_IMG row (SUN AP50 31.0 vs 33.5 full) cannot be attributed to backbone pre-training.","section":"§3.2.2, Eqs. (4)-(8)"},{"comment":"The abstract and Section 5.2 claim consistent improvements over existing pre-training methods, but Table 3 shows Ours+3DETR at 59.2 AP25 on SUN RGB-D, below PiMAE+3DETR's 59.4, and the AP50 advantage over PiMAE is only +0.3 (33.5 vs 33.2). All numbers are from a single run with no error bars or significance tests; differences of 0.2-0.3 AP are likely within run-to-run variation. The authors should report multiple seeds with standard deviation and either soften the consistency claim or provide statistical support.","section":"§5.2, Table 3"},{"comment":"The ablation narrative is contradicted by the table entries. In Table 4, the GS_IMG-only row reports 63.3 AP25 on ScanNetV2, higher than the full model's 62.9, and GS_PC-only reports 63.0; the text acknowledges only that GS_IMG-only achieves the highest AP25. In Table 5, the combined-branch row without cross-modal reconstruction has ScanNetV2 AP50 42.4, which is better than the IMG-only row's 40.4, contradicting the claim that combining branches degrades AP50 'on both datasets' and is 'worse than either isolated branch.' The tables and text need to be reconciled, and the conclusion that the full model is best should be supported by consistent numbers.","section":"§5.3, Tables 4 and 5"},{"comment":"The paper's central motivation is that 3DGS is superior to volume-rendering SSL, but no quantitative comparison with volume-rendering methods (Ponder, Ponderv2, UniPAD) is provided on the same downstream benchmarks. The only comparison is PiMAE, which is a cross-modal MAE rather than a volume-rendering method. In addition, the claimed computational advantage ('alleviates the computational burden') is not substantiated with runtime or memory measurements. The authors should either add such comparisons or qualify the claim as conceptual.","section":"§2.3 and §5.2"},{"comment":"The closely related work GS-PT (Liu et al., 2025), which also exploits 3D Gaussian Splatting for point cloud self-supervised learning, is cited only as an inspiration and is not discussed or compared. Given the paper's novelty claim of using 3DGS for SSL, the relationship to GS-PT must be clarified, including differences in architecture, objectives, and results.","section":"§2, related work and references"}],"minor_comments":[{"comment":"The phrasing 'AP50 gains of 33.5%' in the introduction is inaccurate; it should read 'AP50 of 33.5%'. Similarly, Section 5.2 states 'boosting AP50 of 1.2% and 5.4%', but Table 3 indicates improvements of +3.2% on SUN RGB-D and +5.4% on ScanNetV2.","section":"§1 and §5.2"},{"comment":"The hyperparameters lambda_ssim and gamma in Eq. (4) and alpha and beta in Eq. (7) are never given numerical values, and the code is not released. This makes the reported results hard to reproduce.","section":"§3.2.2 and §4"},{"comment":"There are numerous typos and formatting issues, including 'imcomplete', 'volumn-based', 'seperated', 'representaion', and 'Improments', as well as the missing space in the PiMAE row of Table 3 ('59.433.2'). A careful proofread is needed.","section":"Throughout"},{"comment":"The PSNR comparison with Zhu et al. [2023] is qualitative; no baseline PSNR value is given, so the claim of 'surpassing' volume rendering is not verifiable from the text.","section":"§5.1 and Figure 3"},{"comment":"The t-SNE visualization is qualitative; adding a quantitative clustering metric (e.g., adjusted Rand index or silhouette score) would strengthen the claim of improved feature separability.","section":"§5.2 and Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a high density of self-citations (Fei et al. 2022-2024, Liu et al. 2025), and the closely related GS-PT is not compared or discussed beyond a citation. The editor may wish to consider novelty disclosure and whether the difference from GS-PT is sufficiently established. The manuscript fits the journal's scope, but the missing gradient-path specification and the inconsistent ablation tables are substantial concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuine idea: replace volume rendering with 3D Gaussian Splatting as the scene-level supervision signal in a two-stage masked autoencoder. That combination is new, the motivation against implicit rendering is sensible, and the two-stage recipe is concrete enough to reproduce if the code were released. The data-efficiency sweep and the attempted 2D-vs-3D ablations are also useful checks. I'll give it credit for that.\n\nThe soft spot is real and load-bearing. Section 3.2.2 says Gaussians are seeded from reconstructed point clouds, then optimized, then the joint loss L_GS-branch is applied. Nowhere does it say which parameters receive gradients from L_GS-image. If the Scaffold-GS anchors are leaf tensors—as in the standard implementation—and the rendering loss is computed after fitting Gaussians and detaching them, then L_GS-image has no path to the encoder. The paper claims 'end-to-end multimodal optimization' but never specifies stop-gradient scopes or a reparameterization that would make the rendered image depend on the backbone. This is not a minor omission; it is the claimed contribution. The stress-test note is correct on this point.\n\nThe empirical claims also overreach. Table 3 shows Ours+3DETR at AP25 59.2 on SUN RGB-D versus PiMAE at 59.4—so 'consistent improvements' is false on the paper's own numbers. The real gains are AP50 on ScanNetV2 (43.3 vs 39.4). The ablations in Tables 4 and 5 are hard to parse, and some single-branch rows beat the full model on AP25 or AP50. No error bars, no multiple seeds, no significance tests. The circularity concern is not an issue: downstream detection is a true external benchmark, and self-citations don't make the result circular. That part of the reader's take holds up.\n\nIf the gradient path is absent, the contribution reduces to an extra view-reconstruction finetuning stage. If it is present, the method is a plausible incremental improvement for indoor scene-level pre-training. The paper needs a major revision: specify the gradient flow explicitly, release code, add a baseline that isolates the rendering stage, and present results with error bars and a corrected narrative.\n\nThis deserves a serious referee despite my skepticism, because the core idea is plausible and the issues are addressable. I would not cite it in its current form.","headline":"Real idea—3DGS as explicit scene-level SSL supervision in a two-stage MAE—but the second-stage gradient path is underspecified and the paper's own tables do not support 'consistent improvements.'","tokens_in":12382,"tokens_out":2931,"would_cite":false,"duration_ms":37614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Gaussian2Scene uses explicit 3D Gaussian splatting as a self-supervised pre-training signal for scene-level point clouds, and reports that it improves downstream 3D object detection.","keywords":["self-supervised learning","3D Gaussian splatting","point cloud pre-training","masked autoencoder","3D object detection","cross-modal learning","scene representation learning"],"falsifier":"Measure the gradient norm of $L_{\\mathrm{GS-point}}$ and $L_{\\mathrm{GS-image}}$ with respect to the parameters of the point-cloud decoder during stage-2 training; if those norms are zero (because the reconstructed points are detached before seeding Scaffold-GS or the encoder is frozen), the reported AP50 gains cannot be caused by the joint supervision described by Eq. 8. A complementary check is to freeze the backbone during stage-2 and re-run fine-tuning; if the gains persist unchanged, the contribution is not the claimed joint optimization.","tokens_in":11342,"feed_emoji":"🎯","tokens_out":6601,"duration_ms":66006,"temperature":0.7,"pith_summary":"Gaussian2Scene is a scene-level self-supervised pre-training method that replaces implicit volume rendering with explicit 3D Gaussian Splatting (3DGS) as the reconstruction signal for point-cloud encoders. The paper claims that supervising a masked autoencoder with the geometric locations of Gaussian primitives and rendered RGB images produces representations that transfer to 3D object detection better than volume-rendering pre-training. Concretely, the authors report AP50 of 33.5% on SUN RGB-D and 43.3% on ScanNetV2 when the pre-trained backbone is transferred to 3DETR, versus 30.3% and 37.9% for the untrained 3DETR baseline and 33.2% and 39.4% for PiMAE pre-training. The method matters because it targets two known limitations of neural-rendering pre-training: high memory cost and the lack of direct 3D geometric supervision.","feed_headline":"Gaussian splatting pre-training lifts 3D detection to 43.3 AP50","feed_subtitle":"Two-stage masked autoencoder supervises geometry via Gaussian primitives, beating volume-rendering SSL on ScanNetV2.","key_machinery":"The load-bearing object is the two-stage cross-modal masked autoencoder with 3DGS as a differentiable renderer, plus the two losses that attach Gaussian splatting to the backbone: $L_{\\mathrm{GS-point}}$ (Eq. 5), a bidirectional nearest-neighbour Chamfer distance between optimized Gaussian centers $P_{GS}$ and reconstructed points $P_{rec}$, and $L_{\\mathrm{GS-image}}$ (Eq. 6), an L1 plus D-SSIM photometric loss between rendered and ground-truth images. The Scaffold-GS optimizer supplies the Gaussian primitives (center, covariance, color, opacity) whose explicit centers give direct 3D supervision; the photometric loss gives cross-modal 2D supervision; the stage-1 MAE provides modality-specific reconstruction and cross-attention fusion. The claim is that this combination forces the backbone to encode scene geometry rather than only view-dependent appearance.","core_discovery":"On its own terms, the paper establishes that the explicit, parameterized representation of 3DGS is a better pre-training target than the implicit radiance fields used by volume-rendering SSL. In the first stage, a dual-branch masked autoencoder reconstructs masked 3D point clouds and masked 2D images together with a cross-modal consistency loss. In the second stage, the reconstructed point cloud seeds Scaffold-GS Gaussians; the positions of the optimized Gaussian primitives are pulled toward the reconstructed points by a bidirectional nearest-neighbour loss $L_{\\mathrm{GS-point}}$, and rendered images are pulled toward ground-truth views by an L1 plus D-SSIM loss $L_{\\mathrm{GS-image}}$. The total stage-2 loss adds these 3DGS terms to the stage-1 losses, so the encoder-decoder is supervised jointly by geometry and appearance. The reported detection results are the evidence: pre-training with this recipe improves AP50 by 3.2 points on SUN RGB-D and 5.4 points on ScanNetV2 over 3DETR, and by 0.3 and 3.9 points over PiMAE.","pith_inferences":["If the central claim holds, the recipe should transfer to outdoor LiDAR scenes, where volume-rendering pre-training is costly; the same two-stage MAE plus Gaussian seeding could be run on autonomous-driving data, with the expectation that geometric supervision helps sparse long-range detection.","The ablations suggest a tunable trade-off between the image weight $\\alpha$ and the point weight $\\beta$; a task-aware weighting could yield further gains when the downstream metric is known in advance.","The paper pre-trains on a SUN RGB-D subset only; scaling pre-training to more scenes and more views per scene should improve transfer further, and the framework predicts larger AP50 gains on ScanNetV2, where the geometry is denser and better reconstructed."],"forward_implications":["Pre-training with explicit Gaussian primitives raises strict-overlap detection accuracy (AP50) while keeping coarse detection (AP25) competitive: +3.2 AP50 on SUN RGB-D and +5.4 on ScanNetV2 over the 3DETR baseline.","The two supervision channels contribute differently: 3D point supervision improves fine-grained box geometry (AP50), while 2D rendering supervision improves semantic recall (AP25), so the two losses are complementary rather than redundant.","The pre-trained backbone retains most of its coarse detection performance under reduced fine-tuning labels: at 70% of the training data, AP25 drops only 0.1 points on SUN RGB-D and 1.1 on ScanNetV2.","Because 3DGS renders at real-time speed with explicit geometry, the same pre-training recipe avoids the memory footprint of volume rendering, making scene-level SSL more practical at scale."],"supporting_citations":[{"why":"Introduces 3D Gaussian Splatting and differentiable rasterization, the rendering machinery the method uses for explicit scene reconstruction.","marker":"[Kerbl et al., 2023]"},{"why":"Scaffold-GS is the specific 3DGS implementation used to fit scenes and produce the Gaussian primitives that supply geometric supervision.","marker":"[Lu et al., 2024]"},{"why":"PiMAE provides the two-branch cross-modal masked autoencoder structure that stage one builds on and is the strongest pre-training baseline compared.","marker":"[Chen et al., 2023]"},{"why":"Ponder is the volume-rendering SSL method the paper contrasts with its explicit Gaussian representation.","marker":"[Huang et al., 2023]"},{"why":"3DETR is the downstream transformer detector used to measure transfer performance in all experiments.","marker":"[Misra et al., 2021]"},{"why":"Point-M2AE supplies the point-branch masked autoencoder backbone and multi-scale point tokenization.","marker":"[Zhang et al., 2022]"},{"why":"ScanNetV2 is one of the two benchmark datasets used to evaluate downstream 3D object detection.","marker":"[Dai et al., 2017]"},{"why":"SUN RGB-D is the pre-training dataset and the second downstream evaluation benchmark.","marker":"[Song et al., 2015]"}],"fun_headline_variants":["Explicit Gaussians beat implicit fields for 3D pretraining","Gaussian2Scene: +5.4 AP on ScanNet, better than volume rendering","Two-stage Gaussian pretraining lifts 3D detection scores","3DGS pretraining outperforms implicit radiance fields","Self-supervised 3D pretraining with explicit splats improves detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result depends on the 3DGS branch actually updating the shared encoder-decoder, yet the paper never states which parameters are frozen while Scaffold-GS is fitted and which receive gradients from $L_{\\mathrm{GS-point}}$ and $L_{\\mathrm{GS-image}}$.","fun_headline_variants_meta":{"raw":{"variants":["Explicit Gaussians beat implicit fields for 3D pretraining","Gaussian2Scene: +5.4 AP on ScanNet, better than volume rendering","Two-stage Gaussian pretraining lifts 3D detection scores","3DGS pretraining outperforms implicit radiance fields","Self-supervised 3D pretraining with explicit splats improves detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1743,"prompt_tokens":1094,"completion_tokens":649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":556}},"tokens_in":710,"tokens_out":649,"duration_ms":7527,"temperature":1.0,"reasoning_tokens":556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:02:06.517517+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the gradient norm of $L_{\\mathrm{GS-point}}$ and $L_{\\mathrm{GS-image}}$ with respect to the parameters of the point-cloud decoder during stage-2 training; if those norms are zero (because the reconstructed points are detached before seeding Scaffold-GS or the encoder is frozen), the reported AP50 gains cannot be caused by the joint supervision described by Eq. 8. A complementary check is to freeze the backbone during stage-2 and re-run fine-tuning; if the gains persist unchanged, the contribution is not the claimed joint optimization.","supporting_citations":[{"cited_title":"Pimae: Point cloud and image interactive masked autoencoders for 3d object detection","cited_arxiv_id":null,"evidence_quote":"PiMAE provides the two-branch cross-modal masked autoencoder structure that stage one builds on and is the strongest pre-training baseline compared."},{"cited_title":"Ponder: Point cloud pre-training via neural rendering","cited_arxiv_id":null,"evidence_quote":"Ponder is the volume-rendering SSL method the paper contrasts with its explicit Gaussian representation."},{"cited_title":"An end-to-end transformer model for 3d object detection","cited_arxiv_id":null,"evidence_quote":"3DETR is the downstream transformer detector used to measure transfer performance in all experiments."},{"cited_title":"Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training","cited_arxiv_id":null,"evidence_quote":"Point-M2AE supplies the point-branch masked autoencoder backbone and multi-scale point tokenization."},{"cited_title":"Sun rgb-d: A rgb-d scene understanding benchmark suite","cited_arxiv_id":null,"evidence_quote":"SUN RGB-D is the pre-training dataset and the second downstream evaluation benchmark."}],"review_version":1}