{"id":"5d62e73f-4931-4fb2-afad-62c63945add1","arxiv_id":"2607.05906","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"GaussFusion improves 3D Gaussian pre-training by adding image-text semantic alignment and a salience-guided multi-scale hole masking strategy, outperforming Gaussian-MAE on downstream 3D tasks.","lead":"This paper adds image and text supervision to 3D Gaussian Splatting pre-training, teaching the model semantic information alongside geometric reconstruction. A smart generalist might read it to understand how 3D scene representations can absorb knowledge from 2D vision-language models without needing extra inputs at inference time.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Text supervision uses category names from pre-training data, but category overlap with downstream classification benchmarks is unanalyzed — gains may partly reflect category-level supervision transfer rather than genuine cross-modal representation learning.","rationale":"The reader's weakest assumption — that single query tokens may lose too much geometric information — is a reasonable but secondary concern. The ablation (Table 5) shows the alignment tokens do capture useful signal, so the information bottleneck through two tokens is not the binding constraint. The more load-bearing issue is whether the observed gains reflect genuine cross-modal representation learning or category-level supervision transfer. The paper's own ablation evidence (text > image contribution, much larger gains on classification than segmentation) is consistent with category-level supervision being a primary driver. This does not invalidate the method — the gains are real and consistent across tasks — but it does mean the paper overstates the contribution of cross-modal alignment per se. The CONDITIONAL verdict is appropriate: the method is sound and clearly described, but the interpretation of gains would be strengthened by the category-overlap analysis and non-category text ablation proposed above. No code release further limits independent verification of the reproduced Gaussian-MAE baseline (†), which is the sole comparison point. These are incremental but legitimate gaps that keep the verdict at CONDITIONAL rather than moving it to ACCEPT.","tokens_in":17596,"tokens_out":4820,"duration_ms":374693,"concrete_test":"Run two checks: (1) Compute the category overlap between ShapeSplat's training objects and the ModelNet40/ScanObjectNN category sets, reporting the fraction of downstream categories present in the pre-training data. (2) Re-run the pre-training with text descriptions that exclude category names (using only generic attribute phrases like 'has four legs, flat surface') and re-evaluate on ScanObjectNN PB-T50-RS. If the gain over Gaussian-MAE drops by more than 50% (i.e., below ~1.9%), the improvement is largely driven by category-level supervision rather than the cross-modal alignment mechanism itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that cross-modal semantic alignment enables the Gaussian encoder to learn visual and language-level semantics, improving transferability. However, Section 4.1 states that text descriptions are 'composed of category names and templated attribute phrases.' The downstream tasks (ModelNet40, ScanObjectNN) are themselves category classification tasks. If ShapeSplat's object categories overlap with these downstream categories, the text branch is effectively providing category labels during pre-training, which directly benefits classification but is not necessarily evidence of learned transferable semantic structure. Two pieces of evidence are consistent with this concern: (1) Table 5 shows text supervision contributes more than image supervision (+1.93 vs +1.61 on OBJ-BG without GSHM), which the paper itself attributes to text providing 'direct object-level semantic constraints' (Section 4.3.1); (2) the gains on part segmentation (Table 3, +0.78 mIoU_C) are an order of magnitude smaller than on classification (+3.85% on PB-T50-RS), suggesting the benefits concentrate in category-level tasks rather than fine-grained structural understanding. The paper does not report category overlap between ShapeSplat and the downstream datasets, nor does it include an ablation with non-category text (e.g., attribute-only descriptions). Without this, it is unclear how much of the improvement comes from the proposed cross-modal alignment mechanism versus simple category-level supervision leaking into pre-training.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper proposes GaussFusion, a multimodal self-supervised pre-training framework for 3D Gaussian Splatting (3DGS) representations. The method extends masked Gaussian modeling (as in Gaussian-MAE) by introducing (1) cross-modal semantic alignment, where learnable image and text query tokens aggregate 3D Gaussian features via self-attention and are aligned to frozen ViT and CLIP text encoder outputs, and (2) Gaussian Salience-guided Multi-scale Hole Masking (GSHM), which constructs spatially continuous masked regions based on opacity and scale salience at multiple scales. Experiments on ScanObjectNN, ModelNet40, ShapeNetPart, and few-shot classification show consistent improvements over Gaussian-MAE, with the largest gain on the PB-T50-RS split (+3.85%). The ablation study (Table 5) isolates the contributions of image supervision, text supervision, and GSHM. The central claim—that integrating image and text supervision with salience-guided masking improves transferability of Gaussian representations—is defensible and supported by the experiments.","tokens_in":18359,"tokens_out":1113,"duration_ms":358231,"significance":"The paper addresses a timely problem: extending 3DGS pre-training beyond pure reconstruction to incorporate semantic supervision from frozen vision-language models. The GSHM masking strategy is a reasonable adaptation for the non-uniform distribution of Gaussian primitives, and the ablation in Table 5 provides useful isolation of each component. The use of learnable query tokens (rather than direct feature injection) for cross-modal alignment is a clean design choice that allows the Gaussian encoder to operate without multimodal inputs at inference time. The experimental gains over Gaussian-MAE are consistent across tasks. However, the comparison is limited to a single direct baseline, and the magnitude of gains on fine-grained tasks (part segmentation: +0.78 mIoU_C) is much smaller than on classification (+3.85% on PB-T50-RS), which raises questions about what is being learned that the paper does not fully address.","major_comments":[{"comment":"Section 4.1 states that text descriptions are 'composed of category names and templated attribute phrases.' The downstream tasks (ModelNet40, ScanObjectNN) are category classification tasks. If ShapeSplat's object categories overlap with these downstream categories, the text branch is effectively providing category-level supervision during pre-training that directly benefits classification. The paper does not report category overlap between ShapeSplat and the downstream datasets, nor does it include an ablation with non-category text (e.g., attribute-only descriptions). This is load-bearing for the central claim that cross-modal alignment improves 'transferable semantic structure,' because the gains may partly reflect category-label transfer rather than genuine representation learning. An analysis of category overlap and/or an ablation with attribute-only text would substantially clarify","section":null},{"comment":"The direct comparison is limited to Gaussian-MAE as the sole Gaussian-representation baseline. While the paper acknowledges a performance gap with point cloud methods (Section 4.2.1), the absence of comparison with other cross-modal 3D methods (e.g., ACT, ReCon, ULIP) adapted to the Gaussian setting, or with SceneSplat, makes it difficult to assess whether the gains come from the specific design choices in Section 3.2 or simply from adding any external semantic supervision. At minimum, the paper should discuss why these baselines were not included and whether the gains are expected to hold relative to them.","section":null}],"minor_comments":[{"comment":"Section 4.1: The paper does not specify which viewpoints are used for rendering images during pre-training (16 viewpoints are mentioned but not how many are sampled per object per epoch). Clarifying this would aid reproducibility.","section":null},{"comment":"Table 3: The improvement on part segmentation is modest (+0.78 mIoU_C) and GaussFusion underperforms Gaussian-MAE on several categories (bag, car, e-phone, laptop, pistol, table). The paper briefly attributes this to 'simple categories' but does not explain which categories are affected or why.","section":null},{"comment":"References [16]–[21], [26] appear to be by overlapping author groups and several are dated 2025–2026. These should be verified for availability and properly contextualized, especially if they are concurrent work.","section":null},{"comment":"Equation (3): The salience score uses the geometric mean of scale parameters, but it is unclear whether scales are in the original or log space. This should be stated explicitly.","section":null},{"comment":"Figure 2: The caption mentions 'frozen image and text encoders' but the figure itself does not clearly indicate which components are frozen vs. trainable. Adding freeze symbols would improve clarity.","section":null},{"comment":"Section 3.3, Eq. (17): The reconstruction loss is listed as a sum of five terms, then Eq. (18) rewrites it as L_xyz + L_attr. This is slightly redundant; consider merging.","section":null}],"recommendation":"major_revision","confidential_remarks":"The category-overlap concern is the most important issue. If ShapeSplat contains the same object categories as ModelNet40/ScanObjectNN, the text supervision is essentially providing labels for the downstream task, which would substantially weaken the contribution. The authors need to address this transparently. The paper is otherwise technically sound and well-written, but the single-baseline comparison and the unanalyzed category overlap are significant enough to warrant major revision."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful reading and constructive feedback. Both major comments raise legitimate concerns about the strength of the evidence supporting our central claim. Below we address each point.","responses":[{"response":"The referee raises a valid and important concern. We acknowledge that ShapeSplat's object categories (drawn from ShapeNetCore and other sources) do overlap with categories in ModelNet40, ScanObjectNN, and ShapeNetPart — for example, chairs, tables, airplanes, and lamps appear in both pre-training and downstream data. Because our text descriptions include category names, the text alignment branch does provide some degree of category-level semantic supervision during pre-training, and it is possible that part of the classification gains reflects category-label transfer rather than purely generalizable representation learning. We agree that the current manuscript does not adequately address this confound. To clarify the contribution of the text branch, we will add the following in the revision: (1) A quantitative analysis of category overlap between ShapeSplat and each downstream dataset, reported in the pre-training setup section. (2) An ablation experiment using attribute-only text descriptions (i.e., templated attribute phrases without category names) as the text supervision target, compared against the current category+attribute text. This will directly isolate whether category-name transfer is the primary driver of the text branch's gains. We note that even if some gains are attributable to category-level transfer, the image-only ablation row in Table 5 (85.89% vs. 84.30% baseline) shows that non-category visual supervision also contributes, and the part segmentation gains (+0.78 mIoU_C), while smaller, are consistent with the text branch providing structural rather than purely categorical information. Nevertheless, we agree the attribute-only ablation is necessary to make the central claim precise, and we will include it.","revision_made":"yes","referee_comment":"Category overlap between ShapeSplat and downstream datasets may mean the text branch provides category-level supervision that inflates classification gains. Requests category overlap analysis and/or ablation with attribute-only text."},{"response":"We agree that the comparison is narrower than ideal and that additional baselines would strengthen the paper. We offer the following clarifications and commit to partial revisions. Regarding SceneSplat: SceneSplat operates at the scene level and uses a different pre-training data regime (scene-level Gaussian splats rather than object-level). A direct comparison on object-level downstream tasks (ScanObjectNN, ModelNet40, ShapeNetPart) would require retraining SceneSplat on object-level ShapeSplat data, which is not its intended setting. We will add a discussion of this distinction and clarify why a direct comparison is not straightforward. Regarding ACT, ReCon, and ULIP: these methods are designed for point cloud inputs and rely on point-cloud-specific architectures (e.g., Point-MAE or Point-BERT backbones) and point-cloud-specific cross-modal alignment objectives. Adapting them to the Gaussian setting would require non-trivial modifications: replacing the point cloud encoder with a Gaussian encoder, redefining the masking and reconstruction targets for Gaussian attributes, and re-engineering the cross-modal alignment to operate over Gaussian tokens rather than point patches. This adaptation is itself a research question rather than a straightforward baseline reproduction. That said, we recognize that the absence of such comparisons limits the reader's ability to assess whether our specific design choices (learnable query tokens, GSHM) are necessary or whether any cross-modal supervision would suffice. To partially address this, we will: (1) Add a discussion in the experiments section explaining why these baselines were not directly included and what modifications would be required. (2) If feasible within the revision period, implement a simplified cross-modal baseline —","revision_made":"partial","referee_comment":"Direct comparison limited to Gaussian-MAE; no comparison with other cross-modal 3D methods (ACT, ReCon, ULIP) adapted to the Gaussian setting, or with SceneSplat."}],"tokens_in":17441,"tokens_out":1386,"duration_ms":132687,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Here's what you need to know: GaussFusion adds image and text cross-modal alignment to masked Gaussian reconstruction, plus a salience-guided masking strategy (GSHM) designed for the non-uniform distribution of 3DGS primitives. The gains over Gaussian-MAE are real but modest — +3.85% on ScanObjectNN PB-T50-RS, +0.61% on ModelNet40, +0.78 mIoU on part segmentation. The method still trails point cloud baselines like Point-MAE, but the paper is upfront about this and frames the comparison as same-representation (Gaussian vs. Gaussian), which is fair.","headline":"Solid incremental work on Gaussian pretraining with one real concern about category leakage from text supervision.","tokens_in":18411,"tokens_out":213,"would_cite":false,"duration_ms":90603,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Adding image and text supervision to 3D Gaussian pretraining","keywords":["3D Gaussian Splatting","self-supervised pre-training","multimodal learning","cross-modal alignment","masked autoencoder","point cloud understanding","vision-language models"],"falsifier":"If the alignment tokens learn to predict frozen image/text features through a shortcut that depends only on global object shape (e.g., overall bounding-box proportions visible from a single viewpoint), then removing the alignment tokens at fine-tuning time would yield no improvement over Gaussian-MAE on tasks requiring local geometric discrimination, such as part segmentation on categories with thin structures. A direct test: pre-train with alignment tokens but evaluate part-segmentation IoU per category; if gains concentrate on categories whose parts are distinguishable by global shape alone,","tokens_in":17884,"feed_emoji":"🎯","tokens_out":1149,"duration_ms":174218,"temperature":0.7,"pith_summary":"This paper argues that pre-training methods for 3D Gaussian Splatting representations, which have so far relied solely on reconstructing masked Gaussian attributes, learn local geometry but miss high-level semantics. The authors propose GaussFusion, a framework that adds cross-modal supervision from frozen 2D image and text encoders to the masked Gaussian modeling pipeline. Two learnable query tokens are prepended to the Gaussian token sequence inside a shared Transformer encoder; these tokens aggregate 3D features through standard self-attention and are then projected to match the output of a frozen ViT image encoder and a frozen CLIP text encoder via Smooth L1 loss. The image and text encoders are used only during pre-training and are discarded at inference, so downstream tasks receive a purely 3D model with no extra modal input. To make the masked reconstruction objective compatible with cross-modal alignment, the paper also introduces Gaussian Salience-guided Multi-scale Hole Masking (GSHM), which replaces random per-group masking with spatially continuous, multi-scale masked regions selected by a salience score derived from each Gaussian group's average opacity and geometric-mean scale. The combined system is pre-trained on the ShapeSplat dataset (~52,000 objects) and evaluated by transferring the Gaussian encoder to point-cloud classification on ScanObjectNN and ModelNet40, part segmentation on ShapeNetPart, and few-shot classification on ModelNet40. Under the same Gaussian-representation setting, GaussFusion outperforms the prior Gaussian-MAE baseline by 3.85 percent on the hardest ScanObjectNN split and by 0.61 percent on ModelNet40, with consistent gains across segmentation and few-shot tasks.","feed_headline":"Image and text supervision boost 3D Gaussian pretraining","feed_subtitle":"Learnable alignment tokens let a 3D Gaussian encoder absorb vision-language semantics during pre-training, with no extra cost at inference.","key_machinery":"Two learnable alignment tokens (image and text) prepended to a Gaussian Transformer encoder sequence, aligned via Smooth L1 loss to frozen ViT and CLIP outputs; GSHM masking using opacity-scale salience scores, salience-weighted farthest-point sampling for hole centers, and per-hole scale perturbation for multi-scale spatially continuous masks; joint loss combining Gaussian attribute reconstruction (Chamfer Distance for xyz, L1 for other attributes) with image and text alignment losses.","core_discovery":"The central claim is that a 3D Gaussian encoder can absorb category-level and appearance-level semantics from frozen 2D vision-language models during pre-training through two learnable alignment tokens, and that this cross-modal supervision is more effective when the masking strategy produces spatially coherent, salience-weighted holes rather than random scattered masks. The mechanism carrying the argument is the combination of (1) prepended image/text query tokens that collect 3D context via self-attention and are aligned to external frozen features, and (2) GSHM, which uses opacity-and-scale salience scores with farthest-point sampling to build multi-scale contiguous masked regions adapted","pith_inferences":[],"forward_implications":["If cross-modal alignment tokens can inject semantic priors into a Gaussian encoder without any inference-time cost, the same prepended-token alignment design could extend to other 3D representations such as point clouds, voxels, or neural radiance fields, turning any frozen vision-language model into a zero-cost teacher for 3D pre-training.","GSHM's salience-guided masking principle could generalize beyond Gaussians to any representation with non-uniform information density, such as adaptive meshes or sparse voxel octrees, where random masking similarly wastes capacity on low-information regions.","The finding that ViT image features and CLIP text features are more complementary than using CLIP for both suggests that diversity of teacher feature spaces matters more than alignment of teacher feature spaces for cross-modal 3D supervision.","The gains being largest on the hardest ScanObjectNN split and in few-shot settings imply that multimodal pre-training primarily helps robustness and data efficiency rather than ceiling performance on clean data, which would narrow the practical scope of the method to resource-constrained or noisy deployment scenarios."],"fun_headline_variants":["Frozen vision-language models boost 3D Gaussian pretraining","Cross-modal alignment tokens improve 3D Gaussian pretraining","Salience-guided hole masking and multimodal alignment enhance 3D Gaussians","Multimodal supervision improves 3D Gaussian feature transferability","Aligning 3D Gaussians with 2D image and text features improves transfer"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that compressing all visible Gaussian tokens into two single alignment tokens (one for image, one for text) preserves enough 3D geometric information for the alignment loss to teach meaningful structural semantics rather than just category-level labels. If the self-attention aggregation into two tokens is too lossy, the cross-modal supervision may only improve coarse classification without genuinely enriching the local 3D representations that downstream part","fun_headline_variants_meta":{"raw":{"variants":["Frozen vision-language models boost 3D Gaussian pretraining","Cross-modal alignment tokens improve 3D Gaussian pretraining","Salience-guided hole masking and multimodal alignment enhance 3D Gaussians","Multimodal supervision improves 3D Gaussian feature transferability","Aligning 3D Gaussians with 2D image and text features improves transfer"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1055,"prompt_tokens":541,"completion_tokens":514,"prompt_tokens_details":null},"tokens_in":541,"tokens_out":514,"duration_ms":31134,"temperature":1.0,"reasoning_tokens":442,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T20:44:52.919879+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the alignment tokens learn to predict frozen image/text features through a shortcut that depends only on global object shape (e.g., overall bounding-box proportions visible from a single viewpoint), then removing the alignment tokens at fine-tuning time would yield no improvement over Gaussian-MAE on tasks requiring local geometric discrimination, such as part segmentation on categories with thin structures. A direct test: pre-train with alignment tokens but evaluate part-segmentation IoU per category; if gains concentrate on categories whose parts are distinguishable by global shape alone,","supporting_citations":[],"review_version":1}