{"id":"7dd77f31-8d0b-40d7-b07d-a5c198caaaad","arxiv_id":"2608.07144","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"InstanceSplat predicts 3D Gaussians with per-object identity and language-aligned semantics from unposed multi-view images in a single forward pass, enabling rendering, instance segmentation, and open-vocabulary queries.","lead":"InstanceSplat turns a few unposed photos of a scene into a 3D model whose Gaussian blobs carry object identity and language-aligned meaning in one forward pass. A generalist should read it because it points toward 3D scene understanding in seconds instead of per-scene optimization, with downstream uses in robotics and augmented reality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The generalization claim is bounded by uncharacterized VGGT pseudo-geometry supervision, since poses and depths are trained only via L_distill and all downstream instance/semantic features inherit these errors.","rationale":"The reader identified VGGT pseudo-geometry as the weakest assumption; I agree. This is the single most load-bearing condition because geometry is the substrate for all three tasks: appearance via Gaussian centers, instance grounding via shared 3D primitives, and semantic aggregation via rendered feature maps. The architecture is otherwise internally consistent: the losses are well-specified, the ablations support the proposed coupling modules, and the frozen-3DGS comparison in Table 3 gives a controlled check of the Gaussian representation. The table formatting and missing code/error bars are real but secondary; they do not identify a specific mechanism by which the central claim could fail. The VGGT concern does identify such a mechanism: a systematic pose/depth bias would be inherited by every reported metric, and the observed 16-view LERF degradation is consistent with this mechanism. I therefore keep the reader's conditional verdict: the paper should not be fully accepted until pseudo-label sensitivity is characterized, e.g., by the GT-supervision checkpoint above. In the absence of evidence that VGGT errors are negligible, the SOTA generalization claim is not yet fully established.","tokens_in":12989,"tokens_out":8728,"duration_ms":81742,"concrete_test":"Train an identical InstanceSplat variant on a subset of ScanNet++/ScanNet with L_distill retargeted to ground-truth camera poses and depth (or high-quality multi-view stereo fused depth where GT depth is unavailable), keeping all other losses, hyperparameters, and the evaluation protocol fixed. Evaluate this variant and the published VGGT-supervised variant on the same 50 held-out ScanNet scenes and on the LERF unseen scenes. If the GT-supervised variant yields no meaningful improvement in PSNR/depth and downstream instance/semantic metrics, the pseudo-label concern is not load-bearing; if it does improve, the paper should report a sensitivity analysis and qualify its cross-domain generalization claims. A compact complementary check is to inject increasing rotation/translation noise into the VGGT pseudo-poses during training and measure the slope of downstream mIoU/PSNR.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central generalization claim rests on the assumption that VGGT pseudo-ground-truth geometry is reliable across domains. In Eq. (17), L_distill is the only geometric term supervising camera poses and depths; the photometric term L_rgb cannot independently resolve pose/depth ambiguity. Since Gaussian centers are back-projected from predicted depths and poses (Sec. 3.1), any systematic VGGT error propagates into instance grounding (Sec. 3.2), boundary-aware reconstruction (Eq. 11), and semantic aggregation (Eq. 16). The paper's own LERF result, where performance drops from 8 to 16 views because 'domain shift reduces pose accuracy and causes cross-view misalignment to accumulate,' concedes that pseudo-label reliability is a live bottleneck. No experiment characterizes the model's sensitivity to VGGT noise or compares against supervision from ground-truth poses/depth available in ScanNet/ScanNet++. Thus the SOTA generalization claim is supported only up to an unquantified upper bound set by VGGT.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"InstanceSplat proposes a unified feed-forward 3D Gaussian Splatting framework that, from pose-free multi-view images, constructs Gaussians jointly encoding appearance, geometry, instance identity, and language-aligned semantics in a single forward pass. The method introduces a 3D-Consistent Instance Grounding module with rendered-space contrastive losses and an Instance-Centric Coupling module comprising boundary-aware RGB supervision, semantic-guided hard-negative reweighting, and instance-grounded semantic aggregation. Experiments cover novel-view synthesis, instance segmentation, and open-vocabulary understanding on ScanNet, LERF, and the IGGT benchmark, claiming state-of-the-art performance, roughly 3-second inference, and cross-dataset generalization.","tokens_in":13235,"tokens_out":4209,"duration_ms":37769,"significance":"If fully substantiated, InstanceSplat would provide a practical unified representation for reconstruction and instance-level scene understanding, and the shared-Gaussian design is a sensible way to obtain cross-view-consistent instance features. Strengths of the manuscript include a coherent architecture, four ablations that individually test each coupling mechanism, a concrete training budget, and a zero-shot evaluation on LERF. However, the central state-of-the-art claim is undercut by incomplete reporting in Table 1, an ablation result that contradicts the claimed benefit of hard-negative reweighting, and the absence of any characterization of the model's sensitivity to the VGGT pseudo-geometry supervision on which all downstream geometry and features are built.","major_comments":[{"comment":"The Ours row reports only mIoU and mAcc for the 2-view and 4-view settings, omitting PSNR, SSIM, and LPIPS, yet the text states that InstanceSplat achieves the best PSNR and LPIPS among methods reporting reconstruction metrics. Without these values, the NVS SOTA claim is not verifiable in the two primary settings; please report the full metrics and, ideally, error bars or a significance statement for all tables.","section":"Section 4.2, Table 1"},{"comment":"In the 4-view setting, the full model achieves mIoU 43.51 and mAcc 73.68, which are lower than the Ours w/o Hard Neg. row (mIoU 45.96, mAcc 73.77). This directly contradicts the Section 4.4 claim that semantic-guided hard-negative reweighting improves instance discrimination and weakens one of the central reciprocal-interaction claims; please explain this result or correct the ablation.","section":"Section 4.2, Table 1 (4-view columns)"},{"comment":"L_distill is the only term supervising camera poses and depths, and every downstream component (Gaussian centers, instance grounding, boundary-aware RGB loss, semantic aggregation) inherits errors from the VGGT pseudo-ground-truth geometry. The paper does not quantify this dependence, and the LERF experiment itself shows a drop from 8 to 16 views attributed to pose-accuracy degradation under domain shift. Please add an analysis of sensitivity to pseudo-label noise and, if feasible, compare against training or evaluation with ground-truth poses/depth on ScanNet/ScanNet++ to bound the generalization claim.","section":"Section 3.1, Eq. (17) and Section 4.2"}],"minor_comments":[{"comment":"The row for Ours appears merged with the preceding ablation row in the printed text; please format each row with clear separators so readers can distinguish the full model from the ablations.","section":"Table 1"},{"comment":"The boundary-aware RGB loss divides by the sum of boundary weights, which is undefined if all weights in a view are zero; please specify a numerical safety term or define the loss to be zero in that case.","section":"Section 3.3, Eq. (11)"},{"comment":"The bracket notation [x]_+ is used in Eq. (5) without definition, while Eq. (15) defines [x]^1_0; please define both notations explicitly at first use.","section":"Section 3.2, Eqs. (5) and (15)"},{"comment":"The text refers to T-SR while the table caption specifies T-SR@0.5; please make the threshold explicit in the text and table body for reproducibility.","section":"Section 4.3, Table 3"},{"comment":"Several references are dated 2026 with arXiv identifiers (e.g., Ren et al., Yoo et al., Zheng et al.) and should be consistently marked as preprints or, if already published, updated with venue information.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real integration, not just a wrapper. A single feed-forward network turns pose-free multi-view images into 3D Gaussians that carry instance identity and language-aligned semantics, and the training objective couples reconstruction, instance learning, and semantic learning. The ScanNet mIoU numbers (65.23 at 2 views, 58.76 at 8) are well above the listed baselines, and the LERF table shows a genuine practical win: roughly 3 seconds per scene against 55–280 minutes for per-scene optimization, with higher mask IoU. That is the headline, and it earns a serious look.\n\nThe individual parts are known: AnySplat/VGGT for the backbone, IGGT-style rendered-space contrastive grounding, LSeg-style semantics. The contribution is the instance-centric coupling: boundary-aware RGB loss derived from instance features, semantic-aware hard-negative reweighting, and instance-grounded semantic aggregation. The ablations support the claimed reciprocal interactions; removing each module hurts in the expected direction, and Table 3 shows the Gaussian carrier and the cross-view term both matter for track consistency.\n\nThe soft spots are real but not disqualifying. Table 1 is malformed: the Ours row omits PSNR/SSIM/LPIPS in the 2-view and 4-view columns, so the reconstruction SOTA claim for those settings is unverifiable as printed. There are no error bars or significance tests, and the 4-view mAcc explanation (class support) is plausible but not quantified. The LERF comparison also needs protocol detail: did the per-scene baselines use the exact same selected frames and evaluation annotations? They estimate their own SfM poses, which is fair, but it should be stated explicitly.\n\nThe load-bearing concern is the VGGT pseudo-geometry. L_distill is the only supervision for poses and depth, and Gaussian centers are back-projected from those predictions, so any systematic VGGT error propagates into instance grounding, boundary weighting, and semantic aggregation. The paper itself concedes the 16-view LERF drop is due to domain shift hurting pose accuracy. There is no experiment measuring sensitivity to pseudo-label noise, nor a comparison against training with ground-truth ScanNet poses and depth. That is the biggest open question for generalization. It is not fatal—the LERF results show VGGT transfers reasonably—but it needs a targeted experiment or at least an explicit discussion.\n\nRecommendation: send it to peer review. It is important enough and the core construction is sound, but the revision should fix Table 1, add error bars, spell out the LERF protocol, and address the pseudo-label sensitivity question. I would cite it and bring it to reading group.","headline":"A credible feed-forward 3DGS system that couples reconstruction with instance and semantic learning, with strong results but a few verification gaps—most notably the unquantified reliance on VGGT pseudo-geometry.","tokens_in":13779,"tokens_out":3347,"would_cite":true,"duration_ms":27817,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InstanceSplat constructs, in a single forward pass, a 3D Gaussian representation that jointly encodes appearance, geometry, instance identity, and language-aligned semantics from pose-free multi-view images.","keywords":["feed-forward 3D Gaussian splatting","instance-aware scene understanding","pose-free multi-view reconstruction","cross-view instance consistency","open-vocabulary semantic segmentation","novel-view synthesis","instance grounding","joint learning"],"falsifier":"Swapping the pseudo-geometry distillation loss for ground-truth poses and depths on a benchmark that provides them, and observing whether instance segmentation mIoU rises substantially, would settle whether the method's ceiling is set by the pretrained geometry model's error.","tokens_in":12770,"feed_emoji":"🧩","tokens_out":9968,"duration_ms":76872,"temperature":0.7,"pith_summary":"The paper attempts to establish that instance identity can be built directly into a feed-forward 3D Gaussian representation rather than appended after reconstruction. From pose-free multi-view images, a single network outputs Gaussians that carry not only color and shape but also a compact instance embedding and a language-aligned semantic feature, so novel views, consistent object masks, and open-vocabulary labels all come from one representation. The authors claim that reconstruction, instance learning, and semantic learning reinforce each other when coupled through shared instance structure, yielding better rendering near object boundaries and more coherent object-level semantics. This matters because previous feed-forward methods are category-oriented, while instance-aware methods require slow per-scene optimization; if true, the approach brings instance-level 3D understanding to near-interactive speeds.","feed_headline":"One pass builds 3D scenes with instance IDs and language semantics","feed_subtitle":"Feed-forward network turns pose-free photos into renderable, instance-aware, queryable 3D scenes in about 3 seconds.","key_machinery":"The central object is an instance-aware Gaussian primitive, a 3D Gaussian that augments standard appearance and geometry attributes with an 8-dimensional instance embedding and a compact semantic feature. The mechanism that carries the argument is differentiable rendering of these feature fields from the shared Gaussians into the input views: because the same 3D primitives serve every view, image-space instance supervision is grounded in a common, renderable carrier, which yields cross-view-consistent identities. The supporting machinery is a set of three coupled loss families: a prototype-based contrastive objective (pull, push, and cross-view alignment) for instance grounding; an instance-boundary-weighted RGB loss for reconstruction; and a semantic-alignment loss whose similarity weights reweight the instance push term while instance regions pool semantic features for inference.","core_discovery":"The central discovery is that instance identity is treated as an intrinsic attribute of each 3D Gaussian, not as a feature learned on top of a finished reconstruction. By differentiably rendering the instance embeddings from the shared Gaussians into supervised views and applying prototype-based contrastive losses with explicit cross-view alignment, the model learns a renderable, cross-view-consistent instance field. That grounded instance structure then acts as an interface: boundary-aware weighting focuses the reconstruction loss on sharp depth changes between objects, semantic similarity increases the separation pressure between confusing same-category instances, and the predicted instance regions aggregate language-aligned semantic features into stable object-level predictions. The result is a single feed-forward model that, on held-out indoor scenes and an unseen dataset, outperforms both feed-forward and per-scene-optimized baselines on novel-view synthesis, instance segmentation, and open-vocabulary understanding while taking roughly three seconds per scene.","pith_inferences":["An immediate extension the authors leave implicit is interactive 3D editing: because each object's identity is a Gaussian attribute, editing or deleting an instance could propagate consistently to all views, not just the edited one.","The heavy reliance on pseudo-geometry suggests that swapping in a geometry foundation model with better pose and depth accuracy could lift all downstream results; a controlled comparison would quantify that ceiling.","The instance embeddings, being renderable and language-aligned, could serve directly as a 3D prior for robotics tasks such as grasping or navigation, where object identity must persist across viewpoints.","The paper itself notes that the clustering readout becomes more expensive as views and Gaussians grow; replacing it with a learned grouping head would likely remove that scaling bottleneck."],"forward_implications":["A single feed-forward pass on unseen scenes yields renderable novel views, cross-view-consistent instance segmentation, and open-vocabulary semantic queries without per-scene optimization.","Instance boundaries sharpen the reconstructed depth and color near object contours, so reconstruction quality and instance quality improve together.","Cross-view instance tracking becomes more consistent because instance features are rendered from the same 3D Gaussians, and removing the cross-view alignment term measurably hurts tracking.","The representation supports object-level semantic aggregation, so open-vocabulary queries return coherent labels per instance rather than noisy per-pixel predictions.","On an unseen dataset the per-scene runtime is about three seconds, orders of magnitude faster than per-scene optimized instance-aware methods."],"supporting_citations":[{"why":"Supplies the pretrained geometry foundation model that predicts camera poses, depth maps, and point maps used as pseudo-supervision and as the reconstruction backbone.","marker":"(Wang et al. 2025)"},{"why":"Contributes the confidence-aware voxelization that merges per-pixel Gaussians into a compact voxel-aligned set, and serves as a feed-forward baseline.","marker":"(Jiang et al. 2025a)"},{"why":"Provides the instance decoder design and the cross-view instance-consistency benchmark, and its contrastive learning is the direct predecessor the paper extends.","marker":"(Li et al. 2025b)"},{"why":"Supplies the language-aligned semantic encoder whose feature space is the teacher for the semantic alignment loss.","marker":"(Li et al. 2022)"},{"why":"Defines the 3D Gaussian splatting representation and differentiable alpha-compositing used for RGB, instance, and semantic feature rendering.","marker":"(Kerbl et al. 2023)"},{"why":"Provides the pretrained visual backbone that initializes the geometry transformer encoder.","marker":"(Oquab et al. 2023)"},{"why":"Supplies the density-based clustering algorithm used to turn rendered instance embeddings into object regions at inference.","marker":"(McInnes et al. 2017)"},{"why":"Provides the ScanNet++ indoor scenes with refined instance masks used for training and benchmarking.","marker":"(Yeshwanth et al. 2023)"},{"why":"Provides the ScanNet indoor scenes and instance mask supervision used in the mixed training set.","marker":"(Dai et al. 2017)"}],"fun_headline_variants":["InstanceSplat: one forward pass to 3D scene understanding","Instance-aware 3D splatting with joint geometry and semantics","Feed-forward 3DGS that grounds instance IDs in every Gaussian","See 3D scenes with instance and language in 3 seconds","Instance-centric 3DGS: reconstruction and understanding unified"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All geometry — camera poses, depths, and point maps — is learned from pseudo-labels produced by a pretrained geometry foundation model, so any systematic error in those pseudo-labels is inherited by the reconstructed Gaussians and by every instance or semantic feature built on them.","fun_headline_variants_meta":{"raw":{"variants":["InstanceSplat: one forward pass to 3D scene understanding","Instance-aware 3D splatting with joint geometry and semantics","Feed-forward 3DGS that grounds instance IDs in every Gaussian","See 3D scenes with instance and language in 3 seconds","Instance-centric 3DGS: reconstruction and understanding unified"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1462,"prompt_tokens":979,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":595,"tokens_out":483,"duration_ms":4636,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:01:03.445407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Swapping the pseudo-geometry distillation loss for ground-truth poses and depths on a benchmark that provides them, and observing whether instance segmentation mIoU rises substantially, would settle whether the method's ceiling is set by the pretrained geometry model's error.","supporting_citations":[],"review_version":1}