Pith. sign in

REVIEW 4 major objections 5 minor 63 references

The paper claims that a view-synthesis model trained only for RGB reconstruction can propagate panoptic labels to novel views because its learned cross-view correspondence is geometric and input-agnostic, eliminating the need for 3D reconst

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:46 UTC pith:MZWNJYWK

load-bearing objection A genuinely interesting observation about frozen NVS models propagating non-RGB labels, but the 'novel view' framing leaks because the target image is used at inference and the obvious 2D baseline is missing. the 4 major comments →

arxiv 2607.19765 v1 pith:MZWNJYWK submitted 2026-07-22 cs.CV

Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation

Kwonyoung Ryu , In-Jae Lee , Jonghyun Jin , Hyunjee Lee , Jongmin Lee , Jaesik Park This is my paper
classification cs.CV
keywords novel view synthesispanoptic segmentationlabel propagationimplicit correspondencecross-view attentionbinary encodingfrozen modelsparse views
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that the cross-view correspondence learned by a large view-synthesis model—trained solely to reconstruct RGB images from sparse, unposed input views—is not an appearance-specific artifact. It encodes panoptic labels as three-bit binary maps and passes them through the same frozen model, yielding target-view segmentation with consistent spatial structure and no explicit 3D reconstruction or segmentation-specific training of the renderer. The authors support this claim with gradient saliency: the model attends to geometrically corresponding source regions whether the input is RGB or a binary instance encoding. If correct, this separates scene understanding from geometry reconstruction, turning a renderer into a general label-propagation backbone. The reported result is segmentation quality on par with reconstruction-based methods while keeping rendering quality more than 7 dB higher.

Core claim

The central claim is that the implicit attention correspondence inside a transformer-based view-synthesis model carries geometric, not photometric, information. Passing binary-encoded panoptic labels through the same encoder and decoder renders target-view segmentation; the paper shows that the propagated labels align with the rendered image and remain instance-consistent across views. The only segmentation-specific component is a shared-query decoder that produces cross-view-consistent labels on the input views. On an indoor RGB-D benchmark, the method reaches 33.56 dB PSNR and 0.5949 novel-view mIoU, exceeding a reconstruction-based baseline by more than 7 dB in rendering while matching it

What carries the argument

The load-bearing mechanism is the view-synthesis transformer's cross-view attention operating over source and target tokens, together with a latent ray embedding that specifies the target viewpoint. The paper probes this correspondence with gradient saliency—the average absolute gradient of a target patch's output with respect to source tokens—and finds the same geometric source regions are causally important for RGB and for binary instance encodings. Two simple components carry the pipeline: a shared-query decoder that assigns one instance identity to each object across all source views, and a 3-bit binary encoding whose maximally separated codewords make boundary blending converge to 0.5,

Load-bearing premise

The load-bearing premise is that the frozen model's cross-view correspondence, learned from natural images, survives the distribution shift to binary label encodings; the paper itself flags, in its final limitation paragraph, that the pipeline assumes the transferred signal is viewpoint-invariant and requires a target RGB image to specify the viewpoint, while the deeper premise is supported only by qualitative saliency and end-to-end metrics rather than a formal failure analy

What would settle it

Perturb the input by assigning inconsistent instance IDs to the same object across source views, or replace one source view with a binary encoding of random labels while keeping the other unchanged, and measure whether the target-view propagated labels still track geometric correspondence; if the model's correspondence depends on appearance statistics rather than geometry, the saliency or the propagated labels would degrade sharply. A second check: increase the instance count beyond the eight codewords and measure phantom-instance rates near boundaries, testing whether the binary-margin robust

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Any viewpoint-invariant per-pixel label—semantic class, instance ID, object category—can in principle be propagated through the same frozen model, because the correspondence itself is input-agnostic.
  • Rendering quality is preserved exactly because the segmentation path shares weights with the frozen RGB path; the reported more-than-7 dB advantage over coupled reconstruction baselines follows directly from this decoupling.
  • The pipeline is modular: swapping the source-view segmenter for an off-the-shelf predictor requires no retraining and transfers across datasets, outperforming reconstruction-based transfer on a new indoor dataset.
  • Under sparse overlap between source and target views, attention-based propagation degrades more gracefully than geometry-dependent rasterization; the gap to a reconstruction-based baseline narrows from 0.047 to 0.001 in panoptic quality.
  • Task-specific supervision is confined to source-view labeling, so improvements in 2D segmentation can be plugged into the system without touching the propagation stage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same binary-encoding recipe could be tested on other viewpoint-invariant dense signals—surface material classes, object affordances, static object parts—provided the signal is defined per surface point rather than per viewpoint; the paper's own limitation notes that viewpoint-dependent quantities such as depth do not satisfy this condition.
  • Because the method requires an actual target RGB image to estimate the latent viewpoint, a natural next step is to condition propagation on an explicit pose or interpolated latent path so that an agent can predict labels before reaching a viewpoint; the supplementary video's pose-mapping experiment points in that direction.
  • The per-target forward-pass bottleneck suggests an amortization: reuse the scene latent across many target views to turn label propagation into a real-time operation for embodied agents.
  • If the input-agnostic correspondence claim is robust, it offers a cheap upgrade path for systems that already carry a view-synthesis model, giving them panoptic foresight without adding a reconstruction module.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a pipeline for multi-view panoptic segmentation that reuses a frozen large view synthesis model (Less3Depend) trained only for RGB novel-view synthesis. Input views are segmented with a shared-query Mask2Former-style decoder; panoptic labels are encoded as 3-bit binary maps and passed through the same NVS encoder/decoder to render target-view panoptic labels. The target viewpoint is represented by a latent Plücker embedding estimated from the target image. Experiments on ScanNet report PSNR 33.56 and mIoU 0.5949/PQ 0.6092 on novel views, outperforming Gaussian-based SIU3R in rendering and mIoU, with cross-dataset transfer to Replica when using PanSt3R as source segmenter. The paper also analyzes gradient saliency to argue the NVS model's cross-view correspondence is input-agnostic.

Significance. If the result holds, it identifies a useful new capability of large view synthesis models: their implicit correspondence can propagate view-invariant dense labels, decoupling segmentation from explicit 3D reconstruction. The paper's strengths include a clean ablation (Table 3) that isolates propagation from alternative integration strategies, preservation of rendering quality through frozen weights, and modular composition with different segmenters. The cross-dataset transfer result with PanSt3R is also valuable. However, the current evaluation omits a critical baseline and conflates 'novel view' with an observed target image, so the central claim is not yet established at the reported strength.

major comments (4)
  1. [§4.1, Table 1, Table 3] The evaluation protocol does not include the most natural baseline: since the pose estimator in Eq. (1) consumes the target RGB image I_t, every 'novel view' is observed at inference. The direct baseline D(I_t) (the same Mask2Former segmenter) should be reported. From Table 1, D on input views obtains 0.6186 mIoU; target frames are from the same distribution, so D(I_t) is expected to be near this value, exceeding the proposed propagation's 0.5949 mIoU. Table 3 already shows that Mask2Former on the rendered image ('Segment rendered image') reaches 0.6239 mIoU, higher than propagation. The only clear advantage of the proposed method over 2D segmentation is PQ (0.6092 vs 0.5925 in Table 1). Without this baseline, the mIoU claim is not established.
  2. [§3.3, Eq. (1), Table 4] The central claim concerns propagation to unobserved viewpoints, but the only pose signal used is a latent Plücker embedding estimated from the target image itself. Table 4 shows that replacing the latent pose with GT pose for PanSt3R drops ScanNet mIoU from 0.593 to 0.366, indicating that the target image contributes substantially beyond camera geometry. To support the 'novel view' claim, the paper should evaluate with target pose supplied externally (e.g., GT pose or pose from other views) and/or explicitly report the setting where I_t is used only for pose estimation. As written, the experiments demonstrate label transfer to an observed view, not to a truly unobserved viewpoint.
  3. [§4.1, Table 1, §4.2] All headline comparisons lack error bars or significance tests. The mIoU difference vs SIU3R is 0.0055 (0.5949 vs 0.5894) and the PQ difference is 0.0473; a few view pairs can change these rankings. Please report standard deviations over evaluation pairs or over repeated training runs, and use paired tests where appropriate. In addition, τ=0.2 and the G=1 binary encoding are selected on the ScanNet validation set (Sec. 3.4, Supp. C) then evaluated on the same set; a separate validation split or sensitivity analysis with fixed hyperparameters is needed to quantify selection bias.
  4. [§3.4, Supp. C.1] The main paper states that pixels with |b-0.5|≤τ are 'left unlabeled' but does not specify how unlabeled pixels enter mIoU/PQ in Tables 1-4. Supplementary Table 3 reports both include/exclude policies and shows differences (e.g., at τ=0.20, mIoU_i=0.593 vs mIoU_e=0.612). Since baselines do not have an unlabeled class, the comparison is ambiguous. State the protocol used in the main tables; this is necessary for the numbers to be interpretable.
minor comments (5)
  1. [Fig. 2, Supp. F] The saliency analysis is only qualitative. A quantitative agreement measure between RGB-input and binary-input saliency maps (e.g., rank correlation or overlap) would strengthen the input-agnostic claim.
  2. [§3.4] Notation is inconsistent: Eq. (3) uses b^s_i while the text introduces b_i; also z_seg is defined but not used later. Please unify notation.
  3. [Table 2] The 'PQ gap' rows would be clearer as a separate column. Also specify whether the low-overlap evaluation uses the same 1,860 pairs as the standard-overlap evaluation.
  4. [Supp. Table 5] The Vi=3 row is honestly labeled as compatibility evidence rather than a controlled scaling study; consider moving this caveat to the main text if the Vi=3 result is summarized there.
  5. [Abstract and §1] The phrase 'first work to extend large view synthesis models beyond appearance rendering to 3D scene understanding' is strong given related feed-forward semantic 3D methods such as LSM [14] and the fact that the proposed method propagates 2D labels rather than building a 3D scene model. Consider softening the novelty claim.

Circularity Check

1 steps flagged

No structural circularity in the label-propagation derivation; minor selection of decoding hyperparameters on the ScanNet evaluation set modestly inflates the headline numbers.

specific steps
  1. fitted input called prediction [Sec. 3.4 and Supplementary C/C.1 (Tables 2-3), applied in Table 1]
    "We set τ=0.2 for the best performance. ... We sweep τ using Ours (PanSt3R, latent) on ScanNet and report metrics ... At our default τ=0.20, only 3.4% of pixels are marked uncertain."

    The binary encoding scheme (G=1) and the uncertainty threshold τ are not derived from the frozen NVS model or from an independent validation principle; they are selected by comparing encoding strategies and threshold values on the same ScanNet evaluation set (Supplementary Tables 2 and 3). The reported ScanNet mIoU/PQ numbers in Table 1 are then produced using exactly those selected values. This is selection-on-test: the headline ScanNet segmentation metrics are partly optimized on the evaluation data rather than being parameter-free predictions. It is a mild form of fitted-input-called-prediction, but it does not force the central propagation result, which is independently supported by rendering-quality preservation, design ablations, RayZer backbone replacement, and zero-shot Replica tra

full rationale

The paper's core claim—that a frozen RGB-trained NVS transformer propagates binary-encoded panoptic labels to a target view—is an empirical finding, not a derivation that reduces to its inputs. In Eqs. 1-3 the target image is used only to estimate a latent Plücker pose embedding; the panoptic output is produced by a separate forward pass over binary source-view labels and is decoded from that output. The paper even reports the 'segment rendered image' baseline as achieving higher mIoU (0.6239 vs 0.5949), so the propagation result is not secretly copying target-image labels. No load-bearing self-citation or imported uniqueness theorem appears: Less3Depend and RayZer are cited as prior models, not as proof of the present claim. The only circularity-adjacent practice is selecting τ and the binary encoding on the ScanNet evaluation set before reporting ScanNet numbers; this is a modest benchmarking caveat, not a structural circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical entities are introduced. The binary channel representation is a design choice, not a new postulated entity. The main load-bearing premises are empirical assumptions about the generality of NVS correspondence and about the method's pose-estimation setting.

free parameters (3)
  • Uncertainty threshold tau = 0.2
    Set 'for best performance' in Sec. 3.4 and swept on ScanNet in Appendix C.1; it directly changes how many boundary pixels are left unlabeled and moves mIoU/PQ.
  • Binary codeword levels = unspecified (0.1/0.3 vs 0/1 inconsistent)
    Appendix C says the two levels are 0.1 and 0.3 with tau=0.2 rejecting [0.3,0.7], while Sec. 3.4 assumes the ambiguous midpoint is near 0.5. The exact values matter for boundary robustness and were chosen via ablation (G=1).
  • Perceptual loss weight for NVS fine-tuning = 0.5
    Taken from Less3Depend and applied during ScanNet RGB fine-tuning; affects the quality of the frozen correspondence, though not fit to panoptic metrics.
axioms (5)
  • domain assumption Cross-view correspondence learned from RGB photometric loss is input-agnostic and persists when source tokens are replaced by binary label encodings.
    Core premise of Sec. 3.2; supported only by qualitative gradient saliency and end-to-end metrics, not proven.
  • domain assumption The target viewpoint is specified by estimating a latent Plucker embedding from an actual target RGB image.
    Eq. (1) uses P(I_t, z); this limits the method to views for which an RGB image exists and is available at inference.
  • domain assumption Panoptic instance IDs are consistent across source views, as produced by the shared query decoder.
    Sec. 3.3 multi-view segmentation; conflicting source labels would make propagation ambiguous.
  • standard math DINOv2 features and transformer cross-view attention encode enough geometric structure to transfer non-RGB signals.
    The method relies on pretrained DINOv2 and the transformer architecture; no formal guarantee of geometric fidelity is given.
  • domain assumption Rendered binary outputs are calibrated enough that rounding and thresholding recover valid instance IDs.
    Empirical assumption behind Eq. (5); boundary blending is assumed to fall near 0.5, but the supplementary values of 0.1/0.3 are inconsistent with that assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 18363 in / 18059 out tokens · 176957 ms · 2026-08-01T11:46:46.555452+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation." pith.science (2026). https://pith.science/paper/MZWNJYWK

@misc{pith2026260719765,
  author       = {Pith},
  title        = {Pith review of: Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MZWNJYWK}},
  note         = {Machine review of arXiv:2607.19765}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large view synthesis models synthesize novel views through cross-view attention without explicit 3D representations, and recent studies have shown that they learn accurate spatial correspondence from RGB supervision alone. We observe that this correspondence generalizes beyond appearance. When non-photorealistic signals such as binary encoded panoptic labels are passed through the model, they are propagated to novel views with consistent spatial structure. These results indicate that the correspondence learned for RGB view synthesis can also propagate view-independent per-pixel labels. From this observation, we present the first work to extend large view synthesis models beyond appearance rendering to 3D scene understanding. We propose a panoptic segmentation pipeline that reuses a frozen view synthesis model to propagate panoptic labels from input views to novel views, without 3D reconstruction or any segmentation-specific training of the view synthesis model. Given panoptic labels on the input views, we encode them into binary channel representations and pass them through the same model to render target-view segmentation. On ScanNet, our method achieves segmentation quality on par with Gaussian based approaches requiring explicit 3D reconstruction, while outperforming them in novel view synthesis by more than 7 dB. The label propagation also transfers across datasets, surpassing these approaches on Replica without any fine-tuning.

Figures

Figures reproduced from arXiv: 2607.19765 by Hyunjee Lee, In-Jae Lee, Jaesik Park, Jonghyun Jin, Jongmin Lee, Kwonyoung Ryu.

Figure 1
Figure 1. Figure 1: Illustration of a large-view synthesis model and its extension to multi [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of gradient saliency from target to source views. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overall pipeline. (a) RGB NVS training and source-view segmentation: the view synthesis model (blue) is trained to encode input-view images into a scene la￾tent and render the target-view RGB image from the target-view latent Plücker em￾bedding. A source-view segmenter (red), implemented either as our trainable shared query decoder or an off-the-shelf model, predicts panoptic labels for the input views. (b… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results of novel-view panoptic segmentation on Scan [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Design alternatives. (a) Joint training attaches a segmentation head to in￾termediate features of the NVS rendering decoder. (b) NVS→Segmentation applies a per-view segmenter, Mask2Former [11], to the rendered novel-view image. (c) Our method first segments the input views, encodes the panoptic maps into binary chan￾nel representations, and propagates them through the frozen NVS model to produce target-vie… view at source ↗
Figure 6
Figure 6. Figure 6: Replacing the input view segmentation model. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 1
Figure 1. Figure 1: Additional visualization of gradient saliency from target to input [PITH_FULL_IMAGE:figures/full_fig_p023_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative results on outdoor scenes from RealEstate10K [60]. [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Additional qualitative for SAM2 propagation. [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Additional qualitative comparison with SIU3R [52] in low-overlapped [PITH_FULL_IMAGE:figures/full_fig_p026_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 1 linked inside Pith

  1. [1]

    In: Proceed- ings of the 58th annual meeting of the association for computational linguistics

    Abnar, S., Zuidema, W.: Quantifying attention flow in transformers. In: Proceed- ings of the 58th annual meeting of the association for computational linguistics. pp. 4190–4197 (2020)

    2020

  2. [2]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srini- vasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5855–5864 (2021)

    2021

  3. [3]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Zip-nerf: Anti-aliased grid-based neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19697–19705 (2023)

    2023

  4. [4]

    In: Advances in Neural Information Processing Systems (2023)

    Bhalgat, Y., Laina, I., Henriques, J.F., Zisserman, A., Vedaldi, A.: Contrastive lift: 3d object instance segmentation by slow-fast contrastive fusion. In: Advances in Neural Information Processing Systems (2023)

    2023

  5. [5]

    In: European Conference on Computer Vision

    Bhalgat, Y., Laina, I., Henriques, J.F., Zisserman, A., Vedaldi, A.: N2f2: Hierarchi- cal scene understanding with nested neural feature fields. In: European Conference on Computer Vision. pp. 197–214. Springer (2024)

    2024

  6. [6]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9650–9660 (2021)

    2021

  7. [7]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Charatan, D., Li, S., Tagliasacchi, A., Sitzmann, V.: pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    2024

  8. [8]

    In: International Conference on Learning Representations (2015)

    Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected crfs. In: International Conference on Learning Representations (2015)

    2015

  9. [9]

    In: European Conference on Computer Vision (2024)

    Chen, Y., Xu, H., Zheng, C., Zhuang, B., Pollefeys, M., Geiger, A., Cham, T.J., Cai, J.: Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In: European Conference on Computer Vision (2024)

    2024

  10. [10]

    In: International Conference on Learning Represen- tations (2023)

    Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai, J., Qiao, Y.: Vision transformer adapter for dense predictions. In: International Conference on Learning Represen- tations (2023)

    2023

  11. [11]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention Mask Transformer for Universal Image Segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)

    2022

  12. [12]

    In: Advances in Neural Information Processing Systems

    Cheng, B., Schwing, A., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. In: Advances in Neural Information Processing Systems. vol. 34, pp. 17864–17875 (2021)

    2021

  13. [13]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2017)

    Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scan- Net: Richly-Annotated 3D Reconstructions of Indoor Scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2017)

    2017

  14. [14]

    In: Advances in Neural Information Processing Systems (2024)

    Fan, Z., Zhang, J., Cong, W., Wang, P., Li, R., Wen, K., Zhou, S., Kadambi, A., Wang, Z., Xu, D., Ivanovic, B., Pavone, M., Wang, Y.: Large spatial model: End-to-end unposed images to semantic 3d. In: Advances in Neural Information Processing Systems (2024)

    2024

  15. [15]

    In: International Conference on 3D Vision

    Fu, X., Zhang, S., Chen, T., Lu, Y., Zhu, L., Zhou, X., Geiger, A., Liao, Y.: Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In: International Conference on 3D Vision. pp. 1–11. IEEE (2022) Extending LVSM for Panoptic Seg 13

    2022

  16. [16]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2961–2969 (2017)

    2017

  17. [17]

    In: International Conference on Learning Representations (2024)

    Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F., Sunkavalli, K., Bui, T., Tan, H.: Lrm: Large reconstruction model for single image to 3d. In: International Conference on Learning Representations (2024)

    2024

  18. [18]

    In: ACM SIGGRAPH 2024 conference papers

    Huang, B., Yu, Z., Chen, A., Geiger, A., Gao, S.: 2d gaussian splatting for geo- metrically accurate radiance fields. In: ACM SIGGRAPH 2024 conference papers. pp. 1–11 (2024)

    2024

  19. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Jain, J., Li, J., Chiu, M.T., Hassani, A., Orlov, N., Shi, H.: Oneformer: One trans- former to rule universal image segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2989–2998 (2023)

    2023

  20. [20]

    In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision (2025)

    Jiang, H., Tan, H., Wang, P., Jin, H., Zhao, Y., Bi, S., Zhang, K., Luan, F., Sunkavalli, K., Huang, Q., Pavlakos, G.: Rayzer: A self-supervised large view syn- thesis model. In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision (2025)

    2025

  21. [21]

    Jiao, S., Dong, H., Yin, Y., Jie, Z., Qian, Y., Zhao, Y., Shi, H., Wei, Y.: Clip-gs: Unifyingvision-languagerepresentationwith3dgaussiansplatting.In:Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4670–4680 (2025)

    2025

  22. [22]

    In: International Conference on Learning Representations (2025)

    Jin, H., Jiang, H., Tan, H., Zhang, K., Bi, S., Zhang, T., Luan, F., Snavely, N., Xu, Z.: Lvsm: A large view synthesis model with minimal 3d inductive bias. In: International Conference on Learning Representations (2025)

    2025

  23. [23]

    ACM Transactions on Graphics42(4), 1–14 (2023)

    Kerbl, B., Kopanas, G., Leimkuehler, T., Drettakis, G.: 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics42(4), 1–14 (2023)

    2023

  24. [24]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kerr, J., Kim, C.M., Goldberg, K., Kanazawa, A., Tancik, M.: Lerf: Language em- bedded radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19729–19739 (October 2023)

    2023

  25. [25]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kirillov, A., He, K., Girshick, R., Rother, C., Dollár, P.: Panoptic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9404–9413 (2019)

    2019

  26. [26]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023)

    2023

  27. [27]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kolodiazhnyi, M., Vorontsova, A., Konushin, A., Rukhovich, D.: Oneformer3d: One transformer for unified point cloud segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20943– 20953 (2024)

    2024

  28. [28]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kundu, A., Genova, K., Yin, X., Fathi, A., Pantofaru, C., Guibas, L.J., Tagliasac- chi, A., Dellaert, F., Funkhouser, T.: Panoptic neural fields: A semantic object- aware neural scene representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12871–12881 (2022)

    2022

  29. [29]

    In: European Conference on Computer Vision

    Leroy, V., Cabon, Y., Revaud, J.: Grounding image matching in 3d with mast3r. In: European Conference on Computer Vision. pp. 71–91. Springer (2024)

    2024

  30. [30]

    In: International Conference on Learning Representations (2022)

    Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: International Conference on Learning Representations (2022)

    2022

  31. [31]

    In: Eu- ropean Conference on Computer Vision (2020) 14 K

    Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: Eu- ropean Conference on Computer Vision (2020) 14 K. Ryu et al

    2020

  32. [32]

    ACM transactions on graphics (TOG)41(4), 1–15 (2022)

    Müller,T.,Evans,A.,Schied,C.,Keller,A.:Instantneuralgraphicsprimitiveswith a multiresolution hash encoding. ACM transactions on graphics (TOG)41(4), 1–15 (2022)

    2022

  33. [33]

    Transactions on Machine Learning Research3(1) (2024)

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W.,Howes,R.,Huang,P.Y.,Li,S.W.,Misra,I.,Rabbat,M.,Sharma,V.,Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning Robust Visual Features without Su...

    2024

  34. [34]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    Peng, S., Genova, K., Jiang, C.M., Tagliasacchi, A., Pollefeys, M., Funkhouser, T.: Openscene: 3d scene understanding with open vocabularies. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    2023

  35. [35]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Qin, M., Li, W., Zhou, J., Wang, H., Pfister, H.: Langsplat: 3d language gaussian splatting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    2024

  36. [36]

    In: International Conference on Learning Representations (2025)

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., Wu, C.Y., Girshick, R., Dollár, P., Feichtenhofer, C.: Sam 2: Segment anything in images and videos. In: International Conference on Learning Representations (2025)

    2025

  37. [37]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sajjadi, M.S.M., Meyer, H., Pot, E., Bergmann, U., Greff, K., Radwan, N., Vora, S., Lučić, M., Duckworth, D., Dosovitskiy, A., Uszkoreit, J., Funkhouser, T., Tagliasacchi, A.: Scene representation transformer: Geometry-free novel view syn- thesis through set-latent scene representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2022

  38. [38]

    In: Advances in Neural Information Processing Systems

    Sajjadi, M.S., Duckworth, D., Mahendran, A., Van Steenkiste, S., Pavetic, F., Lu- cic, M., Guibas, L.J., Greff, K., Kipf, T.: Object scene representation transformer. In: Advances in Neural Information Processing Systems. vol. 35, pp. 9512–9524 (2022)

    2022

  39. [39]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sajjadi, M.S., Mahendran, A., Kipf, T., Pot, E., Duckworth, D., Lučić, M., Greff, K.: Rust: Latent neural scene representations from unposed imagery. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17297–17306 (2023)

    2023

  40. [40]

    In: IEEE International Conference on Robotics and Automation (2023)

    Schult, J., Engelmann, F., Hermans, A., Litany, O., Tang, S., Leibe, B.: Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. In: IEEE International Conference on Robotics and Automation (2023)

    2023

  41. [41]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    Siddiqui, Y., Porzi, L., Bulò, S.R., Müller, N., Nießner, M., Dai, A., Kontschieder, P.: Panoptic Lifting for 3D Scene Understanding With Neural Fields. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    2023

  42. [42]

    In: International Con- ference on Learning Representations Workshop (2014)

    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. In: International Con- ference on Learning Representations Workshop (2014)

    2014

  43. [43]

    arXiv preprint arXiv:2408.13912 (2024)

    Smart, B., Zheng, C., Laina, I., Prisacariu, V.A.: Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs. arXiv preprint arXiv:2408.13912 (2024)

    Pith/arXiv arXiv 2024

  44. [44]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)

    Straub, J., Whelan, T., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J.J., Mur- Artal, R., Ren, C., Verma, S., Clarkson, A., Yan, M., Budge, B., Yan, Y., Pan, X., Yon, J., Zou, Y., Leon, K., Carter, N., Briales, J., Gillingham, T., Mueggler, E., Pesqueira, L., Savva, M., Batra, D., Strasdat, H.M., Nardi, R.D., Goesele, M., Extending LVSM for Panoptic ...

    2019

  45. [45]

    In: International Conference on Learning Representations (2026)

    Wang, H., Ye, K., Li, Y., Chen, W., Chen, B.: The less you depend, the more you learn: Synthesizing novel views from sparse, unposed images without any 3d knowledge. In: International Conference on Learning Representations (2026)

    2026

  46. [46]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition

    Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: Vggt: Visual geometry grounded transformer. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. pp. 5294–5306 (2025)

    2025

  47. [47]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, Q., Wang, Z., Genova, K., Srinivasan, P.P., Zhou, H., Barron, J.T., Martin- Brualla, R., Snavely, N., Funkhouser, T.: Ibrnet: Learning multi-view image-based rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4690–4699 (2021)

    2021

  48. [48]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., Revaud, J.: Dust3r: Geometric 3d vision made easy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20697–20709 (2024)

    2024

  49. [49]

    IEEE Transactions on Image Processing (2025)

    Wang,Y.,Wei,X.,Lu,M.,Kang,G.:Plgs:Robustpanopticliftingwith3dgaussian splatting. IEEE Transactions on Image Processing (2025)

    2025

  50. [50]

    In: Advances in Neural Information Processing Systems (2022)

    Weinzaepfel, P., Leroy, V., Lucas, T., Brégier, R., Cabon, Y., Arora, V., Antsfeld, L., Chidlovskii, B., Csurka, G., Jérôme, R.: CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion. In: Advances in Neural Information Processing Systems (2022)

    2022

  51. [51]

    Wu, Y., Meng, J., Li, H., Wu, C., Shi, Y., Cheng, X., Zhao, C., Feng, H., Ding, E., Wang, J., Zhang, J.: Opengaussian: Towards point-level 3d gaussian-based open vocabularyunderstanding.In:AdvancesinNeuralInformationProcessingSystems. pp. 19114–19138 (2024)

    2024

  52. [52]

    In: Advances in Neural Information Processing Systems (2025)

    Xu, Q., Wei, D., Zhao, L., Li, W., Huang, Z., Ji, S., Liu, P.: SIU3R: Simultane- ous Scene Understanding and 3D Reconstruction Beyond Feature Alignment. In: Advances in Neural Information Processing Systems (2025)

    2025

  53. [53]

    In: International Conference on Learning Representations (2025)

    Ye, B., Liu, S., Xu, H., Xueting, L., Pollefeys, M., Yang, M.H., Songyou, P.: No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed im- ages. In: International Conference on Learning Representations (2025)

    2025

  54. [54]

    In: European Conference on Computer Vision (2024)

    Ye, M., Danelljan, M., Yu, F., Ke, L.: Gaussian grouping: Segment and edit any- thing in 3d scenes. In: European Conference on Computer Vision (2024)

    2024

  55. [55]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields from one or few images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4578–4587 (2021)

    2021

  56. [56]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yu, Z., Chen, A., Huang, B., Sattler, T., Geiger, A.: Mip-splatting: Alias-free 3d gaussian splatting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19447–19456 (2024)

    2024

  57. [57]

    In: European Conference on Computer Vision

    Zhang, K., Bi, S., Tan, H., Xiangli, Y., Zhao, N., Sunkavalli, K., Xu, Z.: Gs-lrm: Large reconstruction model for 3d gaussian splatting. In: European Conference on Computer Vision. pp. 1–19. Springer (2024)

    2024

  58. [58]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)

    Zhi,S.,Laidlow,T.,Leutenegger,S., Davison,A.J.: In-placescenelabellingandun- derstanding with implicit scene representation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)

    2021

  59. [59]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhou,S.,Chang,H.,Jiang,S.,Fan,Z.,Zhu,Z.,Xu,D.,Chari,P.,You,S.,Wang,Z., Kadambi, A.: Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21676–21685 (2024) 16 K. Ryu et al

    2024

  60. [60]

    In: ACM SIGGRAPH (2018)

    Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: Learn- ing view synthesis using multiplane images. In: ACM SIGGRAPH (2018)

    2018

  61. [61]

    In: European Conference on Computer Vision

    Zhu, R., Qiu, S., Wu, Q., Hui, K.H., Heng, P.A., Fu, C.W.: Pcf-lift: Panoptic lifting by probabilistic contrastive fusion. In: European Conference on Computer Vision. pp. 92–108. Springer (2024)

    2024

  62. [62]

    International Journal of Computer Vision133(2), 611–627 (2025)

    Zuo, X., Samangouei, P., Zhou, Y., Di, Y., Li, M.: Fmgs: Foundation model em- bedded 3d gaussian splatting for holistic 3d scene understanding. International Journal of Computer Vision133(2), 611–627 (2025)

    2025

  63. [63]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2025)

    Zust, L., Cabon, Y., Marrie, J., Antsfeld, L., Chidlovskii, B., Revaud, J., Csurka, G.: Panst3r: Multi-view consistent panoptic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2025)

    2025