pith. sign in

arxiv: 2606.21292 · v1 · pith:NHR6G3Q3new · submitted 2026-06-19 · 💻 cs.CV

Lightweight 3D Feature Pretraining by Bayesian Inversion of 2D Foundation Models

Marwane Hariat , Gianni Franchi , David Filliat , Antoine Manzanera This is my paper

Pith reviewed 2026-06-26 14:48 UTC · model grok-4.3

classification 💻 cs.CV
keywords Casper3D3D semantic representationBayesian inversion2D foundation modelsmulti-view reasoningvariational inferenceopen-vocabulary 3D understanding
0
0 comments X

The pith

Casper3D converts noisy multi-view 2D foundation model features into a latent 3D semantic representation via Bayesian inversion.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Casper3D as a lightweight probabilistic framework that turns multi-view 2D embeddings from foundation models into a consistent 3D semantic state. It treats the 2D features as noisy observations of an underlying 3D state and recovers that state with a set-based variational model that uses relative pose during inference. Training happens by predicting semantic features from held-out novel viewpoints while keeping the output aligned with both visual and text spaces. The approach is backbone-agnostic and applies to language-aligned or self-supervised 2D embeddings, yielding more stable 3D semantics than simple pooling especially under ambiguous or noisy conditions.

Core claim

Casper3D models view-level semantic features as noisy observations of an underlying 3D semantic state and infers this state with a set-based variational model that incorporates relative pose, trained by predicting held-out semantic observations from novel viewpoints while remaining aligned with visual and text semantic spaces.

What carries the argument

set-based variational model that incorporates relative pose to infer the latent 3D semantic state from noisy 2D observations

If this is right

  • Casper3D produces more stable 3D semantics than simple multi-view pooling in ambiguous and noisy settings.
  • The method applies to both language-aligned and self-supervised 2D embeddings.
  • It supports open-vocabulary 3D understanding without 3D-specific training data.
  • The framework remains backbone-agnostic.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The inversion approach could extend to lifting other 2D probabilistic models to higher-dimensional spaces if similar noisy observations are available.
  • Stability gains in noisy views point toward use in robotics or AR pipelines where camera data is imperfect.
  • If the model generalizes, it offers a route to 3D pretraining that relies mainly on existing 2D data and models.

Load-bearing premise

View-level semantic features from 2D foundation models can be modeled as noisy observations of an underlying 3D semantic state that a set-based variational model can reliably infer using relative pose.

What would settle it

An experiment in which Casper3D 3D semantics show no stability gain over simple multi-view pooling on metrics for ambiguous or noisy view sets.

Figures

Figures reproduced from arXiv: 2606.21292 by Antoine Manzanera, David Filliat, Gianni Franchi, Marwane Hariat.

Figure 1
Figure 1. Figure 1: Overview of Casper3D supervised predictors with pretrained 2D models: ConceptFusion [16] lifts SAM-based [19] and global image features into 3D; FeatureRealisticFusion [25] learns a neural field with an iMAP-like backend [33]; OpenMask3D [34], OpenIns3D [15], and OV3D [45] lift SAM-guided masks [19] to 3D instances. Multi-view recognition methods such as PointCLIP [42] and MV-CLIP [32] aggregate CLIP featu… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our proposed architecture. More details in Appendix. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Robustness to sparse views. Compared with directly averaging 2D features across views. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison between ground-truth semantic maps and Casper3D predictions. [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Additional qualitative comparison between ground-truth semantic maps and Casper3D [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: t-SNE visualizations of point embeddings on representative ScanNet scenes. Compared [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
read the original abstract

We present Casper3D, a lightweight probabilistic framework for converting noisy multi-view 2D foundation-model embeddings into a latent 3D semantic representation. We model view-level semantic features as noisy observations of an underlying 3D semantic state and infer this state with a set-based variational model that incorporates relative pose during multi-view reasoning. Casper3D is trained by predicting held-out semantic observations from novel viewpoints, while remaining aligned with visual and text semantic spaces for open-vocabulary 3D understanding. The framework is backbone-agnostic and applies to both language-aligned and self-supervised embeddings. Experiments show that Casper3D produces more stable 3D semantics than simple multi-view pooling, especially in ambiguous and noisy settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces Casper3D, a lightweight probabilistic framework that models multi-view 2D foundation-model embeddings as noisy observations of a latent 3D semantic state. Inference uses a set-based variational model incorporating relative pose; training is performed by held-out viewpoint prediction while maintaining alignment with visual and text spaces. The method is backbone-agnostic and claims to yield more stable 3D semantics than simple multi-view pooling, particularly in ambiguous or noisy settings.

Significance. If the modeling assumption holds and the stability gains are reproducible, the approach would provide an efficient route to lift existing 2D foundation models into consistent 3D representations without large-scale 3D pretraining. The open-vocabulary alignment and backbone independence are practical strengths for downstream 3D vision tasks.

major comments (3)
  1. [Abstract] Abstract: the central claim that Casper3D produces more stable 3D semantics than multi-view pooling (especially in ambiguous/noisy settings) is asserted without any quantitative metrics, baselines, datasets, or experimental details. This absence prevents evaluation of whether the reported gains are load-bearing or dataset-specific.
  2. [Modeling / generative assumptions] Core generative model (view features as noisy observations of a single latent 3D semantic state): foundation-model embeddings routinely exhibit large view-dependent shifts due to occlusion, lighting, and part visibility. If these shifts are not well-approximated by the assumed additive noise model, the variational inversion cannot be guaranteed to outperform pooling; the manuscript must supply a direct ablation or diagnostic that isolates this mismatch.
  3. [Training procedure] Training objective (held-out viewpoint prediction): while this is a standard self-supervised pattern, the paper must demonstrate that the inferred 3D state generalizes beyond the training views and that the stability advantage persists when the 2D backbone is frozen versus fine-tuned.
minor comments (2)
  1. [Method] Notation for the set-based variational posterior and the relative-pose conditioning should be introduced with explicit variable definitions and a diagram of the inference graph.
  2. [Implementation details] The claim of being 'parameter-free' or 'lightweight' needs a precise accounting of additional parameters introduced by the variational model relative to the frozen 2D backbone.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below, clarifying the existing experimental support while committing to targeted revisions for greater transparency.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that Casper3D produces more stable 3D semantics than multi-view pooling (especially in ambiguous/noisy settings) is asserted without any quantitative metrics, baselines, datasets, or experimental details. This absence prevents evaluation of whether the reported gains are load-bearing or dataset-specific.

    Authors: We agree the abstract is too high-level. The Experiments section provides quantitative comparisons against multi-view pooling on standard 3D datasets, reporting stability metrics under controlled noise and viewpoint ambiguity. We will revise the abstract to include one sentence summarizing the key datasets, baselines, and relative gains. revision: yes

  2. Referee: [Modeling / generative assumptions] Core generative model (view features as noisy observations of a single latent 3D semantic state): foundation-model embeddings routinely exhibit large view-dependent shifts due to occlusion, lighting, and part visibility. If these shifts are not well-approximated by the assumed additive noise model, the variational inversion cannot be guaranteed to outperform pooling; the manuscript must supply a direct ablation or diagnostic that isolates this mismatch.

    Authors: The concern about view-dependent shifts is valid. The variational formulation is intended to marginalize such effects via the latent 3D state, but we will add an explicit ablation that isolates the contribution of the Bayesian inversion (versus pooling) on subsets with high occlusion/lighting variation, together with a simple diagnostic measuring residual view-dependence after inference. revision: yes

  3. Referee: [Training procedure] Training objective (held-out viewpoint prediction): while this is a standard self-supervised pattern, the paper must demonstrate that the inferred 3D state generalizes beyond the training views and that the stability advantage persists when the 2D backbone is frozen versus fine-tuned.

    Authors: Held-out viewpoint prediction is the training objective precisely to enforce generalization to novel views; the reported stability results are obtained with frozen 2D backbones. We will add a short paragraph and table entry explicitly confirming generalization metrics on held-out views and noting that all main results use frozen encoders (with an optional fine-tuning comparison if space permits). revision: partial

Circularity Check

0 steps flagged

No circularity detected in derivation or training setup

full rationale

The paper presents Casper3D as a variational model trained by predicting held-out viewpoint observations, which follows a standard self-supervised pattern without reducing any claimed prediction to a fitted parameter or self-citation by construction. No self-definitional equations, load-bearing self-citations, uniqueness theorems from the same authors, or ansatz smuggling appear in the abstract or description. The framework remains self-contained against external benchmarks via its experimental comparisons to pooling.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

Abstract introduces a latent 3D semantic state as a central modeling construct with no independent evidence supplied. No free parameters or standard axioms are explicitly listed.

invented entities (1)
  • latent 3D semantic state no independent evidence
    purpose: Underlying 3D representation of which 2D view features are noisy observations
    Core modeling assumption stated in the abstract

pith-pipeline@v0.9.1-grok · 5656 in / 1069 out tokens · 27519 ms · 2026-06-26T14:48:15.486705+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 14 canonical work pages · 7 internal anchors

  1. [1]

    S. Amir, Y . Gandelsman, S. Bagon, and T. Dekel. Deep vit features as dense visual descriptors.arXiv preprint arXiv:2112.05814, 2(3):4, 2021

    work page arXiv 2021

  2. [2]

    B. Cao, K. Chen, K.-K. Maninis, K. Chen, A. Karpur, Y . Xia, S. Dua, T. Dabral, G. Han, B. Han, et al. Tipsv2: Advancing vision-language pretraining with enhanced patch-text alignment.arXiv preprint arXiv:2604.12012, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  3. [3]

    Caron, H

    M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021

    2021

  4. [4]

    Matterport3D: Learning from RGB-D Data in Indoor Environments

    A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y . Zhang. Matterport3d: Learning from rgb-d data in indoor environments.arXiv preprint arXiv:1709.06158, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  5. [5]

    R. Chen, Y . Liu, L. Kong, X. Zhu, Y . Ma, Y . Li, Y . Hou, Y . Qiao, and W. Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020–7030, 2023

    2023

  6. [6]

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017

    2017

  7. [7]

    R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi. Pla: Language-driven open-vocabulary 3d scene understanding. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7010–7019, 2023

    2023

  8. [8]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  9. [9]

    Ghiasi, X

    G. Ghiasi, X. Gu, Y . Cui, and T.-Y . Lin. Scaling open-vocabulary image segmentation with image-level labels. InEuropean conference on computer vision, pages 540–557. Springer, 2022

    2022

  10. [10]

    Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

    X. Gu, T.-Y . Lin, W. Kuo, and Y . Cui. Open-vocabulary object detection via vision and language knowledge distillation.arXiv preprint arXiv:2104.13921, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  11. [11]

    Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models

    H. Ha and S. Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models.arXiv preprint arXiv:2207.11514, 2022

    work page Pith review arXiv 2022

  12. [12]

    Unsupervised semantic segmentation by dis- tilling feature correspondences

    M. Hamilton, Z. Zhang, B. Hariharan, N. Snavely, and W. T. Freeman. Unsupervised semantic segmentation by distilling feature correspondences.arXiv preprint arXiv:2203.08414, 2022

    work page arXiv 2022

  13. [13]

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    2022

  14. [14]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  15. [15]

    Huang, X

    Z. Huang, X. Wu, X. Chen, H. Zhao, L. Zhu, and J. Lasenby. Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation. InEuropean Conference on Computer Vision, pages 169–185. Springer, 2024

    2024

  16. [16]

    K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, A. Maalouf, S. Li, G. Iyer, S. Saryazdi, N. Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping.arXiv preprint arXiv:2302.07241, 2023

    work page arXiv 2023

  17. [17]

    Jiang, S

    L. Jiang, S. Shi, and B. Schiele. Open-vocabulary 3d semantic segmentation with foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21284– 21294, 2024

    2024

  18. [18]

    J. Kim, J. Yoo, J. Lee, and S. Hong. Setvae: Learning hierarchical composition for generative modeling of set-structured data. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15059–15068, 2021

    2021

  19. [19]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything. InProceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. 10

    2023

  20. [20]

    Knaebel, K

    K. Knaebel, K. Yilmaz, D. de Geus, A. Hermans, D. Adrian, T. Linder, and B. Leibe. Dino in the room: Leveraging 2d foundation models for 3d segmentation.arXiv preprint arXiv:2503.18944, 2025

    work page arXiv 2025

  21. [21]

    Lambert, Z

    J. Lambert, Z. Liu, O. Sener, J. Hays, and V . Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2879–2888, 2020

    2020

  22. [22]

    J. Lee, Y . Lee, J. Kim, A. Kosiorek, S. Choi, and Y . W. Teh. Set transformer: A framework for attention- based permutation-invariant neural networks. InInternational conference on machine learning, pages 3744–3753. PMLR, 2019

    2019

  23. [23]

    R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. V ondrick. Zero-1-to-3: Zero-shot one image to 3d object. InProceedings of the IEEE/CVF international conference on computer vision, pages 9298–9309, 2023

    2023

  24. [24]

    Lüddecke and A

    T. Lüddecke and A. Ecker. Image segmentation using text and image prompts. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7086–7096, 2022

    2022

  25. [25]

    Mazur, E

    K. Mazur, E. Sucar, and A. J. Davison. Feature-realistic neural fusion for real-time, open set scene understanding. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8201–8207. IEEE, 2023

    2023

  26. [26]

    McCormac, A

    J. McCormac, A. Handa, A. Davison, and S. Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In2017 IEEE International Conference on Robotics and automation (ICRA), pages 4628–4635. IEEE, 2017

    2017

  27. [27]

    DINOv2: Learning Robust Visual Features without Supervision

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  28. [28]

    S. Peng, K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 815–824, 2023

    2023

  29. [29]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  30. [30]

    D. Royo, B. Zhao, A. Muñoz, D. Gutierrez, and K. L. Bouman. Mapping dark-matter clusters via physics-guided diffusion models.arXiv preprint arXiv:2603.14503, 2026

    work page arXiv 2026

  31. [31]

    DINOv3

    O. Siméoni, H. V . V o, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V . Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    D. Song, X. Fu, N. Liu, W.-Z. Nie, W.-H. Li, L.-J. Wang, Y . Yang, and A.-A. Liu. Mv-clip: Multi-view clip for zero-shot 3d shape recognition.IEEE Transactions on Circuits and Systems for Video Technology, 35(9):8767–8779, 2025

    2025

  33. [33]

    Sucar, S

    E. Sucar, S. Liu, J. Ortiz, and A. J. Davison. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6229–6238, 2021

    2021

  34. [34]

    arXiv preprint arXiv:2306.13631 (2023)

    A. Takmaz, E. Fedele, R. W. Sumner, M. Pollefeys, F. Tombari, and F. Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation.arXiv preprint arXiv:2306.13631, 2023

    work page arXiv 2023

  35. [35]

    Sample-efficient evidence estimation of score based priors for model selection

    F. Wang and K. L. Bouman. Sample-efficient evidence estimation of score based priors for model selection. arXiv preprint arXiv:2602.20549, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  36. [36]

    Whelan, R

    T. Whelan, R. F. Salas-Moreno, B. Glocker, A. J. Davison, and S. Leutenegger. Elasticfusion: Real-time dense slam and light source estimation.The International Journal of Robotics Research, 35(14):1697–1716, 2016

    2016

  37. [37]

    Wysocza´nska, O

    M. Wysocza´nska, O. Siméoni, M. Ramamonjisoa, A. Bursuc, T. Trzci´nski, and P. Pérez. Clip-dinoiser: Teaching clip a few dino tricks for open-vocabulary semantic segmentation. InEuropean Conference on Computer Vision, pages 320–337. Springer, 2024

    2024

  38. [38]

    J. Yang, R. Ding, W. Deng, Z. Wang, and X. Qi. Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19823–19832, 2024. 11

    2024

  39. [39]

    Zaheer, S

    M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. Advances in neural information processing systems, 30, 2017

    2017

  40. [40]

    Zhang, W

    B. Zhang, W. Chu, J. Berner, C. Meng, A. Anandkumar, and Y . Song. Improving diffusion inverse problem solving with decoupled noise annealing. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 20895–20905, 2025

    2025

  41. [41]

    Zhang, R

    J. Zhang, R. Dong, and K. Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. InProceedings of the IEEE/CVF international conference on computer vision, pages 2048–2059, 2023

    2048

  42. [42]

    Zhang, Z

    R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y . Qiao, P. Gao, and H. Li. Pointclip: Point cloud understanding by clip. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8552–8562, 2022

    2022

  43. [43]

    Zhang, S

    S. Zhang, S. Zhang, J. Deng, Y . Shen, M. Ma, and Y . Zhang. Pgov3d: Open-vocabulary 3d semantic segmentation with partial-to-global curriculum. InProceedings of the 33rd ACM International Conference on Multimedia, pages 5070–5079, 2025

    2025

  44. [44]

    C. Zhou, C. C. Loy, and B. Dai. Extract free dense labels from clip. InEuropean conference on computer vision, pages 696–712. Springer, 2022

    2022

  45. [45]

    M. Zhou, C. He, R. Wang, and X. Chen. Ov3d-cg: Open-vocabulary 3d instance segmentation with contextual guidance. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 5305–5314, 2025. 12 A Proofs Lemma 1(MAB is Equivariant with respect to its query and Invariant with respect to its value).. Let MAB(Q, V) be the Multihead Attent...

    2025