Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

SAB3R: Semantic-Augmented Backbone in 3D Reconstruction

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One network, SAB3R, outputs a 3D point map and dense per-pixel open-vocabulary semantic features from unposed images in a single forward pass, and it beats a separate MASt3R-plus-CLIP pipeline on the new Map and Locate task.

desk verdict A useful new task and baseline, but the headline Map and Locate comparison is underdetermined because the two-stage baseline is frozen while SAB3R is fine-tuned on indoor data. read the letter →

arxiv 2506.02112 v2 pith:OPSRKU56 submitted 2025-06-02 cs.CV

Xuweiyi Chen , Tian Xia , Sihan Xu , Jianing Yang , Joyce Chai , Zezhou Cheng This is my paper
classification cs.CV
keywords 3Dreconstructionopen-vocabularysegmentationunposedimagesfeaturedistillationpointmapMASt3RCLIPDINOv2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that 3D reconstruction and open-vocabulary segmentation do not need to be separate pipelines, and introduces a task, Map and Locate, where a model receives unposed images and a set of text queries and must output both a coherent point map and per-pixel semantic labels. The authors build SAB3R, a single feed-forward network that predicts the 3D point map and dense CLIP and DINOv2 features at the same time, by fine-tuning the MASt3R reconstruction model with distillation losses from 2D feature teachers. On the new benchmark, SAB3R beats the two-stage approach of running MASt3R and CLIP separately on mIoU, accuracy, and completeness while keeping depth and pose estimates close to MASt3R. The point matters for embodied agents, which need to map and find objects from raw video without pre-scanned point clouds, camera poses, or per-scene optimization.

What carries the argument

The carrying object is the pointmap, a tensor $X^{a,b} \in \mathbb{R}^{H \times W \times 3}$ that maps every pixel of image $I_a$ to a 3D point expressed in the coordinate frame of camera $C_b$. SAB3R keeps MASt3R's Siamese ViT encoder and cross-attention decoder, which produce per-view representations $[H, H']$, and adds a dedicated head that regresses dense 2D features $S_v = \mathrm{Head}^{v}_{2D}([H^v, H'^v])$ alongside the 3D and descriptor heads. The training objective is $L_{\mathrm{total}} = L_{\mathrm{conf}} + \beta L_{\mathrm{match}} + \gamma L_{2D}$, where $L_{2D}$ supervises the new heads with FeatUp-upsampled MaskCLIP and DINOv2 features; FeatUp is used only to generate targets, not at inference.

What would settle it

Run SAB3R and the two-stage baseline on a held-out set of unposed videos from a different distribution, such as outdoor or dynamic scenes, with the same protocol, or recompute the ScanNet benchmark with multiple random splits and error bars after excluding ScanNet-family scenes from fine-tuning; if the baseline matches or beats SAB3R there, the paper's core empirical claim is undercut.

Watch

Extended reading notes

Core claim

SAB3R claims to show that semantic grounding and geometric reconstruction can be carried by the same weights. Starting from MASt3R, which predicts dense point maps from unposed image pairs, the authors add two lightweight heads that regress dense MaskCLIP and DINOv2 features, and fine-tune with a loss that keeps the reconstruction and matching terms while adding a per-pixel regression loss against FeatUp-upsampled teacher features. At inference the extra teacher networks disappear: the model outputs a pointmap, dense CLIP features, and dense DINOv2 features in a single forward pass. In the paper's experiments, this unified model outperforms the separate MASt3R-plus-CLIP baseline across all sparse-view settings on Map and Locate, retains MASt3R-level pose and depth, and improves zero-shot semantic segmentation on ADE20K over its own CLIP teacher.

Load-bearing premise

The strongest result assumes the Map and Locate benchmark is a fair, representative measure of the claimed unification: it uses 24 hand-selected ScanNet indoor scenes, no released splits or error bars, and a model fine-tuned on similar indoor data, so the margin over the two-stage baseline could come from benchmark or distribution choices rather than from unifying the tasks.

Editorial extensions

If this is right

  • An embodied agent could take raw, unposed camera frames and in one forward pass get both a 3D map and masks for arbitrary text queries, with no pre-scanned point cloud and no camera calibration step.
  • Joint training appears to help the semantic side: SAB3R beats its CLIP teacher on ADE20K zero-shot segmentation, suggesting 3D-geometry awareness transfers back to 2D recognition.
  • Adding DINOv2 distillation slightly improves zero-shot segmentation but does not improve pose or depth, so the extra head is a semantic add-on rather than a geometric one.
  • Because SAB3R needs only one forward pass instead of two separate models, it cuts inference cost roughly threefold, making the unified task practical for real-time use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Map and Locate benchmark were extended to outdoor, dynamic, or longer unposed sequences, the indoor-trained SAB3R would likely need joint training on such data; the paper's own KITTI depth results suggest the current indoor bias is a real boundary.
  • The distillation recipe is generic: the same added-head-plus-loss structure could transplant other 2D foundation features, such as depth, surface normal, or material features, into any pointmap-based reconstruction backbone, not just CLIP and DINO.
  • A natural next check would be whether the dense semantic features improve global alignment or loop closure in longer videos, something the 2-to-4-view benchmark does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Map and Locate, a new task that combines open-vocabulary semantic segmentation with 3D reconstruction from unposed multi-view images. To address this task, the authors propose SAB3R, a model fine-tuned from MASt3R with additional heads that regress dense CLIP and DINOv2 features, distilled from FeatUp-upsampled teacher features. The model is evaluated on monocular depth (NYUv2, KITTI), relative pose estimation (CO3Dv2), 2D zero-shot semantic segmentation (Pascal VOC, ADE20K), and a newly curated benchmark built from 24 ScanNet validation scenes. The central claim is that SAB3R performs geometry and semantics in a single forward pass and outperforms a separate two-stage MASt3R-plus-FeatUp-MaskCLIP pipeline on the Map and Locate benchmark, while retaining MASt3R-level reconstruction quality.

Significance. If the central claim is established, the paper would make a useful contribution: the Map and Locate task is a sensible step toward unposed, feed-forward scene understanding for embodied AI, and the single-forward-pass property could offer practical efficiency gains. The manuscript has clear strengths: it validates the model on multiple independent external benchmarks (NYUv2, KITTI, CO3Dv2, VOC, ADE20K), it is honest about under-training and indoor bias in the limitations section, and there is no equation-level circularity in the training objective. However, the headline comparison on Map and Locate is not yet controlled, the benchmark itself is small, unreleased, and lacks error bars, and the metric definitions in the evaluation section contain inconsistencies. These issues are load-bearing for the paper's main empirical conclusion and need to be resolved before the claims can be accepted.

major comments (3)
  1. [Section 3 / Table 3] The central comparison is confounded. SAB3R is fine-tuned from MASt3R on Habitat, ScanNet++, ARKitScenes, Co3Dv2, and BlenderMVS (Section 5.1), while the baseline deploys the original MASt3R checkpoint and the FeatUp-enhanced MaskCLIP teacher in a frozen two-stage pipeline (Section 3). The large reported gains on Map and Locate (e.g., mIoU 17.26 vs 4.57 at 2 views) could therefore be caused by additional optimization on indoor training data that is distributionally close to the ScanNet evaluation scenes, rather than by the unified architecture. The claim in the abstract that a unified model 'achieves superior performance' relative to separately deployed MASt3R and CLIP requires a controlled comparison where the two-stage baseline is fine-tuned on the same data with the same compute budget, or at least where the authors ablate the effect of the additional fine-tuning. As written, Table 3 does not isolate the unification hypothesis.
  2. [Section 3 / Table 3] The metric definitions are internally inconsistent and the reported numbers are ambiguous. The text defines mComp as 'the average distance from each predicted point to its nearest neighbor in the ground truth' and mdComp as the median of such distances, which are error measures for which lower values are better. However, the paper states that 'in all metrics, higher values consistently indicate better performance,' and Table 3 presents SAB3R's higher mComp values (0.73 vs 0.64 at 2 views) as superiority over the baseline. Furthermore, the mIoU definition as 'the ratio of correctly predicted points to the union of predicted and ground truth points' is not a standard intersection-over-union and is never specified at the class level. Because the headline result is read from Table 3, the authors must clarify the exact computation for each metric, correct the direction of the completeness metrics, and ensure that the narrative matches the reported numbers.
  3. [Section B.1 / Section 3] The Map and Locate benchmark is too small and insufficiently specified to support the paper's central quantitative claims on its own. It is built from 24 subjectively selected ScanNet validation scenes, producing 144 groups, with no error bars, no released scene IDs or splits, and no public benchmark code. The training data includes ScanNet++ and ARKitScenes, which are indoor RGB-D datasets with distributional similarity to ScanNet, so the evaluation may favor the method by construction. To establish the benchmark as a credible measure of the task, the authors should release the exact split and selection procedure, report standard deviations or confidence intervals for all metrics, and add at least one held-out or cross-dataset evaluation to demonstrate that the observed gains reflect generalizable joint modeling rather than scene-specific overfitting.
minor comments (5)
  1. [Section 4.2 / Eqs. (6)-(7)] The paper states that the loss weights beta, gamma1, and gamma2 are 'highly sensitive' and that small deviations can cause modality collapse, but no sensitivity analysis or ablation of these weights is provided; a small ablation table would strengthen the practical value of the method.
  2. [References] The same FeatUp paper appears twice in the reference list as [17] and [18]; please merge the duplicate entries.
  3. [Table 3] The LSM comparison is only reported for the 2-view setting, with no 3- or 4-view results, so the table's claim of comprehensive comparison across view counts should be qualified accordingly.
  4. [Section 5.2 / Table 1] The text acknowledges the outdoor KITTI degradation for SAB3R(CD), but the CLIP-only variant also shows a noticeable gap (11.63 Rel vs MASt3R's 8.28); please address both variants when discussing outdoor robustness.
  5. [Section 5.3] The statement that SAB3R 'eliminates the concept of patches' is somewhat overstated because the features are produced by a patch-based ViT backbone and later upsampled to per-pixel resolution; please clarify the distinction from prior patch-level matching mechanisms.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SAB3R's semantic features are distilled from external teachers but validated against independent ground truth, and no load-bearing self-citation or definitional reduction appears in the paper.

full rationale

I walked the paper's derivation chain. SAB3R's semantic heads are supervised by regression to FeatUp-MaskCLIP and DINOv2 features under L2D = ||Sv - S_hat_v|| (Eq. 5), with S_hat_v explicitly defined as target features extracted from foundation models. This is a standard distillation objective, not a prediction of the teacher's output presented as independent evidence. The paper's headline benchmark, Map and Locate, is evaluated against external ScanNet ground-truth labels, depth, and poses, using nearest-predicted-point matching for mIoU and accuracy; these metrics do not reduce to the training targets by construction. The central comparison in Table 3 compares SAB3R, fine-tuned on indoor data, against a frozen MASt3R + FeatUp-MaskCLIP baseline; this is a fairness/confound concern, not circularity, because the student's win is not forced by the training objective and is empirically validated against ground truth. The zero-shot semantic segmentation results in Table 4 are also measured on external VOC and ADE20K labels, so the student's improvement over its own teacher is a genuine empirical result. No load-bearing self-citation is present: MASt3R, DUSt3R, FeatUp, and MaskCLIP are all external prior works, and the only author-overlapping citation (LLM-Grounder) appears in related work and does not support any technical claim. There is no uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known result as a new derivation. The potential issue flagged by a skeptical reader, namely that the baseline is not fine-tuned, is a correctness risk about experimental control, not a circularity of the kind this pass is asked to identify. I therefore find no significant circularity and assign score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The core method relies on three externally provided components: MASt3R for geometry, FeatUp MaskCLIP and DINOv2 for semantic targets, and the selected indoor training datasets. The only free parameters introduced by the authors are the loss weights beta and gamma, which they explicitly say are sensitive. No new physical or conceptual entities are postulated; Map and Locate is an evaluation benchmark, not an invented entity. The main unstated risk is whether the small indoor benchmark and its curation fairly represent the claimed unposed real-world setting.

free parameters (3)
  • beta (Lmatch weight) = 0.75
    Chosen by the authors based on empirical observations; the paper notes hyperparameters are highly sensitive (Sec 5.1).
  • gamma (L2D weight, CLIP-only) = 20
    Set for the MaskCLIP distillation run; small deviations are reported to cause modality collapse (Sec 5.1).
  • gamma1 and gamma2 (joint CLIP+DINO weights) = 20 and 4
    Separate weights for the two-feature training; chosen empirically and reported as sensitive (Sec 5.1, Eq. 7).
assumptions (4)
  • domain assumption Each camera ray intersects only a single 3D point (no translucent surfaces).
    Invoked in Sec 3 Task Setting to define the one-to-one pixel-to-point mapping; excludes a class of real scenes but is standard for dense reconstruction benchmarks.
  • domain assumption FeatUp-upsampled MaskCLIP features and DINOv2 features are reliable dense semantic targets that remain aligned with CLIP's text embedding space.
    The whole distillation (Sec 4.2, Eq. 5) treats these teacher features as ground truth for semantics; if the teachers lack text alignment or local discriminability, the student's open-vocabulary performance is capped.
  • domain assumption MASt3R's pointmap regression and global alignment provide sufficiently accurate geometry for 2 to 4 unposed views.
    SAB3R inherits the MASt3R backbone and does not add geometric supervision beyond the original losses (Sec 4.1, Sec 4.2); any geometric failure of MASt3R propagates to Map and Locate.
  • domain assumption The 24 selected ScanNet validation scenes are representative of real-world embodied unposed settings.
    Task claims in the abstract and Sec 5.4 generalize from this small indoor benchmark; the paper itself reports degraded outdoor depth (Table 1, KITTI) and acknowledges indoor bias in Sec D of the supplement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAB3R: Semantic-Augmented Backbone in 3D Reconstruction." pith.science (2026). https://pith.science/paper/OPSRKU56

@misc{pith2026250602112,
  author       = {Pith},
  title        = {Pith review of: SAB3R: Semantic-Augmented Backbone in 3D Reconstruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPSRKU56}},
  note         = {Machine review of arXiv:2506.02112}
}
read the original abstract

We introduce a new task, Map and Locate, which unifies the traditionally distinct objectives of open-vocabulary segmentation - detecting and segmenting object instances based on natural language queries - and 3D reconstruction, the process of estimating a scene's 3D structure from visual inputs. Specifically, Map and Locate involves generating a point cloud from an unposed video and segmenting object instances based on open-vocabulary queries. This task serves as a critical step toward real-world embodied AI applications and introduces a practical task that bridges reconstruction, recognition and reorganization. To tackle this task, we introduce a simple yet effective baseline, which we denote as SAB3R. Our approach builds upon MASt3R, a recent breakthrough in 3D computer vision, and incorporates a lightweight distillation strategy. This method transfers dense, per-pixel semantic features from 2D vision backbones (eg, CLIP and DINOv2) to enhance MASt3R's capabilities. Without introducing any auxiliary frozen networks, our model generates per-pixel semantic features and constructs cohesive point maps in a single forward pass. Compared to separately deploying MASt3R and CLIP, our unified model, SAB3R, achieves superior performance on the Map and Locate benchmark. Furthermore, we evaluate SAB3R on both 2D semantic segmentation and 3D tasks to comprehensively validate its effectiveness.

Figures

Figures reproduced from arXiv: 2506.02112 by the authors.

Figure 1
Figure 1. Given an unposed input video (a), we show ground truth [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our method, SAB3R , a semantic-augmented backbone for 3D reconstruction, enables zero-shot open-vocabulary segmentation and 3D reconstruction from unposed images in a single forward pass. By jointly performing reconstruction and open-vocabulary semantic segmentation, SAB3R introduces a novel capability that unifies these tasks within a single framework. perceptual similarity. Ideally, these tasks should mutually ben… view at source ↗
Figure 3
Figure 3. Methods Architecture. We distill dense features from CLIP and DINO into the MASt3R framework, enriching it with 2D semantic understanding. Each encoder-decoder pair operates on multi-view images, sharing weights and exchanging information to ensure consistent feature extraction across views. The model simultaneously generates depth, dense DINOv2, and dense CLIP features, which are then used for multi-view 3D reconst… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: mIoU Analysis on Frequently Occurring Objects Across Three Methods (Sparse View = 3). This plot compares mIoU values for frequently appearing objects, illustrating performance differences between our methods and the pipeline approaches and providing insights into the s…
Figure 5
Figure 5. Figure 5: Qualitative Example of Map and Locate. This fig￾ure illustrates an example from our benchmark. In (a), the ground truth annotation for the scene is highlighted in red, with the dresser segmented from the rest of the scene on the left. In (b), the pre￾dictions from SAB3…
Figure 6
Figure 6. Figure 6: Camera Distributions. Camera translation differences and rotation differences at different group levels. Relative Camera Pose We evaluate SAB3R on the task of relative pose estimation using the CO3Dv2 dataset [46]. To assess the relative pose error for each image pair,…
Figure 7
Figure 7. Figure 7: Qualitative Examples of Map and Locate with SAB3R . Panels (a), (b), and (c) illustrate successful examples of 3D scene reconstruction and accurate object segmentation. In each sub-group, the top row shows the ground truth, with the target objects highlighted in red, a…
Figure 8
Figure 8. Figure 8: 3D Feature Visualizations. Additional visualizations of 3D features are presented for DINO and CLIP, alongside the original RGB 3D point map for reference. 4 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction

    cs.CV 2026-01 conditional novelty 6.0 of 10

    TTSA3R adds temporal and spatial adaptive gating to the CUT3R persistent-state update rule, cutting error growth on long sequences from >4x to 1.33x over 50-250 frames.

  2. Unified Semantic Transformer for 3D Scene Understanding

    cs.CV 2025-12 reject novelty 5.0 of 10

    UNITE is a feed-forward transformer that predicts geometry plus semantic, instance, open-vocabulary, and articulation features for indoor 3D scenes from RGB images in one pass.

Reference graph

Works this paper leans on

81 extracted references · 73 canonical work pages · cited by 2 Pith papers

  1. [1]

    Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes

    Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Vision–ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 422–440. Springer, 2020. 1

    2020

  2. [2]

    Building rome in a day

    Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Si- mon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in a day. Communications of the ACM , 54 (10):105–112, 2011. 3

    work page 2011

  3. [3]

    ARK- itscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARK- itscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. InThirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Trac...

    work page 2021

  4. [4]

    Auto-rectify network for unsupervised indoor depth estimation, 2021

    Jia-Wang Bian, Huangying Zhan, Naiyan Wang, Tat-Jun Chin, Chunhua Shen, and Ian Reid. Auto-rectify network for unsupervised indoor depth estimation, 2021. 6

    work page 2021

  5. [5]

    Unsupervised scale-consistent depth learning from video.In- ternational Journal of Computer Vision (IJCV), 2021

    Jia-Wang Bian, Huangying Zhan, Naiyan Wang, Zhichao Li, Le Zhang, Chunhua Shen, Ming-Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth learning from video.In- ternational Journal of Computer Vision (IJCV), 2021. 6

    work page 2021

  6. [6]

    Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, Jose Neira, Ian Reid, and John J. Leonard. Past, present, and future of simultaneous localiza- tion and mapping: Toward the robust-perception age. IEEE Transactions on Robotics, 32(6):1309–1332, 2016. 3

    work page 2016

  7. [7]

    Emerg- ing properties in self-supervised vision transformers, 2021

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers, 2021. 3

    2021

  8. [8]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11165–11174, 2023. 7

    work page 2023

Show all 81 references
  1. [9]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pages 202–221. Springer, 2020. 1

    2020

  2. [10]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017. 4, 2

    2017

  3. [11]

    Superpoint: Self-supervised Interest Point Detection and Description

    Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabi- novich. Superpoint: Self-supervised Interest Point Detection and Description. In CVPR, 2018. 6

    2018

  4. [12]

    Maskclip: Masked self-distillation advances contrastive language-image pretraining, 2023

    Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Maskclip: Masked self-distillation advances contrastive language-image pretraining, 2023. 4, 7, 1

    2023

  5. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  6. [14]

    Everingham, S

    M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual ob- ject classes challenge: A retrospective.International Journal of Computer Vision, 111(1):98–136, 2015. 7

    2015

  7. [15]

    Nerf-sos: Any-view self- supervised object segmentation on complex scenes, 2022

    Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, De- jia Xu, and Zhangyang Wang. Nerf-sos: Any-view self- supervised object segmentation on complex scenes, 2022. 3

    2022

  8. [16]

    Large spatial model: End-to-end unposed images to semantic 3d, 2024

    Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjie Li, Kairun Wen, Shijie Zhou, Achuta Kadambi, Zhangyang Wang, Danfei Xu, Boris Ivanovic, Marco Pavone, and Yue Wang. Large spatial model: End-to-end unposed images to semantic 3d, 2024. 3, 7, 8

    2024

  9. [17]

    Stephanie Fu, Mark Hamilton, Laura Brandt, Axel Feld- man, Zhoutong Zhang, and William T. Freeman. Featup: A model-agnostic framework for features at any resolution,

  10. [18]

    Brandt, Axel Feld- mann, Zhoutong Zhang, and William T

    Stephanie Fu, Mark Hamilton, Laura E. Brandt, Axel Feld- mann, Zhoutong Zhang, and William T. Freeman. Featup: A model-agnostic framework for features at any resolution. In The Twelfth International Conference on Learning Repre- sentations, 2024. 4, 5, 7, 1

    2024

  11. [19]

    Vision meets robotics: The kitti dataset

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. Interna- tional Journal of Robotics Research (IJRR), 2013. 6, 1

    2013

  12. [20]

    Cl ´ement Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth prediction. 2019. 6

    2019

  13. [21]

    3d-llm: Inject- ing the 3d world into large language models, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Inject- ing the 3d world into large language models, 2023. 3

    2023

  14. [22]

    Open- CLIP, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- CLIP, 2021. 1

    2021

  15. [23]

    Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, and Katerina Fragkiadaki

    Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W. Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, and Katerina Fragkiadaki. Odin: A single model for 2d and 3d segmentation, 2024. 3

    2024

  16. [24]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42 (4), 2023. 1, 3

    2023

  17. [25]

    Lerf: Language embedded radiance fields, 2023

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields, 2023. 3, 7 9

    2023

  18. [26]

    Anthony Movshon

    Michael Landy and J. Anthony Movshon. The Plenoptic Function and the Elements of Early Vision , pages 3–20

  19. [27]

    Large-scale localization datasets in crowded indoor spaces, 2021

    Donghwan Lee, Soohyun Ryu, Suyong Yeon, Yonghan Lee, Deokhwa Kim, Cheolho Han, Yohann Cabon, Philippe Weinzaepfel, Nicolas Gu ´erin, Gabriela Csurka, and Martin Humenberger. Large-scale localization datasets in crowded indoor spaces, 2021. 1

    2021

  20. [28]

    Ground- ing image matching in 3d with mast3r, 2024

    Vincent Leroy, Yohann Cabon, and J´erˆome Revaud. Ground- ing image matching in 3d with mast3r, 2024. 5

    2024

  21. [29]

    Ground- ing image matching in 3d with mast3r, 2024

    Vincent Leroy, Yohann Cabon, and Jerome Revaud. Ground- ing image matching in 3d with mast3r, 2024. 3, 4, 5, 6, 7, 1

    2024

  22. [30]

    Weinberger, Serge Belongie, Vladlen Koltun, and Ren ´e Ranftl

    Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and Ren ´e Ranftl. Language-driven semantic seg- mentation, 2022. 3

    2022

  23. [31]

    Megadepth: Learning single- view depth prediction from internet photos

    Zhengqi Li and Noah Snavely. Megadepth: Learning single- view depth prediction from internet photos. In Computer Vision and Pattern Recognition (CVPR), 2018. 1

    2018

  24. [32]

    Zhang, and Shubham Tulsiani

    Amy Lin, Jason Y . Zhang, and Shubham Tulsiani. Rel- pose++: Recovering 6d poses from sparse-view observa- tions. CoRR, abs/2305.04926, 2023. 6, 7

    2023 arXiv

  25. [33]

    Pixel-perfect structure-from-motion with featuremetric refinement

    Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Marc Pollefeys. Pixel-perfect structure-from-motion with featuremetric refinement. In ICCV, 2021. 6

    2021

  26. [34]

    Open-world semantic seg- mentation via contrasting and clustering vision-language embedding

    Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open-world semantic seg- mentation via contrasting and clustering vision-language embedding. In European Conference on Computer Vision , pages 275–292. Springer, 2022. 7

    2022

  27. [35]

    Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation

    Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In In- ternational Conference on Machine Learning, pages 23033– 23044. PMLR, 2023. 7

    2023

  28. [36]

    The three r’s of computer vision: Recognition, recon- struction and reorganization

    Jitendra Malik, Pablo Arbel ´aez, Jo ˜ao Carreira, Katerina Fragkiadaki, Ross Girshick, Georgia Gkioxari, Saurabh Gupta, Bharath Hariharan, Abhishek Kar, and Shubham Tul- siani. The three r’s of computer vision: Recognition, recon- struction and reorganization. Pattern Recogni...

    2016

  29. [37]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. In ECCV, 2020. 1, 3

    2020

  30. [38]

    Raul Mur-Artal, J. M. M. Montiel, and Juan D. Tardos. Orb- slam: A versatile and accurate monocular slam system.IEEE Transactions on Robotics, 31(5):1147–1163, 2015. 3

    2015

  31. [39]

    Indoor segmentation and support inference from rgbd images

    Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 4, 6, 1

    2012

  32. [40]

    Robustmix: Improving robust- ness by regularizing the frequency bias of deep nets, 2024

    Jonas Ngnawe, Marianne Abemgnigni Njifon, Jonathan Heek, and Yann Dauphin. Robustmix: Improving robust- ness by regularizing the frequency bias of deep nets, 2024. 6

    2024

  33. [41]

    Dinov2: Learning robust visual features with- out supervision, 2024

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mah- moud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michae...

    2024

  34. [42]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu ”Max” Jiang, An- drea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 3

    2023

  35. [43]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 5, 1, 2, 3

    2021

  36. [44]

    Per- ceptual grouping in contrastive vision-language models

    Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Per- ceptual grouping in contrastive vision-language models. ICCV , 2023. 7, 2

    2023

  37. [45]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. In ICCV, 2021. 6

    2021

  38. [46]

    Com- mon objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction

    Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Com- mon objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Con- ference on Computer Vision, 2021. 6, 7, 1, 2

    2021

  39. [47]

    Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency

    Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guan- grun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. arXiv preprint arXiv:2302.10307, 2023. 7

    2023 arXiv

  40. [48]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Ni...

    2022

  41. [49]

    SuperGlue: Learning feature matching with graph neural networks

    Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, 2020. 6

    2020

  42. [50]

    Habitat: A platform for embodied ai research, 2019

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research, 2019. 1

    2019

  43. [51]

    Mask3D: Mask Trans- former for 3D Semantic Instance Segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or 10 Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask Trans- former for 3D Semantic Instance Segmentation. 2023. 1

    2023

  44. [52]

    Sch ¨onberger and Jan-Michael Frahm

    Johannes L. Sch ¨onberger and Jan-Michael Frahm. Structure- from-motion revisited. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 4104– 4113, 2016. 3

    2016

  45. [53]

    Distilled feature fields enable few-shot language-guided manipulation, 2023

    William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guided manipulation, 2023. 3

    2023

  46. [54]

    Panoptic lifting for 3d scene understanding with neural fields

    Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bul `o, Nor- man M ¨uller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (C...

    2023

  47. [55]

    Kick back & relax: Learning to reconstruct the world by watching slowtv, 2023

    Jaime Spencer, Chris Russell, Simon Hadfield, and Richard Bowden. Kick back & relax: Learning to reconstruct the world by watching slowtv, 2023. 6

    2023

  48. [56]

    Efm3d: A bench- mark for measuring progress towards 3d egocentric founda- tion models, 2024

    Julian Straub, Daniel DeTone, Tianwei Shen, Nan Yang, Chris Sweeney, and Richard Newcombe. Efm3d: A bench- mark for measuring progress towards 3d egocentric founda- tion models, 2024. 3

    2024

  49. [57]

    Sc-depthv3: Robust self- supervised monocular depth estimation for dynamic scenes

    Libo Sun, Jia-Wang Bian, Huangying Zhan, Wei Yin, Ian Reid, and Chunhua Shen. Sc-depthv3: Robust self- supervised monocular depth estimation for dynamic scenes. IEEE Transactions on Pattern Analysis and Machine Intelli- gence (TPAMI), 2023. 6

    2023

  50. [58]

    Habitat 2.0: Training home assistants to rearrange their habitat

    Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir V ondrus, Sameer Dharur, Franziska Meier, Woj- ciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun,...

  51. [59]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras, 2021

    Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras, 2021. 3

    2021

  52. [60]

    Neural feature fusion fields: 3d distillation of self- supervised 2d image representations, 2022

    Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self- supervised 2d image representations, 2022. 3

    2022

  53. [61]

    Sam-clip: Merging vision foundation models towards semantic and spatial understanding, 2024

    Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding, 2024. 7, 2

    2024

  54. [62]

    PoseDiffusion: Solving pose estimation via diffusion-aided bundle adjustment

    Jianyuan Wang, Christian Rupprecht, and David Novotny. PoseDiffusion: Solving pose estimation via diffusion-aided bundle adjustment. 2023. 6, 7

    2023

  55. [63]

    Vggsfm: Visual geometry grounded deep structure from motion

    Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21686–21697, 2024. 3

    2024

  56. [64]

    Dust3r: Geometric 3d vi- sion made easy, 2023

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vi- sion made easy, 2023. 3, 4, 5, 6, 7

    2023

  57. [65]

    Point transformer v3: Simpler, faster, stronger

    Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xi- hui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. In CVPR, 2024. 3

    2024

  58. [66]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18134–18144, 2022. 7

    2022

  59. [67]

    Learning open-vocabulary semantic segmentation models from natural language supervision

    Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 2935–2944, 2023. 7

    2023

  60. [68]

    Llm- grounder: Open-vocabulary 3d visual grounding with large language model as an agent

    Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Llm- grounder: Open-vocabulary 3d visual grounding with large language model as an agent. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages ...

    2024

  61. [69]

    Sat: 2d semantics assisted training for 3d visual grounding, 2021

    Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semantics assisted training for 3d visual grounding, 2021. 3

    2021

  62. [70]

    Blendedmvs: A large-scale dataset for generalized multi-view stereo net- works

    Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo net- works. Computer Vision and Pattern Recognition (CVPR) ,

  63. [71]

    Scannet++: A high-fidelity dataset of 3d indoor scenes, 2023

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes, 2023. 6, 1

    2023

  64. [72]

    New crfs: Neural window fully-connected crfs for monocular depth estimation, 2022

    Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. New crfs: Neural window fully-connected crfs for monocular depth estimation, 2022. 6

    2022

  65. [73]

    Improving 2d feature representations by 3d-aware fine-tuning, 2024

    Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, and Jan Eric Lenssen. Improving 2d feature representations by 3d-aware fine-tuning, 2024. 3

    2024

  66. [74]

    Zamir, Tilman Wekel, Pulkit Agrawal, Colin Wei, Jitendra Malik, and Silvio Savarese

    Amir R. Zamir, Tilman Wekel, Pulkit Agrawal, Colin Wei, Jitendra Malik, and Silvio Savarese. Generic 3D Repre- sentation via Pose Estimation and Matching, page 535–553. Springer International Publishing, 2016. 1

    2016

  67. [75]

    Zhang, Deva Ramanan, and Shubham Tulsiani

    Jason Y . Zhang, Deva Ramanan, and Shubham Tulsiani. Rel- pose: Predicting probabilistic relative rotation for single ob- jects in the wild. In ECCV, 2022. 6, 7

    2022

  68. [76]

    Cameras as rays: Pose estimation via ray diffusion

    Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. InInternational Con- ference on Learning Representations (ICLR), 2024. 6, 7

    2024

  69. [77]

    Monovit: Self-supervised monocular depth estimation with a vision transformer

    Chaoqiang Zhao, Youmin Zhang, Matteo Poggi, Fabio Tosi, Xianda Guo, Zheng Zhu, Guan Huang, Yang Tang, and Ste- fano Mattoccia. Monovit: Self-supervised monocular depth estimation with a vision transformer. In International Con- ference on 3D Vision, 2022. 6

    2022

  70. [78]

    Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and An- drew J. Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV, 2021. 3 11

    2021

  71. [79]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302–321, 2019. 7

    2019

  72. [80]

    Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields

    Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Ze- hao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. InPro- ceedings of the IEEE/CVF Conference on Compu...

    2024

  73. [81]

    Unifying 3d vision-language understanding via prompt- able queries

    Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via prompt- able queries. ECCV, 2024. 3 12 SAB3R: Semantic-Augmented Backbone in 3D Reconstruction Supplementary Mate...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.