Pith. sign in

REVIEW 4 major objections 5 minor 21 references

Pre-training surgical vision models on RGB-D pairs makes them beat RGB-only models on every tested task, including with far fewer labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 07:49 UTC pith:LX57AL2C

load-bearing objection Worth engaging: a genuinely useful benchmark with an over-claimed data-efficiency headline and a depth attribution that is real but partly confounded. the 4 major comments →

arxiv 2601.18929 v2 pith:LX57AL2C submitted 2026-01-26 cs.CV

On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training

John J. Han , Adam Schmidt , Muhammad Abdullah Jamal , Chinedu Nwoye , Anita Rau , Jie Ying Wu , Omid Mohareri This is my paper
classification cs.CV
keywords surgical vision foundation modelsRGB-D pre-trainingmasked autoencodersself-supervised learningdata efficiencygeometric representation learningrobotic surgerymulti-modal pre-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that surgical vision foundation models learn better representations when RGB images are paired with depth maps during self-supervised pre-training, and that the gains are large enough to make label-hungry tasks far cheaper. The authors compare eight ViT-based models across eight datasets and report that MultiMAE—a masked autoencoder trained to reconstruct both RGB and depth from masked tokens—beats every RGB-only baseline on object detection, segmentation, pose estimation, and depth estimation. They further report that a MultiMAE fine-tuned on 25% of labeled data consistently outperforms RGB-only models fine-tuned on the full dataset, and that this comes at no inference cost because depth is used only in pre-training. The paper's main practical claim is that depth-aware pre-training, not model scaling or extra data collection, is the effective next step for surgical scene understanding.

Core claim

The paper's central claim is that feeding surgical vision transformers paired RGB and depth images during self-supervised pre-training—specifically through architectures that tokenize depth explicitly and reconstruct masked tokens across modalities—yields representations that outperform RGB-only pre-training on object detection, semantic segmentation, pose estimation, and depth estimation across eight surgical datasets. The strongest version of the claim is quantitative: a MultiMAE model pre-trained on 1.4 million da Vinci frames with pseudo-depth, fine-tuned on 25% of downstream labels, consistently beats RGB-only baselines fine-tuned on 100%. The authors further claim that the effect is no

What carries the argument

The load-bearing object is MultiMAE's cross-modal masked reconstruction: RGB and depth images are tokenized into a shared latent space, patches are masked, and the model must reconstruct missing tokens of both modalities from the unmasked tokens of either. This forces the encoder to learn correspondences between appearance and geometry. The paper contrasts it with DINOv2-RGBD, which randomly swaps RGB and depth crops in a self-distillation objective without architectural change—that variant gains almost nothing—and with Mask3D, which reconstructs only depth and thereby favors dense pixel tasks over object-level semantics.

Load-bearing premise

The paper attributes its gains to depth information, but the models being compared differ not only in modality but also in pre-training objective, epoch count, batch size, and architecture; if depth is not what drives the improvement, the central claim collapses.

What would settle it

Run MultiMAE pre-training on the same 1.4M surgical frames with the same objective, epochs, and batch size, but replace the depth maps with identically shaped random noise or a constant image. If downstream performance matches the RGB-D model, then the geometric signal itself is not responsible for the reported gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Surgical foundation models can be built from existing monocular or stereo video: pseudo-depth from an off-the-shelf stereo matcher or monocular depth net suffices; no new sensor hardware is needed.
  • Downstream deployment is unchanged: models that used RGB now take RGB at inference with identical architecture and compute; the depth signal is baked into the weights.
  • Label-hungry surgical tasks become cheaper: fine-tuning on 25% of annotations can exceed RGB-only full-data performance, and the effect persists at extreme scarcity (2.5% on EndoVis18).
  • A single frozen geometry-aware backbone can serve multiple tasks, since frozen multimodal backbones often beat fully fine-tuned RGB-only models.
  • Geometry-aware pre-training partly compensates for narrow, homogeneous surgical pre-training data, narrowing the usual ImageNet-versus-domain gap.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Reader's inference: the causal story would be strengthened by a controlled ablation in which the same architecture, objective, epoch budget, and batch size are trained with and without depth; as published, MultiMAE differs from MAE in all of these, so part of the margin could come from the multi-task reconstruction objective itself rather than the depth modality.
  • If depth is the active ingredient, the recipe should transfer to other endoscopic and interventional settings where stereo or monocular depth can be synthesized, such as colonoscopy, arthroscopy, or cataract surgery, where annotation is even scarcer.
  • A testable extension: corrupt the depth channel (e.g., random noise or permuted patches) and re-run MultiMAE pre-training; if performance stays high, the model is learning from reconstruction geometry rather than from actual depth.
  • The data-efficiency result suggests a practical rule for surgical AI teams: before collecting more annotations, try re-pre-training an RGB-D masked autoencoder on unlabeled video with pseudo-depth, since the 25%-beats-100% pattern held across four tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript reports a large-scale empirical study of RGB-D pre-training for surgical vision. The authors pre-train eight ViT-B models on 1.4M da Vinci frames paired with FoundationStereo pseudo-depth, then evaluate frozen-backbone and end-to-end fine-tuning transfer on eight datasets spanning object detection, semantic segmentation, depth estimation, and pose estimation. The central claims are (i) that models with explicit geometric tokenization, especially MultiMAE, substantially outperform RGB-only baselines, and (ii) that geometry-aware pre-training yields 'remarkable data efficiency,' with models fine-tuned on 25% of labeled data consistently surpassing RGB-only models trained on the full dataset. The paper also introduces a naive DINOv2-RGBD baseline to separate architectural from data effects and releases a COLMAP-corrected version of the SCARED depth benchmark.

Significance. If the central causal claim were established, the paper would have clear practical value: it would show that depth-only-at-pre-training improves downstream spatial tasks without inference-time cost, and that annotation-scarce surgical settings could benefit substantially. The study has genuine strengths: it is large-scale; it includes an external structured-light benchmark (SCARED) in addition to pseudo-labeled in-domain depth; it ablates task-head architecture (Table 8) and pseudo-depth source (Table 7); and the authors promise release of the corrected SCARED data. However, the paper's central attribution of the gains to the geometric content of depth is confounded, and the headline 'consistently surpass' claim is contradicted by the paper's own Table 11a. The DV depth benchmark is partly circular because both pre-training and evaluation use FoundationStereo pseudo-depth. These issues affect the paper's main conclusions, so major revision is required.

major comments (4)
  1. [§3.2, Table 3 and Table 10] The causal attribution of the gains to depth is not supported by the experimental design. MultiMAE differs from the RGB-only baselines not only in input modality but also in pretext objective (joint RGB+depth reconstruction vs. RGB reconstruction), tokenization, and training recipe. Table 3 shows different batch sizes and learning rates (MultiMAE: batch 3072, LR 1e-4; MAE: batch 5120, LR 1e-5), and the models also differ in epochs and initialization. No RGB-only MultiMAE control is provided. The paper's most controlled comparison, DINOv2-RGBD vs. DINOv2-RGB (Finding 5), shows only a 3.08% average gain and a degradation on SurgPose (Table 10: 49.4 vs. 51.4 mAP). This is the paper's own evidence that depth alone does not explain the large MultiMAE improvements. The abstract's phrasing that 'geometric-aware pre-training' drives the results overstates what this matrix can establish. Please a
  2. [§4, Finding 2 and Table 11a] The abstract and Finding 2 claim that models fine-tuned on 25% of labeled data 'consistently surpass' RGB-only models trained on the full dataset. This is contradicted by Table 11a (CholecTrack20): MultiMAE (Surg) at 25% achieves 26.7 mAP, while MAE (Surg) at 100% achieves 29.6 mAP. Thus the claim holds on 3 of the 4 reported tasks but not on object detection. The phrase 'consistently' should be replaced with a qualified statement, e.g., 'on most tasks,' and the exception should be reported in the abstract and in Finding 2. Figure 4 should also be annotated so the reader can see the exception rather than inferring a universal trend.
  3. [§3.1 and §3.3, DV depth benchmark] The DV depth-estimation benchmark is partly circular. Pre-training depth maps are generated by FoundationStereo (Section 3.1), and the DV downstream depth labels are also generated by FoundationStereo (Section 3.3). Gains on this benchmark therefore partly reflect learning the generator's bias rather than general geometric understanding. The external SCARED benchmark partially mitigates this, and Table 10 shows MultiMAE also improves there (0.624 vs. 0.526 for MAE), but the effect is smaller and the ranking is not identical. The paper should (a) explicitly acknowledge this circularity in Section 4.1, which currently mentions only pseudo-depth errors, and (b) present SCARED as the primary external depth evidence, or additionally evaluate on depth labels from a different generator (e.g., Depth Anything) for the DV split.
  4. [§4, Fig. 4 and Table 11] The data-efficiency experiments report only means over three seeds, with no error bars or standard deviations. Several central comparisons rest on differences that could be within seed noise, and the 'consistently' language depends on these rankings. Please report per-seed results or standard deviations, and ideally a paired significance test, for the 25%/50%/75% comparisons. This is particularly important because the matched DINOv2-RGBD comparison (Finding 5) shows only a small 3.08% average gain, and the reader cannot assess whether that gain is robust.
minor comments (5)
  1. [Table 11 vs. Table 1/10] The naming is inconsistent: 'MAE (Surg)' in Table 11 refers to the same model called 'MAE' in Tables 1 and 10. Similarly, Figure 4 uses 'MAE' with a caption mentioning 'MAE (DV)'. Please unify notation throughout.
  2. [References] References 'Yang et al. 2024a' and 'Yang et al. 2024b' appear to refer to the same Depth Anything paper with identical titles. Please remove the duplicate or correct the citation.
  3. [§3.1] The depth maps are 'normalized to [0,1] during pre-training,' but it is unclear whether this normalization is per-image or dataset-wide. This detail affects the pretext task and should be stated precisely.
  4. [Figure 3] The legend and caption describe 'solid inner bars' and 'hatched outer bars,' but the figure is small and the hatching is not visible in the reproduction. Consider a larger figure or a different visual encoding.
  5. [Appendix C.2] The sentence 'The number of visible patches is set constant throughout training' is truncated or incomplete. Please complete the description of the masking procedure.

Circularity Check

1 steps flagged

Only partial circularity: the DV depth benchmark reuses the same FoundationStereo pseudo-depth generator used to create pre-training targets; external benchmarks keep the central claim independent.

specific steps
  1. fitted input called prediction [Section 3.1 ('Pre-training Data') and Section 3.3 ('Downstream Tasks'), Table 2; DV depth results in Fig. 3/4 and Tables 9-11.]
    "To generate depth images, we leverage the recently released FoundationStereo (Wen et al., 2025) model, which generates disparity maps given stereo rectified pairs of images. ... For the remaining depth estimation dataset (DV), we generate pseudo-labeled depth maps using disparity maps obtained from FoundationStereo (Wen et al., 2025). Notably, among the eight datasets evaluated, DV is the only one that is in-domain with respect to the pre-training."

    The depth maps used as pre-training targets are normalized disparity maps produced by FoundationStereo, and the DV downstream depth-estimation 'ground truth' is also produced by FoundationStereo. Thus a model like MultiMAE or Mask3D, pre-trained to reconstruct exactly this pseudo-depth distribution, is being evaluated on its ability to reproduce the same generator's outputs. The DV depth-estimation numbers, including the data-efficiency curves in Fig. 4d, therefore partly measure faithfulness to FoundationStereo's biases rather than genuine 3D geometric understanding. This is a real but partial circularity: SCARED uses structured-light depth and still shows gains, and Appendix F.1 shows similar trends when pre-training with Depth Anything labels. The central claim therefore retains indepen

full rationale

The paper is an empirical comparison rather than a derivation, so most of the analysis is not circular in the formal sense. The main weakness—confounded model comparisons (MultiMAE vs. MAE differs in objective, tokenization, epochs, and batch size)—is a validity concern, not a circularity concern. The one genuine circular element is the DV depth-estimation benchmark: the same off-the-shelf FoundationStereo model generates both the pre-training depth targets and the evaluation labels, and the paper explicitly notes DV is in-domain with respect to pre-training. This makes the DV-specific depth results partly a self-consistency check against the pseudo-label generator. However, this is not load-bearing for the paper's overall conclusion: SCARED (structured-light ground truth) shows depth-estimation gains, the detection/segmentation/pose benchmarks are independent of depth-label generation, and Appendix F.1 demonstrates robustness across a different pseudo-depth source. No load-bearing self-citation chain or imported uniqueness theorem appears. Therefore the appropriate score is 2: minor, localized circularity with the central claim still independently supported.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim relies on pseudo-depth quality, faithful reuse of prior architectures, fair task-head evaluation, and scale-invariance of the effect. No new physical entities are introduced. The most consequential free choices are the unequal pre-training budgets and the hand-set DINOv2-RGBD sampling probabilities, which prevent a clean modality-only ablation.

free parameters (3)
  • Pre-training compute budget (epochs) = MAE: 400, MultiMAE: 400, DINOv2 variants: 100, Mask3D: 100
    Epoch counts differ across models due to compute constraints (Appendix B), and these differences confound the comparison between RGB-only and RGB-D pre-training.
  • DINOv2-RGBD modality sampling probabilities = p_teacher = p_global_student = p_local_student = 0.5
    Chosen by hand in Appendix A; the behavior of the naive multimodal variant depends on these probabilities, and no sensitivity analysis is provided.
  • SCARED COLMAP scale recovery factor = s = median(D / D_hat), per sequence
    Used in Appendix D to metricize COLMAP camera poses; estimated from keyframe depth maps rather than measured directly.
axioms (4)
  • domain assumption FoundationStereo pseudo-depth is a sufficiently accurate geometric signal for pre-training and for evaluating the DV depth benchmark.
    Section 3.1 uses FoundationStereo to generate all pre-training depth maps, and Section 3.3 uses the same generator for DV pseudo-labels. If these depth maps are systematically biased, the geometric-prior explanation is weakened.
  • domain assumption The implementations of MAE, DINOv2, MultiMAE, and the re-implemented Mask3D faithfully match the original methods.
    Section 3.2 states that original codebases are used where available, but Mask3D is re-implemented because the authors did not release code or weights; any faithful-reimplementation failure would change comparisons.
  • domain assumption OpenMMLab task heads and fine-tuning protocols are fair and do not favor particular backbones.
    Section 3.4 uses mmdetection, mmsegmentation, and mmpose consistently; however, the authors note in Appendix F that some heads (e.g., RTMPose) interact strongly with backbone quality, so head choice is not entirely neutral.
  • domain assumption ViT-Base is a sufficient backbone for the conclusions to generalize to larger foundation models.
    All pre-trained models use ViT-Base (Table 1); the paper does not test whether the RGB-D advantage persists at ViT-L or ViT-g scale.

pith-pipeline@v1.3.0-alltime-deepseek · 19208 in / 11472 out tokens · 132847 ms · 2026-08-03T07:49:18.464526+00:00 · methodology

0 comments
read the original abstract

Vision foundation models (VFMs) have emerged as powerful tools for surgical scene understanding. However, current approaches predominantly rely on unimodal RGB pre-training, overlooking the complex 3D geometry inherent to surgical environments. Although several architectures support multimodal or geometry-aware inputs in general computer vision, the benefits of incorporating depth information in surgical settings remain underexplored. We conduct a large-scale empirical study comparing eight ViT-based VFMs that differ in pre-training domain, learning objective, and input modality (RGB vs. RGB-D). For pre-training, we use a curated dataset of 1.4 million robotic surgical images paired with depth maps generated from an off-the-shelf network. We evaluate these models under both frozen-backbone and end-to-end fine-tuning protocols across eight surgical datasets spanning object detection, segmentation, depth estimation, and pose estimation. Our experiments yield several consistent findings. Models incorporating explicit geometric tokenization, such as MultiMAE, substantially outperform unimodal baselines across all tasks. Notably, geometric-aware pre-training enables remarkable data efficiency: models fine-tuned on just 25% of labeled data consistently surpass RGB-only models trained on the full dataset. Importantly, these gains require no architectural or runtime changes at inference; depth is used only during pre-training, making adoption straightforward. These findings suggest that multimodal pre-training offers a viable path towards building more capable surgical vision systems.

Figures

Figures reproduced from arXiv: 2601.18929 by Adam Schmidt, Anita Rau, Chinedu Nwoye, Jie Ying Wu, John J. Han, Muhammad Abdullah Jamal, Omid Mohareri.

Figure 1
Figure 1. Figure 1: An overview of our contribution. (a) Previous work in surgical VFMs typically use [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Samples from the pre-training dataset. We curate an in-house RGB-D dataset from [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of downstream task performance on eight surgical datasets. Solid inner bars [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Data efficiency analysis comparing model performance at 25%, 50%, 75%, and 100% [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Demonstration of a naive multimodal DINOv2 training scheme. During training, both [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Feature map visualizations via PCA of the multimodally and unimodally trained DI [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Reconstruction visualization for MultiMAE Bachmann et al. (2022). The model receives [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Reconstruction visualization for Mask3D (Hou et al., 2023). The model receives unmasked [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Reconstruction visualization for MAE (Hou et al., 2023). The model receives unmasked [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Samples of the original SCARED dataset (left column) and COLMAP-corrected [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Extreme data scarcity experiments with EndoVis18 for MultiMAE and MultiMAE (IN [PITH_FULL_IMAGE:figures/full_fig_p028_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative results on object detection datasets. GT is ground truth, IN is ImageNet-1k, [PITH_FULL_IMAGE:figures/full_fig_p030_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative results on segmentation datasets. GT is ground truth, IN is ImageNet-1k, [PITH_FULL_IMAGE:figures/full_fig_p031_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative results on depth estimation datasets. GT is ground truth, IN is ImageNet [PITH_FULL_IMAGE:figures/full_fig_p032_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Qualitative results on pose estimation datasets. GT is ground truth, IN is ImageNet-1k, [PITH_FULL_IMAGE:figures/full_fig_p033_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 12 linked inside Pith

  1. [5]

    Rethinking rgb-d fusion for semantic segmenta- tion in surgical datasets.arXiv preprint arXiv:2407.19714,

    Muhammad Abdullah Jamal and Omid Mohareri. Rethinking rgb-d fusion for semantic segmenta- tion in surgical datasets.arXiv preprint arXiv:2407.19714,

    Pith/arXiv arXiv

  2. [6]

    Scaling up self-supervised learning for improved surgical foundation models

    Tim JM Jaspers, Ronald LPD de Jong, Yiping Li, Carolus HJ Kusters, Franciscus HA Bakker, Romy C van Jaarsveld, Gino M Kuiper, Richard van Hillegersberg, Jelle P Ruurda, Willem M Brinkman, et al. Scaling up self-supervised learning for improved surgical foundation models. arXiv preprint arXiv:2501.09436,

    arXiv

  3. [7]

    Rtmpose: Real-time multi-person pose estimation based on mmpose.arXiv preprint arXiv:2303.07399,

    Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen. Rtmpose: Real-time multi-person pose estimation based on mmpose.arXiv preprint arXiv:2303.07399,

    Pith/arXiv arXiv

  4. [8]

    The multimodal DINOv2, despite its simple data augmentation strategy, extracts similar representations between the two modalities as opposed to the unimodal’s features

    20 RGB-Depth Input DINOv2-RGBD DINOv2-RGB Figure 6: Feature map visualizations via PCA of the multimodally and unimodally trained DI- NOv2. The multimodal DINOv2, despite its simple data augmentation strategy, extracts similar representations between the two modalities as opposed to the unimodal’s features. Input Prediction Ground Truth RGB Depth RGB Dept...

    2022

  5. [9]

    in Allan et al

    D Correcting the SCARED Dataset The original authors of the SCARED dataset clarify that the non-keyframe labels are inaccurate (see Section VI. in Allan et al. (2021)) due to calibration and kinematics error. However, only using keyframes is not a viable approach for training depth estimation models since there are only 35 labeled training samples. As a r...

    2021

  6. [10]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193,

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193,

    Pith/arXiv arXiv

  7. [11]

    Surglavi: Large-scale hierarchical dataset for surgical vision-language representation learning.arXiv preprint arXiv:2509.10555,

    Alejandra Perez, Chinedu Nwoye, Ramtin Raji Kermani, Omid Mohareri, and Muhammad Abdul- lah Jamal. Surglavi: Large-scale hierarchical dataset for surgical vision-language representation learning.arXiv preprint arXiv:2509.10555,

    arXiv

  8. [13]

    General surgery vi- sion transformer: A video pre-trained foundation model for general surgery.arXiv preprint arXiv:2403.05949,

    Samuel Schmidgall, Ji Woong Kim, Jeffrey Jopling, and Axel Krieger. General surgery vi- sion transformer: A video pre-trained foundation model for general surgery.arXiv preprint arXiv:2403.05949,

    Pith/arXiv arXiv

  9. [14]

    Aaron Van Den Oord, Oriol Vinyals, et al

    doi: 10.1109/TMI.2016.2593957. Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advances in neural information processing systems, 30,

    arXiv 2016

  10. [15]

    Surgpose: a dataset for articulated robotic surgical tool pose estimation and tracking.arXiv preprint arXiv:2502.11534,

    Zijian Wu, Adam Schmidt, Randy Moore, Haoying Zhou, Alexandre Banks, Peter Kazanzides, and Septimiu E Salcudean. Surgpose: a dataset for articulated robotic surgical tool pose estimation and tracking.arXiv preprint arXiv:2502.11534,

    Pith/arXiv arXiv

  11. [17]

    Dino in the room: Leveraging 2d foundation models for 3d segmentation

    Karim Abou Zeid, Kadir Yilmaz, Daan de Geus, Alexander Hermans, David Adrian, Timm Linder, and Bastian Leibe. Dino in the room: Leveraging 2d foundation models for 3d segmentation. arXiv preprint arXiv:2503.18944,

    arXiv

  12. [18]

    In contrast, DINOv2-RGB is inconsistent in its visualized features between the two modalities since it has only seen one modality during training

    As can be seen, the feature maps of the DINOv2-RGBD model are identical regardless of modality because their features were implicitly aligned during pre-training. In contrast, DINOv2-RGB is inconsistent in its visualized features between the two modalities since it has only seen one modality during training. C.2 MultiMAE Reconstruction Visualization We di...

    2022

  13. [21]

    head) Config Value Epochs 100 Optimizer AdamW Learning Rate1×10 −4 Weight Decay 0.05 Optimizer Momentumβ 1,β 2 = 0.9,0.95 Batch Size 64 Learning Rate Schedule Cosine Decay Warmup Epochs 10 (b) Hyperparameters for segmentation and depth estimation linear probing, which both use a single linear layer after the last feature map. Config Value Epochs 10 Optimi...

    2024

  14. [2004]

    Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916,

    15 Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks.arXiv preprint arXiv:2206.08916,

    Pith/arXiv arXiv

  15. [2016]

    Video dataset for surgical phase, keypoint, and instrument recognition in laparoscopic surgery (phakir).arXiv preprint arXiv:2511.06549,

    Tobias Rueckert, Raphaela Maerkl, David Rauber, Leonard Klausmann, Max Gutbrod, Daniel Rueckert, Hubertus Feussner, Dirk Wilhelm, and Christoph Palm. Video dataset for surgical phase, keypoint, and instrument recognition in laparoscopic surgery (phakir).arXiv preprint arXiv:2511.06549,

    arXiv

  16. [2018]

    Surgical robot transformer (srt): Imitation learning for surgical tasks.arXiv preprint arXiv:2407.12998,

    Ji Woong Kim, Tony Z Zhao, Samuel Schmidgall, Anton Deguet, Marin Kobilarov, Chelsea Finn, and Axel Krieger. Surgical robot transformer (srt): Imitation learning for surgical tasks.arXiv preprint arXiv:2407.12998,

    Pith/arXiv arXiv

  17. [2020]

    Stereo correspondence and recon- struction of endoscopic data challenge.arXiv preprint arXiv:2101.01133,

    Max Allan, Jonathan Mcleod, Congcong Wang, Jean Claude Rosenthal, Zhenglei Hu, Niklas Gard, Peter Eisert, Ke Xue Fu, Trevor Zeffiro, Wenyao Xia, et al. Stereo correspondence and recon- struction of endoscopic data challenge.arXiv preprint arXiv:2101.01133,

    Pith/arXiv arXiv

  18. [2022]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10371–10381, 2024a. 17 Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth a...

    Pith/arXiv arXiv

  19. [2023]

    Depth anything in medical images: A comparative study.arXiv preprint arXiv:2401.16600,

    John J Han, Ayberk Acar, Callahan Henry, and Jie Ying Wu. Depth anything in medical images: A comparative study.arXiv preprint arXiv:2401.16600,

    Pith/arXiv arXiv

  20. [2024]

    MMDetection: Open mmlab detection toolbox and benchmark.arXiv preprint arXiv:1906.07155,

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and b...

    Pith/arXiv arXiv 1906

  21. [2025]

    2018 robotic scene segmentation challenge.arXiv preprint arXiv:2001.11190,

    Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, et al. 2018 robotic scene segmentation challenge.arXiv preprint arXiv:2001.11190,

    Pith/arXiv arXiv 2018