{"id":"8edd4972-b70e-4e4c-b1ce-45ac801c1bee","arxiv_id":"2601.18929","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RGB-D pre-training with explicit cross-modal objectives (MultiMAE) improves surgical detection, segmentation, pose, and depth estimation over RGB-only pre-training, with gains persisting when fine-tuned on 25% of labeled data.","lead":"This paper tests whether adding depth maps during pre-training makes surgical vision models better, using 1.4 million robot-surgery images with computer-generated depth. Across eight benchmarks, the depth-pretrained MultiMAE model generally beats RGB-only models, often needing only a quarter of the labeled data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Depth attribution is confounded: MultiMAE differs from RGB baselines in objective, tokenization, and training recipe, not just in input modality; the matched DINOv2-RGBD ablation shows little depth benefit.","rationale":"The reader's weakest assumption—that the comparisons are controlled ablations isolating depth—is also the most load-bearing concern. The paper's headline conclusion and data-efficiency recommendation depend on depth being the causal variable. The current design does not isolate depth: MultiMAE is the only high-performing model, and it differs from RGB baselines in objective, architecture, and training details. The only depth-matched comparison (DINOv2-RGBD vs DINOv2-RGB) shows weak or negative effects, which actually undermines the 'depth alone' narrative. The internal inconsistencies between Table 10 and Table 11 (e.g., CholecTrack20 100% MAE 30.1 vs 29.6; SurgPose MultiMAE 69.7 vs 67.6) are real reproducibility concerns but are secondary to the causal attribution problem. The proposed RGB-RGB MultiMAE control would directly settle whether the geometric content of depth is necessary. Since the reader already assigned CONDITIONAL largely on this basis, my concern does not move the verdict; it reinforces the need for the stated condition.","tokens_in":19545,"tokens_out":6461,"duration_ms":78184,"concrete_test":"Pre-train a MultiMAE variant on the same in-house DV dataset with two input streams both being the RGB image (or RGB plus a color-jittered/grayscale copy), maintaining the identical cross-modal reconstruction objective, tokenizer design, 400 epochs, batch size, and learning rate as the RGB-D MultiMAE. Then fine-tune this RGB-RGB MultiMAE on the same downstream tasks (e.g., CholecTrack20, EndoVis18, SurgPose, DV depth). If it matches or exceeds the RGB-D MultiMAE, the reported gains are not attributable to depth; if it performs substantially worse, the geometric signal is implicated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that geometric depth, rather than other training differences, drives the large downstream gains. But the strongest model, MultiMAE, differs from the RGB-only baselines in at least three coupled ways: (1) input modality (RGB+depth vs RGB only), (2) pretext objective (joint RGB+depth reconstruction vs RGB-only reconstruction), and (3) architecture/tokenization (shared multimodal encoder with modality-specific tokens). Table 3 shows additional training-recipe differences: MultiMAE uses batch size 3072 and LR 1e-4, while MAE uses batch size 5120 and LR 1e-5; DINOv2 uses a different objective and only 100 epochs. The paper's own matched comparison, DINOv2-RGBD vs DINOv2-RGB (Finding 5), yields only a 3.08% average gain and a degradation on SurgPose. Thus the large MultiMAE advantage may stem from the multi-task reconstruction objective or the shared tokenizer, not from the geometric content of depth. The abstract's causal phrasing—'geometric-aware pre-training enables remarkable data efficiency'—overstates what this experimental matrix can establish, because no RGB-only MultiMAE control is provided. The DV depth benchmark is also partly circular (both pre-training and evaluation use FoundationStereo pseudo-depth), though SCARED provides an external check; the more fundamental issue remains the missing ablation of the depth signal itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a large-scale empirical study of RGB-D pre-training for surgical vision. The authors pre-train eight ViT-B models on 1.4M da Vinci frames paired with FoundationStereo pseudo-depth, then evaluate frozen-backbone and end-to-end fine-tuning transfer on eight datasets spanning object detection, semantic segmentation, depth estimation, and pose estimation. The central claims are (i) that models with explicit geometric tokenization, especially MultiMAE, substantially outperform RGB-only baselines, and (ii) that geometry-aware pre-training yields 'remarkable data efficiency,' with models fine-tuned on 25% of labeled data consistently surpassing RGB-only models trained on the full dataset. The paper also introduces a naive DINOv2-RGBD baseline to separate architectural from data effects and releases a COLMAP-corrected version of the SCARED depth benchmark.","tokens_in":19841,"tokens_out":5183,"duration_ms":60661,"significance":"If the central causal claim were established, the paper would have clear practical value: it would show that depth-only-at-pre-training improves downstream spatial tasks without inference-time cost, and that annotation-scarce surgical settings could benefit substantially. The study has genuine strengths: it is large-scale; it includes an external structured-light benchmark (SCARED) in addition to pseudo-labeled in-domain depth; it ablates task-head architecture (Table 8) and pseudo-depth source (Table 7); and the authors promise release of the corrected SCARED data. However, the paper's central attribution of the gains to the geometric content of depth is confounded, and the headline 'consistently surpass' claim is contradicted by the paper's own Table 11a. The DV depth benchmark is partly circular because both pre-training and evaluation use FoundationStereo pseudo-depth. These issues affect the paper's main conclusions, so major revision is required.","major_comments":[{"comment":"The causal attribution of the gains to depth is not supported by the experimental design. MultiMAE differs from the RGB-only baselines not only in input modality but also in pretext objective (joint RGB+depth reconstruction vs. RGB reconstruction), tokenization, and training recipe. Table 3 shows different batch sizes and learning rates (MultiMAE: batch 3072, LR 1e-4; MAE: batch 5120, LR 1e-5), and the models also differ in epochs and initialization. No RGB-only MultiMAE control is provided. The paper's most controlled comparison, DINOv2-RGBD vs. DINOv2-RGB (Finding 5), shows only a 3.08% average gain and a degradation on SurgPose (Table 10: 49.4 vs. 51.4 mAP). This is the paper's own evidence that depth alone does not explain the large MultiMAE improvements. The abstract's phrasing that 'geometric-aware pre-training' drives the results overstates what this matrix can establish. Please a","section":"§3.2, Table 3 and Table 10"},{"comment":"The abstract and Finding 2 claim that models fine-tuned on 25% of labeled data 'consistently surpass' RGB-only models trained on the full dataset. This is contradicted by Table 11a (CholecTrack20): MultiMAE (Surg) at 25% achieves 26.7 mAP, while MAE (Surg) at 100% achieves 29.6 mAP. Thus the claim holds on 3 of the 4 reported tasks but not on object detection. The phrase 'consistently' should be replaced with a qualified statement, e.g., 'on most tasks,' and the exception should be reported in the abstract and in Finding 2. Figure 4 should also be annotated so the reader can see the exception rather than inferring a universal trend.","section":"§4, Finding 2 and Table 11a"},{"comment":"The DV depth-estimation benchmark is partly circular. Pre-training depth maps are generated by FoundationStereo (Section 3.1), and the DV downstream depth labels are also generated by FoundationStereo (Section 3.3). Gains on this benchmark therefore partly reflect learning the generator's bias rather than general geometric understanding. The external SCARED benchmark partially mitigates this, and Table 10 shows MultiMAE also improves there (0.624 vs. 0.526 for MAE), but the effect is smaller and the ranking is not identical. The paper should (a) explicitly acknowledge this circularity in Section 4.1, which currently mentions only pseudo-depth errors, and (b) present SCARED as the primary external depth evidence, or additionally evaluate on depth labels from a different generator (e.g., Depth Anything) for the DV split.","section":"§3.1 and §3.3, DV depth benchmark"},{"comment":"The data-efficiency experiments report only means over three seeds, with no error bars or standard deviations. Several central comparisons rest on differences that could be within seed noise, and the 'consistently' language depends on these rankings. Please report per-seed results or standard deviations, and ideally a paired significance test, for the 25%/50%/75% comparisons. This is particularly important because the matched DINOv2-RGBD comparison (Finding 5) shows only a small 3.08% average gain, and the reader cannot assess whether that gain is robust.","section":"§4, Fig. 4 and Table 11"}],"minor_comments":[{"comment":"The naming is inconsistent: 'MAE (Surg)' in Table 11 refers to the same model called 'MAE' in Tables 1 and 10. Similarly, Figure 4 uses 'MAE' with a caption mentioning 'MAE (DV)'. Please unify notation throughout.","section":"Table 11 vs. Table 1/10"},{"comment":"References 'Yang et al. 2024a' and 'Yang et al. 2024b' appear to refer to the same Depth Anything paper with identical titles. Please remove the duplicate or correct the citation.","section":"References"},{"comment":"The depth maps are 'normalized to [0,1] during pre-training,' but it is unclear whether this normalization is per-image or dataset-wide. This detail affects the pretext task and should be stated precisely.","section":"§3.1"},{"comment":"The legend and caption describe 'solid inner bars' and 'hatched outer bars,' but the figure is small and the hatching is not visible in the reproduction. Consider a larger figure or a different visual encoding.","section":"Figure 3"},{"comment":"The sentence 'The number of visible patches is set constant throughout training' is truncated or incomplete. Please complete the description of the masking procedure.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical resource, but the abstract and conclusion claim more than the controlled comparisons support. The lack of an RGB-only MultiMAE control is the key gap; if the authors can provide that run, or alternatively reframe the conclusions as 'MultiMAE-style multimodal pre-training' rather than 'depth,' the paper could become publishable. The DV circularity and the missing error bars should also be addressed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, useful empirical study and probably the largest controlled comparison of RGB vs RGB-D pre-training for surgical vision. It deserves a serious referee, but the abstract overreaches and the tables need cleanup before I would trust the strong claims.\n\nWhat's new: a 1.4M-image surgical RGB-D pre-training dataset, eight ViT-B models, four tasks across eight datasets, frozen and end-to-end protocols, data-efficiency curves, and a new DINOv2-RGBD variant. They also correct the SCARED non-keyframe depths with COLMAP and plan to release that. The SCARED external benchmark is a real plus, and the Depth Anything vs FoundationStereo ablation is a good robustness check. Credit where due: MultiMAE's gains are large and consistent on most datasets, the frozen-backbone finding is practically valuable, and the head ablations suggest the benefit lives in the representations rather than one decoder.\n\nSoft spots, in order of importance:\n\n1. The abstract overclaims data efficiency. Table 11a shows MultiMAE at 25% data (26.7 mAP) does not beat MAE(Surg) at 100% (29.6) on CholecTrack20. On that benchmark the headline claim is false; on the other three it holds. Easy fix: say \"often\" or \"on most tasks.\"\n\n2. Depth attribution is confounded. MultiMAE differs from MAE in objective, tokenization, epochs, batch size, and learning rate, not just input modality. There is no RGB-only MultiMAE control, so the paper cannot isolate depth. The author's own matched comparison, DINOv2-RGBD vs DINOv2-RGB, shows only a 3.08% average gain and a degradation on SurgPose. The paper partially acknowledges this by emphasizing explicit geometric tokenization, but the abstract's \"geometric-aware pre-training enables\" language implies causal depth. The stress-test note is right on this.\n\n3. Internal inconsistencies. Table 10 and Table 11 disagree on MAE(Surg) CholecTrack20 at 100% (30.1 vs 29.6), and some data-efficiency curves are non-monotonic (e.g., MultiMAE on EndoVis18 at 75% is below 50%). With only three seeds and no error bars, that's hard to evaluate.\n\n4. DV depth evaluation is partly circular. Both pre-training and the DV depth benchmark use FoundationStereo pseudo-depth. SCARED helps, and the Depth Anything swap shows the result isn't tied to one generator, but the shared-error component remains unquantified.\n\n5. Reproducibility. The pre-training data is proprietary; they promise code and the corrected SCARED dataset, but not model weights. For a paper whose main deliverable is pre-trained backbones, that limits adoption.\n\nWho this is for: surgical vision researchers deciding whether to invest in depth pre-training. The practical recipe likely works, but the mechanism question stays open. I would send this to peer review with major-revision expectations, and my own verdict is conditional rather than reject.","headline":"Worth engaging: a genuinely useful benchmark with an over-claimed data-efficiency headline and a depth attribution that is real but partly confounded.","tokens_in":20401,"tokens_out":2714,"would_cite":true,"duration_ms":30732,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training surgical vision models on RGB-D pairs makes them beat RGB-only models on every tested task, including with far fewer labels.","keywords":["surgical vision foundation models","RGB-D pre-training","masked autoencoders","self-supervised learning","data efficiency","geometric representation learning","robotic surgery","multi-modal pre-training"],"falsifier":"Run MultiMAE pre-training on the same 1.4M surgical frames with the same objective, epochs, and batch size, but replace the depth maps with identically shaped random noise or a constant image. If downstream performance matches the RGB-D model, then the geometric signal itself is not responsible for the reported gains.","tokens_in":19383,"feed_emoji":"🩺","tokens_out":4131,"duration_ms":45626,"temperature":0.7,"pith_summary":"This paper claims that surgical vision foundation models learn better representations when RGB images are paired with depth maps during self-supervised pre-training, and that the gains are large enough to make label-hungry tasks far cheaper. The authors compare eight ViT-based models across eight datasets and report that MultiMAE—a masked autoencoder trained to reconstruct both RGB and depth from masked tokens—beats every RGB-only baseline on object detection, segmentation, pose estimation, and depth estimation. They further report that a MultiMAE fine-tuned on 25% of labeled data consistently outperforms RGB-only models fine-tuned on the full dataset, and that this comes at no inference cost because depth is used only in pre-training. The paper's main practical claim is that depth-aware pre-training, not model scaling or extra data collection, is the effective next step for surgical scene understanding.","feed_headline":"RGB-D pre-training beats RGB-only on every tested surgical task","feed_subtitle":"Models fine-tuned on just a quarter of labels still outscore RGB-only models trained on full datasets.","key_machinery":"The load-bearing object is MultiMAE's cross-modal masked reconstruction: RGB and depth images are tokenized into a shared latent space, patches are masked, and the model must reconstruct missing tokens of both modalities from the unmasked tokens of either. This forces the encoder to learn correspondences between appearance and geometry. The paper contrasts it with DINOv2-RGBD, which randomly swaps RGB and depth crops in a self-distillation objective without architectural change—that variant gains almost nothing—and with Mask3D, which reconstructs only depth and thereby favors dense pixel tasks over object-level semantics.","core_discovery":"The paper's central claim is that feeding surgical vision transformers paired RGB and depth images during self-supervised pre-training—specifically through architectures that tokenize depth explicitly and reconstruct masked tokens across modalities—yields representations that outperform RGB-only pre-training on object detection, semantic segmentation, pose estimation, and depth estimation across eight surgical datasets. The strongest version of the claim is quantitative: a MultiMAE model pre-trained on 1.4 million da Vinci frames with pseudo-depth, fine-tuned on 25% of downstream labels, consistently beats RGB-only baselines fine-tuned on 100%. The authors further claim that the effect is no","pith_inferences":["Reader's inference: the causal story would be strengthened by a controlled ablation in which the same architecture, objective, epoch budget, and batch size are trained with and without depth; as published, MultiMAE differs from MAE in all of these, so part of the margin could come from the multi-task reconstruction objective itself rather than the depth modality.","If depth is the active ingredient, the recipe should transfer to other endoscopic and interventional settings where stereo or monocular depth can be synthesized, such as colonoscopy, arthroscopy, or cataract surgery, where annotation is even scarcer.","A testable extension: corrupt the depth channel (e.g., random noise or permuted patches) and re-run MultiMAE pre-training; if performance stays high, the model is learning from reconstruction geometry rather than from actual depth.","The data-efficiency result suggests a practical rule for surgical AI teams: before collecting more annotations, try re-pre-training an RGB-D masked autoencoder on unlabeled video with pseudo-depth, since the 25%-beats-100% pattern held across four tasks."],"forward_implications":["Surgical foundation models can be built from existing monocular or stereo video: pseudo-depth from an off-the-shelf stereo matcher or monocular depth net suffices; no new sensor hardware is needed.","Downstream deployment is unchanged: models that used RGB now take RGB at inference with identical architecture and compute; the depth signal is baked into the weights.","Label-hungry surgical tasks become cheaper: fine-tuning on 25% of annotations can exceed RGB-only full-data performance, and the effect persists at extreme scarcity (2.5% on EndoVis18).","A single frozen geometry-aware backbone can serve multiple tasks, since frozen multimodal backbones often beat fully fine-tuned RGB-only models.","Geometry-aware pre-training partly compensates for narrow, homogeneous surgical pre-training data, narrowing the usual ImageNet-versus-domain gap."],"fun_headline_variants":["Depth pre-training boosts surgical AI on all tasks","Surgical AI sees better with depth pre-training","RGB-D pretraining outperforms RGB-only in surgery","25% data with depth beats 100% RGB-only in surgery","Depth-aware pretraining wins in surgical vision"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper attributes its gains to depth information, but the models being compared differ not only in modality but also in pre-training objective, epoch count, batch size, and architecture; if depth is not what drives the improvement, the central claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Depth pre-training boosts surgical AI on all tasks","Surgical AI sees better with depth pre-training","RGB-D pretraining outperforms RGB-only in surgery","25% data with depth beats 100% RGB-only in surgery","Depth-aware pretraining wins in surgical vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1649,"prompt_tokens":770,"completion_tokens":879,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":814}},"tokens_in":514,"tokens_out":879,"duration_ms":7530,"temperature":1.0,"reasoning_tokens":814,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T07:49:18.464526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MultiMAE pre-training on the same 1.4M surgical frames with the same objective, epochs, and batch size, but replace the depth maps with identically shaped random noise or a constant image. If downstream performance matches the RGB-D model, then the geometric signal itself is not responsible for the reported gains.","supporting_citations":[],"review_version":1}