Pith. sign in

REVIEW 4 major objections 4 minor 25 references

Adding monocular depth cues lets a lightweight Segment Anything Model surpass a strong RGB-only baseline while training on less than 0.1% of the original dataset.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:57 UTC pith:RLFBHUQZ

load-bearing objection Intriguing multi-click gains, but the headline 'data-efficient' claim is unsupported because the RGB tower almost certainly inherits full SA-1B pretraining and no same-data RGB-only baseline is run. the 4 major comments →

arxiv 2602.11804 v1 pith:RLFBHUQZ submitted 2026-02-12 cs.CV eess.IV

Efficient Segment Anything with Depth-Aware Fusion and Limited Training Data

Yiming Zhou , Xuenjie Xie , Panfeng Li , Albrecht Kunz , Ahmad Osman , Xavier Maldague This is my paper
classification cs.CV eess.IV
keywords Segment Anything Modeldepth-aware fusionmonocular depth priorsdata-efficient segmentationzero-shot segmentationinteractive segmentationEfficientViT-SAMRGB-D fusion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Depth-Aware EfficientViT-SAM is a lightweight Segment Anything variant that adds monocular depth as a second input stream, fused with RGB features by a simple learnable sum. The paper's central claim is that this depth prior, combined with a training recipe on just 11.2k images, lets the model surpass the RGB-only EfficientViT-SAM-L2 on zero-shot box-prompted and point-prompted segmentation, and even beat the original SAM and a larger efficient variant when users provide several clicks. If true, this would mean geometric depth cues can substitute for millions of extra training images in adapting a universal segmenter to practical, resource-limited settings. The work matters because it points to a cheap way to regain the accuracy of heavy SAM models without their data and compute appetite.

Core claim

The authors claim that augmenting EfficientViT-SAM with a parallel depth encoder—fed by monocular depth maps and fused as F_fuse = F_rgb + alpha*F_dep—produces higher zero-shot interactive segmentation accuracy than the RGB-only model while training on only 11.2k images for four epochs. In point-prompted evaluation with three or five clicks, the approach outperforms EfficientViT-SAM-XL1 and SAM-ViT-H on COCO and LVIS; box-prompted results consistently improve on small objects. The depth branch roughly doubles parameters and computation but remains far lighter than SAM-ViT-H.

What carries the argument

The mechanism is a second encoder with the same architecture as the RGB encoder, taking monocular depth maps (replicated to three channels) and outputting embeddings that are added to RGB embeddings with a learnable scaling factor alpha. A five-term loss (mask, dice, IoU regression, direct intermediate supervision, and a boundary-focused auxiliary term) trains first the depth encoder alone, then the whole model end-to-end. The depth signal supplies geometric structure that RGB textures lack, and this is what carries the boundary and small-object improvements.

Load-bearing premise

The comparison assumes the EfficientViT-SAM backbone is already pretrained on the full 11M-image dataset, so the 11.2k images are a fine-tuning budget; if the backbone were trained from scratch on those 11.2k images, the reported gains would likely not appear.

What would settle it

Train an RGB-only EfficientViT-SAM on the same 11.2k images with the same schedule and loss; if it matches the depth-aware model's scores, the depth branch is not carrying the improvement. Likewise, re-running with the backbone randomly initialized would show whether the system is truly learning from 11.2k images or merely relying on pretrained weights.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If depth priors supply the geometric structure that RGB fine-tuning lacks, SAM variants can be adapted to new domains with orders of magnitude fewer annotated images.
  • The consistent gains on small and medium objects imply depth fusion is most valuable where boundaries are ambiguous, so boundary-sensitive downstream tasks could use the same recipe.
  • Point-prompted gains that grow with click count suggest depth-aware fusion is especially effective in interactive segmentation workflows.
  • Because training completes in a few hours on commodity GPUs, the approach lowers the compute barrier for customizing universal segmentation models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's data-efficiency claim is sensitive to initialization: the setup appears to start from a pretrained EfficientViT-SAM-L2 and fine-tune with the depth branch, making the 11.2k-image budget a fine-tuning cost rather than a from-scratch training cost; a true from-scratch comparison would clarify this.
  • A natural test is to train an RGB-only EfficientViT-SAM-L2 on the same 11.2k images with the same schedule and loss; if the RGB-only model matches or exceeds these scores, depth fusion is not the active ingredient.
  • Extending beyond the paper, the same additive-fusion-with-alpha recipe could apply to other tasks where monocular depth is a cheap prior, such as video segmentation or object detection, provided the depth estimator generalizes to the target domain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Depth-Aware EfficientViT-SAM, a variant of EfficientViT-SAM-L2 that fuses monocular depth maps (from a pretrained DepthAnything estimator) with RGB features through a separate depth encoder, using simple additive fusion with a learned scaling coefficient. The framework is trained end-to-end for four epochs on an 11.2k-image subset of SA-1B and evaluated zero-shot on COCO and LVIS in both box-prompted and point-prompted settings. The central claim is that this limited-data training, aided by depth priors, yields a model that surpasses EfficientViT-SAM and even larger SAM variants in accuracy while remaining lightweight. The paper also reports runtime, parameter counts, and qualitative examples. The main evidence consists of comparisons in Tables 1–3, but the most striking point-prompted gains appear only at 3 and 5 clicks, and the comparison protocol does not isolate the effect of depth from the effect of additional fine-tuning.

Significance. If substantiated, the claimed result would be practically interesting: it would show that monocular depth priors can substantially reduce the data required to adapt an efficient SAM to interactive segmentation, and the proposed architecture is simple and easy to build on. The paper also reports a useful runtime/parameter analysis and targets a real deployment concern. However, the current version does not provide enough evidence to support the central claim. The key missing piece is an RGB-only baseline trained on the same 11.2k images with the same loss and protocol; without it, the observed improvements at 3/5 clicks cannot be attributed to depth. In addition, the unqualified abstract statement of superiority is contradicted by the 1-click results in Table 2. These are correctness-level issues, not presentation issues. The paper is a reasonable empirical exploration, but its claims outrun its experiments.

major comments (4)
  1. [Abstract / §4.5, Table 2] The abstract states that the method 'achieves higher accuracy than EfficientViT-SAM' without qualification. Table 2 shows the opposite at 1 click: COCO 52.6 vs. 56.1 and LVIS 56.3 vs. 57.8 for our method versus EfficientViT-SAM-L2. The claim must be restricted to the 3/5-click settings, and the 1-click behavior needs discussion. As written, the central claim is internally contradicted by the paper's own main point-prompted table.
  2. [§4.1 / §4.4, Tables 2 and 3] The manuscript never discloses whether the RGB encoder and SAM head are initialized from the released EfficientViT-SAM-L2 weights (pre-trained on the full SA-1B) or trained from scratch. Section 4.1 only says 'we build on EfficientViT-SAM-L2' and then 'the entire model is trained end-to-end.' If the RGB tower retains full-dataset pretrained weights, the 11.2k images are a fine-tuning budget, not a training-from-scratch budget. The comparison to EfficientViT-SAM-L2 then conflates two variables: additional fine-tuning on 11.2k images and the added depth branch. The paper needs an RGB-only baseline fine-tuned on the same 11.2k images with the same loss, optimizer, and click protocol, in order to isolate the contribution of depth. Without this baseline, the 'depth cues enable data-efficient training' conclusion is unsupported.
  3. [§4.4, Table 1 / Table 3] The claimed gains in box-prompted settings are very small and are reported without error bars or multiple seeds. For example, on COCO-VitDet the gain over EfficientViT-SAM-L2 is 0.3 mAP (44.3 vs. 44.0), while on COCO-YOLOv8 our method is slightly worse (41.8 vs. 41.9). In the ground-truth box setting (Table 3), the small-object gain is 2.5 points on COCO but the large-object gain is negative. With no statistical testing or variance estimates, 'consistent improvements' cannot be distinguished from noise. The ablation mentioned in §4.4 ('removing depth fusion reduces accuracy by about 1% on small objects') is also reported only as a single prose sentence with no table, no setting, and no numbers. A full ablation table separating RGB-only, depth-fusion, and each auxiliary loss term is required.
  4. [§4.2 / Abstract] The 'efficient' and 'lightweight' framing should be calibrated. The proposed model doubles parameters (61.3M→118.7M), doubles MACs (69G→137G), and halves throughput (62.8→31.9 images/s) relative to the EfficientViT-SAM-L2 baseline. It is lightweight only in comparison to SAM-ViT-H (>600M, ~3000G). Since the paper's stated goal is to improve EfficientViT-SAM, the efficiency claim should be quantified against that baseline more carefully, and the computational cost of the monocular depth estimator should be reported. As written, the efficiency contribution is overstated.
minor comments (4)
  1. [§4.3 / Fig. 2] The qualitative examples are not quantified and no failure cases are shown. Since the paper claims sharper boundaries and better small-object handling, a few quantitative boundary/F-score examples or failure-mode discussion would strengthen the presentation.
  2. [§3.1 / Eq. (1)] The text says 'depth maps are generated by DepthAnything' and 'depth images are replicated across three channels,' but the depth pre-processing (normalization, resizing, whether the same image transform is applied) is not specified. This detail is needed for reproducibility.
  3. [§4.1] Training details are missing some standard items: learning rate schedule, warm-up, weight decay, and whether the RGB encoder is frozen in stage 1. The stage-1 stage-2 description is also ambiguous about which components are trained in each stage.
  4. [References] References [8]–[11] are about 3D reconstruction and SLAM and are only loosely related to the segmentation argument. The authors may wish to replace them with more directly relevant RGB-D segmentation or efficient SAM works, or justify the connection more explicitly.

Circularity Check

0 steps flagged

No significant circularity: the reported results are measured on held-out benchmarks and the method is an empirical fine-tuning framework, not a derivation that reduces to its inputs.

full rationale

The paper makes an empirical claim: training Depth-Aware EfficientViT-SAM on 11.2k images yields competitive or better zero-shot segmentation than EfficientViT-SAM. The evidence is measured on COCO/LVIS with box prompts, point prompts, and external detectors (Tables 1–3). No equation or parameter in the method is defined in terms of the benchmark outcome, and no fitted quantity is renamed as a prediction. The loss function (Eq. 3) supervises masks directly and does not encode the test-set result. The cited related works by overlapping authors ([9]–[11]) are background examples of SAM-based 3D/SLAM systems and are not load-bearing for the central architecture or results. The acknowledged concern that the RGB encoder may inherit full SA-1B pretrained weights and that no RGB-only limited-data baseline is provided is a fairness/attribution issue about what causes the gains, not circularity: the comparison is still an external measurement, not a construction. Therefore no circular step can be exhibited, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The central empirical claims rely on a learned fusion coefficient, hand-set loss weights, and two pretrained components (EfficientViT-SAM and DepthAnything) whose pretraining data are far larger than the 11.2k fine-tuning set. None of these are independently justified inside the paper, and no sensitivity analysis is given. No invented physical or mathematical entities are introduced.

free parameters (6)
  • learnable fusion scale α = not reported
    Eq. 1 controls the strength of the depth branch; learned during training with no final value or sensitivity analysis.
  • λ_mask = 20
    Hand-chosen loss weight in Eq. 2 and Eq. 3.
  • λ_dice = 1
    Hand-chosen loss weight in Eq. 2 and Eq. 3.
  • λ_iou = 1.0
    Hand-chosen loss weight in Eq. 3.
  • λ_direct = 0.5
    Hand-chosen auxiliary direct prediction loss weight in Eq. 3.
  • λ_aux = 0.2
    Hand-chosen auxiliary boundary loss weight in Eq. 3.
axioms (4)
  • domain assumption Pretrained monocular depth estimator (DepthAnything) yields depth maps with useful geometric cues for segmenting COCO/LVIS objects.
    Assumed throughout Section 4; no analysis of depth quality or failure cases is provided.
  • ad hoc to paper Replicating the depth map into 3 channels and passing it through the same architecture as RGB is an adequate representation of depth.
    Section 3.1; no architectural search or comparison to alternative depth encodings.
  • domain assumption The EfficientViT-SAM backbone is initialized from its original full SA-1B pretrained weights (implicit).
    The paper says only 'we build on EfficientViT-SAM-L2' (Section 4.1); without this assumption the data-efficiency claim is unsupported.
  • domain assumption The 11.2k-image SA-1B sample is a sufficient training distribution for zero-shot transfer to COCO/LVIS.
    Section 4.1; no analysis of sample representativeness is given.

pith-pipeline@v1.3.0-alltime-deepseek · 5519 in / 16057 out tokens · 141158 ms · 2026-08-02T23:57:24.295914+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Efficient Segment Anything with Depth-Aware Fusion and Limited Training Data." pith.science (2026). https://pith.science/paper/RLFBHUQZ

@misc{pith2026260211804,
  author       = {Pith},
  title        = {Pith review of: Efficient Segment Anything with Depth-Aware Fusion and Limited Training Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RLFBHUQZ}},
  note         = {Machine review of arXiv:2602.11804}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Segment Anything Models (SAM) achieve impressive universal segmentation performance but require massive datasets (e.g., 11M images) and rely solely on RGB inputs. Recent efficient variants reduce computation but still depend on large-scale training. We propose a lightweight RGB-D fusion framework that augments EfficientViT-SAM with monocular depth priors. Depth maps are generated with a pretrained estimator and fused mid-level with RGB features through a dedicated depth encoder. Trained on only 11.2k samples (less than 0.1\% of SA-1B), our method achieves higher accuracy than EfficientViT-SAM, showing that depth cues provide strong geometric priors for segmentation.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 6 linked inside Pith

  1. [1]

    Its success is largely attributed to large-scale pretraining on SA-1B [1] dataset which contains 11M images and 1B masks

    INTRODUCTION The Segment Anything Model (SAM) [1] has recently demon- strated strong zero-shot generalization in tasks such as AR/VR, medical imaging, and interactive editing. Its success is largely attributed to large-scale pretraining on SA-1B [1] dataset which contains 11M images and 1B masks. However, SAM’s reliance on massive datasets and heavy trans...

  2. [2]

    RELATED WORK 2.1. Efficient Segment Anything Models The Segment Anything Model (SAM) [1] is a major advance in universal segmentation, achieving strong zero-shot perfor- mance with minimal user input. It has been applied to diverse tasks such as object tracking [7], medical imaging, and acces- sibility, and more recently extended to 3D perception tasks li...

    Pith/arXiv arXiv 2026

  3. [3]

    EfficientViT [4] further accelerates SAM with a lightweight vision transformer

    replaces the transformer with a convolutional encoder, while MobileSAM [2] applies distillation for edge devices. EfficientViT [4] further accelerates SAM with a lightweight vision transformer. Building on this line of work, we adopt EfficientViT as the backbone and integrate depth maps to guide boundary refinement, aiming for both efficiency and improved...

  4. [4]

    We keep the original SAM prompt encoder and mask decoder unchanged, while replac- ing SAM’s heavy image encoder with EfficientViT to achieve efficiency

    METHOD We proposeDepth-Aware EfficientViT-SAM, which aug- ments EfficientViT-SAM with a depth-aware encoder and a simple fusion mechanism. We keep the original SAM prompt encoder and mask decoder unchanged, while replac- ing SAM’s heavy image encoder with EfficientViT to achieve efficiency. 3.1. Model Architecture Figure 1 illustrates the framework. The R...

  5. [5]

    Experimental Settings We build on EfficientViT-SAM-L2, which offers the best trade-off between accuracy and computation cost, and extend it with our depth-aware framework

    EXPERIMENTS 4.1. Experimental Settings We build on EfficientViT-SAM-L2, which offers the best trade-off between accuracy and computation cost, and extend it with our depth-aware framework. Training is conducted in two stages. In the first stage, the depth encoder is randomly initialized and trained independently for 2 epochs on the SA- 1B example dataset ...

  6. [6]

    CONCLUSION In this work, we introduced Depth-Aware EfficientViT-SAM, a lightweight framework that integrates monocular depth with EfficientViT-SAM through a dedicated encoder. This fusion improves segmentation quality, especially on object boundaries and small objects, and in point-prompted settings surpasses larger variants such as EfficientViT-SAM-XL1 a...

  7. [7]

    Segment anything,

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al., “Segment anything,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015–4026

    2023

  8. [8]

    Faster segment anything: Towards lightweight sam for mobile applications,

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong, “Faster segment anything: Towards lightweight sam for mobile applications,”arXiv preprint arXiv:2306.14289, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Fast segment anything,

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang, “Fast segment anything,”arXiv preprint arXiv:2306.12156, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Efficientvit: Mem- ory efficient vision transformer with cascaded group at- tention,

    Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, and Yixuan Yuan, “Efficientvit: Mem- ory efficient vision transformer with cascaded group at- tention,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 14420–14430

    2023

  11. [11]

    Efficientvit- sam: Accelerated segment anything model without per- formance loss,

    Zhuoyang Zhang, Han Cai, and Song Han, “Efficientvit- sam: Accelerated segment anything model without per- formance loss,” inProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2024, pp. 7859–7863

    2024

  12. [12]

    Depth anything: Unleashing the power of large-scale unlabeled data,

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 10371–10381

    2024

  13. [13]

    Segment and track anything,

    Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang, “Segment and track anything,”arXiv preprint arXiv:2305.06558, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Vggt: Visual geometry grounded transformer,

    Jianyuan Wang, Minghao Chen, Nikita Karaev, An- drea Vedaldi, Christian Rupprecht, and David Novotny, “Vggt: Visual geometry grounded transformer,” inPro- ceedings of the Computer Vision and Pattern Recogni- tion Conference, 2025, pp. 5294–5306

    2025

  15. [15]

    Evaluating mod- ern approaches in 3d scene reconstruction: Nerf vs gaussian-based methods,

    Yiming Zhou, Zixuan Zeng, Andi Chen, Xiaofan Zhou, Haowei Ni, Shiyao Zhang, Panfeng Li, Liangxi Liu, Mengyao Zheng, and Xupeng Chen, “Evaluating mod- ern approaches in 3d scene reconstruction: Nerf vs gaussian-based methods,” in2024 6th International Conference on Data-driven Optimization of Complex Systems (DOCS), 2024, pp. 926–931

    2024

  16. [16]

    Ddn-slam: Real time dense dynamic neural implicit slam,

    Mingrui Li, Zhetao Guo, Tianchen Deng, Yiming Zhou, Yuxiang Ren, and Hongyu Wang, “Ddn-slam: Real time dense dynamic neural implicit slam,”IEEE Robotics and Automation Letters, vol. 10, no. 5, pp. 4300–4307, 2025

    2025

  17. [17]

    Dy3dgs-slam: Monocular 3d gaussian splatting slam for dynamic environments,

    Mingrui Li, Yiming Zhou, Hongxing Zhou, Xinggang Hu, Florian Roemer, Hongyu Wang, and Ahmad Os- man, “Dy3dgs-slam: Monocular 3d gaussian splatting slam for dynamic environments,” in2025 IEEE Interna- tional Conference on Robotics and Automation (ICRA), 2025, pp. 14572–14578

    2025

  18. [18]

    Geminifusion: Efficient pixel-wise multimodal fusion for vision transformer,

    Ding Jia, Jianyuan Guo, Kai Han, Han Wu, Chao Zhang, Chang Xu, and Xinghao Chen, “Geminifusion: Efficient pixel-wise multimodal fusion for vision transformer,” arXiv preprint arXiv:2406.01210, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Dformer: Rethink- ing rgbd representation learning for semantic segmen- tation,

    Bowen Yin, Xuying Zhang, Zhongyu Li, Li Liu, Ming- Ming Cheng, and Qibin Hou, “Dformer: Rethink- ing rgbd representation learning for semantic segmen- tation,”arXiv preprint arXiv:2309.09668, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Dformerv2: Geometry self-attention for rgbd semantic segmentation,

    Bo-Wen Yin, Jiao-Long Cao, Ming-Ming Cheng, and Qibin Hou, “Dformerv2: Geometry self-attention for rgbd semantic segmentation,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 19345–19355

    2025

  21. [21]

    Microsoft coco: Common objects in context,

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick, “Microsoft coco: Common objects in context,” inEuropean conference on computer vision. Springer, 2014, pp. 740–755

    2014

  22. [22]

    Lvis: A dataset for large vocabulary instance segmentation,

    Agrim Gupta, Piotr Dollar, and Ross Girshick, “Lvis: A dataset for large vocabulary instance segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 5356–5364

    2019

  23. [23]

    Exploring plain vision transformer backbones for object detection,

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He, “Exploring plain vision transformer backbones for object detection,” inEuropean conference on computer vision. Springer, 2022, pp. 280–296

    2022

  24. [24]

    Uav-yolov8: A small-object- detection model based on improved yolov8 for uav aerial photography scenarios,

    Gang Wang, Yanfei Chen, Pei An, Hanyu Hong, Jinghu Hu, and Tiange Huang, “Uav-yolov8: A small-object- detection model based on improved yolov8 for uav aerial photography scenarios,”Sensors, vol. 23, no. 16, pp. 7190, 2023

    2023

  25. [25]

    Grounding dino: Marrying dino with grounded pre-training for open-set object de- tection,

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al., “Grounding dino: Marrying dino with grounded pre-training for open-set object de- tection,” inEuropean conference on computer vision. Springer, 2024, pp. 38–55

    2024