REVIEW 3 major objections 6 minor 38 references
When SAM2 Meets Video Shadow and Mirror Detection
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A first-frame mask prompt lets SAM2 beat specialized video detectors
desk verdict SAM2 mask prompts track video shadows/mirrors well, point prompts fail—that's a useful, believable result—but the 'outperforms SOTA' claim is confounded by giving SAM2 the first-frame ground truth. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is SAM2's mask-prompted video segmentation: the user supplies a segmentation mask for the first frame, and the model's streaming memory propagates that mask through the rest of the video using a transformer-based architecture. The mask prompt matters because it bypasses SAM2's point-to-mask conversion, which the paper identifies as the failure point for shadow and mirror regions with complex textures. The comparison is carried by this prompt-conditioned propagation together with four standard metrics: MAE, F1-score, IoU, and BER.
What would settle it
Run a specialized video shadow detector (for example, CVSD or SSTINet) on the ViSha test set with the same first-frame ground-truth mask provided as initialization; if its F1-score and IoU match or exceed SAM2's mask-prompted scores, the paper's claim that SAM2 outperforms the state of the art collapses. Alternatively, feed SAM2 only automatically generated prompts, without first-frame ground-truth information, and check whether it still beats the specialized methods.
Extended reading notes
Core claim
On its own terms, the paper establishes that SAM2, initialized with the ground-truth mask of the first frame, achieves state-of-the-art results on video shadow detection (ViSha) and video mirror detection (VMD) without task-specific training. On ViSha, the mask-prompted SAM2 variants reach an F1-score of up to 0.878 and an IoU of up to 0.755, surpassing the previous best published method (F1 0.866, IoU 0.746) while tying its MAE of 0.017. On VMD, the same models reach an F1-score of up to 0.961 and an IoU of up to 0.884, well above the prior best (F1 0.869, IoU 0.725). Point prompts, with 5 to 50 positive and negative clicks sampled from the ground truth, produce markedly worse results, and the paper attributes this to SAM2's inaccurate point-to-mask conversion in the first frame. The paper also observes that mask-prompted accuracy declines on later frames, indicating a temporal-drift limitation.
Load-bearing premise
The comparison assumes that published state-of-the-art numbers from specialized methods, which were not given the first-frame ground-truth mask, are directly comparable to SAM2's scores, which used that mask as privileged input; if that assumption fails, the state-of-the-art conclusion weakens, though the mask-versus-point finding would remain.
Editorial extensions
If this is right
- If the claim is correct, SAM2 with a first-frame mask prompt becomes a zero-training baseline that future video shadow and mirror detectors must be compared against.
- The large gap between mask and point prompts implies that improving SAM2's point-to-mask conversion, or its automatic prompt generation, could unlock practical point-prompt performance.
- The observed decline in accuracy on later frames suggests that SAM2's streaming memory alone does not fully solve long-video propagation, motivating hybrid memory designs for rare-object tracking.
- The result extends across two different rare-object video tasks, suggesting that mask-prompted SAM2 generalizes beyond the common-object videos it was trained on.
Reading between the lines
- The headline comparison is not apples-to-apples: the published specialized methods were evaluated without the first-frame ground-truth mask, so SAM2's advantage partly reflects privileged test-set information; a fairer benchmark would supply competing methods with the same initial mask.
- A concrete boundary test is to run SAM2 with prompts generated automatically by an image-level shadow or mirror detector; the paper lists this as future work, and the outcome would show whether the mask-prompt advantage survives without human-annotated initialization.
- The point-prompt failure mode likely extends to other rare or highly textured video objects, meaning SAM2's practical utility in the wild depends more on the prompt interface than on the model weights alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates SAM2 on video shadow detection (ViSha) and video mirror detection (VMD), using either ground-truth point prompts or ground-truth mask prompts to initialize the first frame and then propagating masks to subsequent frames. The central empirical findings are that mask prompts yield strong results, while point prompts perform poorly and often collapse to degenerate masks. The paper compares these results against published state-of-the-art methods and concludes that SAM2 with mask prompts achieves state-of-the-art performance on both datasets. The code is released.
Significance. If the central claim were supported by an apples-to-apples comparison, the paper would provide a useful reference point for the community: a zero-shot, promptable video segmentation model as a strong baseline for shadow and mirror detection, and a clear warning about the brittleness of point prompting on these rare-object tasks. The paper explicitly ships code and is transparent about the prompt settings, which is creditworthy. However, the headline SOTA comparison is confounded by the oracle first-frame mask, so the main significance hinges on reframing the contribution as a prompt-propagation study rather than a detector comparison. The mask-versus-point finding is internally consistent and survives the confound, because both prompt types receive the same privileged information.
major comments (3)
- [Section 2.4, Tables 1 and 2] The claim that SAM2 'outperforms the state-of-the-art methods' is not a like-for-like comparison. In Section 2.3, mask prompts are defined as the ground-truth mask of the first frame, while the published baselines listed in Tables 1 and 2 (e.g., CVSD, SSTINet, MG-VMD) are fully automatic methods that must locate the shadow or mirror in the first frame without any annotation. The reported margins (e.g., F1 0.878 vs. 0.866 on ViSha; F1 0.961 vs. 0.869 on VMD) therefore measure an oracle-initialized propagation pipeline against an unassisted detector. To support the SOTA claim, the authors should either add a fully automatic SAM2 variant (e.g., with auto-generated prompts) and compare that, or explicitly reframe the conclusions as evaluating prompt-based propagation performance rather than detector-level SOTA.
- [Section 2.3, Tables 1 and 2] Point prompts are generated by randomly sampling N positive and N negative points from the ground-truth mask, but no seeds, number of trials, or error bars are reported. The point-prompt results are highly sensitive to the sampled positions, as evidenced by the erratic IoU values (e.g., SAM2-tiny on ViSha: IoU 0.474 at 5 points, 0.191 at 20, 0.0 at 40; SAM2-small on VMD: IoU 0.565 at 10, 0.59 at 20, 0.148 at 50). Without multiple random trials or a fixed shared seed, the reader cannot determine whether the reported point-prompt numbers are stable phenomena or artifacts of a particular random draw. Please report mean and standard deviation over at least three trials, or fix and disclose the seed.
- [Abstract] The abstract states 'we evaluate SAM2 on three distinct video segmentation tasks: Video Shadow Detection (VSD) and Video Mirror Detection (VMD),' but only two tasks and two datasets appear in the paper. This is internally inconsistent. Either a third task is missing from the manuscript, or the abstract should say 'two.'
minor comments (6)
- [Section 2.2] Typo: 'peformance' should be 'performance.'
- [Tables 1 and 2 captions] Typos: 'postive' should be 'positive,' and the phrase 'When point mask is adopted' is ambiguous; it should be 'When point prompts are adopted.'
- [Figures 1, 2, 4, 5 captions] Typos: 'Visha' should be 'ViSha,' and 'groud truth' should be 'ground truth' in multiple captions. Figure 4's caption says 'predicted shadow points' but the figure shows predicted masks; the wording should be corrected.
- [Figure 3 caption] Typo: 'organe areas' should be 'orange areas.'
- [References] References [34] and [38] appear incomplete: [34] lacks a venue and year, and [38] lacks a year or venue. Please provide complete bibliographic information.
- [Section 2.4] The sentence 'SAM2 outperforms the state-of-the-art methods in terms of MAE, F1-score, and IoU, except for the BER metric' is confusing because BER is not a member of the preceding list; it would be clearer to say 'SAM2 achieves the best MAE, F1, and IoU, but not the best BER.'
Circularity Check
No circularity: SAM2 is a frozen external model, the evaluation is extrinsic and unfitted, and the self-citations are contextual; the oracle first-frame prompt is a fairness confound, not a circular step.
full rationale
The paper contains no derivation chain that reduces to its inputs. It evaluates a fixed, externally released model (SAM2) on two public datasets using standard metrics; no parameter is fitted to the test data and no reported number is defined in terms of another reported number. The use of the first-frame ground-truth mask as a prompt is an oracle-style evaluation protocol, which makes the comparison with fully automatic SOTA methods unfair (a correctness-risk confound), but it does not make the subsequent-frame predictions circular: those masks are produced by SAM2's frozen weights and are not equal to the input by construction. The mask-prompt-versus-point-prompt result is an empirical observation, and although it is unsurprising that a mask prompt outperforms point prompts, the paper does not define 'mask prompt performance' in terms of 'point prompt performance' or vice versa. The only self-citations ([5], [12]) appear in the introduction as background on SAM adapters; they are not load-bearing for the paper's claims, no uniqueness theorem is invoked, and no ansatz is imported through them. Accordingly, no circular step meeting the quoted-evidence standard can be exhibited.
Assumptions & free parameters
assumptions (3)
- domain assumption SAM2 is treated as a fixed pretrained function with no fine-tuning.
- domain assumption The published SOTA metrics in Tables 1 and 2 are directly comparable to the author's runs.
- domain assumption Ground-truth first-frame masks and points sampled from them are valid prompts for evaluation.
Cite this review
Pith. "Pith review of When SAM2 Meets Video Shadow and Mirror Detection." pith.science (2026). https://pith.science/paper/RQZ4R74Z
@misc{pith2026241219293,
author = {Pith},
title = {Pith review of: When SAM2 Meets Video Shadow and Mirror Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/RQZ4R74Z}},
note = {Machine review of arXiv:2412.19293}
}
read the original abstract
As the successor to the Segment Anything Model (SAM), the Segment Anything Model 2 (SAM2) not only improves performance in image segmentation but also extends its capabilities to video segmentation. However, its effectiveness in segmenting rare objects that seldom appear in videos remains underexplored. In this study, we evaluate SAM2 on three distinct video segmentation tasks: Video Shadow Detection (VSD) and Video Mirror Detection (VMD). Specifically, we use ground truth point or mask prompts to initialize the first frame and then predict corresponding masks for subsequent frames. Experimental results show that SAM2's performance on these tasks is suboptimal, especially when point prompts are used, both quantitatively and qualitatively. Code is available at \url{https://github.com/LeipingJie/SAM2Video}
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023
arXiv 2023
-
[2]
T. Chen, L. Zhu, C. Ding, R. Cao, S. Zhang, Y . Wang, Z. Li, L. Sun, P. Mao, and Y . Zang, “Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more,” arXiv preprint arXiv:2304.09148, 2023
arXiv 2023
-
[3]
Can sam segment anything? when sam meets camouflaged object detection,
L. Tang, H. Xiao, and B. Li, “Can sam segment anything? when sam meets camouflaged object detection,” arXiv preprint arXiv:2304.04709 , 2023
arXiv 2023
-
[4]
Towards underwater camouflaged object tracking: An experimental evaluation of sam and sam 2,
C. Zhang, L. Liu, G. Huang, H. Wen, X. Zhou, and Y . Wang, “Towards underwater camouflaged object tracking: An experimental evaluation of sam and sam 2,” arXiv preprint arXiv:2409.16902, 2024
arXiv 2024
-
[5]
When SAM Meets Shadow Detection
L. Jie and H. Zhang, “When sam meets shadow detection,” arXiv preprint arXiv:2305.11513, 2023
work page Pith review arXiv 2023
-
[6]
Segment anything in medical images,
J. Ma, Y . He, F. Li, L. Han, C. You, and B. Wang, “Segment anything in medical images,” Nature Communications, vol. 15, no. 1, p. 654, 2024
2024
-
[7]
Segment anything model for medical image analysis: an experimental study,
M. A. Mazurowski, H. Dong, H. Gu, J. Yang, N. Konz, and Y . Zhang, “Segment anything model for medical image analysis: an experimental study,” Medical Image Analysis, vol. 89, p. 102918, 2023
work page 2023
-
[8]
Samrs: Scaling-up remote sensing segmentation dataset with segment anything model,
D. Wang, J. Zhang, B. Du, M. Xu, L. Liu, D. Tao, and L. Zhang, “Samrs: Scaling-up remote sensing segmentation dataset with segment anything model,” Advances in Neural Information Processing Systems , vol. 36, 2024
work page 2024
Show all 38 references
-
[9]
The segment anything model (sam) for remote sensing applications: From zero to one shot,
L. P. Osco, Q. Wu, E. L. de Lemos, W. N. Gonc ¸alves, A. P. M. Ramos, J. Li, and J. M. Junior, “The segment anything model (sam) for remote sensing applications: From zero to one shot,” International Journal of Applied Earth Observation and Geoinformation , vol. 124, p. 103540, 2023
2023
-
[10]
Towards a unified view of parameter-efficient transfer learning,
J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,” arXiv preprint arXiv:2110.04366, 2021
2021 arXiv
-
[11]
Lora: Low-rank adaptation of large language models,
E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021
2021 arXiv
-
[12]
Adaptershadow: Adapting segment anything model for shadow detection,
L. Jie and H. Zhang, “Adaptershadow: Adapting segment anything model for shadow detection,” arXiv preprint arXiv:2311.08891, 2023
2023 arXiv
-
[13]
Make seg- ment anything model perfect on shadow detection,
X.-D. Chen, W. Wu, W. Yang, H. Qin, X. Wu, and X. Mao, “Make seg- ment anything model perfect on shadow detection,” IEEE Transactions on Geoscience and Remote Sensing, 2023
2023
-
[14]
Medical sam adapter: Adapting segment anything model for medical image segmentation,
J. Wu, W. Ji, Y . Liu, H. Fu, M. Xu, Y . Xu, and Y . Jin, “Medical sam adapter: Adapting segment anything model for medical image segmentation,” arXiv preprint arXiv:2304.12620, 2023
2023 arXiv
-
[15]
3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable tumor segmentation,
S. Gong, Y . Zhong, W. Ma, J. Li, Z. Wang, J. Zhang, P.-A. Heng, and Q. Dou, “3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable tumor segmentation,” Medical Image Analysis , vol. 98, p. 103324, 2024
2024
-
[16]
Sam 2: Segment anything in images and videos,
N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R¨adle, C. Rolland, L. Gustafson et al., “Sam 2: Segment anything in images and videos,” arXiv preprint arXiv:2408.00714, 2024
2024 arXiv
-
[17]
Sam2-adapter: Evaluating & adapting segment anything 2 in downstream tasks: Camouflage, shadow, medical image segmentation, and more,
T. Chen, A. Lu, L. Zhu, C. Ding, C. Yu, D. Ji, Z. Li, L. Sun, P. Mao, and Y . Zang, “Sam2-adapter: Evaluating & adapting segment anything 2 in downstream tasks: Camouflage, shadow, medical image segmentation, and more,” arXiv preprint arXiv:2408.04579, 2024
2024 arXiv
-
[18]
Evaluating sam2’s role in camouflaged object detection: From sam to sam2,
L. Tang and B. Li, “Evaluating sam2’s role in camouflaged object detection: From sam to sam2,” arXiv preprint arXiv:2407.21596, 2024
2024 arXiv
-
[19]
Sam2-unet: Segment anything 2 makes strong encoder for natural and medical image segmentation,
X. Xiong, Z. Wu, S. Tan, W. Li, F. Tang, Y . Chen, S. Li, J. Ma, and G. Li, “Sam2-unet: Segment anything 2 makes strong encoder for natural and medical image segmentation,” arXiv preprint arXiv:2408.08870, 2024
2024
-
[20]
When sam2 meets video camouflaged object segmentation: A comprehensive evaluation and adaptation,
Y . Zhou, G. Sun, Y . Li, L. Benini, and E. Konukoglu, “When sam2 meets video camouflaged object segmentation: A comprehensive evaluation and adaptation,” arXiv preprint arXiv:2409.18653, 2024
2024 arXiv
-
[21]
Evaluation of segment anything model 2: The role of sam2 in the underwater environment,
S. Lian and H. Li, “Evaluation of segment anything model 2: The role of sam2 in the underwater environment,” arXiv preprint arXiv:2408.02924, 2024
2024 arXiv
-
[22]
Triple- cooperative video shadow detection,
Z. Chen, L. Wan, L. Zhu, J. Shen, H. Fu, W. Liu, and J. Qin, “Triple- cooperative video shadow detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 2715–2724
2021
-
[23]
Learning to detect mirrors from videos via dual correspondences,
J. Lin, X. Tan, and R. W. Lau, “Learning to detect mirrors from videos via dual correspondences,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9109–9118
2023
-
[24]
Temporal feature warping for video shadow detection,
S. Hu, H. Le, and D. Samaras, “Temporal feature warping for video shadow detection,” arXiv preprint arXiv:2107.14287, 2021
2021 arXiv
-
[25]
Spatial-temporal fusion network for fast video shadow detection,
J. Lin and L. Wang, “Spatial-temporal fusion network for fast video shadow detection,” in Proceedings of the 18th ACM SIGGRAPH Inter- national Conference on Virtual-Reality Continuum and its Applications in Industry, 2022, pp. 1–5
2022
-
[26]
Video shadow detection via spatio-temporal interpolation consistency training,
X. Lu, Y . Cao, S. Liu, C. Long, Z. Chen, X. Zhou, Y . Yang, and C. Xiao, “Video shadow detection via spatio-temporal interpolation consistency training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3116–3125
2022
-
[27]
Learning shadow correspondence for video shadow detection,
X. Ding, J. Yang, X. Hu, and X. Li, “Learning shadow correspondence for video shadow detection,” in European Conference on Computer Vision . Springer, 2022, pp. 705–722
2022
-
[28]
Semi-supervised video shadow detection via image-assisted pseudo-label generation,
Z. Chen, X. Lu, L. Zhang, and C. Xiao, “Semi-supervised video shadow detection via image-assisted pseudo-label generation,” in Proceedings of the 30th acm international conference on multimedia , 2022, pp. 2700– 2708
2022
-
[29]
Scotch and soda: A transformer video shadow detection framework,
L. Liu, J. Prost, L. Zhu, N. Papadakis, P. Li `o, C.-B. Sch ¨onlieb, and A. I. Aviles-Rivero, “Scotch and soda: A transformer video shadow detection framework,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10 449–10 458
2023
-
[30]
Detect any shadow: Segment anything for video shadow detection,
Y . Wang, W. Zhou, Y . Mao, and H. Li, “Detect any shadow: Segment anything for video shadow detection,” IEEE Transactions on Circuits and Systems for Video Technology, 2023
2023
-
[31]
Timeline and boundary guided diffusion network for video shadow detection,
H. Zhou, H. Wang, T. Ye, Z. Xing, J. Ma, P. Li, Q. Wang, and L. Zhu, “Timeline and boundary guided diffusion network for video shadow detection,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 166–175
2024
-
[32]
Learning motion-guided multi-scale memory features for video shadow detection,
J. Lin, J. Shen, X. Yang, H. Fu, Q. Zhang, P. Li, B. Sheng, L. Wang, and L. Zhu, “Learning motion-guided multi-scale memory features for video shadow detection,” IEEE Transactions on Circuits and Systems for Video Technology, 2024
2024
-
[33]
Two-stage video shadow detection via temporal-spatial adaption,
X. Ding, J. Yang, X. Hu, and X. Li, “Two-stage video shadow detection via temporal-spatial adaption,” in European Conference on Computer Vision. Springer, 2024, pp. 196–214
2024
-
[34]
Structure-aware spatial- temporal interaction network for video shadow detection
H. Wei, G. Xing, J. Liao, Y . Zhang, and Y . Liu, “Structure-aware spatial- temporal interaction network for video shadow detection.”
-
[35]
Effective video mirror detection with inconsistent motion cues,
A. Warren, K. Xu, J. Lin, G. K. Tam, and R. W. Lau, “Effective video mirror detection with inconsistent motion cues,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 17 244–17 252
2024
-
[36]
Zoom: Learning video mirror detection with extremely-weak supervision,
K. Xu, T. W. Siu, and R. W. Lau, “Zoom: Learning video mirror detection with extremely-weak supervision,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 6, 2024, pp. 6315–6323
2024
-
[37]
Fusion of short-term and long- term attention for video mirror detection,
M. Xu, J. Wu, Y . Lai, and Z. Ji, “Fusion of short-term and long- term attention for video mirror detection,” in 2024 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2024, pp. 1–9
2024
-
[38]
Farther than mirror: Explore pattern-compensated depth of mirror with temporal changes for video mirror detection
Z. Xing, L. Liu, T. Ye, S. Chen, Y . Yang, G. Liu, X. Xu, and L. Zhu, “Farther than mirror: Explore pattern-compensated depth of mirror with temporal changes for video mirror detection.” 8 (a) RGB/GT (b) 11th (c) 21th (d) 31th (e) 41th (f) 51th (g) 61th (h) 71th (i) 81th (j) 9...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.