REVIEW 3 major objections 4 minor 12 references
Universal Concept Disruption for SAM3 Image Segmentation
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Universal perturbation drops SAM3 mask AP from 59.4 to 18.7
desk verdict First attack on SAM3's presence-gated concept segmentation, with solid ablations and honest defense analysis, but the evaluation sampling is too thin to support the claim of consistent superiority over GRAT on every dataset. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the presence-gated final concept score $s_i = \sigma(\ell_i)\cdot\sigma(p)$, where $\ell_i$ is the query logit for candidate $i$ and $p$ is a prompt-level presence logit that gates all candidates. Because post-processing keeps only candidates above a threshold on this product, suppressing the presence signal alone can discard geometrically valid masks. UCD's machinery is a four-term objective—score suppression, mask area collapse, Dice disruption against clean masks, and negative feature divergence—optimized jointly into one $\ell^\infty$-bounded perturbation $\delta$ via projected Adam.
What would settle it
Re-run the matched protocol with, say, ten evaluation seeds per dataset and report per-seed ranges; if on PhraseCut the best GRAT seed beats the worst UCD seed in mask AP, or if the perturbation fails to transfer when a checkpoint's presence gate is computed per-candidate, the paper's central advantage claim would not hold.
Extended reading notes
Core claim
The central discovery is that SAM3's presence token—a prompt-level existence logit that globally multiplies every candidate's score—is the load-bearing weakness. UCD optimizes a single universal perturbation that simultaneously activates the same text-conditioned path used at inference by training on real image–noun-phrase pairs, maximizes divergence in the prompt-shared visual backbone to encourage cross-concept and cross-dataset transfer, and suppresses the final presence-gated scores while collapsing or displacing retained masks through area and Dice losses. The result is that a single 8/255-bounded perturbation, trained on 100 images, disrupts concept segmentation on five datasets more than adapted existing attacks, and it transfers to a newer SAM3 checkpoint and to video without retraining.
Load-bearing premise
The comparison assumes that three randomly drawn 100-image evaluation subsets per dataset give stable estimates of attack strength, and that SAM3's released inference really computes each final score as a global presence probability multiplied into every candidate.
Editorial extensions
If this is right
- A single reusable perturbation can degrade SAM3 across five datasets with different prompt types (categories, referring expressions, phrases, open-vocabulary), implying the attack surface is not specific to one benchmark or prompt style.
- The same image-trained perturbation transfers to SAM3.1 without re-optimization, implying prompt-shared visual features and presence-gated scoring are stable across the SAM3 family.
- Injecting the perturbation into only the first frame of a video propagates through SAM3's memory to later clean frames, with tail-frame drops up to 10.36 mask AP in the 14/1 setting.
- Prompt ensembling reduces the measured attack drop only by also reducing clean performance; head fine-tuning preserves clean utility but UCD retrained against it still leaves performance below clean, implying lightweight defenses do not restore robustness.
Reading between the lines
- The presence-gated score product is a general design pattern in open-vocabulary segmentation; UCD's success suggests that any model multiplying per-candidate confidence by a global existence estimate inherits a similar single-point attack surface, so the finding may extend beyond SAM3.
- Since feature divergence is the component whose removal most weakens the attack, prompt-shared backbone features may be the most transferable carrier; this hints that input-space universal perturbations trained to maximize feature divergence could generalize to other vision-language grounding models.
- The defense results imply that robustification should target the presence decision itself, for example by making the presence gate less sensitive to small perturbations, rather than pruning low-confidence predictions, which trades away clean accuracy.
- A testable extension: training UCD on negative prompts as well to activate absent concepts could create a combined attack that both erases present concepts and hallucinates absent ones; the paper explicitly leaves that threat model out.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Universal Concept Disruption (UCD), a universal adversarial perturbation for SAM3's open-vocabulary concept segmentation. UCD is trained on (image, noun-phrase) pairs with an untargeted objective that combines suppression of the presence-gated final concept score, mask area collapse, clean-mask Dice disruption, and divergence of prompt-shared image features. The method is evaluated on five datasets against SAM/SAM2-style and general universal-perturbation baselines under a matched protocol, with ablations of each loss component, transfer experiments to SAM3.1 and to SAM3 video inference, and a defense-oriented evaluation of prompt ensembling, head fine-tuning, and temporal filtering. The headline result is a reduction of average mask AP from 59.43 to 18.73 and average cgF1 from 50.32 to 20.49 across the five datasets.
Significance. If the comparative claim holds, UCD is the first attack specifically targeting SAM3's presence-gated concept segmentation, and the paper's identification of the presence token as a universal attack surface is a useful conceptual contribution. The experimental design has real strengths: baselines are optimized under the same budget and evaluation pipeline, the perturbation is evaluated on held-out test images disjoint from the training set, ablations isolate each loss term, and the transfer tests to SAM3.1 and video are performed without re-optimization. The defense-oriented evaluation is also honest about the clean-performance cost of the tested mitigations. There is no evidence of circularity: the loss does not reduce to the reported metrics by construction, and the baselines are optimized under the same budget. The main weakness is statistical: on two datasets the margin over the strongest baseline is small relative to expected sampling variation, and no error bars or paired statistics are reported.
major comments (3)
- [Experimental Setup / Table 1] The central comparative claim that UCD 'consistently outperforms' all baselines is not statistically supported as reported. Each evaluation uses three random 100-image subsets and only the averages are given; no per-seed values, standard deviations, or confidence intervals are reported. On PhraseCut the UCD/GRAT margin is 0.57 mask AP (9.44 vs 10.01), and on LVIS it is 1.32 points (30.51 vs 31.83); the LVIS cgF1 margin is only 0.31 points (23.57 vs 23.88). These gaps are small relative to expected subset-to-subset variation of instance-level AP over 100 images, so the ranking on these datasets is plausibly sampling noise. Because all methods are evaluated on identical subsets, paired differences or confidence intervals would address this concern; please report per-seed values together with paired statistics, or increase the evaluation subset size.
- [Comparison Study / Experimental Setup] The paper states that all baselines are adapted to the same universal-perturbation setting but does not describe how any baseline is adapted. For example, DarkSAM and UAP-SAM2 are originally designed for geometric-prompt or video settings, and it is unclear how they are converted to a single bounded image perturbation trained on text prompts. Without a precise description of each baseline's objective, prompt form, and training schedule, the claim of a 'matched evaluation protocol' cannot be verified. Please provide these details for every baseline, or release code.
- [Methodology, Eq. (2)] The output-level attack design rests on Eq. (2), which states that the final concept score is the product of the sigmoid of the query logit and the sigmoid of the presence logit. The paper describes this as a 'simplified image inference path,' but it does not verify that the released SAM3 model implements exactly this global multiplication. If the actual combination differs (for example, an additive or learned fusion), the presence-suppression and score-targeting terms may be attacking an abstraction rather than the real decision. Please verify Eq. (2) against the released code or via a direct activation probe, and report the result.
minor comments (4)
- [Table 1] Only the average Box AP across five datasets is reported; per-dataset box AP values are absent and should be added for completeness.
- [Experimental Setup] The paper should state explicitly whether the three random 100-image evaluation subsets are identical across all methods and whether the reported averages are computed over the same image sets, so that paired comparisons are meaningful.
- [Experimental Setup / Datasets and Metrics] The paper notes that cgF1 is not applicable when a sampled set contains only positive prompts, but it does not specify how negative prompts are sampled for SACo-Gold and LVIS or whether the same negative-prompt set is used for every method; please clarify.
- [Abstract and Introduction] Several passages in the provided manuscript text have missing spaces between words (for example, 'Inthispaper'), which should be corrected in the final version.
Circularity Check
No significant circularity: UCD's perturbation is optimized on held-out training pairs and evaluated on disjoint test sets, with no reported metric reducible to a fitted input by construction.
full rationale
The paper's central empirical claim is an attack-effectiveness comparison, not a derived law. UCD optimizes Eq. (5) with loss terms (6), (7), (10), and (11) on a small training set of image–NP pairs, and all reported mask AP, box AP, and cgF1 values are computed on disjoint evaluation subsets (100 images per dataset, three repeats). No evaluation metric is defined as a term of the training objective on the same samples: the score loss targets the final concept score in Eq. (2), but the reported metric aggregates binarized masks over held-out images and prompts, so the attack-evaluation relation is causal rather than definitional. The paper also includes negative-prompt cases where cgF1 applies and reports clean baselines, so the numbers are not constructed from the attack objective. The baselines are adapted under a matched protocol, and no load-bearing argument relies on a self-citation by the present authors. The absence of error bars on the three-repeat averages is a statistical-evidence weakness, correctly flagged by the skeptic, but it is not a circularity: it concerns the strength of an empirical comparison, not a reduction of a claimed prediction to its input. Therefore no circular step meets the quoted-evidence standard, and the appropriate score is 0.
Assumptions & free parameters
free parameters (5)
- Perturbation budget epsilon =
8/255
- Loss weights lambda_s, lambda_a, lambda_d, lambda_f =
0.5, 0.15, 0.5, 1.0
- TopK counts k, k_a, k_d =
10, 3, 3
- Training set composition =
50 LVIS-train + 50 SACo-Silver images, up to 4 positive NPs per image
- Score and mask thresholds tau =
0.5
assumptions (4)
- domain assumption SAM3's inference path matches the description in Eq. (2): candidate score equals query probability times presence probability, and the presence token globally gates all masks.
- domain assumption White-box access to the released SAM3 model weights is available to the attacker.
- domain assumption Three random 100-image subsets per dataset provide stable metric estimates for the comparison.
- standard math Adam with projected gradient descent finds a perturbation that generalizes across the five evaluation datasets.
Cite this review
Pith. "Pith review of Universal Concept Disruption for SAM3 Image Segmentation." pith.science (2026). https://pith.science/paper/XSNDY25L
@misc{pith2026260805983,
author = {Pith},
title = {Pith review of: Universal Concept Disruption for SAM3 Image Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/XSNDY25L}},
note = {Machine review of arXiv:2608.05983}
}
read the original abstract
SAM3 extends promptable segmentation from geometry-driven mask prediction to open-vocabulary concept segmentation, where a text-conditioned grounding model decides whether a concept is present and segments all matching instances. While this presence-gated design improves concept-level prediction, its adversarial robustness remains unexplored. In this paper, we introduce Universal Concept Disruption (UCD), the first universal cross-concept adversarial attack tailored to SAM3 image segmentation. UCD learns a single bounded image perturbation from (image, noun-phrase) pairs and attacks SAM3 as an integrated concept-grounding system. It jointly disrupts the text-conditioned input path, maximizes divergence in prompt-shared visual features, suppresses the final presence-gated concept scores, and corrupts the spatial validity of retained masks through area collapse and clean-mask Dice disruption. Across SACo-Gold, LVIS, RefCOCO, PhraseCut, and OpenImages datasets, UCD consistently outperforms all baselines under a matched evaluation protocol, reducing average mask AP from 59.43 to 18.73 and average cgF1 from 50.32 to 20.49. The learned perturbation also transfers to SAM3.1 and to SAM3 video inference without re-optimization, while prompt ensembling, lightweight head fine-tuning, and temporal filtering provide limited recovery.
Figures
Reference graph
Works this paper leans on
-
[1]
Bolya,D.;Huang,P.-Y.;Sun,P.;Cho,J.H.;Madotto,A.;Wei, C.;Ma,T.;Zhi,J.;Rajasegaran,J.;Bangalath,H.;etal.2026. Perception encoder: The best visual embeddings are not at the output of the network.Advances in Neural Information Processing Systems, 38: 60884–60937. Carion, N.; Gustafson, L.; Hu, Y.-T.; Debnath, S.; Hu, R.; Suris, D.; Ryali, C.; Alwala, K. V.; K...
work page 2026
-
[5]
Kazemzadeh,S.;Ordonez,V.;Matten,M.;andBerg,T.2014
Segment Shards: Cross-Prompt Adversarial At- tacksagainsttheSegmentAnythingModel.AppliedSciences (2076-3417), 14(8). Kazemzadeh,S.;Ordonez,V.;Matten,M.;andBerg,T.2014. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methodsinnaturallanguageprocessing(EMNLP),787–798. Kingma,D.P.;andBa,J....
arXiv 2014
-
[10]
InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 194–203
Practical region-level attack against segment anything models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 194–203. Szegedy,C.;Zaremba,W.;Sutskever,I.;Bruna,J.;Erhan,D.; Goodfellow,I.;andFergus,R.2013. Intriguingpropertiesof neural networks.Computer Science. Wu,C.;Lin,Z.;Cohen,S.;Bui,T.;andMaji,S.2020.Phrase- cut:L...
work page 2013
-
[11]
Attack-sam: Towards evaluating adver- sarialrobustnessofsegmentanythingmodel.arXivpreprint arXiv:2305.00866, 1(3):
-
[12]
Black-BoxTargeted Adversarial Attack on Segment Anything (SAM).Trans
Zheng,S.;Zhang,C.;andHao,X.2025. Black-BoxTargeted Adversarial Attack on Segment Anything (SAM).Trans. Multi., 27: 1901–1913. Zhou, Z.; Hu, Y.; Song, Y.; Li, Z.; Hu, S.; Zhang, L. Y.; Yao, D.; Zheng, L.; and Jin, H
work page 2025
-
[2014]
Gupta,A.;Dollar,P.;andGirshick,R.2019
Ex- plaining and Harnessing Adversarial Examples.Computer Science. Gupta,A.;Dollar,P.;andGirshick,R.2019. Lvis:Adataset for large vocabulary instance segmentation. InProceedings oftheIEEE/CVFconferenceoncomputervisionandpattern recognition, 5356–5364. Han, D.; Zhang, C.; Zheng, S.; Lu, C.; Yang, Y.; and Shen, H. T
work page 2019
-
[2020]
The open images dataset v4: Unified image classification, object detection, and visual re- lationship detection at scale.International journal of com- puter vision, 128(7): 1956–1981. Li, L. H.; Zhang, P.; Zhang, H.; Yang, J.; Li, C.; Zhong, Y.; Wang, L.; Yuan, L.; Zhang, L.; and Hwang, J. N
work page 1956
-
[2021]
InInternational conference on machine learning, 8748–8763
Learning transferable visual models from natural language supervision. InInternational conference on machine learning, 8748–8763. PmLR. Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr,H.;Rädle,R.;Rolland,C.;Gustafson,L.;etal.2025. Sam 2: Segment anything in images and videos. InInter- national Conference on Learning Representations, volu...
work page 2025
Show all 12 references
-
[2023]
Radford, A.; Kim, J
Robustness of sam: Segment anything under corruptions and beyond.arXiv preprint arXiv:2306.07713. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al
-
[2024]
arXiv:2310.12431
SAM Meets UAP: Attacking Segment Anything Model With Universal Adversarial Perturbation. arXiv:2310.12431. Huang, S.; Fan, Q.; Zhang, Z.; Liu, X.; Song, G.; and Qin, J
-
[2025]
Carlini, N.; and Wagner, D
Sam 3: Segment anything with concepts.arXiv preprint arXiv:2511.16719. Carlini, N.; and Wagner, D
-
[2026]
Long,J.;Xu,Z.;Jiang,T.;Yao,W.;Jia,S.;Ma,C.;andChen, X.2025.RobustSAM:ontheadversarialrobustnessofvision foundation models
Region-guided attack on the segment anything model.Neural Networks, 193(c): 108058. Long,J.;Xu,Z.;Jiang,T.;Yao,W.;Jia,S.;Ma,C.;andChen, X.2025.RobustSAM:ontheadversarialrobustnessofvision foundation models. InProceedings of the AAAI Conference on Artificial Intelligence, volum...
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.