REVIEW 3 major objections 4 minor 13 references
EOVSAM removes SAM 3's prompt conditioning to turn it into a single-pass open-vocabulary segmenter, pooling SigLIP features with decoder attention maps to classify masks in the same forward pass.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
By removing SAM 3's text prompts and pooling SigLIP features with decoder attention maps, EOVSAM performs open-vocabulary segmentation in one pass, beating vanilla SAM 3 in accuracy while running up to 338× faster.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection Solid single-pass SAM 3 adaptation with real speed and accuracy gains; the attentional-pooling mechanism is plausible but the alignment premise deserves a direct check. the 3 major comments →
EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that the multi-pass, per-category loop of SAM 3 is an architectural accident, not a necessity. EOVSAM turns SAM 3's detector decoder into a prompt-free mask proposal engine: it drops text cross-attention from the fusion encoder and decoder, keeps the pretrained SAM 3 weights, and uses a vision backbone that emits SAM-oriented and SigLIP-oriented features on the same grid. The decoder's learned attention maps then aggregate SigLIP 2 features into per-object embeddings, which are scored against a SigLIP text classifier. Because the aggregation is a soft, differentiable average rather than binarized mask pooling, the recognition loss flows into mask localization, and the pa
What carries the argument
Attentional Aggregation is the load-bearing mechanism: across the six decoder layers, a learned set of weights pools the multi-head attention maps into one soft attention map per object query, and that map averages the SigLIP 2 feature grid into an object embedding (Eq. 3). The embedding is classified by cosine similarity to SigLIP text embeddings, with a learnable temperature. This differs from prior mask-cropping/mask-pooling schemes because the attention map is never binarized, so the gradient from the open-vocabulary classification loss reaches the mask decoder. The other half of the machinery is the prompt-free reuse of SAM 3: removing text cross-attention and keeping the pretrained wei
Load-bearing premise
The entire classification branch assumes the decoder's attention maps and the SigLIP feature map are aligned pixel-to-pixel spatially; the paper does not report a direct measurement of that alignment.
What would settle it
Compute the overlap between each attention map and its final mask (e.g., attention-to-mask IoU) in SigLIP feature space; if overlap is low yet classification still works, the aggregation mechanism is not doing what the paper claims. Alternatively, shift the SigLIP grid by a few pixels at inference and measure the drop in mIoU: a sharp drop would confirm dependence on exact alignment, while little change would imply another mechanism.
If this is right
- Inference cost stops growing with vocabulary size; the only added cost for a larger vocabulary is encoding the text labels once.
- A single checkpoint trained on COCO Panoptic covers both semantic and panoptic tasks, and one trained model serves arbitrary unseen category lists without fine-tuning.
- Resolution can be dropped (e.g., 512 vs 1152) with only small mIoU loss, giving a latency-performance knob for deployment.
- The fixed 200 object queries bound the number of masks per image; scenes with more than 200 objects would exceed the proposal budget.
- The approach avoids multi-stage pipelines and post-processing heuristics used by existing methods, since masks and labels come from the same forward pass.
Where Pith is reading between the lines
- A direct test of the alignment premise—measuring how well each attention map overlaps its corresponding mask in SigLIP feature space—would separate the benefit of the aggregation rule from the benefit of the end-to-end training signal.
- The geometric ensemble with the SAM classifier is nearly unnecessary at α=β=1.0, so a simpler single-classifier variant could be deployed; the ensemble mainly adds robustness for seen categories.
- The method suggests a recipe for other DETR-like segmenters: freeze a strong mask proposer, attach a dual-feature backbone, and let attention maps pool a vision-language feature map. One could test whether this transfers to SAM 2 or other prompt-based segmenters.
- Since the model uses 200 queries regardless of scene content, adaptive query selection or iterative refinement could be explored for scenes with many objects.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. EOVSAM adapts SAM 3 for single-pass open-vocabulary segmentation. It replaces the SAM 3 image encoder with C-RADIOv4, which provides both SAM-3-like features (F_sam) and SigLIP 2 features (F_siglip); removes text cross-attention from the fusion encoder and detector decoder; and trains a Mask2Former-style decoder with 200 object queries to produce masks and attention maps. The attention maps pool SigLIP features into object embeddings (Eq. 3), which are classified with a SigLIP text classifier; a frozen SAM classifier provides objectness and supplementary classification. Training is on COCO Panoptic. The paper reports mIoU of 39.0/16.6/60.9/20.4/97.0 on A-150/A-847/PC-59/PC-459/PAS-20 and ADE20K PQ of 30.9, with large speedups over vanilla SAM 3. Ablations attribute substantial gains to Attentional Aggregation and to inheriting pretrained SAM 3 weights.
Significance. If the claims hold, EOVSAM is a valuable contribution to efficient open-vocabulary segmentation: a single COCO-Panoptic-trained model sets a new state of the art on ADE20K panoptic segmentation among open-vocabulary methods and avoids the vocabulary-length-dependent inference of SAM 3. The paper is well structured, with explicit equations, a useful set of component ablations (Table 4), a direct comparison of aggregation strategies (Table 5), and sufficient training details to enable reproduction. The main caveat is that the core Attentional Aggregation mechanism rests on a spatial-alignment assumption that is asserted rather than directly demonstrated; the empirical gains should be accepted with caution until that link is verified.
major comments (3)
- [Efficient Open-Vocabulary Recognition via Attentional Aggregation, Eq. (3)] Eq. (3) pools SigLIP features F_siglip with attention maps A produced by the detector decoder from the SAM-path features F_enh. The paper asserts that A 'inherently maintains a precise point-to-point spatial correspondence' with F_siglip, but no direct evidence is given. C-RADIOv4's SAM-3 and SigLIP-2 adapter heads are distilled from different teachers; identical grid geometry and semantic alignment do not follow. Table 4 shows that removing Attentional Aggregation drops A-150 mIoU from 39.0 to 22.6, so the +16.4 mIoU gain—the main evidence for the proposed novelty—rests on this unverified premise. Please add direct alignment analysis (e.g., attention-to-mask IoU in SigLIP feature space, grid-offset/receptive-field statistics, or explicit spatial resampling/alignment) or otherwise demonstrate that Eq. (3) does not pool spatially misaligned features.
- [Objective / Additional Experiments, Table 6] The final prediction procedure is incompletely specified. Eq. (4) defines only the SigLIP-classifier probability P_i,k; the paper also introduces a SAM classifier, objectness scores, and a geometric ensemble with weights alpha and beta, but no equation describes how these signals are combined. It is also unclear how the background token is used and how the N=200 masks are converted into the semantic mIoU and panoptic PQ numbers in Tables 1-2. Additionally, alpha=0.7, beta=1.0 in Table 6 is selected by sweeping on the same test benchmarks, so the reported final numbers include test-set selection pressure; the effect is small relative to the alpha=beta=1.0 row, but the protocol should be disclosed and, ideally, the coefficients chosen on a validation split. Please provide the ensemble formula, the inference/post-processing protocol, and clarification of the selection procedure.
- [Experiments / Table 3] The 'up to 338x' speedup claim is not fully supported. Table 3 reports FPS but does not state the exact measurement protocol (e.g., batch size, warm-up, whether text encoding is included, whether timings are end-to-end or model-only). Since text features are precomputed, the comparison should specify what is timed. Also, EOVSAM uses C-RADIOv4-H and a different resolution than SAM 3's official setting, so the speed advantage may be dominated by backbone/resolution rather than the single-pass design. Please provide a latency breakdown (image encoder / fusion encoder / decoder / post-processing) and the exact setting that yields 338x.
minor comments (4)
- [Adapting SAM 3 for Single-Pass Full-Image Segmentation] The term 'prompt-free' should be qualified: category text embeddings are still used for classification via the SigLIP/SAM classifiers; only the mask-generation path removes per-category prompts.
- [Experimental Setup / Implementation Details] The learning-rate decay milestones are not specified, and 'C-RADIOv4-H' is not defined in the manuscript. Please provide the exact schedule and model configuration.
- [Experiments / Table 3] Add a note identifying which row/dataset corresponds to the 'up to 338x' speedup, since the reported FPS values vary by dataset and resolution.
- [Figures 2 and 3] Mark which components are frozen and which are trainable. This would clarify the 'end-to-end' claim, because the visual backbone and both text encoders are frozen.
Circularity Check
No circularity: EOVSAM's claims are benchmark-grounded and its derivation does not reduce to its inputs or to a self-citation chain.
full rationale
EOVSAM's core derivation is an architectural transformation: remove text cross-attention, route C-RADIOv4's SAM/SigLIP features through the fusion encoder, detector decoder, and pixel decoder, pool SigLIP features with attention maps, and train with Mask2Former-style losses plus a SigLIP cosine classification loss. None of these steps define the target quantity in terms of the input. Equation (3) computes object embeddings o_i as A-weighted sums of F_siglip, where A is produced by the detector decoder over F_enh; the two feature streams are distinct outputs of different C-RADIOv4 adapter heads, and the classification target is the external SigLIP text encoder's embeddings of dataset categories. The end-to-end loss L_cls supervises the whole chain; no output is reintroduced as an input to define itself. The asserted 'precise point-to-point spatial correspondence' between attention maps and SigLIP features is an unverified architectural premise, but it is not circular: if alignment fails, benchmark accuracy would drop, and the paper's measured external benchmark numbers are the evidence. The alpha/beta geometric ensemble sweep is model selection on benchmark datasets, not a fitted parameter renamed as a prediction; Table 6 shows the SigLIP-only variant still reaches 38.4 mIoU on A-150, so the central single-pass claim does not rely on the swept coefficients. Self-citations to Mask-Adapter and MAFT+ appear only as baselines or related work and do not carry the load of the central claim. There is no imported uniqueness theorem and no ansatz smuggled in via citation. Thus no circular step reaches the evidence threshold; score 0.
Axiom & Free-Parameter Ledger
free parameters (7)
- α (SigLIP classifier weight, seen categories) =
0.7
- β (SigLIP classifier weight, unseen categories) =
1.0
- Temperature τ (Eq. 4) =
learnable
- Loss weights λce, λbce, λdice, λcls, λL1, λgiou =
3, 5, 5, 1, 5, 2
- Object query count N and decoder depth L =
N=200, L=6
- Input resolution / crop size =
1152 (512-1152 at inference)
- Per-head attention pooling coefficients =
learned
axioms (6)
- domain assumption C-RADIOv4's SAM-3 and SigLIP-2 adapter heads output features on identical spatial grids with aligned semantics at the same stride.
- domain assumption Removing text cross-attention from SAM 3's fusion encoder and detector decoder preserves its localization priors.
- domain assumption SAM 3 pre-trained weights transfer into the modified architecture.
- domain assumption SigLIP 2 text embeddings form a suitable open-vocabulary classifier space.
- domain assumption A fixed set of 200 learnable queries can localize an arbitrary number and type of objects without text guidance.
- domain assumption Zero-shot transfer from COCO Panoptic training to ADE20K/Pascal benchmarks measures open-vocabulary capability.
Cite this review
Pith. "Pith review of EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass." pith.science (2026). https://pith.science/paper/OA4TULKV
@misc{pith2026260802284,
author = {Pith},
title = {Pith review of: EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass},
year = {2026},
howpublished = {\url{https://pith.science/paper/OA4TULKV}},
note = {Machine review of arXiv:2608.02284}
}
read the original abstract
Open-vocabulary segmentation identifies and segments objects from arbitrary textual descriptions. SAM 3 supports noun-phrase-guided segmentation and achieves competitive open-vocabulary performance through exhaustive vocabulary traversal, yet suffers from prohibitive computational overhead as target categories scale. In this paper, we propose an Efficient Open-Vocabulary segmentation framework with SAM 3 (EOVSAM), which adapts SAM 3 for single-pass prediction. EOVSAM removes prompt conditioning to turn SAM 3 into an efficient mask generator and introduces a new Attentional Aggregation strategy to optimize open-vocabulary classification end-to-end. This formulation avoids the multi-stage pipelines and post-processing heuristics commonly used by existing methods, while mitigating the closed-set collapse that can arise when classification is optimized directly. EOVSAM consistently improves segmentation accuracy over vanilla SAM 3 on all evaluated datasets and accelerates inference by up to 338$\times$. Furthermore, EOVSAM maintains high accuracy at lower resolutions while achieving even more remarkable inference speeds. Experiments on standard semantic and panoptic segmentation benchmarks show that EOVSAM combines competitive or state-of-the-art accuracy with a substantial speed advantage over existing open-vocabulary segmentation models. Code and models are available at https://github.com/hustvl/EOVSAM.
Figures
Reference graph
Works this paper leans on
-
[4]
In2024 IEEEInternationalConferenceonImageProcessing(ICIP), 2494–2500
Open- Vocabulary Panoptic Segmentation Using Bert Pre-Training of Vision-Language Multiway Transformer Model. In2024 IEEEInternationalConferenceonImageProcessing(ICIP), 2494–2500. IEEE. Cheng,B.;Misra,I.;Schwing,A.G.;Kirillov,A.;andGird- har,R.2022. Masked-attentionmasktransformerforuniver- sal image segmentation. InProceedings of the IEEE/CVF conference ...
2022
-
[9]
C-RADIOv4 (Tech Report).arXiv preprint arXiv:2601.17237. Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; Rädle, R.; Rolland, C.; Gustafson, L.; Mintun, E.; Pan, J.; Alwala, K. V.; Carion, N.; Wu, C.-Y.; Girshick, R.; Dollar, P.; and Feichtenhofer, C
-
[10]
In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds.,International Conference on Learning Representations, volume 2025, 28085–28128
SAM 2: Segment Anything in Images and Videos. In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds.,International Conference on Learning Representations, volume 2025, 28085–28128. Shan, X.; Wu, D.; Zhu, G.; Shao, Y.; Sang, N.; and Gao, C
2025
-
[11]
Tschannen, M.; Gritsenko, A.; Wang, X.; Naeem, M
Dinov3.arXiv preprint arXiv:2508.10104. Tschannen, M.; Gritsenko, A.; Wang, X.; Naeem, M. F.; Alabdulmohsin, I.; Parthasarathy, N.; Evans, T.; Beyer, L.; Xia, Y.; Mustafa, B.; et al
-
[12]
Xiao, S.; Kabra, R.; Li, Y.; Lee, D.; Carreira, J.; and Panda, P
Siglip 2: Multilingual vision-language encoders with improved semantic under- standing, localization, and dense features.arXiv preprint arXiv:2502.14786. Xiao, S.; Kabra, R.; Li, Y.; Lee, D.; Carreira, J.; and Panda, P
-
[2014]
InEuropean conference on computer vision, 740–755
Microsoft coco: Common objects in context. InEuropean conference on computer vision, 740–755. Springer. Loshchilov,I.;andHutter,F.2017. Decoupledweightdecay regularization.arXiv preprint arXiv:1711.05101. Mottaghi, R.; Chen, X.; Liu, X.; Cho, N.-G.; Lee, S.-W.; Fidler, S.; Urtasun, R.; and Yuille, A
Pith/arXiv arXiv 2017
-
[2015]
Han,C.;Zhong,Y.;Li,D.;Han,K.;andMa,L.2023
The pascal vi- sual object classes challenge: A retrospective.International journal of computer vision, 111(1): 98–136. Han,C.;Zhong,Y.;Li,D.;Han,K.;andMa,L.2023. Open- vocabulary semantic segmentation with decoupled one-pass network. InProceedings of the IEEE/CVF International Conference on Computer Vision, 1086–1096. He,K.;Gkioxari,G.;Dollár,P.;andGirsh...
2023
-
[2018]
InProceedings of the IEEEconferenceoncomputervisionandpatternrecognition, 1209–1218
Coco-stuff: Thing and stuff classes in context. InProceedings of the IEEEconferenceoncomputervisionandpatternrecognition, 1209–1218. Carion, N.; Gustafson, L.; Hu, Y.-T.; Debnath, S.; Hu, R.; Suris, D.; Ryali, C.; Alwala, K. V.; Khedr, H.; Huang, A.; Lei, J.; Ma, T.; Guo, B.; Kalla, A.; Marks, M.; Greer, J.; Wang, M.; Sun, P.; Rädle, R.; Afouras, T.; Mavr...
Pith/arXiv arXiv 2025
-
[2021]
InInternational conference on ma- chine learning, 4904–4916
Scaling up visual and vision-language representation learning with noisy text supervision. InInternational conference on ma- chine learning, 4904–4916. PMLR. Jiao, S.; Zhu, H.; Huang, J.; Zhao, Y.; Wei, Y.; and Shi, H. 2024.Collaborativevision-textrepresentationoptimizingfor open-vocabulary segmentation. InEuropean Conference on Computer Vision, 399–416. ...
2024
-
[2023]
InProceed- ingsoftheIEEE/CVFinternationalconferenceoncomputer vision, 11975–11986
Sigmoid loss for language image pre-training. InProceed- ingsoftheIEEE/CVFinternationalconferenceoncomputer vision, 11975–11986. Zhou, B.; Zhao, H.; Puig, X.; Xiao, T.; Fidler, S.; Barriuso, A.;andTorralba,A.2019. Semanticunderstandingofscenes through the ade20k dataset.International journal of com- puter vision, 127(3): 302–321
2019
-
[2024]
Chen, Y.-C.; Li, W.-H.; and Chen, C.-S
Frozenseg:Harmonizingfrozenfoundationmodelsforopen- vocabularysegmentation.arXivpreprintarXiv:2409.03525. Chen, Y.-C.; Li, W.-H.; and Chen, C.-S
-
[2025]
InProceedings of the Computer Vision and Pattern Recognition Conference, 26081–26090
Effective SAM combination for open-vocabulary semantic segmentation. InProceedings of the Computer Vision and Pattern Recognition Conference, 26081–26090. Li, B.; Zhang, D.; Zhao, Z.; Gao, J.; and Li, X. 2025a. FGAseg: Fine-Grained Pixel-Text Alignment for Open-Vocabulary Semantic Segmentation.arXiv preprint arXiv:2501.00877. Li,J.;Lu,Y.;Zhang,Y.;Wang,F.;...
-
[2026]
Chen,L.-C.;Zhu,Y.;Papandreou,G.;Schroff,F.;andAdam, H.2018
SAM-MI:AMask-injectedFrameworkforEnhancing Open-vocabulary Semantic Segmentation with SAM.Ma- chine Intelligence Research, 1–17. Chen,L.-C.;Zhu,Y.;Papandreou,G.;Schroff,F.;andAdam, H.2018. Encoder-decoderwithatrousseparableconvolution for semantic image segmentation. InProceedings of the Europeanconferenceoncomputervision(ECCV),801–818. Chen, X.; Yang, H....
2018
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.