Pith. sign in

REVIEW 3 major objections 4 minor 13 references

EOVSAM removes SAM 3's prompt conditioning to turn it into a single-pass open-vocabulary segmenter, pooling SigLIP features with decoder attention maps to classify masks in the same forward pass.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

By removing SAM 3's text prompts and pooling SigLIP features with decoder attention maps, EOVSAM performs open-vocabulary segmentation in one pass, beating vanilla SAM 3 in accuracy while running up to 338× faster.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Solid single-pass SAM 3 adaptation with real speed and accuracy gains; the attentional-pooling mechanism is plausible but the alignment premise deserves a direct check. the 3 major comments →

arxiv 2608.02284 v1 pith:OA4TULKV submitted 2026-08-03 cs.CV

EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass

classification cs.CV
keywords open-vocabulary segmentationSAM 3single-pass inferenceattentional aggregationpanoptic segmentationzero-shot recognitionvision-language modelsefficient inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EOVSAM claims that SAM 3—which currently runs one forward pass per searched noun phrase—can be rewritten as a single-pass open-vocabulary segmenter without losing localization strength. It removes text cross-attention, making the model prompt-free, and classifies the resulting masks by pooling SigLIP 2 image features through the decoder's own attention maps. Trained only on COCO Panoptic, the model reports mIoU of 39.0/16.6/60.9/20.4/97.0 on A-150/A-847/PC-59/PC-459/PAS-20, a panoptic PQ of 30.9 on ADE20K (state of the art among open-vocabulary panoptic models), and inference up to 338× faster than vanilla SAM 3. The practical stake is that open-vocabulary segmentation need not choose between accuracy and latency: one frozen backbone with a light trainable decoder can do both.

Core claim

The central claim is that the multi-pass, per-category loop of SAM 3 is an architectural accident, not a necessity. EOVSAM turns SAM 3's detector decoder into a prompt-free mask proposal engine: it drops text cross-attention from the fusion encoder and decoder, keeps the pretrained SAM 3 weights, and uses a vision backbone that emits SAM-oriented and SigLIP-oriented features on the same grid. The decoder's learned attention maps then aggregate SigLIP 2 features into per-object embeddings, which are scored against a SigLIP text classifier. Because the aggregation is a soft, differentiable average rather than binarized mask pooling, the recognition loss flows into mask localization, and the pa

What carries the argument

Attentional Aggregation is the load-bearing mechanism: across the six decoder layers, a learned set of weights pools the multi-head attention maps into one soft attention map per object query, and that map averages the SigLIP 2 feature grid into an object embedding (Eq. 3). The embedding is classified by cosine similarity to SigLIP text embeddings, with a learnable temperature. This differs from prior mask-cropping/mask-pooling schemes because the attention map is never binarized, so the gradient from the open-vocabulary classification loss reaches the mask decoder. The other half of the machinery is the prompt-free reuse of SAM 3: removing text cross-attention and keeping the pretrained wei

Load-bearing premise

The entire classification branch assumes the decoder's attention maps and the SigLIP feature map are aligned pixel-to-pixel spatially; the paper does not report a direct measurement of that alignment.

What would settle it

Compute the overlap between each attention map and its final mask (e.g., attention-to-mask IoU) in SigLIP feature space; if overlap is low yet classification still works, the aggregation mechanism is not doing what the paper claims. Alternatively, shift the SigLIP grid by a few pixels at inference and measure the drop in mIoU: a sharp drop would confirm dependence on exact alignment, while little change would imply another mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Inference cost stops growing with vocabulary size; the only added cost for a larger vocabulary is encoding the text labels once.
  • A single checkpoint trained on COCO Panoptic covers both semantic and panoptic tasks, and one trained model serves arbitrary unseen category lists without fine-tuning.
  • Resolution can be dropped (e.g., 512 vs 1152) with only small mIoU loss, giving a latency-performance knob for deployment.
  • The fixed 200 object queries bound the number of masks per image; scenes with more than 200 objects would exceed the proposal budget.
  • The approach avoids multi-stage pipelines and post-processing heuristics used by existing methods, since masks and labels come from the same forward pass.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the alignment premise—measuring how well each attention map overlaps its corresponding mask in SigLIP feature space—would separate the benefit of the aggregation rule from the benefit of the end-to-end training signal.
  • The geometric ensemble with the SAM classifier is nearly unnecessary at α=β=1.0, so a simpler single-classifier variant could be deployed; the ensemble mainly adds robustness for seen categories.
  • The method suggests a recipe for other DETR-like segmenters: freeze a strong mask proposer, attach a dual-feature backbone, and let attention maps pool a vision-language feature map. One could test whether this transfers to SAM 2 or other prompt-based segmenters.
  • Since the model uses 200 queries regardless of scene content, adaptive query selection or iterative refinement could be explored for scenes with many objects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. EOVSAM adapts SAM 3 for single-pass open-vocabulary segmentation. It replaces the SAM 3 image encoder with C-RADIOv4, which provides both SAM-3-like features (F_sam) and SigLIP 2 features (F_siglip); removes text cross-attention from the fusion encoder and detector decoder; and trains a Mask2Former-style decoder with 200 object queries to produce masks and attention maps. The attention maps pool SigLIP features into object embeddings (Eq. 3), which are classified with a SigLIP text classifier; a frozen SAM classifier provides objectness and supplementary classification. Training is on COCO Panoptic. The paper reports mIoU of 39.0/16.6/60.9/20.4/97.0 on A-150/A-847/PC-59/PC-459/PAS-20 and ADE20K PQ of 30.9, with large speedups over vanilla SAM 3. Ablations attribute substantial gains to Attentional Aggregation and to inheriting pretrained SAM 3 weights.

Significance. If the claims hold, EOVSAM is a valuable contribution to efficient open-vocabulary segmentation: a single COCO-Panoptic-trained model sets a new state of the art on ADE20K panoptic segmentation among open-vocabulary methods and avoids the vocabulary-length-dependent inference of SAM 3. The paper is well structured, with explicit equations, a useful set of component ablations (Table 4), a direct comparison of aggregation strategies (Table 5), and sufficient training details to enable reproduction. The main caveat is that the core Attentional Aggregation mechanism rests on a spatial-alignment assumption that is asserted rather than directly demonstrated; the empirical gains should be accepted with caution until that link is verified.

major comments (3)
  1. [Efficient Open-Vocabulary Recognition via Attentional Aggregation, Eq. (3)] Eq. (3) pools SigLIP features F_siglip with attention maps A produced by the detector decoder from the SAM-path features F_enh. The paper asserts that A 'inherently maintains a precise point-to-point spatial correspondence' with F_siglip, but no direct evidence is given. C-RADIOv4's SAM-3 and SigLIP-2 adapter heads are distilled from different teachers; identical grid geometry and semantic alignment do not follow. Table 4 shows that removing Attentional Aggregation drops A-150 mIoU from 39.0 to 22.6, so the +16.4 mIoU gain—the main evidence for the proposed novelty—rests on this unverified premise. Please add direct alignment analysis (e.g., attention-to-mask IoU in SigLIP feature space, grid-offset/receptive-field statistics, or explicit spatial resampling/alignment) or otherwise demonstrate that Eq. (3) does not pool spatially misaligned features.
  2. [Objective / Additional Experiments, Table 6] The final prediction procedure is incompletely specified. Eq. (4) defines only the SigLIP-classifier probability P_i,k; the paper also introduces a SAM classifier, objectness scores, and a geometric ensemble with weights alpha and beta, but no equation describes how these signals are combined. It is also unclear how the background token is used and how the N=200 masks are converted into the semantic mIoU and panoptic PQ numbers in Tables 1-2. Additionally, alpha=0.7, beta=1.0 in Table 6 is selected by sweeping on the same test benchmarks, so the reported final numbers include test-set selection pressure; the effect is small relative to the alpha=beta=1.0 row, but the protocol should be disclosed and, ideally, the coefficients chosen on a validation split. Please provide the ensemble formula, the inference/post-processing protocol, and clarification of the selection procedure.
  3. [Experiments / Table 3] The 'up to 338x' speedup claim is not fully supported. Table 3 reports FPS but does not state the exact measurement protocol (e.g., batch size, warm-up, whether text encoding is included, whether timings are end-to-end or model-only). Since text features are precomputed, the comparison should specify what is timed. Also, EOVSAM uses C-RADIOv4-H and a different resolution than SAM 3's official setting, so the speed advantage may be dominated by backbone/resolution rather than the single-pass design. Please provide a latency breakdown (image encoder / fusion encoder / decoder / post-processing) and the exact setting that yields 338x.
minor comments (4)
  1. [Adapting SAM 3 for Single-Pass Full-Image Segmentation] The term 'prompt-free' should be qualified: category text embeddings are still used for classification via the SigLIP/SAM classifiers; only the mask-generation path removes per-category prompts.
  2. [Experimental Setup / Implementation Details] The learning-rate decay milestones are not specified, and 'C-RADIOv4-H' is not defined in the manuscript. Please provide the exact schedule and model configuration.
  3. [Experiments / Table 3] Add a note identifying which row/dataset corresponds to the 'up to 338x' speedup, since the reported FPS values vary by dataset and resolution.
  4. [Figures 2 and 3] Mark which components are frozen and which are trainable. This would clarify the 'end-to-end' claim, because the visual backbone and both text encoders are frozen.

Circularity Check

0 steps flagged

No circularity: EOVSAM's claims are benchmark-grounded and its derivation does not reduce to its inputs or to a self-citation chain.

full rationale

EOVSAM's core derivation is an architectural transformation: remove text cross-attention, route C-RADIOv4's SAM/SigLIP features through the fusion encoder, detector decoder, and pixel decoder, pool SigLIP features with attention maps, and train with Mask2Former-style losses plus a SigLIP cosine classification loss. None of these steps define the target quantity in terms of the input. Equation (3) computes object embeddings o_i as A-weighted sums of F_siglip, where A is produced by the detector decoder over F_enh; the two feature streams are distinct outputs of different C-RADIOv4 adapter heads, and the classification target is the external SigLIP text encoder's embeddings of dataset categories. The end-to-end loss L_cls supervises the whole chain; no output is reintroduced as an input to define itself. The asserted 'precise point-to-point spatial correspondence' between attention maps and SigLIP features is an unverified architectural premise, but it is not circular: if alignment fails, benchmark accuracy would drop, and the paper's measured external benchmark numbers are the evidence. The alpha/beta geometric ensemble sweep is model selection on benchmark datasets, not a fitted parameter renamed as a prediction; Table 6 shows the SigLIP-only variant still reaches 38.4 mIoU on A-150, so the central single-pass claim does not rely on the swept coefficients. Self-citations to Mask-Adapter and MAFT+ appear only as baselines or related work and do not carry the load of the central claim. There is no imported uniqueness theorem and no ansatz smuggled in via citation. Thus no circular step reaches the evidence threshold; score 0.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The central claim rests on two families of pulled-in content: (1) frozen pretrained components (C-RADIOv4, SAM 3 encoder/decoder weights, SigLIP 2, SAM 3 text encoder) whose quality is assumed, and (2) design hyperparameters (α, β, loss weights, N, L, resolution) chosen by hand or tuned on the evaluated benchmarks. The paper's own ablations empirically defend the SAM-3-transfer and text-cross-attention-removal assumptions; the spatial-alignment axiom for C-RADIOv4 features is the least defended. No new physical or architectural entities are postulated: the learnable background token and per-head attention weights are trained parameters, not invented entities.

free parameters (7)
  • α (SigLIP classifier weight, seen categories) = 0.7
    Exponential weight in geometric ensembling of SAM and SigLIP classifiers; chosen from the Table 6 sweep over the reported benchmarks, with no disclosed held-out split.
  • β (SigLIP classifier weight, unseen categories) = 1.0
    Same sweep as α; the pair (0.7, 1.0) is adopted across all datasets.
  • Temperature τ (Eq. 4) = learnable
    Learned softmax temperature for cosine-similarity classification.
  • Loss weights λce, λbce, λdice, λcls, λL1, λgiou = 3, 5, 5, 1, 5, 2
    Hand-set weighting of Mask2Former losses plus Lcls and box losses (Eq. 5).
  • Object query count N and decoder depth L = N=200, L=6
    Architecture capacity for single-pass localization without text prompts; inherited from DETR-style defaults.
  • Input resolution / crop size = 1152 (512-1152 at inference)
    A single checkpoint is evaluated at multiple resolutions; 1152 is the default and the one used for headline numbers.
  • Per-head attention pooling coefficients = learned
    Learnable weights that combine multi-head attention maps into the classification attention map A.
axioms (6)
  • domain assumption C-RADIOv4's SAM-3 and SigLIP-2 adapter heads output features on identical spatial grids with aligned semantics at the same stride.
    Eq. (3) applies decoder attention maps computed from F_enh directly to F_siglip; alignment is asserted ("inherently maintains a precise point-to-point spatial correspondence") but not measured. This is the load-bearing premise of Attentional Aggregation.
  • domain assumption Removing text cross-attention from SAM 3's fusion encoder and detector decoder preserves its localization priors.
    The whole single-pass design depends on SAM 3 mask quality surviving prompt removal; the 'w/ Text C-A' ablation supports this empirically on their setup.
  • domain assumption SAM 3 pre-trained weights transfer into the modified architecture.
    The 'w/o SAM 3 Weights' ablation (-6.7 PQ) validates transfer on their training/eval protocol.
  • domain assumption SigLIP 2 text embeddings form a suitable open-vocabulary classifier space.
    Classification uses cosine similarity to frozen SigLIP 2 text embeddings; relies on SigLIP 2's zero-shot generalization from prior work rather than a verification in this paper.
  • domain assumption A fixed set of 200 learnable queries can localize an arbitrary number and type of objects without text guidance.
    The prompt-free decoder has no per-concept conditioning; completeness depends on DETR-style query capacity.
  • domain assumption Zero-shot transfer from COCO Panoptic training to ADE20K/Pascal benchmarks measures open-vocabulary capability.
    Standard OVS evaluation protocol (following FC-CLIP); validity assumed.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass." pith.science (2026). https://pith.science/paper/OA4TULKV

@misc{pith2026260802284,
  author       = {Pith},
  title        = {Pith review of: EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OA4TULKV}},
  note         = {Machine review of arXiv:2608.02284}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Open-vocabulary segmentation identifies and segments objects from arbitrary textual descriptions. SAM 3 supports noun-phrase-guided segmentation and achieves competitive open-vocabulary performance through exhaustive vocabulary traversal, yet suffers from prohibitive computational overhead as target categories scale. In this paper, we propose an Efficient Open-Vocabulary segmentation framework with SAM 3 (EOVSAM), which adapts SAM 3 for single-pass prediction. EOVSAM removes prompt conditioning to turn SAM 3 into an efficient mask generator and introduces a new Attentional Aggregation strategy to optimize open-vocabulary classification end-to-end. This formulation avoids the multi-stage pipelines and post-processing heuristics commonly used by existing methods, while mitigating the closed-set collapse that can arise when classification is optimized directly. EOVSAM consistently improves segmentation accuracy over vanilla SAM 3 on all evaluated datasets and accelerates inference by up to 338$\times$. Furthermore, EOVSAM maintains high accuracy at lower resolutions while achieving even more remarkable inference speeds. Experiments on standard semantic and panoptic segmentation benchmarks show that EOVSAM combines competitive or state-of-the-art accuracy with a substantial speed advantage over existing open-vocabulary segmentation models. Code and models are available at https://github.com/hustvl/EOVSAM.

Figures

Figures reproduced from arXiv: 2608.02284 by Haomin Peng, Shiguo Lian, Xiaojie Jin, Xinggang Wang, Yongkang Li, Yunchao Wei, Zhaoxiang Liu.

Figure 1
Figure 1. Figure 1: Inference speed and segmentation performance on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Adapting SAM 3 for open-vocabulary segmentation. EOVSAM replaces the SAM 3 image encoder with an agglom [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of feature aggregation strategies re [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results of our proposed model. We present visualizations of the masks and attention maps predicted by [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 6 linked inside Pith

  1. [4]

    In2024 IEEEInternationalConferenceonImageProcessing(ICIP), 2494–2500

    Open- Vocabulary Panoptic Segmentation Using Bert Pre-Training of Vision-Language Multiway Transformer Model. In2024 IEEEInternationalConferenceonImageProcessing(ICIP), 2494–2500. IEEE. Cheng,B.;Misra,I.;Schwing,A.G.;Kirillov,A.;andGird- har,R.2022. Masked-attentionmasktransformerforuniver- sal image segmentation. InProceedings of the IEEE/CVF conference ...

  2. [9]

    Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; Rädle, R.; Rolland, C.; Gustafson, L.; Mintun, E.; Pan, J.; Alwala, K

    C-RADIOv4 (Tech Report).arXiv preprint arXiv:2601.17237. Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; Rädle, R.; Rolland, C.; Gustafson, L.; Mintun, E.; Pan, J.; Alwala, K. V.; Carion, N.; Wu, C.-Y.; Girshick, R.; Dollar, P.; and Feichtenhofer, C

  3. [10]

    In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds.,International Conference on Learning Representations, volume 2025, 28085–28128

    SAM 2: Segment Anything in Images and Videos. In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds.,International Conference on Learning Representations, volume 2025, 28085–28128. Shan, X.; Wu, D.; Zhu, G.; Shao, Y.; Sang, N.; and Gao, C

  4. [11]

    Tschannen, M.; Gritsenko, A.; Wang, X.; Naeem, M

    Dinov3.arXiv preprint arXiv:2508.10104. Tschannen, M.; Gritsenko, A.; Wang, X.; Naeem, M. F.; Alabdulmohsin, I.; Parthasarathy, N.; Evans, T.; Beyer, L.; Xia, Y.; Mustafa, B.; et al

  5. [12]

    Xiao, S.; Kabra, R.; Li, Y.; Lee, D.; Carreira, J.; and Panda, P

    Siglip 2: Multilingual vision-language encoders with improved semantic under- standing, localization, and dense features.arXiv preprint arXiv:2502.14786. Xiao, S.; Kabra, R.; Li, Y.; Lee, D.; Carreira, J.; and Panda, P

  6. [2014]

    InEuropean conference on computer vision, 740–755

    Microsoft coco: Common objects in context. InEuropean conference on computer vision, 740–755. Springer. Loshchilov,I.;andHutter,F.2017. Decoupledweightdecay regularization.arXiv preprint arXiv:1711.05101. Mottaghi, R.; Chen, X.; Liu, X.; Cho, N.-G.; Lee, S.-W.; Fidler, S.; Urtasun, R.; and Yuille, A

  7. [2015]

    Han,C.;Zhong,Y.;Li,D.;Han,K.;andMa,L.2023

    The pascal vi- sual object classes challenge: A retrospective.International journal of computer vision, 111(1): 98–136. Han,C.;Zhong,Y.;Li,D.;Han,K.;andMa,L.2023. Open- vocabulary semantic segmentation with decoupled one-pass network. InProceedings of the IEEE/CVF International Conference on Computer Vision, 1086–1096. He,K.;Gkioxari,G.;Dollár,P.;andGirsh...

  8. [2018]

    InProceedings of the IEEEconferenceoncomputervisionandpatternrecognition, 1209–1218

    Coco-stuff: Thing and stuff classes in context. InProceedings of the IEEEconferenceoncomputervisionandpatternrecognition, 1209–1218. Carion, N.; Gustafson, L.; Hu, Y.-T.; Debnath, S.; Hu, R.; Suris, D.; Ryali, C.; Alwala, K. V.; Khedr, H.; Huang, A.; Lei, J.; Ma, T.; Guo, B.; Kalla, A.; Marks, M.; Greer, J.; Wang, M.; Sun, P.; Rädle, R.; Afouras, T.; Mavr...

  9. [2021]

    InInternational conference on ma- chine learning, 4904–4916

    Scaling up visual and vision-language representation learning with noisy text supervision. InInternational conference on ma- chine learning, 4904–4916. PMLR. Jiao, S.; Zhu, H.; Huang, J.; Zhao, Y.; Wei, Y.; and Shi, H. 2024.Collaborativevision-textrepresentationoptimizingfor open-vocabulary segmentation. InEuropean Conference on Computer Vision, 399–416. ...

  10. [2023]

    InProceed- ingsoftheIEEE/CVFinternationalconferenceoncomputer vision, 11975–11986

    Sigmoid loss for language image pre-training. InProceed- ingsoftheIEEE/CVFinternationalconferenceoncomputer vision, 11975–11986. Zhou, B.; Zhao, H.; Puig, X.; Xiao, T.; Fidler, S.; Barriuso, A.;andTorralba,A.2019. Semanticunderstandingofscenes through the ade20k dataset.International journal of com- puter vision, 127(3): 302–321

  11. [2024]

    Chen, Y.-C.; Li, W.-H.; and Chen, C.-S

    Frozenseg:Harmonizingfrozenfoundationmodelsforopen- vocabularysegmentation.arXivpreprintarXiv:2409.03525. Chen, Y.-C.; Li, W.-H.; and Chen, C.-S

  12. [2025]

    InProceedings of the Computer Vision and Pattern Recognition Conference, 26081–26090

    Effective SAM combination for open-vocabulary semantic segmentation. InProceedings of the Computer Vision and Pattern Recognition Conference, 26081–26090. Li, B.; Zhang, D.; Zhao, Z.; Gao, J.; and Li, X. 2025a. FGAseg: Fine-Grained Pixel-Text Alignment for Open-Vocabulary Semantic Segmentation.arXiv preprint arXiv:2501.00877. Li,J.;Lu,Y.;Zhang,Y.;Wang,F.;...

  13. [2026]

    Chen,L.-C.;Zhu,Y.;Papandreou,G.;Schroff,F.;andAdam, H.2018

    SAM-MI:AMask-injectedFrameworkforEnhancing Open-vocabulary Semantic Segmentation with SAM.Ma- chine Intelligence Research, 1–17. Chen,L.-C.;Zhu,Y.;Papandreou,G.;Schroff,F.;andAdam, H.2018. Encoder-decoderwithatrousseparableconvolution for semantic image segmentation. InProceedings of the Europeanconferenceoncomputervision(ECCV),801–818. Chen, X.; Yang, H....

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.