REVIEW 4 major objections 5 minor 32 references
Beyond-Labels: Advancing Open-Vocabulary Segmentation With Vision-Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that a lightweight fusion module trained only on base classes reaches 49.4 mIoU on PASCAL-5i, beating one-shot baselines.
desk verdict Plausible ablation, unsupported headline: the zero-vs-one-shot comparison undermines the 49.4 claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the "Beyond-Labels" multimodal embedding fusion module: a stack of transformer encoder layers that takes channel-aligned visual patches and text-category embeddings and updates both through self-attention, so each modality's features are refined by the other. The second mechanism is the Fourier positional embedding $f_{emb}(x,y)$, added to each image patch before the frozen vision encoder, which the paper claims gives smoother, more generalizable spatial encoding than learned position embeddings. The final mechanism is a hierarchical visual decoder that upsamples fused features and compares them with text features by cosine similarity to produce per-category masks.
What would settle it
Run the full BL2 configuration but replace the unspecified $f_{emb}(x,y)$ with a standard, fully specified sinusoidal positional encoding at the same patch grid; if the mIoU stays near 49.4, the claimed Fourier-embedding benefit is not what carries the result, and if it drops toward BL0's 34.0, the missing specification becomes the decisive variable. Either way, the comparison would be settled only after the paper releases the exact definition of $f_{emb}$.
Extended reading notes
Core claim
In the paper's own terms, the central discovery is that a simple, lightweight multimodal fusion module can turn frozen CLIP visual and textual features into an open-vocabulary segmenter that outperforms established few-shot baselines on PASCAL-5i. The visual encoder is frozen, positional information comes from Fourier embeddings rather than pre-trained position embeddings, and the fusion module uses alternating self-attention to update visual and text features so they align in a shared space. Segmentation masks are produced by cosine similarity between upsampled visual features and text embeddings, followed by thresholded, temperature-scaled sigmoid predictions. The reported mIoU of 49.4 across the four folds is the paper's headline evidence that this combination generalizes to unseen categories.
Load-bearing premise
The whole reported advantage rests on the Fourier positional embedding $f_{emb}(x,y)$ in Eq. (2), whose formula, frequency scale, and dimension are never given in the paper; if that embedding is not in fact the source of the 49.4 mIoU, the ablation story and the headline comparison lose their support.
Editorial extensions
If this is right
- A frozen CLIP backbone can be adapted to open-vocabulary segmentation with only a small amount of segmentation data; the fusion module is the only part that needs training.
- Fourier positional embeddings alone raise mIoU from 34.0 to 39.5 in the paper's ablation, suggesting they generalize better than learned positional embeddings in this setting.
- The zero-shot Beyond-Labels model beats one-shot baselines like PANet on PASCAL-5i, implying support-set examples are not necessary for this benchmark's novel classes.
- Because both encoders stay frozen, the same fusion module could be reused with future, stronger vision-language backbones without re-training the whole model.
Reading between the lines
- The paper never specifies the formula, frequency scale, or feature dimension of the Fourier embedding in Eq. (2); the reported gain is therefore not reproducible from the text, and the 10-point ablation jump could be sensitive to those hidden choices.
- The comparison in Table I pits a zero-shot model against one-shot methods; a more direct test would compare against zero-shot baselines trained on the same frozen CLIP features, which the paper does not provide.
- If the recipe transfers to other backbones and datasets, it would suggest a general pattern: large frozen multimodal encoders plus a small learned alignment layer may be enough for dense open-vocabulary prediction, which would lower the cost of adapting foundation models to new tasks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Beyond-Labels, a lightweight transformer-based fusion module that combines frozen CLIP visual and text features for open-vocabulary semantic segmentation. The method replaces learned positional embeddings with Fourier embeddings and uses a hierarchical decoder to produce segmentation masks via cosine similarity between visual and textual features. On the PASCAL-5i benchmark the paper reports a mean IoU of 49.4, claiming to outperform both zero-shot and one-shot baselines. Ablation studies show consistent improvements when adding Fourier embeddings and the fusion module, from 34.0 mIoU (BL0) to 39.5 (BL1) to 49.4 (BL2).
Significance. If the reported results were fully substantiated, the work would offer a simple and parameter-efficient way to adapt frozen vision-language models for open-vocabulary segmentation with small amounts of segmentation data. The ablation design is a strength: it isolates the contribution of Fourier embeddings and fusion, and the method is explicitly zero-shot, which is a useful point of comparison. However, the significance is currently not established because the headline comparison in Table I mixes one-shot and zero-shot protocols, the Fourier embedding is never specified, and essential implementation details are missing. The manuscript also appears to select the final configuration on test-fold mIoU, further weakening the central claim. The ideas are plausible, but as submitted the evidence does not support the claimed state-of-the-art result.
major comments (4)
- [Section IV-C, Table I] The headline claim that the proposed zero-shot model 'outperforms all baselines' is not supported by the table as presented, because the comparison mixes one-shot baselines (LogReg, Siamese, Finetuning, 1-NN, OSLSM, CO-FCN, AMP-2, SG-One, PANet) with zero-shot methods (SPNet, ZS3Net) and reports 'Ours' as zero-shot. The paper does not state whether the baseline numbers were re-run under the proposed zero-shot protocol, whether the query images and class splits are identical to those used by the one-shot baselines, or how the support set is handled for the one-shot methods while the proposed method never consumes a support set. Without this information, the comparison is not apples-to-apples and the 49.4 mIoU does not substantiate the claim.
- [Section III-B, Eq. (2)] The Fourier embedding f_emb(x,y) is never defined: there is no formula, no specification of the frequency scale or number of frequency components, no output dimension, and no statement of how it is combined with the patch embedding. This term is essential to the method's claimed generalization benefit (BL1 vs. BL0 in Table III), and a reader cannot implement or reproduce the method from the text. Please provide the exact definition and architectural details.
- [Section III-E, Section IV-D, Table III] Several essential experimental details are missing: the temperature tau in the sigmoid, the class-wise thresholds, the fusion transformer depth/heads/MLP dimensions, the decoder architecture, the CLIP backbone variant, patch size, and input resolution. In addition, the final configuration BL2 appears to be selected on the test-fold mIoU reported in Table III, which is a form of test-set fitting. Please specify the validation procedure and report means and variances over multiple runs or episodic samples.
- [Section IV-A] The dataset section describes PASCAL-5i as a few-shot episodic benchmark with support and query sets, but the proposed method has no support branch and is evaluated as zero-shot. The paper should state exactly how the zero-shot evaluation is conducted: which images are used as queries, whether all images in a fold are used, how the class-wise threshold is set without validation labels, and how the reported fold numbers relate to the episodic protocol. This ambiguity makes the reported mIoU difficult to interpret or compare.
minor comments (5)
- [Section IV-C] The statement 'The worst scores belonged to Folds 2 and 3' is inconsistent with Table I, where SPNet's lowest fold score is 14.1 on fold 2 and fold 3 is 18.3.
- [Appendix A] The template list contains a duplicate of 'An image of the {category} within the context', and the phrase 'This falls under one {category} within the context' is grammatically awkward.
- [References] Reference [6] is garbled: the author list 'Jacob Devlin, Ming-Wei Chang, Kenton, and Lee Kristina Toutanova' should be corrected to the standard BERT citation.
- [Sections III-B and III-C] CLIP is described as 'a self-supervised language model' when it is in fact a contrastive vision-language model; the terminology should be corrected.
- [Author affiliations] The email address for Seung-Ik Lee appears as 'the silee@etri.re.kr' with a stray 'the '.
Circularity Check
No circularity found: the paper is an empirical learning-system evaluation; its benchmark results and ablations are not forced by construction or by self-citation.
full rationale
The paper proposes a fusion module trained on PASCAL-5i base classes and reports mIoU on held-out folds; this is an empirical generalization claim, not a derivation. The only explicit formula, Eq. (2), defines the position-aware patch embedding Xi = xi + f_emb(x,y), an architectural construction; the paper never claims this equation by itself predicts the reported 49.4 mIoU. The headline comparison in Table I is open to a protocol critique -- one-shot baselines receive labeled support images while the proposed method is labeled zero-shot, and the Fourier embedding, temperatures, and thresholds are underspecified -- but a mixed or underspecified evaluation protocol is a correctness/comparability risk, not circularity, because the reported numbers are not constructed from the inputs. Table III is an ablation: each configuration is trained and evaluated, and the best configuration is reported; selecting the empirically best variant after evaluation is model selection, not a fitted parameter renamed as a prediction. The paper invokes no load-bearing self-citation: its citations to Fourier features (Tancik et al.), CLIP, PANet, etc., are external prior work, and none of the paper's premises are justified by prior work of the same authors. No equation in the paper equates an output with a fitted input, and no uniqueness theorem or self-definitional construction is used to force the choice of Fourier embeddings or fusion. Consequently, no circular step meets the required evidence bar.
Assumptions & free parameters
free parameters (4)
- Temperature tau in sigmoid
- Fourier feature scale and dimension
- Fusion transformer architecture (layers, heads, MLP dims)
- Class-wise threshold
assumptions (3)
- domain assumption CLIP contrastive training aligns visual and text features in a shared embedding space.
- standard math Fourier feature mappings improve generalization for spatial encoding.
- domain assumption Training on PASCAL-5i base classes yields a fusion module that transfers to novel classes.
Cite this review
Pith. "Pith review of Beyond-Labels: Advancing Open-Vocabulary Segmentation With Vision-Language Models." pith.science (2026). https://pith.science/paper/6JZ2QQSY
@misc{pith2026250116769,
author = {Pith},
title = {Pith review of: Beyond-Labels: Advancing Open-Vocabulary Segmentation With Vision-Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/6JZ2QQSY}},
note = {Machine review of arXiv:2501.16769}
}
read the original abstract
Open-vocabulary semantic segmentation attempts to classify and outline objects in an image using arbitrary text labels, including those unseen during training. Self-supervised learning resolves numerous visual and linguistic processing problems when effectively trained. This study investigates simple yet efficient methods for adapting previously learned foundation models for open-vocabulary semantic segmentation tasks. Our research proposes "Beyond-Labels", a lightweight transformer-based fusion module that uses a small amount of image segmentation data to fuse frozen visual representations with language concepts. This strategy allows the model to leverage the extensive knowledge of pre-trained models without requiring significant retraining, making the approach data-efficient and scalable. Furthermore, we capture positional information in images using Fourier embeddings, improving generalization and enabling smooth and consistent spatial encoding. We perform thorough ablation studies to examine the main components of our proposed method. On the standard benchmark PASCAL-5i, the method performs better despite being trained on frozen vision and language representations. Index Terms: Beyond-Labels, open-vocabulary semantic segmentation, Fourier embeddings, PASCAL-5i
Figures
Reference graph
Works this paper leans on
-
[1]
Flamingo: A visual language model for few- shot learning
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, and Malcolm Reynolds et al. Flamingo: A visual language model for few- shot learning. arXiv preprint arXiv:2204.14198 , 2022
arXiv 2022
-
[2]
Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell et al
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell et al. Language models are few-shot learners. In Adv. Neural Inform. Process. Syst. , volume 33, pages 1877–1901, 2020
work page 1901
-
[3]
Zero-shot semantic segmentation
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick P ´erez. Zero-shot semantic segmentation. In Adv. Neural Inform. Process. Syst., volume 32, 2019
work page 2019
-
[4]
Emerging properties in self-supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J ´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Int. Conf. Comput. Vis. , pages 9650–9660, 2021
work page 2021
-
[5]
An empirical study of training self-supervised vision transformers
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Int. Conf. Comput. Vis. , pages 9640–9649, 2021
work page 2021
-
[6]
Bert: Pre-training of deep bidirectional transformers for language un- derstanding
Jacob Devlin, Ming-Wei Chang, Kenton, and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language un- derstanding. In NAACL-HLT, pages 4171–4186, 2019
work page 2019
-
[7]
Palm: Scaling language modeling with pathways
Aakanksha Chowdhery et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022
arXiv 2022
-
[8]
On the opportunities and risks of foundation models
Rishi Bommasani et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021
arXiv 2021
Show all 32 references
-
[9]
The pascal visual object classes (voc) challenge
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision , 88(2):303–338, 2010
2010
-
[10]
Bootstrap your own latent-a new approach to self-supervised learning
Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, and Mohammad Gheshlaghi Azar et al. Bootstrap your own latent-a new approach to self-supervised learning. In Adv. Neural ...
2020
-
[11]
Context-aware feature generation for zero-shot semantic segmentation
Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In ACM Int. Conf. Multimedia , pages 1921–1929, 2020
1921
-
[12]
Simultaneous detection and segmentation
Bharath Hariharan, Pablo Arbelaez, Ross Girshick, and Jitendra Ma- lik. Simultaneous detection and segmentation. In Proceedings of the European Conference on Computer Vision , pages 297–312, 2014
2014
-
[13]
Perceiver io: A general architecture for structured inputs & outputs
Andreas Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Jo ˜ao Carreira. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint, arXiv:2107.14795, 2021
2021 arXiv
-
[14]
Scaling up visual and vision-language representation learning with noisy text supervision
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Int. Conf. Learn. Represent. , pages 4904–4916, 2021
2021
-
[15]
Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization
Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In IEEE Conf. Comput. Vis. Pattern Recog., 2022
2022
-
[16]
Srinivasan, Matthew Tancik, Jonathan T
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Bar- ron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neu- ral radiance fields for view synthesis. arXiv preprint, arXiv:2003.08934, 2020
2003 arXiv
-
[17]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark et al. Learning transferable visual models from natural language supervision. In Int. Conf. Learn. Represent., pages 8748–8763, 2021
2021
-
[18]
Conditional networks for few-shot semantic segmenta- tion
Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, and Sergey Levine. Conditional networks for few-shot semantic segmenta- tion. In International Conference on Learning Representations Workshop (ICLRW), 2018
2018
-
[19]
Vision trans- formers for dense prediction
Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision trans- formers for dense prediction. In Int. Conf. Comput. Vis. , pages 12159– 12168, 2021
2021
-
[20]
U-net: Convolu- tional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolu- tional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention , pages 234–241, 2015
2015
-
[21]
One-shot learning for semantic segmentation
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. 2017
2017
-
[22]
Fully convolutional networks for semantic segmentation
Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 39:640–651, 2017
2017
-
[23]
Unsupervised salient object detection with spectral cluster voting
Gyungin Shin, Samuel Albanie, and Weidi Xie. Unsupervised salient object detection with spectral cluster voting. In CVPR L3DIVU Work- shop, 2022
2022
-
[24]
Oreshkin, and Martin Jagersand
Mennatullah Siam, Boris N. Oreshkin, and Martin Jagersand. Amp: Adaptive masked proxies for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 5249–5258, 2019
2019
-
[25]
Implicit neural representations with periodic activation functions
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 7531–7542, 2020
2020
-
[26]
Fourier features let networks learn high frequency functions in low dimensional domains
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Adv. Neural Inform. Process. Syst., 2020
2020
-
[27]
Menick, Serkan Cabi, S.M
Maria Tsimpoukelli, Jacob L. Menick, Serkan Cabi, S.M. Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Adv. Neural Inform. Process. Syst. , volume 34, pages 200–212, 2021
2021
-
[28]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017
2017
-
[29]
Panet: Few-shot image semantic segmentation with prototype alignment
Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 9197–9206, 2019
2019
-
[30]
Semantic projection network for zero-and few-label semantic segmentation
Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 8256–8265, 2019
2019
-
[31]
Socratic models: Com- posing zero-shot multimodal reasoning with language
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, and Vincent Vanhoucke et al. Socratic models: Com- posing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022
2022 arXiv
-
[32]
Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S. Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions on Cybernetics , 2020
2020
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.