{"id":"e9adec58-a8a3-47c2-8486-8b165902b90a","arxiv_id":"2501.16769","paper_version":5,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A frozen-CLIP fusion module with Fourier positional embeddings reports 49.4 mIoU on PASCAL-5i, above the zero-shot baselines but under-specified and not compared to modern open-vocabulary methods.","lead":"This paper introduces a lightweight fusion module that connects a frozen CLIP image encoder and a frozen CLIP text encoder to segment objects from categories never seen during training. It reports a 49.4 mean IoU on the PASCAL-5i benchmark, beating the zero-shot baselines it compares against, but the evaluation mixes one-shot methods and lacks error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 49.4 mIoU claim depends on comparing a zero-shot method against one-shot baselines without a shared evaluation protocol; the comparison is not established as apples-to-apples.","rationale":"The reader's weakest assumption, the under-specified Fourier embedding in Eq. (2), is real: without a definition of f_emb, the method cannot be reproduced, and the reported gains could hinge on that unspecified choice. However, the more load-bearing concern for the paper's central claim is the evaluation protocol. The strongest claim is about outperforming all baselines on PASCAL-5i. Table I lists baselines that use 1-shot support examples; the proposed method is labeled zero-shot and the text never describes using support. Comparing zero-shot against one-shot is potentially valid only if the query set and metric are identical, but the paper does not establish this. It does not say how the query images for 'Ours' were selected, whether the baseline numbers are re-used from their original papers under their own episodic protocols, or how thresholding/temperature are chosen. This ambiguity directly affects whether the reported 49.4 mIoU is comparable to PANet's 49.1 or ZS3Net's 38.3. If the proposed method is evaluated on a different set of images or with a more permissive protocol, the claimed superiority could be an artifact. I therefore agree with the reader's REJECT verdict, but for a reason that is more fundamental than the missing Fourier formula. The concrete test I propose—recomputing mIoU under a shared evaluation protocol—would settle whether the concern lands. If the number survives that test, the paper would still need the Fourier details and training hyperparameters to be reproducible, but the central comparison would be on firmer ground. This is why I mark agreement as partial: the reader correctly identified a severe reproducibility gap, but the protocol mismatch is the more load-bearing flaw.","tokens_in":8548,"tokens_out":7045,"duration_ms":65425,"concrete_test":"Obtain the official PASCAL-5i evaluation toolkit and the authors' code or predicted masks; recompute per-fold mIoU using exactly the same query images, class splits, and metric code used for the 1-shot baselines (e.g., PANet's 49.1). If the recomputed mIoU differs from 49.4 or falls below PANet's 49.1, the protocol mismatch is confirmed and the headline claim fails. If no code or masks are available, the claim is not verifiable and the comparison remains uncontrolled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Beyond-Labels outperforms all baselines rests on Table I, which mixes protocols: PANet, SG-One, AMP-2, CO-FCN, and others are 1-shot methods that receive labeled support images for the novel classes at inference, while 'Ours' is labeled zero-shot and the paper's method description never uses a support set. The paper does not state whether the baseline numbers were re-run under the proposed method's protocol, whether the query images and class splits are identical to those used by the one-shot baselines, or how the class-wise threshold and temperature are set. The dataset section describes a support set, but the method never consumes it, so the evaluation setting for the proposed method is ambiguous. If the zero-shot evaluation is done on all images containing novel classes without episodic support, it is not directly comparable to the reported 1-shot mIoU values. The missing Fourier embedding formula is a genuine reproducibility problem, but even a fully specified Fourier embedding would not settle whether the headline comparison is valid. The protocol mismatch is logically prior: without a shared evaluation protocol, the 49.4 mIoU does not substantiate 'outperforms all baselines.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Beyond-Labels, a lightweight transformer-based fusion module that combines frozen CLIP visual and text features for open-vocabulary semantic segmentation. The method replaces learned positional embeddings with Fourier embeddings and uses a hierarchical decoder to produce segmentation masks via cosine similarity between visual and textual features. On the PASCAL-5i benchmark the paper reports a mean IoU of 49.4, claiming to outperform both zero-shot and one-shot baselines. Ablation studies show consistent improvements when adding Fourier embeddings and the fusion module, from 34.0 mIoU (BL0) to 39.5 (BL1) to 49.4 (BL2).","tokens_in":8754,"tokens_out":8436,"duration_ms":72005,"significance":"If the reported results were fully substantiated, the work would offer a simple and parameter-efficient way to adapt frozen vision-language models for open-vocabulary segmentation with small amounts of segmentation data. The ablation design is a strength: it isolates the contribution of Fourier embeddings and fusion, and the method is explicitly zero-shot, which is a useful point of comparison. However, the significance is currently not established because the headline comparison in Table I mixes one-shot and zero-shot protocols, the Fourier embedding is never specified, and essential implementation details are missing. The manuscript also appears to select the final configuration on test-fold mIoU, further weakening the central claim. The ideas are plausible, but as submitted the evidence does not support the claimed state-of-the-art result.","major_comments":[{"comment":"The headline claim that the proposed zero-shot model 'outperforms all baselines' is not supported by the table as presented, because the comparison mixes one-shot baselines (LogReg, Siamese, Finetuning, 1-NN, OSLSM, CO-FCN, AMP-2, SG-One, PANet) with zero-shot methods (SPNet, ZS3Net) and reports 'Ours' as zero-shot. The paper does not state whether the baseline numbers were re-run under the proposed zero-shot protocol, whether the query images and class splits are identical to those used by the one-shot baselines, or how the support set is handled for the one-shot methods while the proposed method never consumes a support set. Without this information, the comparison is not apples-to-apples and the 49.4 mIoU does not substantiate the claim.","section":"Section IV-C, Table I"},{"comment":"The Fourier embedding f_emb(x,y) is never defined: there is no formula, no specification of the frequency scale or number of frequency components, no output dimension, and no statement of how it is combined with the patch embedding. This term is essential to the method's claimed generalization benefit (BL1 vs. BL0 in Table III), and a reader cannot implement or reproduce the method from the text. Please provide the exact definition and architectural details.","section":"Section III-B, Eq. (2)"},{"comment":"Several essential experimental details are missing: the temperature tau in the sigmoid, the class-wise thresholds, the fusion transformer depth/heads/MLP dimensions, the decoder architecture, the CLIP backbone variant, patch size, and input resolution. In addition, the final configuration BL2 appears to be selected on the test-fold mIoU reported in Table III, which is a form of test-set fitting. Please specify the validation procedure and report means and variances over multiple runs or episodic samples.","section":"Section III-E, Section IV-D, Table III"},{"comment":"The dataset section describes PASCAL-5i as a few-shot episodic benchmark with support and query sets, but the proposed method has no support branch and is evaluated as zero-shot. The paper should state exactly how the zero-shot evaluation is conducted: which images are used as queries, whether all images in a fold are used, how the class-wise threshold is set without validation labels, and how the reported fold numbers relate to the episodic protocol. This ambiguity makes the reported mIoU difficult to interpret or compare.","section":"Section IV-A"}],"minor_comments":[{"comment":"The statement 'The worst scores belonged to Folds 2 and 3' is inconsistent with Table I, where SPNet's lowest fold score is 14.1 on fold 2 and fold 3 is 18.3.","section":"Section IV-C"},{"comment":"The template list contains a duplicate of 'An image of the {category} within the context', and the phrase 'This falls under one {category} within the context' is grammatically awkward.","section":"Appendix A"},{"comment":"Reference [6] is garbled: the author list 'Jacob Devlin, Ming-Wei Chang, Kenton, and Lee Kristina Toutanova' should be corrected to the standard BERT citation.","section":"References"},{"comment":"CLIP is described as 'a self-supervised language model' when it is in fact a contrastive vision-language model; the terminology should be corrected.","section":"Sections III-B and III-C"},{"comment":"The email address for Seung-Ik Lee appears as 'the silee@etri.re.kr' with a stray 'the '.","section":"Author affiliations"}],"recommendation":"reject","confidential_remarks":"The paper is a very early-stage manuscript with substantial reproducibility gaps. The protocol mismatch in Table I is a load-bearing flaw: the central claim of outperforming all baselines is not supported by an apples-to-apples comparison. The missing Fourier embedding definition and the test-set selection issue further undermine confidence. If the authors were to revise, they would need to re-run the evaluation under a clearly defined zero-shot protocol, provide full implementation details, and temper the claims; in its current form, the manuscript does not meet the standard for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The ablation is the most honest part of this paper, but the headline 49.4 mIoU is not trustworthy because it compares zero-shot against one-shot baselines without a shared protocol.\n\nWhat's new: applying Fourier positional embeddings and a small transformer fusion module to frozen CLIP features is a simple recipe; each ingredient is known, but the specific combination on PASCAL-5i is new. The paper is data-efficient and easy to port. The ablation shows a clean monotonic jump: +5.5 mIoU from Fourier embeddings, +9.9 from adding fusion. That's a real signal that the components do something.\n\nSoft spots: the main comparison is muddled. Table 1 lists 1-shot methods like PANet and SG-One that receive a labeled support image at evaluation, while the proposed method is zero-shot. If the baseline numbers are taken from the original papers, this is apples-to-oranges. The paper never explains whether baselines were re-run under its own protocol, and the dataset section's mention of a support set is confusing because the method never uses one. Also, the Fourier embedding in Eq. (2) is never defined—no frequency scale, no dimension, no formula. That makes the method irreproducible from text alone. There are no error bars, no hyperparameters, and no code.\n\nThe result itself is plausible; the gain over PANet is within a few points, and zero-shot methods often lag one-shot. But the missing details make it impossible to verify, and the protocol issue means the central claim is not established.\n\nWho it's for: researchers looking for a quick baseline in open-vocabulary segmentation might find the recipe useful. The ablation is worth reading. But this is not a scientifically rigorous advance.\n\nRecommendation: I'd send it to peer review if I were an editor, because the protocol issue needs an expert eye, but I'd expect heavy revision. For your own reading group, it's a maybe—the ablation is useful, but the protocol muddle is a cautionary tale.","headline":"Plausible ablation, unsupported headline: the zero-vs-one-shot comparison undermines the 49.4 claim.","tokens_in":9321,"tokens_out":2973,"would_cite":false,"duration_ms":25364,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a lightweight fusion module trained only on base classes reaches 49.4 mIoU on PASCAL-5i, beating one-shot baselines.","keywords":["open-vocabulary semantic segmentation","vision-language models","Fourier embeddings","PASCAL-5i","zero-shot segmentation","transformer fusion module","frozen encoders","few-shot segmentation"],"falsifier":"Run the full BL2 configuration but replace the unspecified $f_{emb}(x,y)$ with a standard, fully specified sinusoidal positional encoding at the same patch grid; if the mIoU stays near 49.4, the claimed Fourier-embedding benefit is not what carries the result, and if it drops toward BL0's 34.0, the missing specification becomes the decisive variable. Either way, the comparison would be settled only after the paper releases the exact definition of $f_{emb}$.","tokens_in":8321,"feed_emoji":"🖼️","tokens_out":5097,"duration_ms":41266,"temperature":0.7,"pith_summary":"This paper tries to show that open-vocabulary semantic segmentation—labeling every pixel of an image with categories never seen during training—can be achieved by stitching together frozen, pre-trained vision and language models with a small trainable fusion module. The proposed \"Beyond-Labels\" module takes CLIP's frozen image and text features, fuses them through transformer self-attention layers, and adds Fourier-based positional embeddings instead of learned position embeddings. On the PASCAL-5i benchmark it reports an mIoU of 49.4, higher than the compared one-shot and zero-shot baselines, despite never seeing support-set examples. A sympathetic reader would care because the recipe is data-efficient and backbone-agnostic: it adapts existing foundation models without retraining them.","feed_headline":"Frozen CLIP plus a small fusion module hits 49.4 mIoU on PASCAL-5i","feed_subtitle":"A zero-shot segmenter trained only on base classes beats one-shot baselines by fusing frozen vision and text features.","key_machinery":"The load-bearing object is the \"Beyond-Labels\" multimodal embedding fusion module: a stack of transformer encoder layers that takes channel-aligned visual patches and text-category embeddings and updates both through self-attention, so each modality's features are refined by the other. The second mechanism is the Fourier positional embedding $f_{emb}(x,y)$, added to each image patch before the frozen vision encoder, which the paper claims gives smoother, more generalizable spatial encoding than learned position embeddings. The final mechanism is a hierarchical visual decoder that upsamples fused features and compares them with text features by cosine similarity to produce per-category masks.","core_discovery":"In the paper's own terms, the central discovery is that a simple, lightweight multimodal fusion module can turn frozen CLIP visual and textual features into an open-vocabulary segmenter that outperforms established few-shot baselines on PASCAL-5i. The visual encoder is frozen, positional information comes from Fourier embeddings rather than pre-trained position embeddings, and the fusion module uses alternating self-attention to update visual and text features so they align in a shared space. Segmentation masks are produced by cosine similarity between upsampled visual features and text embeddings, followed by thresholded, temperature-scaled sigmoid predictions. The reported mIoU of 49.4 across the four folds is the paper's headline evidence that this combination generalizes to unseen categories.","pith_inferences":["The paper never specifies the formula, frequency scale, or feature dimension of the Fourier embedding in Eq. (2); the reported gain is therefore not reproducible from the text, and the 10-point ablation jump could be sensitive to those hidden choices.","The comparison in Table I pits a zero-shot model against one-shot methods; a more direct test would compare against zero-shot baselines trained on the same frozen CLIP features, which the paper does not provide.","If the recipe transfers to other backbones and datasets, it would suggest a general pattern: large frozen multimodal encoders plus a small learned alignment layer may be enough for dense open-vocabulary prediction, which would lower the cost of adapting foundation models to new tasks."],"forward_implications":["A frozen CLIP backbone can be adapted to open-vocabulary segmentation with only a small amount of segmentation data; the fusion module is the only part that needs training.","Fourier positional embeddings alone raise mIoU from 34.0 to 39.5 in the paper's ablation, suggesting they generalize better than learned positional embeddings in this setting.","The zero-shot Beyond-Labels model beats one-shot baselines like PANet on PASCAL-5i, implying support-set examples are not necessary for this benchmark's novel classes.","Because both encoders stay frozen, the same fusion module could be reused with future, stronger vision-language backbones without re-training the whole model."],"supporting_citations":[{"why":"Supplies the frozen CLIP vision and text encoders that produce the visual and textual features the fusion module aligns.","marker":"[17]"},{"why":"Defines the PASCAL-5i benchmark, its four folds, and several of the one-shot baselines (LogReg, Siamese, Finetuning, 1-NN, OSLSM) used in Table I.","marker":"[21]"},{"why":"Source of the Fourier feature embedding idea that the paper adopts for positional encoding in place of learned position embeddings.","marker":"[26]"},{"why":"Zero-shot segmentation baseline (ZS3Net) that Beyond-Labels compares against and outperforms on PASCAL-5i.","marker":"[3]"},{"why":"PANet, the strongest one-shot baseline in Table I, whose 49.1 mIoU the paper's 49.4 result is measured against.","marker":"[29]"},{"why":"SPNet, a zero-shot baseline in the comparison table, used as a reference for the open-vocabulary setting.","marker":"[30]"},{"why":"Provides the transformer self-attention layers used in the multimodal fusion module.","marker":"[28]"}],"fun_headline_variants":["Frozen CLIP fusion hits 49.4 mIoU on PASCAL-5i","Lightweight fusion of frozen CLIP beats few-shot baselines","49.4 mIoU: Frozen CLIP plus tiny fusion for open-vocab segmentation","Frozen vision-language fusion beats few-shot segmenters on PASCAL-5i"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole reported advantage rests on the Fourier positional embedding $f_{emb}(x,y)$ in Eq. (2), whose formula, frequency scale, and dimension are never given in the paper; if that embedding is not in fact the source of the 49.4 mIoU, the ablation story and the headline comparison lose their support.","fun_headline_variants_meta":{"raw":{"variants":["Frozen CLIP fusion hits 49.4 mIoU on PASCAL-5i","Lightweight fusion of frozen CLIP beats few-shot baselines","49.4 mIoU: Frozen CLIP plus tiny fusion for open-vocab segmentation","Frozen vision-language fusion beats few-shot segmenters on PASCAL-5i"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2722,"prompt_tokens":891,"completion_tokens":1831,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1740}},"tokens_in":507,"tokens_out":1831,"duration_ms":12034,"temperature":1.0,"reasoning_tokens":1740,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T10:50:02.596739+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full BL2 configuration but replace the unspecified $f_{emb}(x,y)$ with a standard, fully specified sinusoidal positional encoding at the same patch grid; if the mIoU stays near 49.4, the claimed Fourier-embedding benefit is not what carries the result, and if it drops toward BL0's 34.0, the missing specification becomes the decisive variable. Either way, the comparison would be settled only after the paper releases the exact definition of $f_{emb}$.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP vision and text encoders that produce the visual and textual features the fusion module aligns."},{"cited_title":"One-shot learning for semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Defines the PASCAL-5i benchmark, its four folds, and several of the one-shot baselines (LogReg, Siamese, Finetuning, 1-NN, OSLSM) used in Table I."},{"cited_title":"Fourier features let networks learn high frequency functions in low dimensional domains","cited_arxiv_id":null,"evidence_quote":"Source of the Fourier feature embedding idea that the paper adopts for positional encoding in place of learned position embeddings."},{"cited_title":"Zero-shot semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Zero-shot segmentation baseline (ZS3Net) that Beyond-Labels compares against and outperforms on PASCAL-5i."},{"cited_title":"Panet: Few-shot image semantic segmentation with prototype alignment","cited_arxiv_id":null,"evidence_quote":"PANet, the strongest one-shot baseline in Table I, whose 49.1 mIoU the paper's 49.4 result is measured against."},{"cited_title":"Semantic projection network for zero-and few-label semantic segmentation","cited_arxiv_id":null,"evidence_quote":"SPNet, a zero-shot baseline in the comparison table, used as a reference for the open-vocabulary setting."},{"cited_title":"Attention is all you need","cited_arxiv_id":null,"evidence_quote":"Provides the transformer self-attention layers used in the multimodal fusion module."}],"review_version":1}