{"id":"99d54736-e0cc-4724-a501-fca715f2e129","arxiv_id":"2506.16058","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OpenBench, a new benchmark with categories semantically far from the COCO training space, shows that fine-tuning CLIP hurts open-vocabulary segmentation, and the proposed OVSNet method achieves state-of-the-art on both existing and new benchmarks.","lead":"This paper argues that current open-vocabulary segmentation benchmarks are too similar to the training data, so models look better than they are, and introduces OpenBench, a new test set built from categories far from the training semantics. It also proposes OVSNet, which fuses CLIP features with decoder features and augments training with synthetic proxy embeddings, reporting state-of-the-art results on both existing and new benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OpenBench's image-level filtering cannot guarantee category-level novelty; domain shift is a plausible alternative explanation for the observed performance divergence.","rationale":"The reader's weakest assumption is that OpenBench isolates semantic novelty from domain shift, and the paper indeed fails to control for this. My concrete objection sharpens the reader's point: the filtering operates on category text embeddings, not image content, so images from visually distinct domains (food close-ups, object-centric ImageNet-S crops, driving scenes) can enter OpenBench even when the visual content is not semantically novel; moreover the σ2 rule relabels familiar categories to 'others', injecting a candidate-set confound. I disagree slightly with the reader's framing that label granularity is the main extra confound; the relabeling-to-'others' mechanism is more directly tied to the paper's own construction and more cleanly testable. Nevertheless, the vulnerability and the requested remedy (reporting σ1/σ2, releasing or re-analyzing the benchmark) are the same. The OVSNet method is independent of this concern and its ablations (Tables 3 and 4) show consistent gains, so I do not see grounds for REJECT or UNVERDICTED. CONDITIONAL remains the correct verdict: the benchmark claim and the fine-tuning-hurts-generalization claim need the domain-shift control and threshold disclosure before they can be accepted as established.","tokens_in":13984,"tokens_out":1769,"duration_ms":17958,"concrete_test":"Re-split OpenBench into image subsets matched to the source domains (Food103, ImageNet-S, CamVid) and recompute Figure 1(b)'s frozen-versus-finetuned performance gap separately per subset. If the gap is negative (fine-tuning hurts) on the ImageNet-S subset but flips positive on CamVid after controlling for relabeled 'others' pixels, the observed divergence is at least partly a domain effect rather than purely semantic novelty. Additionally, report σ1 and σ2 and rerun the benchmark construction with a text-similarity-only criterion that excludes relabeling to 'others', to see whether the fine-tuning penalty persists.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that standard OVS benchmarks overestimate generalization and that fine-tuning CLIP hurts generalization rests entirely on OpenBench measuring semantic novelty rather than other distribution shifts. Section 3.2 filters each image by the minimum CLIP similarity among the categories present in that image, using unreported σ1 and σ2. This image-level filtering has a critical gap: an image passes the filter if every present category is far from COCO in CLIP text space, regardless of whether the image content itself is visually novel relative to COCO training images. Food103 close-ups, ImageNet-S object-centric crops, and CamVid driving scenes each differ from COCO in domain—capture style, scene layout, object scale, background, camera viewpoint—independently of semantic distance. Concretely, a category such as 'person' appearing in a CamVid driving image would have high CLIP similarity to the COCO class 'person' and would therefore be relabeled 'others' by the σ2 rule; but 'others' is not in the model's training vocabulary, so performance on OpenBench conflates (i) inability to recognize a novel class with (ii) inability to recognize a familiar class in an unfamiliar visual domain and (iii) inability to handle the absence of the true class from the candidate list. The reported mean similarity of 0.61 (Table 1) only establishes that the category set is novel; it says nothing about per-image visual domain shift. Since the finetuned-versus-frozen comparison in Figure 1(b) and the headline divergence claim are computed on OpenBench, the conclusion that fine-tuning CLIP is harmful specifically for semantic novelty is not established unless domain effects are controlled for.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Open-vocabulary segmentation models are typically trained on COCO and evaluated on VOC, Pascal Context, ADE20K, and Cityscapes. The paper argues that these test vocabularies lie close to the COCO semantic space in CLIP text-embedding space, so high scores on them do not demonstrate ability to segment genuinely novel categories. To remedy this, the authors construct OpenBench from Food103, ImageNet-S, and CamVid by keeping images and categories with low CLIP similarity to COCO, yielding 286 classes with an average maximum similarity of 0.61. They report that a fine-tuned-CLIP method (CATSeg) gains on existing benchmarks but falls below a frozen-CLIP method (SAN) on OpenBench, suggesting that fine-tuning harms open-domain generalization. They also propose OVSNet, which uses a gradient-free aggregation of CLIP and decoder features and a Proxy Calibration training strategy, and report state-of-the-art average mIoU on both existing sets and OpenBench.","tokens_in":14245,"tokens_out":12484,"duration_ms":138783,"significance":"The paper makes a useful and falsifiable observation: existing OVS benchmarks are semantically close to the training vocabulary, and Table 1 quantifies this in a transparent way. If OpenBench is validated and released, it can serve as a complementary stress test for OVS. The authors deserve credit for explicitly connecting this observation to the fine-tuning-versus-frozen CLIP behavior, and the method ablations in Table 3 show consistent gains from both proposed components. However, the benchmark's validity currently rests on unreported thresholds and an uncontrolled domain mix, and the fine-tuning conclusion is drawn from a cross-method comparison rather than a controlled ablation; these issues need to be addressed before the central claims can be fully accepted.","major_comments":[{"comment":"The thresholds σ1 and σ2 are never reported, the category list and manual filtering procedure are not described, and no URL or release plan for OpenBench is provided. Since the benchmark is defined by applying these thresholds, the reader cannot audit whether OpenBench truly isolates semantic novelty from other factors. The authors should report the threshold values, provide a sensitivity analysis, publish the complete category list, and release the construction code and annotations.","section":"Section 3.2"},{"comment":"The image-level filtering rule does not guarantee category-level novelty. An image passes the filter if it contains at least one category far from COCO, and all other categories with similarity above σ2 are relabeled 'others'. Because 'others' is not a real semantic concept and does not appear in any evaluated model's training vocabulary, a model that correctly recognizes a familiar COCO category in a novel image is penalized, while a model that merely guesses 'others' can be rewarded. This confounds open-vocabulary comprehension with the ability to abstain, so the mIoU gap in Figure 1(b) cannot be attributed solely to semantic novelty.","section":"Section 3.2"},{"comment":"The three source datasets (Food103, ImageNet-S, CamVid) introduce systematic visual domain shifts—close-up food photography, object-centric ImageNet crops, and driving scenes—that are entangled with semantic distance. A model fine-tuned on COCO may fail on OpenBench because of unfamiliar object scale, background, and viewpoint rather than because the categories are semantically new. The authors should provide a per-source performance breakdown, evaluate familiar COCO categories in shifted image domains, and ideally construct an image-domain-matched control set that varies only semantic distance.","section":"Section 3.2"},{"comment":"The central claim that fine-tuning CLIP hurts on OpenBench is inferred from a comparison of CATSeg (fine-tuned CLIP) with SAN (frozen CLIP). These methods differ in decoder architecture, feature aggregation, training loss, and other implementation choices, so the observed −3.5 gap on OpenBench is not a controlled test of fine-tuning. A within-architecture ablation that toggles only CLIP fine-tuning, reporting both existing and OpenBench results, is needed to support the paper's stated conclusion.","section":"Figure 1(b), Table 2"},{"comment":"The similarity statistics are computed against 'COCO categories', but the evaluated models are trained on different vocabularies (COCO-Stuff-171 vs COCO-Panoptic-133). The exact vocabulary and text templates used for the CLIP-L/14 similarity computation should be specified, and statistics should be reported for both candidate training vocabularies; otherwise the claim that OpenBench is novel relative to 'the training space' is ambiguous.","section":"Section 3.2, Table 1"},{"comment":"The fusion weight ω in Eqs. (1)–(4) is a core hyperparameter of the Gradient-Free Aggregation, but its default value is not reported, and no conditions are given for the convergence of the Neumann series in Eq. (4). The authors should report ω, specify the exact normalization used in Norm(Z), and verify or justify that the spectral radius of ω²A is less than one.","section":"Section 4.2, Section 5.2"}],"minor_comments":[{"comment":"The caption states that γ=1 is equivalent to not using PC, but Beta(1,1) is a uniform distribution and would still generate random convex combinations in Eq. (5) unless explicitly disabled; please clarify the implementation or correct the claim.","section":"Table 4(b)"},{"comment":"The 'Average Score' column is a simple unweighted mean over six datasets with very different class counts and difficulty levels; please report a defined aggregation or present per-dataset numbers so that the average is interpretable.","section":"Table 2"},{"comment":"The abstract states 'state-of-the-art results on both existing datasets and our proposed OpenBench', but Table 2 shows that OVSNet is below SCAN on VOC; the claim should be qualified to match the reported results.","section":"Abstract and Section 5.3"},{"comment":"The claim in Figure 5 that model performance degrades consistently as the number of inference categories increases is based on two methods without error bars; please add standard deviations across runs or clarify that the trend is qualitative.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark construction and validation are the more important contribution; the method part is secondary. The revision should prioritize releasing thresholds and data, adding a controlled fine-tuning ablation, and providing domain-shift analyses. The paper's self-citations are frequent but not overtly problematic; the authors should nevertheless double-check that all related work is covered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the benchmark observation is the real thing here. Table 1 and Figure 1(b) show a clear pattern—existing OVS test sets sit close to COCO in CLIP text space, and fine-tuning CLIP helps on those sets but loses ground on OpenBench. That is a useful caution to the field, and it is stated without overclaiming. The method, OVSNet, is more incremental: Gradient-Free Aggregation is a closed-form feature diffusion (Neumann series of a random-walk update) and Proxy Calibration is convex mixup in embedding space, both known ingredients. But the combination is new to OVS, and the ablations report consistent gains across existing benchmarks and OpenBench. The paper also honestly notes where it loses to SCAN on VOC.\n\nThe soft spots are real but not fatal. The benchmark construction thresholds σ1 and σ2 are not disclosed, and the benchmark itself is not released. That alone prevents anyone from auditing the central claim. More substantively, OpenBench mixes semantic novelty with domain shift. Food103 close-ups, ImageNet-S object-centric crops, and CamVid driving scenes differ from COCO in capture style, layout, scale, and background, independently of category distance. The σ2 rule relabels any image category that is too similar to a COCO class as 'others'—but 'others' is not in the training vocabulary, so performance conflates failing to recognize a new class with failing to recognize a familiar class in an unfamiliar domain and with confusion from an absent true class. The reported mean similarity of 0.61 says the category set is novel; it does not isolate semantic novelty from visual domain shift. So the strong claim that fine-tuning CLIP is specifically harmful for semantic novelty is not established. The paper itself acknowledges the test sets are 'limited in measuring models' comprehension', but the benchmark as constructed cannot yet support the fine-grained conclusion.\n\nOne smaller technical gap: the GFA equations involve ω, but the implementation section only sets λ and γ. The Neumann series requires a convergence condition (spectral radius of ω²A < 1) that is not checked. That is minor and fixable.\n\nThe circularity concern is mild. Yes, the benchmark is defined via CLIP text similarity and the method is designed to preserve CLIP's space, but the empirical divergence is measured on outputs independent of the method's parameters. Self-citations to SCAN and GKC are relevant prior work, not padding.\n\nWho is this for? OVS researchers and anyone building evaluation benchmarks. It deserves a serious referee, but the path to acceptance needs: release the benchmark, report the thresholds, and add a domain-shift analysis—for example, a per-domain breakdown or a control set with novel categories from the same domain as training. I would not desk-reject this; I would send it out with those asks.","headline":"OpenBench flips the usual OVS leaderboard and that is worth taking seriously, but the domain shift in the benchmark makes the headline claim about fine-tuning CLIP unproven; the method half is solid, incremental work.","tokens_in":14837,"tokens_out":2818,"would_cite":true,"duration_ms":31518,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-vocabulary segmentation benchmarks are semantically too close to the training space, so standard scores overstate generalization; the new OpenBench reverses the ranking and shows fine-tuning CLIP hurts on novel categories.","keywords":["open-vocabulary segmentation","benchmark evaluation","CLIP","semantic similarity","generalization","Proxy Calibration","Gradient-Free Aggregation","OpenBench"],"falsifier":"An experiment could use the same OpenBench images with an alternative label set whose class names are equally fine-grained but similar to COCO; if fine-tuned CLIP drops on that version too, the divergence is not about semantic distance, and if it drops only on the original labels, the benchmark's premise is supported.","tokens_in":13776,"feed_emoji":"🧩","tokens_out":10729,"duration_ms":115668,"temperature":0.7,"pith_summary":"The paper argues that current open-vocabulary segmentation benchmarks rank models within a semantic space that is almost the same as the training space, so high scores on them do not show that a model can handle genuinely novel concepts. It introduces OpenBench, a test set whose category names sit much farther from the training categories of COCO, so that every model must rely on retained vision-language generalization rather than memorized alignments. When the paper evaluates existing methods on OpenBench, the ranking reverses: fine-tuning CLIP, which reliably helps on standard benchmarks, hurts on OpenBench. The paper also proposes OVSNet, which preserves frozen-CLIP generalization through a gradient-free feature fusion and a proxy calibration training trick, and reports that this model improves performance both on standard sets and on OpenBench.","feed_headline":"New benchmark exposes overfitting in open-vocabulary segmentation","feed_subtitle":"On categories far from COCO training data, fine-tuned CLIP gets worse while frozen CLIP holds up.","key_machinery":"OpenBench is the central measuring object: candidate images from Food103, ImageNet-S, and CamVid are scored by the minimum CLIP text-similarity of their annotated categories to any COCO category, images above a threshold are dropped, and remaining near-training categories are relabeled 'others'. The method side has two named mechanisms. Proxy Calibration (PC) forms random convex combinations of class embeddings, sampled from a Beta distribution with gamma equal to 2, and applies distance supervision to those synthetic embeddings, widening the training space without new labels. Gradient-Free Aggregation (GFA) treats the decoder's query embeddings and mask-pooled CLIP features as two feature sets and alternately updates them through an affinity matrix, replacing the infinite iteration with a closed-form Neumann-series inverse so that no learned fusion layer can overfit to seen semantics.","core_discovery":"The paper claims that the field's standard evaluation of open-vocabulary segmentation is misleading because the test sets used everywhere live inside the training set's semantic neighborhood. On VOC, Pascal Context-59, and ADE20K-150, the average maximum CLIP similarity between test categories and COCO training categories is 0.97, 0.95, and 0.81 respectively; even ADE-847 and PC-459 score 0.79 and 0.84. OpenBench is built from Food103, ImageNet-S, and CamVid by filtering to images whose categories are far from COCO, giving a mean similarity of 0.61, and it includes an 'others' class so models cannot always guess from the candidate list. On this benchmark, methods that fine-tune CLIP, which gain steadily on standard sets, lose several mIoU points, while frozen-CLIP methods maintain their performance, so the standard ranking reverses. The proposed OVSNet, using proxy-calibrated training and gradient-free aggregation of CLIP and decoder features, achieves the highest average mIoU on both the standard sets and OpenBench.","pith_inferences":["Editorial inference: OpenBench's three source datasets bring their own visual domains, so the reported divergence could partially reflect food close-ups, object-centric images, and driving scenes rather than semantic distance alone; a single-domain control set would separate these factors.","Editorial inference: the filtering thresholds sigma-1 and sigma-2 are named but not reported, so an independent regeneration of OpenBench is not currently possible; publishing them would let the community audit the benchmark's construction.","Editorial inference: a more continuous evaluation would bin OpenBench categories by their COCO similarity and plot mIoU per bin, turning the benchmark into a stress curve for how gracefully a model degrades as semantic distance grows.","Editorial inference: the same similarity-filtering recipe could be used to build open-distribution test splits for other vision-language tasks such as retrieval or detection, not just segmentation."],"forward_implications":["If OpenBench is a fair test, rankings on standard OVS benchmarks should not be read as generalization rankings; a model that wins on VOC, Pascal Context, and ADE20K can still be among the worst on semantically novel categories.","Fine-tuning CLIP on the training distribution is a liability for truly open use: the same recipe that adds mIoU points on near-training benchmarks removes points on OpenBench, so preserving the pretrained vision-language alignment is the safer design choice.","The two proposed mechanisms are complementary: Proxy Calibration contributes +0.8/+2.9/+2.0 mIoU and Gradient-Free Aggregation +1.6/+1.7/+1.4 on ADE-150/PC-459/OpenBench, and together they add +2.7/+4.8/+2.6 over the baseline.","The number of candidate categories during inference changes scores substantially, so benchmark comparisons should control candidate-set size before attributing differences to semantic novelty."],"supporting_citations":[{"why":"Defines the vision-language embedding space used to measure how far each candidate category sits from the COCO training vocabulary.","marker":"[46]"},{"why":"COCO is the training semantic space; the benchmark filters by similarity to its categories.","marker":"[32]"},{"why":"CamVid supplies driving images whose categories are far from COCO training classes.","marker":"[3]"},{"why":"ImageNet-S supplies object-centric categories used to build OpenBench.","marker":"[14]"},{"why":"Food103 supplies food categories whose semantics and visual style differ from the training set.","marker":"[53]"},{"why":"Pascal VOC is one of the standard test sets whose high similarity to the training space the paper documents.","marker":"[13]"},{"why":"Pascal Context provides both the 59-class and 459-class test sets whose similarity statistics anchor the comparison.","marker":"[44]"},{"why":"ADE20K provides the 150-class and 847-class test sets whose semantic overlap with training is analyzed.","marker":"[66]"},{"why":"Represents the CLIP-fine-tuning route that improves on standard benchmarks but loses on OpenBench.","marker":"[10]"},{"why":"Represents the frozen-CLIP route that holds up on OpenBench, supplying the divergence shown in the paper.","marker":"[61]"}],"fun_headline_variants":["OpenBench: New test exposes CLIP fine-tuning pitfalls in segmentation","Benchmark shows fine-tuned CLIP fails on far-from-training categories","Open-vocabulary segmentation: Standard tests miss the real challenge","New benchmark reverses rankings: frozen CLIP beats fine-tuned on open sets","OVSNet pushes open-vocabulary segmentation beyond COCO's semantic shadow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"OpenBench's conclusions stand or fall on the assumption that CLIP similarity to COCO isolates semantic novelty; if the food, object-centric, and driving image domains themselves explain the performance drops, existing benchmarks may not overstate generalization.","fun_headline_variants_meta":{"raw":{"variants":["OpenBench: New test exposes CLIP fine-tuning pitfalls in segmentation","Benchmark shows fine-tuned CLIP fails on far-from-training categories","Open-vocabulary segmentation: Standard tests miss the real challenge","New benchmark reverses rankings: frozen CLIP beats fine-tuned on open sets","OVSNet pushes open-vocabulary segmentation beyond COCO's semantic shadow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0008,"raw_usage":{"total_tokens":3536,"prompt_tokens":983,"completion_tokens":2553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":2458}},"tokens_in":599,"tokens_out":2553,"duration_ms":18386,"temperature":1.0,"reasoning_tokens":2458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:44:12.066734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment could use the same OpenBench images with an alternative label set whose class names are equally fine-grained but similar to COCO; if fine-tuned CLIP drops on that version too, the divergence is not about semantic distance, and if it drops only on the original labels, the benchmark's premise is supported.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Defines the vision-language embedding space used to measure how far each candidate category sits from the COCO training vocabulary."},{"cited_title":"Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C","cited_arxiv_id":null,"evidence_quote":"COCO is the training semantic space; the benchmark filters by similarity to its categories."},{"cited_title":"Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla","cited_arxiv_id":null,"evidence_quote":"CamVid supplies driving images whose categories are far from COCO training classes."},{"cited_title":"Large-scale unsu- pervised semantic segmentation","cited_arxiv_id":null,"evidence_quote":"ImageNet-S supplies object-centric categories used to build OpenBench."},{"cited_title":"A large-scale benchmark for food im- age segmentation","cited_arxiv_id":null,"evidence_quote":"Food103 supplies food categories whose semantics and visual style differ from the training set."},{"cited_title":"The pascal visual object classes challenge: A retrospective.IJCV, 111:98–136, 2015","cited_arxiv_id":null,"evidence_quote":"Pascal VOC is one of the standard test sets whose high similarity to the training space the paper documents."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pascal Context provides both the 59-class and 459-class test sets whose similarity statistics anchor the comparison."},{"cited_title":"Scene parsing through ADE20K dataset","cited_arxiv_id":null,"evidence_quote":"ADE20K provides the 150-class and 847-class test sets whose semantic overlap with training is analyzed."},{"cited_title":"Cat-seg: Cost aggregation for open-vocabulary semantic segmenta- tion","cited_arxiv_id":null,"evidence_quote":"Represents the CLIP-fine-tuning route that improves on standard benchmarks but loses on OpenBench."},{"cited_title":"Side adapter network for open-vocabulary semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Represents the frozen-CLIP route that holds up on OpenBench, supplying the divergence shown in the paper."}],"review_version":1}