{"id":"569c9be8-6d02-4444-9e2c-e3aa30d351dc","arxiv_id":"2412.19650","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Weakly supervised segmentation with CLIP improves by learning per-class prototypes in vision space instead of relying on text prototypes.","lead":"This paper argues that CLIP's text and image feature spaces do not line up perfectly, and that segmentation models should use image-side prototypes rather than text prompts. Adding this idea to two existing methods improves the accuracy of weakly supervised object segmentation on standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The KL objective in Eq. (7) makes vision prototypes mimic text-prototype distributions, so the learned W should inherit text-prototype bias; Theorem 3's guarantee that W'* approaches W* is unsupported because the optimized objective differs from the one defining W*.","rationale":"After reading the manuscript, the empirical results are credible: the ablations in Table 3 show a clear gain from switching text to vision prototypes, and the improvements on VOC/COCO are substantial. The concern is not that the method fails; it is that the mechanism claimed in the title and contributions—that learning prototypes in vision space under text supervision overcomes an inherent modality gap—is not supported by the equations. The KL objective (Eq. 7) only distills the text-prototype distribution into W. The pseudo-mask m defining the optimal W* in Theorem 3 never appears in the optimization. Theorem 3's bound would require P' (text distribution) to be close to the true mask Y, which the paper itself disputes in the co-occurrence examples. Therefore the central theoretical claim is unproven. This is exactly the weakest assumption the reader identified: text supervision is the only signal, and if it is biased, the learned W should be biased too. The proposed test—comparing KL-based learning against direct pseudo-mask supervision—would isolate whether the text-distribution matching is the actual source of improvement or whether other components (temperature, contrastive learning) carry the gain. We agree with the reader's conditional verdict: the paper should be revised to either provide a rigorous proof, or be reframed as an empirical method without the theoretical claims.","tokens_in":14013,"tokens_out":12503,"duration_ms":405247,"concrete_test":"Replace the KL loss in Eq. (7) with a direct soft-target cross-entropy loss using the GradCAM pseudo-masks m_i (normalized to a distribution) as supervision for W, keeping the temperature τI and all subsequent pipeline elements identical. Evaluate the resulting vision prototypes and final mIoU on VOC 2012 val against the current VPL. If direct mask supervision performs at least as well as the KL version, then Eq. (7) is not recovering the mask-defined optimum W*, and the theoretical narrative is unsupported. If direct mask supervision performs worse, then the text-distribution matching provides a beneficial inductive bias that Theorem 3 does not characterize; either outcome forces a reframing of the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that VPL learns vision prototypes that outperform text prototypes for localization. The learning objective, Eq. (7), is min_W Σ_i KL(P'_i || P_i), where P'_i is the class distribution produced by the text prototypes Z and P_i by the learnable vision prototypes W. Minimizing this KL is equivalent to cross-entropy distillation from Z to W: it forces W to reproduce the text-prototype predictions. Since the paper's own motivation (Introduction, Fig. 1) is that text prototypes activate co-occurring categories (e.g., railroad and train) and miss complete objects, any W matching P' inherits that bias. Theorem 3 defines W* as minimizer of L(m,W) using GradCAM pseudo-masks m, but the algorithm never optimizes L(m,W); it optimizes L(P',W). The proof (Appendix B.6, missing) allegedly bounds ||W'* - W*|| by a term involving P' - Y, but no argument establishes that P' is close to Y; in fact, the paper argues the opposite for co-occurrence cases. Thus the equations do not explain how VPL improves over text prototypes. The empirical gains (Tables 1–3) are plausible but could come from the temperature choice τI > τT, the regional semantic contrast, or implicit feature averaging, not from the stated theoretical mechanism. This is a load-bearing gap because the paper's contribution explicitly includes a theoretical demonstration that optimal vision prototypes cannot be achieved in text space.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Vision Prototype Learning (VPL), a weakly-supervised semantic segmentation framework built on CLIP. The authors argue that text prototypes suffer from an inherent modality gap and therefore cannot optimally localize objects; they propose learning class-specific vision prototypes in vision space, supervised by text-prototype distributions via a KL-divergence objective. Vision prototypes replace text prototypes for GradCAM-based pseudo-label generation, and a regional semantic contrast module aligns region embeddings with vision prototypes during decoder training. Experiments on PASCAL VOC 2012 and MS COCO 2014 report state-of-the-art or improved mIoU over strong baselines such as CLIP-ES and CLIP-CPAL, with ablations showing gains from the vision prototypes, the contrast module, and the temperature setting.","tokens_in":14463,"tokens_out":3539,"duration_ms":36245,"significance":"If the theoretical and empirical claims hold, the paper would make a useful contribution: it identifies a real practical issue, the text-vision modality gap in CLIP-based WSSS, and it offers a simple plug-and-play module that improves several existing pipelines on held-out VOC and COCO benchmarks. The empirical evaluation is externally benchmarked, the ablations isolate the proposed components, and the temperature prediction tau_I > tau_T is concretely tested and confirmed. However, the theoretical core that motivates the method is not currently verifiable: the proofs are placed in appendices B.1-B.6 that are absent from the submitted version, and the stated theorems contain notational and logical gaps. Because the paper explicitly claims a theoretical demonstration that optimal vision prototypes cannot be achieved in text space, this is a load-bearing weakness rather than a purely cosmetic one.","major_comments":[{"comment":"The manuscript repeatedly relies on proofs in Appendices B.1, B.3, B.4, B.5, and B.6 (for example, the claim that small temperature does not pull text and vision spaces together, Proposition 1, Theorems 1-3), but none of these appendices appears in the submitted v1. Without those proofs, the theoretical statements cannot be checked, and the central claim that the modality gap is inherent and that optimal vision prototypes cannot be achieved in text space is unsupported as written.","section":"Preliminary; Methodology (Hypothesis 1 through Theorem 3)"},{"comment":"Equation (6) states Delta_gap = ||Z - W*||_F^2 >= c_perp + epsilon, but c_perp is described as a constant vector while epsilon is a scalar and Delta_gap is a scalar Frobenius norm. Adding a vector and a scalar is not defined, and no operational definition is given for c_perp or epsilon. This makes the lower-bound theorem ill-typed and prevents the claimed conclusion that 'minimizing the distance between Z and W is difficult due to the inherent modality gap' from being established.","section":"Methodology, Theorem 1, Eq. (6)"},{"comment":"Proposition 1 assumes z^x_n = w*_n, which is exactly the condition that the vision component of the text prototype equals the optimal vision classifier. The subsequent equality p^{n'}_{i,k} = p^n_{i,k} then follows by construction from the assumed identity and the temperature relation, rather than demonstrating that the text space can or cannot recover optimal vision prototypes. The remark that the assumptions are 'difficult to hold' is an assertion, not a proof, so the paper does not actually establish that optimal vision prototypes cannot be achieved in text space.","section":"Methodology, Proposition 1, Eq. (5)"},{"comment":"There is a mismatch between the objective used in the algorithm and the objective used to define the target W*. Equation (9) defines W* as the minimizer of L(m, W), where m is a GradCAM pseudo-mask distribution generated through text prototypes Z, but the algorithm optimizes the KL objective L(P', W) in Eq. (7), where P' is the distribution estimated by the text prototypes. The bound in Eq. (10) involves a term with P' - Y and needs an argument that P' is close to the true labels Y; the paper's own motivation is that text prototypes misactivate co-occurring categories such as railroad and train, so no such closeness is established. Consequently, the theorem does not explain how the learned W' approaches the optimal W*, and the stated theoretical guarantee is unsupported.","section":"Methodology, Theorem 3, Eqs. (7)-(11)"}],"minor_comments":[{"comment":"The opening sentence 'research powerful cross-modal semantic understanding capabilities' is grammatically incomplete; it should be something like 'exhibit powerful cross-modal semantic understanding capabilities'.","section":"Abstract"},{"comment":"Table 4 contains a typo: one row is labeled 'Baselinee' and another 'Baselines', and the notation VPLs/VPLe for single versus ensemble prompts is not defined in the caption.","section":"Experiments, Table 4"},{"comment":"The 'Gap' metric used in Table 5 is not formally defined; the text says it measures the difference between prototypes and features, but the exact computation should be stated for reproducibility.","section":"Experiments, Table 5"},{"comment":"The sentence 'The proposed CLIP-CPAL does not modify the architecture of the CLIP network' appears to refer to VPL, not CLIP-CPAL, since CLIP-CPAL is a prior method; this should be corrected.","section":"Experiments, Comparisons With State-of-the-Art Methods"},{"comment":"The implementation section mentions 'standard projected gradient descent' and the algorithm is described as 'convex optimization', but Algorithm 1 line 4 only performs an unconstrained gradient step and includes no projection or bounded-norm operation; this discrepancy should be clarified.","section":"Algorithm 1 and Implementation Details"},{"comment":"Equation (10) uses the notation nabla_{W'*} L(W) to denote ||W'* - W*||_2, which is not a gradient; the notation should be changed to avoid confusion with the actual gradient used in Eq. (11).","section":"Methodology, Eq. (10)"}],"recommendation":"major_revision","confidential_remarks":"The empirical results and the plug-and-play nature of VPL appear genuinely useful, but the paper currently presents a large theoretical apparatus whose proofs are entirely in missing appendices. The theorem statements also have type errors and circular assumptions. The authors should either supply a complete, self-contained theoretical section or substantially reframe the paper as an empirical method with heuristic motivation. Given that the experiments are externally validated, major revision seems appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the empirical contribution is real. VPL as a plug-in improves CLIP-ES and CLIP-CPAL by 4–6 mIoU points on VOC and about 3–4 on COCO, and the ablations show each component adds something. The temperature analysis in Table 5 is a nice touch and supports the claim that a larger tau_I helps. This is a useful result for the WSSS subfield.\n\nThe idea of learning class-specific vision prototypes in vision space under text supervision is new in this literature, and the regional semantic contrast term is a sensible addition. The paper cites the relevant prior work on the modality gap, including Liang et al. and intra-modal proxy learning.\n\nNow the soft spots, which are mostly in the theory section. The appendices B.1–B.6 are missing from this arXiv version, so the proofs cannot be checked. Theorem 1's bound says Delta_gap >= c_perp + epsilon where c_perp is called a 'constant vector' — that is dimensionally odd, and the statement is not operational. Proposition 1's equality requires z^x_n = w*_n, which is exactly the condition the paper is trying to establish, so it is circular as a justification. And Theorem 3 defines W* as the minimizer of L(m,W), but the algorithm optimizes the KL objective in Eq. (7); the gradient bound says it approaches W* but the objective being optimized is different, so the link is unsupported, especially since the KL distillation from text prototypes likely inherits the co-occurrence bias the paper says is the problem. The stress-test note is right: the theory does not explain the empirical gain.\n\nThat does not sink the paper, because the empirical evidence stands on its own. But the authors should either supply the missing proofs, fix the theorem statements and align the optimization objective with the analysis, or reframe the paper as an empirical contribution and drop the theory to a remark. As is, the theory section should not be taken at face value.\n\nWho is this for? Anyone working on CLIP-based WSSS will find the method and the plug-in results worth reading. It deserves a serious referee — the empirical result is significant enough — but the referee should push for a corrected or trimmed theory and for the appendices. I would not cite the theoretical claims, but I would cite the method and results.","headline":"Solid empirical WSSS paper with real plug-in gains, but the theory section is under-developed and should be fixed or cut before publication.","tokens_in":14915,"tokens_out":3127,"would_cite":true,"duration_ms":28320,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that CLIP's text prototypes cannot optimally localize visual objects because of an inherent modality gap, and that learning class-specific vision prototypes under text supervision plus regional semantic contrast produces…","keywords":["weakly supervised semantic segmentation","CLIP","modality gap","vision prototype learning","regional semantic contrast","pseudo-mask generation","PASCAL VOC","MS COCO"],"falsifier":"On PASCAL VOC 2012 val, compare VPL against a text-only baseline with identical ensemble prompts, CRF post-processing, and decoder training, but with text prototypes projected into the vision space through a linear map learned on a small labeled split. If the projected-text baseline reaches or exceeds VPL's 78.5% mIoU, the claim that optimal prototypes must be learned within the vision space would be falsified.","tokens_in":13819,"feed_emoji":"🎯","tokens_out":7283,"duration_ms":64561,"temperature":0.7,"pith_summary":"CLIP-based weakly supervised semantic segmentation has relied on optimizing text prompts to obtain text prototypes that activate object regions, but this paper claims that the text and vision spaces of CLIP are separated by an inherent modality gap that contrastive loss cannot close. The authors try to establish that the optimal category prototypes for pixel-level localization must therefore live in the vision space, and they propose Vision Prototype Learning (VPL): learn class-specific vision prototypes by gradient descent under a KL-divergence constraint with text prototypes, then use those vision prototypes to generate pseudo-masks and supervise a decoder. A regional semantic contrast loss aligns masked-average-pooled region embeddings with the vision prototypes. If correct, this shifts CLIP-based WSSS from prompt engineering to vision-space prototype learning and yields state-of-the-art pseudo-mask and segmentation results on PASCAL VOC 2012 and MS COCO 2014.","feed_headline":"Vision prototypes beat text prompts for CLIP segmentation","feed_subtitle":"Moving prototypes into vision space lifts weakly-supervised segmentation by up to 5.9 mIoU points on VOC and COCO.","key_machinery":"The central object is the set of learnable vision prototypes W = {w_n} in the CLIP vision embedding space, replacing text prototypes Z as the source of class activation. Theorems 2 and 3 argue that a strongly convex KL objective in W can be minimized by gradient descent, with Eq. 10 bounding the gradient update and Eq. 11 iterating it, and the choice of temperature tau_I larger than CLIP's tau_T (0.03 vs 0.01) follows from Proposition 1 and is confirmed by the gap metric in Table 5. The second mechanism is regional semantic contrast, which pools region embeddings by masked average pooling (Eq. 12) and applies a bidirectional contrastive loss (Eq. 13) so that region embeddings and vision prototypes are pulled together and pushed away from negatives. Together, these turn text-derived activation maps into a seed for learning vision-space representations rather than an end product.","core_discovery":"The paper's central claim is that the modality gap between CLIP text and vision embeddings is inherent: Theorem 1 bounds the distance between an optimal vision prototype W* and a text prototype Z from below by a constant plus alignment noise, so minimizing contrastive loss cannot eliminate it, and the best prototypes for vision tasks are therefore not text prototypes but vision prototypes. The method learns W' by minimizing KL divergence between the distribution induced by text prototypes and the distribution induced by learnable vision prototypes (Eq. 7), updates it by projected gradient descent using Eq. 10-11, and uses the resulting W' to generate GradCAMs and pseudo-masks. A regional semantic contrast loss (Eq. 13) then contrasts region embeddings with these vision prototypes during decoder training. Empirically, plugging VPL into CLIP-ES and CLIP-CPAL raises seed mIoU by 5.5 and 5.9 points and final segmentation mIoU to 78.5% and 79.3% on VOC val, 77.8% and 79.0% on VOC test, and 49.2% and 49.8% on COCO val, which the paper reports as state-of-the-art.","pith_inferences":["If the modality gap is truly irreducible by contrastive loss, then similar vision-prototype learning could benefit other CLIP-based dense prediction tasks, such as open-vocabulary segmentation and referring expression segmentation, where text prototypes are currently the default.","Because VPL's vision prototypes are anchored to text-derived pseudo-masks, the method inherits any blind spots of text GradCAM; a natural extension is to break this anchor with multiple prompt sources or self-training iterations that let vision prototypes escape the text distribution.","A cheap comparison, applying a learned linear map from text prototypes into the vision embedding space, would test whether VPL's gains come from crossing modalities or from optimization within the vision space."],"forward_implications":["VPL can be added to existing CLIP-based WSSS pipelines without changing their architecture, improving seed and pseudo-mask quality by several IoU points.","Text-side prompt engineering in CLIP-based WSSS has a ceiling set by the modality gap, so further gains should come from vision-space adaptation rather than better prompts.","The temperature used to learn vision prototypes should be larger than CLIP's text temperature, matching the theoretical relationship tau_T = sqrt(epsilon) tau_I.","Regional semantic contrast with vision prototypes yields more compact and separable feature clusters, which leads to better final segmentation under noisy pseudo-labels."],"supporting_citations":[{"why":"Supplies the CLIP model and its cross-modal text and vision embedding spaces that the paper starts from.","marker":"(Radford et al. 2021)"},{"why":"Provides the observation of an inherent modality gap in contrastive models that the paper formalizes as Theorem 1.","marker":"(Liang et al. 2022)"},{"why":"CLIP-ES, the text-driven baseline that VPL is plugged into in Tables 1-3.","marker":"(Lin et al. 2023)"},{"why":"CLIP-CPAL, the second baseline improved by VPL in Tables 1-2.","marker":"(Tang et al. 2024b)"},{"why":"Defines the CAM and GradCAM activation mechanism used to generate pseudo-masks from prototypes.","marker":"(Zhou et al. 2016)"},{"why":"FMA, the task-specific text-prompt and text-prototype approach that VPL contrasts against.","marker":"(Yang and Gong 2024)"}],"fun_headline_variants":["Vision prototypes beat text prompts for CLIP segmentation","Inherent CLIP gap makes vision prototypes the better choice","Learn vision prototypes, not text, for stronger WSSS with CLIP","Modality gap forces vision prototypes in CLIP segmentation","Vision Prototype Learning lifts CLIP segmentation by 5.9 mIoU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GradCAM pseudo-masks produced by text prototypes, despite activating co-occurring or background regions, carry enough correct localization signal that learning vision prototypes under KL supervision yields genuinely better prototypes than the text prototypes themselves.","fun_headline_variants_meta":{"raw":{"variants":["Vision prototypes beat text prompts for CLIP segmentation","Inherent CLIP gap makes vision prototypes the better choice","Learn vision prototypes, not text, for stronger WSSS with CLIP","Modality gap forces vision prototypes in CLIP segmentation","Vision Prototype Learning lifts CLIP segmentation by 5.9 mIoU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":3009,"prompt_tokens":1008,"completion_tokens":2001,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":1914}},"tokens_in":624,"tokens_out":2001,"duration_ms":23448,"temperature":1.0,"reasoning_tokens":1914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:01:26.020769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On PASCAL VOC 2012 val, compare VPL against a text-only baseline with identical ensemble prompts, CRF post-processing, and decoder training, but with text prototypes projected into the vision space through a linear map learned on a small labeled split. If the projected-text baseline reaches or exceeds VPL's 78.5% mIoU, the claim that optimal prototypes must be learned within the vision space would be falsified.","supporting_citations":[{"cited_title":"W.; Zhang, Y.; Kwon, Y.; Yeung, S.; and Zou, J","cited_arxiv_id":null,"evidence_quote":"Provides the observation of an inherent modality gap in contrastive models that the paper formalizes as Theorem 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CLIP-ES, the text-driven baseline that VPL is plugged into in Tables 1-3."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CAM and GradCAM activation mechanism used to generate pseudo-masks from prototypes."}],"review_version":1}