Repurposing CLIP to Localize at Pixel Level

Fei Guo; Jiaxiang Fang; Jing Wang; Shengfeng He; Shiqiang Ma; Siyu Chen

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

CLIPix extracts class-specific activation maps from CLIP's classification backpropagation, denoises them via a correction strategy, and embeds them into image features for zero-shot binary semantic segmentation, achieving state-of-the-art on PASCAL-5i and COCO-20i.

2026-07-07 21:16 UTC pith:S6AYIIB2

load-bearing objection CLIPix repurposes CLIP classification gradients for zero-shot binary segmentation — strong results, but the ablation conflates two changes and the baseline is weak. the 3 major comments →

arxiv 2607.05253 v2 pith:S6AYIIB2 submitted 2026-07-06 cs.CV

Repurposing CLIP to Localize at Pixel Level

Jiaxiang Fang , Shiqiang Ma , Jing Wang , Siyu Chen , Fei Guo , Shengfeng He This is my paper

classification cs.CV

keywords cliplocalizationclipixpixel-levelbiasescuesglobalintroduce

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CLIP is a vision-language model that can tell you what objects are in an image, but it was trained on whole-image captions, so it does not naturally produce pixel-accurate masks. CLIPix addresses this by running CLIP's normal classification process (given a text prompt like 'a photo of cat') and then tracing back which spatial regions in the image most influenced that classification decision. This produces a rough heatmap of where the target object is. Because CLIP was trained globally, this heatmap is noisy—it may fire on background or unrelated objects. The paper's Noise-Resistant Correction (NRC) strategy takes the initial heatmap, uses it to reweight the input patches and update the class token, then re-runs the encoder to get a cleaner second heatmap. A Localization Embedding (LE) strategy then takes both heatmaps, builds prototypes from them, concatenates them with image features through convolutional and cross-attention layers, and feeds the result to a lightweight decoder for the final mask. The system is trained on some object categories and tested on entirely unseen ones, demonstrating generalization. Results show large mIoU gains over prior zero-shot methods on PASCAL-5i (80.7%) and COCO-20i (61.8%), and the method also extends to training-free multi-class segmentation.

Core claim

CLIPix achieves state-of-the-art performance on PASCAL-5i (80.7 mIoU) and COCO-20i (61.8 mIoU) for zero-shot binary open-set semantic segmentation by repurposing CLIP's classification backpropagation to extract pixel-level localization cues, refining them with a Noise-Resistant Correction strategy, and embedding them via a Localization Embedding strategy, all while preserving CLIP's generalization to unseen classes.

Load-bearing premise

The method assumes that CLIP's classification gradients (Eq. 2) provide more object-specific localization than the affinity matrix used by prior work like DenseCLIP. This is the load-bearing premise: if the gradient-based activation map L1 is not substantially more precise than the affinity matrix, the entire NRC and LE pipeline built on top of it adds complexity without proportional benefit. The paper provides visual evidence (Fig. 5) and ablation numbers (Table IV: +10 mIoU from NRC), but the baseline is DenseCLIP's affinity matrix, not a strong gradient-based baseline, so the comparison does not isolate whether gradient tracing per se is the source of improvement versus the reweighting and prototype construction steps.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

CLIPix repurposes CLIP classification gradients for zero-shot binary segmentation — strong results, but the ablation conflates two changes and the baseline is weak.

read the letter

The headline: CLIPix extracts pixel-level localization cues from CLIP's classification gradients (rather than the affinity matrix used by DenseCLIP/PMGA-Net), refines them with an iterative patch-reweighting loop (NRC), and feeds them through a lightweight cross-attention decoder (LE). The results are strong — 80.7 mIoU on PASCAL-5i and 61.8 on COCO-20i in the zero-shot setting, with 50 FPS on MobileNetV2. Code is public. The efficiency story is real and well-documented (Table VII). The per-class breakdown (Tables VIII–IX) and failure case analysis (Fig. 8) are honest and useful. The multi-class extension (Table VI) is a nice bonus showing the NRC module generalizes beyond the binary setting. The core mechanism — Eq. 2, extracting L1 from classification backprop — is parameter-free given frozen CLIP, which is clean. The NRC loop (Eqs. 3–4) is a non-trivial construction and the ablation in Table V isolates its sub-components sensibly. The alpha_1/alpha_2 sweep (Fig. 7) shows the 0.5/0.5 choice is near-optimal across folds, which is reassuring. Now the soft spots. The stress-test concern lands. Table IV's ablation baseline (63.0 mIoU, DenseCLIP affinity matrix) is 7.6 points below PMGA-Net (70.6), which uses the same affinity-matrix paradigm. If the baseline were properly tuned, the +10 mIoU NRC gain would shrink to roughly +3. More importantly, there is no ablation row for L1 alone (gradient-based activation map without NRC correction). The +10 gain from NRC bundles two changes: switching from affinity matrix to gradient-based L1, and applying the correction to produce L2. Without isolating L1 as a prior, we cannot tell whether gradient tracing per se is the key innovation or whether the downstream correction and embedding steps carry the weight. This is the load-bearing gap in the ablation. The SOTA claim (80.7/61.8) likely holds regardless — the margin over PMGA-Net is 10.1 points on PASCAL — but the per-component contribution story is materially weakened. Minor issues: the SNN discussion in the Introduction is unrelated to the contribution and should go. The multi-class threshold parameter is unspecified. No statistical significance tests across folds, though the margins are large enough that this is not a serious concern. This paper is for researchers working on open-set segmentation and CLIP-based dense prediction. It deserves a serious referee. The main revision ask should be straightforward: add an L1-only ablation row and either tune the DenseCLIP baseline or explain the gap from PMGA-Net. If the gradient-based prior still shows meaningful improvement over a properly tuned affinity-matrix baseline, the central claim is well-supported.

Referee Report

3 major / 7 minor

Summary. The manuscript introduces CLIPix, a framework that repurposes CLIP's classification backpropagation to extract pixel-level localization cues for binary open-set semantic segmentation. The method traces gradient-based activation maps from CLIP's visual-language logits (L1), refines them via a Noise-Resistant Correction (NRC) strategy that reweights input patches and class tokens to produce a corrected map (L2), and integrates both maps into image features via a Localization Embedding (LE) strategy with a lightweight transformer decoder. Experiments on PASCAL-5i and COCO-20i report state-of-the-art zero-shot mIoU (80.7 and 61.8), and the NRC module is additionally adapted to training-free open-vocabulary segmentation (Table VI). Code is stated to be available.

Significance. The core idea of extracting localization from CLIP's classification gradients rather than from the affinity matrix is a clean, parameter-free mechanism given the frozen backbone, and the reported gains over prior zero-shot methods on PASCAL-5i and COCO-20i are substantial. The extension to training-free multi-class segmentation (Table VI) and the computational efficiency analysis (Table VII) add practical value. The per-class breakdown (Tables VIII-IX) and failure-case analysis (Fig. 8) are commendable for honesty. However, the ablation design does not cleanly isolate the central premise, which weakens the mechanistic claim.

major comments (3)

Table IV: The ablation baseline (no NRC, no LE) reports 63.0 mIoU on PASCAL-5i using the DenseCLIP affinity-matrix paradigm, but PMGA-Net (Table I) achieves 70.6 mIoU using the same affinity-matrix approach. This 7.6-point gap suggests the baseline is not properly tuned, which inflates the apparent +10.0 mIoU gain attributed to NRC. The authors should either reconcile this gap or re-run the ablation with a competitive affinity-matrix baseline to fairly assess NRC's marginal contribution.
Table IV / Eq. (2)-(4): The NRC row bundles two distinct changes simultaneously: (1) replacing the affinity matrix with the gradient-based activation map L1 (Eq. 2), and (2) applying the correction strategy (Eqs. 3-4) to produce L2. There is no ablation row that uses L1 alone as the prior (without NRC correction) to isolate whether gradient extraction per se outperforms the affinity matrix. This conflates the contribution of gradient tracing with that of the correction steps, undermining the paper's central claim that gradient-based localization is the key innovation. An additional ablation row for 'L1 only' is needed.
Fig. 7: The hyperparameters alpha_1 and alpha_2 (Eq. 4) are tuned on PASCAL-5i, which is also the test set. Since the four folds of PASCAL-5i are used for both training and evaluation, selecting alpha_1=alpha_2=0.5 based on PASCAL-5i performance constitutes test-set tuning. The authors should either validate these parameters on a held-out split or demonstrate robustness across a wider range without peak-picking on the test set.

minor comments (7)

Eq. (2): The summation over m, i, j and the ReLU application could be clarified. It is ambiguous whether the gradient is summed over spatial dimensions before or after the ReLU, and whether F_visual^m refers to a specific feature channel or feature map.
Fig. 2: The sub-figures (b), (c), (d) contain inline equations that are difficult to read due to formatting. Consider simplifying or moving these to the main text.
Table I: The backbone column lists 'ResNet' for many methods without specifying the variant (e.g., ResNet-50 vs. ResNet-101). Clarify for reproducibility.
Section IV.G: The claim that 'a frame rate above 90 FPS is typically required' for AR/VR is cited without a reference. Provide a citation or soften the claim.
Table VIII: 'Knite' should be 'Knife'. 'Hairdrier' should be 'Hair dryer'.
Section I: 'Can we repurposes CLIP' should be 'Can we repurpose CLIP'.
Eq. (8): The notation mixes softmax with a cosine similarity bracket <>, but the exact formulation of the attention score is unclear. Clarify whether D is a scaling factor and how the softmax is applied over the spatial dimension.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for a careful and constructive review. The core concern—that our ablation design does not cleanly isolate the central premise—is well-taken, and we address each point below. We agree that two of the three comments warrant revisions to the manuscript.

read point-by-point responses

Referee: Table IV: The ablation baseline (no NRC, no LE) reports 63.0 mIoU on PASCAL-5i using the DenseCLIP affinity-matrix paradigm, but PMGA-Net (Table I) achieves 70.6 mIoU using the same affinity-matrix approach. This 7.6-point gap suggests the baseline is not properly tuned, which inflates the apparent +10.0 mIoU gain attributed to NRC. The authors should either reconcile this gap or re-run the ablation with a competitive affinity-matrix baseline to fairly assess NRC's marginal contribution.

Authors: The referee raises a fair concern, and we agree the gap deserves explicit reconciliation in the manuscript. The key distinction is that our ablation baseline (63.0 mIoU) uses the DenseCLIP affinity-matrix paradigm within our own architectural pipeline—i.e., the affinity matrix is plugged into our LE decoder and training setup, with NRC and LE disabled. PMGA-Net (70.6 mIoU) uses the affinity matrix as part of a different and independently optimized architecture that includes its own mask-assembly modules, multi-scale feature fusion, and training recipe. The two are therefore not directly comparable as 'same approach, different tuning.' Nevertheless, we acknowledge that presenting the 63.0 baseline without this context is misleading. In the revision, we will (1) add a clarifying note explaining that the ablation baseline is our own architecture with the affinity matrix substituted for L1/L2, not a reproduction of PMGA-Net; and (2) add an additional ablation row using PMGA-Net's affinity-matrix prior within our pipeline with their reported hyperparameters, to provide a more competitive reference point. We expect this will narrow but not eliminate the gap, since PMGA-Net's gains come partly from architectural components outside the scope of our ablation. revision: yes
Referee: Table IV / Eq. (2)-(4): The NRC row bundles two distinct changes simultaneously: (1) replacing the affinity matrix with the gradient-based activation map L1 (Eq. 2), and (2) applying the correction strategy (Eqs. 3-4) to produce L2. There is no ablation row that uses L1 alone as the prior (without NRC correction) to isolate whether gradient extraction per se outperforms the affinity matrix. This conflates the contribution of gradient tracing with that of the correction steps, undermining the paper's central claim that gradient-based localization is the key innovation. An additional ablation row for 'L1 only' is needed.

Authors: We thank the referee for this precise observation. We note that the requested ablation actually already exists in Table V (Row 1: E✓, C✓, E_new✗, C_new✗), which isolates L1 without the correction strategy. This row reports 68.0 mIoU on PASCAL-5i and 48.5 on COCO-20i, compared to the affinity-matrix baseline of 63.0 and 40.0 respectively. This means gradient extraction alone (L1) contributes +5.0 mIoU on PASCAL-5i and +8.5 on COCO-20i over the affinity matrix, and the correction strategy (producing L2) adds a further +5.0 and +4.5 respectively. However, we agree that this information is buried in Table V and not visible in Table IV, which is the primary ablation table. In the revision, we will add an explicit 'L1 only (no NRC correction)' row to Table IV so that the decomposition is immediately visible: baseline (affinity) → L1 only → L1+L2 (full NRC) → full method. This will make clear that both gradient extraction and the correction strategy contribute meaningfully, with gradient extraction alone already outperforming the affinity matrix. revision: yes
Referee: Fig. 7: The hyperparameters alpha_1 and alpha_2 (Eq. 4) are tuned on PASCAL-5i, which is also the test set. Since the four folds of PASCAL-5i are used for both training and evaluation, selecting alpha_1=alpha_2=0.5 based on PASCAL-5i performance constitutes test-set tuning. The authors should either validate these parameters on a held-out split or demonstrate robustness across a wider range without peak-picking on the test set.

Authors: The referee is correct that selecting alpha_1 = alpha_2 = 0.5 based on PASCAL-5i performance constitutes test-set tuning under the standard cross-fold protocol, since all four folds serve as test folds. We cannot fully resolve this concern because we do not have a separate held-out dataset with the same class structure to serve as a validation set for hyperparameter selection. However, we can make two honest points. First, Figure 7 already shows that performance is relatively stable across a wide range of alpha values (approximately 0.3–0.7), with variations within ~2 mIoU; the choice of 0.5 is not a sharp peak but a broad plateau. Second, the same alpha values transfer to COCO-20i without retuning (61.8 mIoU), which provides some evidence of robustness. In the revision, we will (1) explicitly acknowledge in the text that alpha was selected on PASCAL-5i and discuss the test-set tuning limitation honestly; (2) add the COCO-20i alpha sensitivity curve to Figure 7 to demonstrate that the same setting generalizes without retuning; and (3) report performance at multiple alpha values (e.g., 0.3, 0.5, 0.7) in a table to show the flatness of the response. We cannot claim this fully resolves the concern, but we believe the robustness evidence, combined with transparent disclosure, is the most honest treatment available to us. revision: partial

standing simulated objections not resolved

The test-set tuning of alpha_1/alpha_2 cannot be fully resolved without a separate held-out validation set of comparable structure, which we do not have. We can demonstrate robustness and disclose the limitation, but we cannot eliminate the concern entirely.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

No new physical entities, particles, or forces are introduced. The method is a computational pipeline built on existing CLIP and CNN backbones. The free parameters are standard ML hyperparameters and learnable weights, not ad hoc constants introduced to force a derivation.

free parameters (5)

alpha_1 = 0.5
Weight for original class token C in Eq. 4; tuned on PASCAL-5i (Fig. 7)
alpha_2 = 0.5
Weight for pseudo-target prototype in Eq. 4; tuned on PASCAL-5i (Fig. 7)
W_q, W_k, W_v, W_o
Learnable projection matrices in cross-attention decoder (Eqs. 8-9); trained on seen classes, generalization to unseen classes is the empirical claim
Key layer selection (final CLIP layer) = final layer
Chosen by ablation (Fig. 6) on PASCAL-5i; used for all reported results
Threshold for multi-class extension
Used in Section IV.F to filter high-response regions; value not stated in the paper

axioms (3)

domain assumption CLIP's classification gradients contain object-specific spatial information that is more precise than affinity-matrix localization
Section III.B: 'We believe that extracting regions of interest from CLIP's mature classification decisions better represents its potential for dense localization.' This is the foundational premise; it is supported empirically (Fig. 5, Table IV) but not derived from first principles.
domain assumption Features learned on seen categories generalize to unseen categories in the LE decoder
Section III.C: The LE module with W_q, W_k, W_v, W_o is trained on D_train categories and applied to D_test with strictly disjoint C_train and C_test. Generalization is assumed and measured, not guaranteed.
standard math The standard PASCAL-5i/COCO-20i fold protocol is a valid proxy for open-set generalization
Section III: follows the established protocol from [47], [48]. Standard in the field.

pith-pipeline@v1.1.0-glm · 25327 in / 3165 out tokens · 270456 ms · 2026-07-07T21:16:38.187567+00:00 · methodology

0 comments

read the original abstract

Large-scale Vision-Language Models like CLIP have demonstrated impressive open-set localization capabilities at the image level. However, adapting this capability to pixel-level dense prediction poses challenges due to global feature biases. In this paper, we introduce CLIPix, a simple yet effective framework that repurposes CLIP to perform pixel-level localization. By tracing back CLIP's classification process, CLIPix identifies object-specific attentive regions and repurposes them as pixel-level localization cues. To address noise introduced by global biases, we propose a Noise-Resistant Correction strategy, refining these cues for more precise segmentation. Additionally, we introduce a Localization Embedding strategy to integrate both localization and enriched detail information, enabling accurate, high-resolution segmentation. Our approach preserves CLIP's generalization strength and unlocks its potential for segmenting arbitrary objects. Extensive experiments on the PASCAL and COCO datasets demonstrate that CLIPix achieves state-of-the-art performance, underscoring its effectiveness.