Pith. sign in

REVIEW 2 major objections 5 minor 33 references

Replacing softmax with α-entmax in CLIP’s final attention layers zeros irrelevant token mass and improves dense open-vocabulary prediction, with gains largest when baseline attention is most diffuse.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 15:50 UTC pith:JRB2HAAG

load-bearing objection Solid training-free demo that swapping softmax for α-entmax in CLIP’s last layers denoises dense features, with gains scaling cleanly to off-class attention mass. the 2 major comments →

arxiv 2607.07135 v2 pith:JRB2HAAG submitted 2026-07-08 cs.CV

Sparse Attention for Dense Open-Vocabulary Prediction in CLIP

Fatimah Zohra , Chen Zhao , Shuming Liu , Bernard Ghanem This is my paper
classification cs.CV
keywords CLIPsparse attentionentmaxopen-vocabulary segmentationdense predictionself-attentionfine-grained retrievaltraining-free
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CLIP is trained with a global image–text contrastive objective, so its late self-attention layers produce dense, strictly positive softmax weights that mix every patch with every other patch. That long tail of low-salience tokens adds noise that blurs the fine spatial cues needed for pixel-level segmentation and region-level attribute recognition. The paper shows that a pure inference-time swap of the final-layer row-wise softmax for the α-entmax transform acts as an implicit denoiser: a data-dependent threshold maps weak scores exactly to zero and redistributes mass onto the most relevant tokens. Across open-vocabulary semantic segmentation (Pascal VOC, Context, ADE20K) and fine-grained region–text retrieval (FG-OVD), the improvement is consistently proportional to how much the original attention already leaked onto non-class patches. The same pattern holds for both standard query–key attention and self-correlation variants, and the relative gain grows with higher resolution and larger backbones that increase the token count.

Core claim

The gain from attention sparsification is proportional to how much the baseline attention spreads off the target class. Substituting α-entmax for softmax in the final visual self-attention layers of a frozen CLIP encoder zeros low-relevance dependencies and concentrates mass on relevant tokens, lifting dense open-vocabulary segmentation and fine-grained retrieval without any training or architectural change.

What carries the argument

The α-entmax transform: a data-dependent threshold that maps low attention scores exactly to zero while remaining on the probability simplex (recovering softmax as α → 1).

Load-bearing premise

The claim assumes that CLIP’s global contrastive training concentrates its localization bias almost entirely in the final one or two layers, so sparsifying only those layers is sufficient.

What would settle it

Compute, for every foreground patch, the fraction of its attention mass that lands on non-class patches (using ground-truth masks downsampled to the patch grid). If that off-class fraction shows no positive correlation with the mIoU or retrieval gain of entmax over softmax across images or classes, the central proportionality claim is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies a training-free inference-time substitution of row-wise softmax by α-entmax in the final visual self-attention layer(s) of frozen CLIP (ViT-B/16 and ViT-L/14). Entmax applies a data-dependent threshold that zeros low-relevance scores while remaining on the simplex, acting as an implicit denoiser. The central empirical claim is that the resulting gains on dense open-vocabulary tasks (semantic segmentation on Pascal VOC/Context/ADE20K and fine-grained region-text retrieval on FG-OVD) are proportional to how diffusely the baseline (softmax) attention mass spreads onto non-class tokens. The substitution is evaluated across native qkv and self-correlation score sources (qqv, kkv, vvv, ensembles), with supporting ablations on depth/α, resolution, residual/FFN pathways, random-mask and temperature controls, and qualitative attention/similarity maps.

Significance. If the proportionality claim holds, the work isolates attention density as a previously under-examined factor in CLIP's known localization bias and supplies a zero-parameter, architecture-preserving intervention that consistently improves self-correlated attention distributions for dense prediction. The multi-dataset, multi-backbone, multi-resolution design, together with the matched-support random-mask and temperature controls and the same-class mass redistribution diagnostics, makes the result immediately usable by practitioners and a clean reference point for future sparse-attention studies in vision-language models. The explicit demonstration that already-concentrated distributions (e.g., vvv on ViT-B/16) receive null or negative gains is a falsifiable strength rather than a caveat.

major comments (2)
  1. The proportionality claim (abstract, §1, Fig. 1) is the paper's main contribution, yet the quantitative support is limited to a single scatter of off-class mass versus ΔmIoU on Pascal VOC for one backbone/distribution pair. Tables 1–2 and 5–7 show consistent directional gains, but a systematic quantification of the same off-class mass statistic across all reported distributions, backbones and datasets would make the claim fully load-bearing rather than illustrative.
  2. §3.2 and Fig. 5 justify sparsifying only the final 1–2 layers by appealing to prior literature on global-objective bias. While the shallow-depth sweeps are consistent with this design choice, the paper never measures how much of the off-class mass actually originates in earlier layers; a brief diagnostic (e.g., cumulative off-class mass by layer) would confirm that the intervention is applied where the noise is concentrated.
minor comments (5)
  1. Several typographical errors remain: "yeilds" (§5.1), "higer" (conclusion), "finallayer" (Table 4 caption), "finallayer Transformer block" (Table 4), and inconsistent spacing around α-entmax.
  2. Figure 1 caption and the surrounding text in §1 refer to "the fraction of a foreground patch's attention that lands on non-class (irrelevant) patches" measured against downsampled VOC GT; the precise downsampling procedure (nearest-neighbour vs. majority vote) should be stated once for reproducibility.
  3. Table 3 reports FG-OVD mHME for the random-mask control but does not list the exact support size or sampling procedure used to match entmax; a one-sentence clarification would strengthen the control.
  4. The background-threshold sweep for VOC is mentioned (§4.1) but the selected thresholds are never reported; adding them to the appendix would aid exact reproduction.
  5. References [5] and [13] are cited for MaskCLIP and CLIP-Surgery, yet the precise implementation details of the "v-only" reference (Tables 5–7) are left implicit; a short note confirming that the residual and FFN are retained would remove ambiguity.

Circularity Check

0 steps flagged

No significant circularity: purely empirical substitution study whose central proportionality claim is measured against external ground-truth masks, not derived from its own inputs.

full rationale

The paper performs a training-free inference-time swap of row-wise softmax for α-entmax (a known transform from Blondel/Martins et al.) in the final visual self-attention layers of frozen CLIP, then evaluates the resulting dense features on external open-vocabulary benchmarks (Pascal VOC/Context, ADE20K mIoU; FG-OVD mHME). The load-bearing claim—that sparsification gain scales with how much baseline attention mass lands off the target class—is an empirical observation obtained by measuring off-class mass against downsampled ground-truth segmentation masks (Fig. 1) and same-class mass redistribution (Fig. 2b), then correlating those quantities with ΔmIoU/ΔmHME across distributions, backbones, resolutions and tasks (Tables 1–7). α is treated as a free hyper-parameter (default 1.2) whose effect is swept, not fitted to produce a “prediction.” Prior self-citations (e.g., β-CLIP) and literature on self-correlation attention are used only for baselines and motivation; none of them force the proportionality result by construction. There is therefore no self-definitional loop, no fitted quantity renamed as prediction, and no load-bearing uniqueness theorem imported from the authors. The derivation chain is simply “apply known sparse normalizer → measure external metrics → report observed correlation,” which is self-contained against external data.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The paper rests on standard transformer mathematics, the known closed-form of α-entmax, and the domain premise that CLIP’s final layers are the primary locus of global-objective bias. The only free choices are the sparsity level α and the number of final layers to sparsify; no new physical or mathematical entities are postulated.

free parameters (2)
  • α (entmax sparsity) = 1.2 (primary)
    Hand-chosen; main tables use α=1.2, depth/α heatmaps explore nearby values. Performance is sensitive to this choice (too large α discards useful mass).
  • number of final layers sparsified = 1–2
    Chosen by sweep; optimum is last 1–2 layers. Deeper application degrades results.
axioms (3)
  • standard math α-entmax recovers softmax as α→1 and maps scores below a data-dependent threshold τ exactly to zero while remaining on the simplex.
    Taken from Blondel et al. / Peters et al.; used throughout §3.1.
  • domain assumption CLIP’s global image-text contrastive objective concentrates localization degradation in the final self-attention layers.
    Stated in §1 and §3.2; justifies restricting the intervention to the last layers.
  • domain assumption Self-correlation score matrices (QQ, KK, VV) already provide better localization than native QK for dense CLIP tasks.
    Taken from prior work (SCLIP, GEM, etc.) and used as the baseline distributions in Tables 1–2.

pith-pipeline@v1.1.0-grok45 · 20097 in / 2397 out tokens · 23121 ms · 2026-07-14T15:50:44.816196+00:00 · methodology

0 comments
read the original abstract

Contrastive Language-Image Pre-training (CLIP) relies on softmax-based self-attention, a strictly positive distribution that assigns probability mass to every pair of tokens-even semantically irrelevant ones. While these dense softmax weights are effective for gathering broad context during pre-training, they spread attention across many low-salience tokens, producing noise that obscures the fine-grained, spatially localized cues required for dense, open-vocabulary prediction. We study an inference-time substitution of the row-wise softmax in the final visual self-attention layers with the $\alpha$-entmax transform, applied across both the standard query-key attention and self-correlation variants. Because entmax applies a data-dependent threshold that maps low scores exactly to zero, it acts as an implicit denoiser, zeroing contextually irrelevant dependencies while redistributing mass onto the most relevant tokens. We evaluate on open-vocabulary tasks-dense semantic segmentation (Pascal VOC, Pascal Context, ADE20K) and fine-grained retrieval (FG-OVD)-and find the gain from attention sparsification is proportional to how much the baseline attention spreads off the target class.

Figures

Figures reproduced from arXiv: 2607.07135 by Bernard Ghanem, Chen Zhao, Fatimah Zohra, Shuming Liu.

Figure 1
Figure 1. Figure 1: Attention Mass in Softmax vs. Entmax Distributions. Left: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-patch attention diagnostics. CLIP ViT-B/16, last block. For a central query patch shown as (green ×), we visualise the attention induced by each distribution— the standard qk and the self-correlated qq, kk, vv—under softmax (top row of each panel) and entmax (α=1.2, bottom row). Titles report the effective number of attended keys. The qk attention is globally diffuse throughout the image, whereas the s… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative Visualization of Segmentation Maps and Heatmaps. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Entmax sparsity vs. depth on Pascal VOC on ViT-L/14 [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Additional per-patch attention diagnostics. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Additional per-patch attention diagnostics. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 13 linked inside Pith

  1. [1]

    arXiv (2023).https://doi.org/10.48550/arxiv.2311.17518

    Bianchi, L., Carrara, F., Messina, N., Gennaro, C., Falchi, F.: The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding. arXiv (2023).https://doi.org/10.48550/arxiv.2311.17518

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.17518 2023

  2. [2]

    In: Chaudhuri, K., Sugiyama, M

    Blondel, M., Martins, A.F.T., Niculae, V.: Learning classifiers with fenchel-young losses: Generalized entropies, margins, and algorithms. In: Chaudhuri, K., Sugiyama, M. (eds.) Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 89, pp. 606–615. PMLR (April16–18 2019),...

    2019

  3. [3]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Bousselham, W., Petersen, F., Ferrari, V., Kuehne, H.: Grounding everything: Emerging localization properties in vision-language transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3828–3837 (June 2024)

    2024

  4. [4]

    In: Proc

    Correia, G.M., Niculae, V., Martins, A.F.: Adaptively sparse transformers. In: Proc. EMNLP-IJCNLP (to appear) (2019)

    2019

  5. [5]

    Dong, X., Bao, J., Zheng, Y., Zhang, T., Chen, D., Yang, H., Zeng, M., Zhang, W., Yuan, L., Chen, D., Wen, F., Yu, N.: Maskclip: Masked self-distillation advances contrastive language-image pretraining (2023)

    2023

  6. [6]

    International Journal of Computer Vision111(1), 98–136 (Jan 2015)

    Everingham, M., Eslami, S.M.A., Van Gool, L., Williams, C.K.I., Winn, J., Zisser- man, A.: The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision111(1), 98–136 (Jan 2015)

    2015

  7. [7]

    Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., Cao, Y.: Eva: Exploring the limits of masked visual representation learning at scale (2022)

    2022

  8. [8]

    In: Proceedings of the Winter Conference on Applications of Computer Vision (WACV)

    Hajimiri, S., Ben Ayed, I., Dolz, J.: Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation. In: Proceedings of the Winter Conference on Applications of Computer Vision (WACV). pp. 5061–5071 (February 2025)

    2025

  9. [9]

    Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q.V., Sung, Y., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision (2021)

    2021

  10. [10]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024), https://openreview.net/forum?id=nExI4FuKWD

    Jing, D., He, X., Luo, Y., Fei, N., Yang, G., Wei, W., Zhao, H., Lu, Z.: FineCLIP: Self-distilled region-based CLIP for better fine-grained understanding. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024), https://openreview.net/forum?id=nExI4FuKWD

    2024

  11. [11]

    In: European Conference on Computer Vision

    Lan, M., Chen, C., Ke, Y., Wang, X., Feng, L., Zhang, W.: Clearclip: Decomposing clip representations for dense vision-language inference. In: European Conference on Computer Vision. pp. 143–160. Springer (2024)

    2024

  12. [12]

    In: CVPR (2022)

    Li, F., et al.: Grounded language-image pre-training. In: CVPR (2022)

    2022

  13. [13]

    Li, Y., Wang, H., Duan, Y., Zhang, J., Li, X.: A closer look at the explainability of contrastive language-image pre-training (2024),https://arxiv.org/abs/2304. 05653

    2024

  14. [14]

    Ma, Z., Hong, J., Gul, M.O., Gandhi, M., Gao, I., Krishna, R.: Crepe: Can vision- language foundation models reason compositionally? (2023),https://arxiv.org/ abs/2212.07796

    Pith/arXiv arXiv 2023

  15. [15]

    02068 16 F

    Martins, A.F.T., Astudillo, R.F.: From softmax to sparsemax: A sparse model of attention and multi-label classification (2016),https://arxiv.org/abs/1602. 02068 16 F. Zohra et al

    2016

  16. [16]

    Martins, P.H., Marinho, Z., Martins, A.F.T.: Sparse text generation (2020),https: //arxiv.org/abs/2004.02644

    Pith/arXiv arXiv 2020

  17. [17]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2014)

    Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., Yuille, A.: The role of context for object detection and semantic segmentation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2014)

    2014

  18. [18]

    Peters, B., Niculae, V., Martins, A.F.T.: Sparse sequence-to-sequence models (2019), https://arxiv.org/abs/1905.05702

    Pith/arXiv arXiv 2019

  19. [19]

    Qiu, C., Wu, Y., Ke, W., Bai, X., Zhang, T.: Refining clip’s spatial awareness: A visual-centric perspective (2025),https://arxiv.org/abs/2504.02328

    Pith/arXiv arXiv 2025

  20. [20]

    ICML (2021)

    Radford, A., Kim, J.W., Hallacy, C., et al.: Learning transferable visual models from natural language supervision. ICML (2021)

    2021

  21. [21]

    Ranasinghe, K., McKinzie, B., Ravi, S., Yang, Y., Toshev, A., Shlens, J.: Perceptual grouping in contrastive vision-language models (2023),https://arxiv.org/abs/ 2210.09996

    Pith/arXiv arXiv 2023

  22. [22]

    Shao,T.,Tian,Z.,Zhao,H.,Su,J.:Explorethepotentialofclipfortraining-freeopen vocabulary semantic segmentation (2024),https://arxiv.org/abs/2407.08268

    Pith/arXiv arXiv 2024

  23. [23]

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need (2023)

    2023

  24. [24]

    Voita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I.: Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned (2019), https://arxiv.org/abs/1905.09418

    Pith/arXiv arXiv 2019

  25. [25]

    arXiv preprint arXiv:2312.01597 (2023)

    Wang, F., Mei, J., Yuille, A.: Sclip: Rethinking self-attention for dense vision- language inference. arXiv preprint arXiv:2312.01597 (2023)

    Pith/arXiv arXiv 2023

  26. [26]

    Wu, S., Zhang, W., Xu, L., Jin, S., Li, X., Liu, W., Loy, C.C.: Clipself: Vision transformer distills itself for open-vocabulary dense prediction (2024),https:// arxiv.org/abs/2310.01403

    Pith/arXiv arXiv 2024

  27. [27]

    In: CVPR (2022)

    Wu, X., et al.: Regionclip: Region-based language Image pre-training. In: CVPR (2022)

    2022

  28. [28]

    Xie, C., Wang, B., Kong, F., Li, J., Liang, D., Zhang, G., Leng, D., Yin, Y.: Fg-clip: Fine-grained visual and textual alignment (2025),https://arxiv.org/abs/2505. 05071

    2025

  29. [29]

    Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.: When and why vision-language models behave like bags-of-words, and what to do about it? (2023), https://arxiv.org/abs/2210.01936

    Pith/arXiv arXiv 2023

  30. [30]

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training (2023),https://arxiv.org/abs/2303.15343

    Pith/arXiv arXiv 2023

  31. [31]

    Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.: Semanticunderstandingofscenesthroughtheade20kdataset.Int.J.Comput.Vision 127(3), 302–321 (Mar 2019).https://doi.org/10.1007/s11263- 018- 1140- 0 , https://doi.org/10.1007/s11263-018-1140-0

    doi:10.1007/s11263- 2019

  32. [32]

    Zhou, Y., Fritz, M., Keuper, M.: Multimax: Sparse and multi-modal attention learning (2025),https://arxiv.org/abs/2406.01189

    Pith/arXiv arXiv 2025

  33. [33]

    arXiv (2026).https://doi

    Zohra, F., Zhao, C., Itani, H., Ghanem, B.:beta-CLIP: Text-conditioned contrastive learning for multi-granular vision-language alignment. arXiv (2026).https://doi. org/10.48550/arxiv.2512.12678 Sparse Attention in CLIP 17 A Extended Quantitative Analysis A.1 Resolution Scaling for Dense Segmentation on ViT-B/16 Table 7 extends the resolution sweep to dens...

    doi:10.48550/arxiv.2512.12678 2026