Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

CLIP's own output predictions, fed back into its last-layer attention, improve open-vocabulary segmentation without training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A feedback self-adaptive attention module uses CLIP's own output predictions as a spatial coherence prior to reweight intermediate attention, improving training-free open-vocabulary segmentation across 8 benchmarks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Genuinely useful plug-in for training-free CLIP segmentation with strong experiments, but the 'consistent improvement' claim is overstated and the mechanism needs a logit-space control to separate attention feedback from output self-smoothing. the 3 major comments →

arxiv 2508.20265 v1 pith:KFP7OYYO submitted 2025-08-27 cs.CV cs.LG

Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation

Zhixiang Chi , Yanan Wu , Li Gu , Huan Liu , Ziqiang Wang , Yang Zhang , Yang Wang , Konstantinos N. Plataniotis This is my paper
classification cs.CV cs.LG
keywords open-vocabulary segmentationtraining-free adaptationCLIPfeedback attentionself-adaptive attentionconfidence-based pruningattention isolationsemantic segmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-vocabulary segmentation inherits CLIP's poor patch-level localization: improved intermediate attention often does not reach the final predictions, and attention maps never see class information. This paper proposes a training-free plug-in, FSA, that treats the model's own patch-level class predictions as a spatial coherence prior and feeds pairwise similarities back into the last attention block. The core claim is that this feedback loop makes internal attention and final output semantically consistent, so patches belonging to the same class aggregate more strongly. The paper shows the plug-in improves four existing training-free methods (MaskCLIP, SCLIP, ClearCLIP, ProxyCLIP) on three CLIP backbones and across Q-K, self-self, and proxy attention types on eight benchmarks. If correct, this gives a parameter-preserving way to upgrade any CLIP-based open-vocabulary segmentation pipeline by reusing information the model already computes.

Core claim

The paper's central claim is that the final patch logits carry the most complete visual and textual semantics, so pairwise KL divergence between those logits is a stronger spatial-coherence prior than any intermediate attention map. To feed that prior back faithfully, the paper isolates the initial attention's contribution by subtracting logits produced by a parallel uniform-attention branch (Eq. 6), then sparsifies the resulting similarity map with a cumulative-confidence threshold and exponential scaling to form Af. Three adaptations—Af applied before, after, or instead of Attninit·V—are ensembled as (Af·Attninit + Attninit·Af + Af)/3 · V. The paper reports this ensemble improves MaskCLIP,

What carries the argument

The feedback attention Af is the load-bearing object. For every patch pair, it converts the KL divergence between their class-prediction distributions (after uniform-attention isolation) into a similarity, keeps only the highest-confidence entries by cumulative probability threshold p, and scales survivors exponentially with λ before a softmax. It is then inserted into the last attention block as an ensemble, (Af·Attninit + Attninit·Af + Af)/3 · V, so that value aggregation follows semantic cues derived from the model's own outputs rather than only from Q-K or proxy patterns.

Load-bearing premise

The load-bearing premise is that subtracting the uniform-attention branch's logits from the original logits isolates exactly what the learned attention contributes, so the feedback similarity reflects only the initial attention maps and not downstream projection, FFN, or text-alignment effects.

What would settle it

Replace the isolated logits Y_iso in Eq. 7 with logits whose patch order has been randomly permuted per image, keeping the marginal class distribution intact; if mIoU gains persist or barely drop, the feedback is not carrying class-coherent spatial information. A second check: on an image set where intermediate attention and final predictions already agree (retention near 100%), FSA should give near-zero improvement; large gains there would indicate it is fixing something other than attention-output inconsistency.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Adding FSA to a training-free CLIP segmentation method requires no training and no change to model weights, so CLIP's zero-shot generalization is preserved.
  • The reported 3–5% per-image overhead means the feedback loop can serve as a drop-in last-layer upgrade in deployment.
  • Methods with weak intermediate priors (plain Q-K attention, MaskCLIP) gain the most, suggesting feedback compensates for missing spatial coherence rather than re-ranking already-good attention.
  • Because the feedback is computed from class predictions, it injects text-aligned class information into the attention layer, which pure self-self or proxy attention cannot do.
  • Consistent gains across ViT-B/L/H and VFM proxies (MAE, SAM, DINO) indicate the principle transfers across model scales and auxiliary feature sources.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same output-to-attention feedback at multiple layers or iteratively, though the paper reports no gain from iteration; deeper layers may need a different sparsification schedule.
  • The isolation-by-subtraction trick could be reused in other zero-shot dense tasks such as depth, saliency, or referring segmentation, where CLIP logits are the only supervision, to separate attention's contribution from downstream projections.
  • The feedback metric need not be KL divergence; replacing it with a learned or calibrated similarity could test whether the gains come from distributional comparison or simply from patch-pair class agreement.
  • Since FSA only touches the last block, it should compose with fine-tuned or prompt-tuned CLIP variants, giving a test-time adaptation layer that requires no gradient updates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Feedback-driven Self-adaptive Attention (FSA), a training-free plug-in for CLIP-based open-vocabulary segmentation. FSA computes patch-level semantic similarities from the model's own output logits after an 'attention isolation' step that subtracts a uniform-attention forward pass (Eq. 6), prunes this similarity map using cumulative-confidence thresholding (Eqs. 9-10), and re-injects the resulting feedback attention Af into the last-layer attention block through three adaptations that are ensembled in Eq. 14. The method is evaluated by plugging it into MaskCLIP, SCLIP, ClearCLIP, and ProxyCLIP across ViT-B/L/H, multiple attention types (Q-K, self-self, Proxy with MAE/SAM/DINO), and eight benchmarks. The paper claims consistent improvement across all configurations and datasets.

Significance. If the causal claim holds, FSA is a useful and practical plug-in: it is training-free, has only two fixed hyperparameters, adds modest overhead, and shows consistent average gains, with particularly large improvements on weak baselines such as MaskCLIP. The evaluation is unusually broad — 4 methods, 3 backbones, 8 benchmarks, 3 attention families — and the authors provide code. The main weakness is that the experiments do not include a logit-space control, so the distinctive claim that feedback improves the intermediate attention, rather than simply smoothing output logits, is not yet supported. The paper also overstates consistency, as several individual benchmark cells decrease. These are correctable issues, but they affect the paper's central interpretation and should be addressed before publication.

major comments (3)
  1. [Sec. 4.3, Eqs. 11-14] Missing logit-space control. Af is derived from the output logits and then applied to V or Attninit·V. The reported gains could equally be obtained by treating Af as a spatial smoothing kernel on the output probabilities themselves, e.g., Y_final = softmax(Af · softmax(Y_dense)). Such a baseline would directly test whether the improvement comes from re-injecting feedback into the attention block or from output-space self-smoothing. Without this control, the central claim that FSA 'enhances semantic consistency between internal representations and final predictions' is underdetermined. The large gains on MaskCLIP (+18.7 mIoU on ViT-L) are exactly what a strong output-space post-processor would produce.
  2. [Sec. 4.1, Eq. 6] The attention-isolation assumption is not justified. Because Eq. 2 contains a residual connection and a nonlinear FFN, Y_dense − Y_dense_uni is not a pure function of Attn_init; it also depends on x and on the interaction between the attention output and the FFN. The statement that the subtraction 'allows us to isolate the learned attention's net effect' is therefore an assumption, not a consequence of the equations. The ablation in Table 6 shows that the isolation procedure helps empirically, but it does not verify that the subtracted logits isolate the contribution of the initial attention map. A direct test — for example, perturbing Attn_init and measuring how Y_dense_iso changes, or comparing with a randomized-attention control — would be needed to support the mechanism.
  3. [Abstract, Sec. 5.2, Tables 2-4] The claim of 'consistent improvement across all datasets' is contradicted by the reported numbers. In Table 2, ClearCLIP with ViT-B/16 drops on Context59 (35.9 → 33.8). In Table 3, ProxyCLIP with SAM and ViT-H/14 drops on ADE (22.0 → 21.9). In Table 4, ProxyCLIP K-K with ViT-L/14 drops on Stuff (20.5 → 20.4) and Cityscapes (31.8 → 31.7). Please revise the abstract and Section 5.2 to describe average improvements or to report per-cell exceptions accurately.
minor comments (4)
  1. [Eq. 8] The text says 'KL divergence, bounded in [0, −∞]'. KL divergence is nonnegative and unbounded above, so the range should be [0, ∞); consequently S = 1/(D+1) lies in (0, 1].
  2. [Sec. 2, Related work] Duplicate word: 'minimizes entropy to to adapt models' should be 'to adapt models'.
  3. [Table 8 vs. Table S11] The main text says FSA adds only 3-5% overhead, but Table S11 reports up to 11.8% overhead for MaskCLIP ViT-B/16 (5.1 → 5.7 ms). Clarify that the 3-5% figure applies to ProxyCLIP, or update the cost claim.
  4. [Sec. 4.1, Eq. 6] Y_dense_uni is described in words but never defined in an equation. Please write the uniform-attention forward pass explicitly, including the softmax normalization, to make the isolation step reproducible.

Circularity Check

1 steps flagged

No central circularity: external mIoU benchmarks validate the method; one supporting 'semantic retention' metric is increased by construction because the feedback attention is defined from output predictions.

specific steps
  1. self definitional [Sec. 3 Eq. (5) and Fig. 3; Sec. 4.2 Eq. (10); Sec. 4.3 Eq. (14)]
    "Retention = 1/L Σ_i max_{j∈Top-k(Attninit_i)} I(M_i = M_j) ... we propose an ensemble approach by isolating V in Eq. 11-13: (Af·Attninit + Attninit·Af + Af)/3 · V."

    The retention metric counts whether the patches most attended by a row of the attention map share the predicted class of the reference patch. The feedback matrix Af is constructed directly from pairwise KL divergence of output class predictions (Eqs. 6-10), and Eq. 14 injects Af into the adapted attention. Consequently, the improved retention reported in Fig. 3 is a definitional consequence: the adapted attention is, by construction, biased toward patch pairs whose output predictions already agree. It is therefore not independent evidence for the paper's semantic-consistency mechanism. The central mIoU improvements on eight external benchmarks are not circular, since they are measured against ground-truth labels and no parameters are fit to them.

full rationale

The main derivation chain is self-contained. FSA computes a feedback attention Af from the model's own output logits and re-enters it into the last-layer attention (Eqs. 11-14); this is an explicit feedback/self-adaptation loop, not a disguised fit. No learned parameters are fitted to the target labels; the two hyperparameters (λ, p) are fixed globally and tested for sensitivity, and the evaluation is against external ground truth on eight benchmarks, so the central claim of improved open-vocabulary segmentation is not circular. The paper's own citations to prior author work (e.g., test-time adaptation references) are contextual and not load-bearing; no uniqueness theorem or ansatz is imported from self-citations. The only noticeable circular-adjacent element is the retention metric in Fig. 3, which is mechanically increased by the way Af is defined from output predictions; this affects a supporting explanatory metric, not the benchmark results. Overall circularity is therefore low.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The method introduces a computation object, the feedback attention A_f, but no new physical entity, particle, or dimension. Free parameters are the pruning and scaling constants. The key unstated assumptions are the linearity of the isolation step and the validity of the output-logit similarity as a semantic prior.

free parameters (2)
  • lambda = 2.0
    Scaling sharpness for exponential amplification in Eq. 10; set empirically and stable between 1.0 and 2.5 per Fig. 7.
  • p = 0.45
    Cumulative confidence cutoff for pruning in Eq. 10; set empirically, sensitivity shown in Fig. 7.
axioms (4)
  • domain assumption Output logits of CLIP dense prediction capture refined patch-level semantic understanding and can serve as a spatial coherence prior.
    Stated in Sec. 3 motivation; the entire method relies on this.
  • domain assumption Patches with similar class-prediction distributions are likely semantically related.
    Used to compute pairwise similarity via KL divergence in Sec. 4.2.
  • ad hoc to paper Subtracting uniform-attention logits isolates the contribution of the initial attention map through the rest of the pipeline.
    Introduced in Sec. 4.1, Eq. 6; assumes linearity of the effect through projections, FFN, and text alignment, not proven.
  • ad hoc to paper The inverse-KL similarity S = 1/(D+1) monotonically reflects patch semantic correspondence.
    Chosen metric in Eq. 8; no theoretical justification, though KL versus cosine is compared in Table S9.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation." pith.science (2026). https://pith.science/paper/KFP7OYYO

@misc{pith2026250820265,
  author       = {Pith},
  title        = {Pith review of: Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KFP7OYYO}},
  note         = {Machine review of arXiv:2508.20265}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

CLIP exhibits strong visual-textual alignment but struggle with open-vocabulary segmentation due to poor localization. Prior methods enhance spatial coherence by modifying intermediate attention. But, this coherence isn't consistently propagated to the final output due to subsequent operations such as projections. Additionally, intermediate attention lacks direct interaction with text representations, such semantic discrepancy limits the full potential of CLIP. In this work, we propose a training-free, feedback-driven self-adaptive framework that adapts output-based patch-level correspondences back to the intermediate attention. The output predictions, being the culmination of the model's processing, encapsulate the most comprehensive visual and textual semantics about each patch. Our approach enhances semantic consistency between internal representations and final predictions by leveraging the model's outputs as a stronger spatial coherence prior. We design key modules, including attention isolation, confidence-based pruning for sparse adaptation, and adaptation ensemble, to effectively feedback the output coherence cues. Our method functions as a plug-in module, seamlessly integrating into four state-of-the-art approaches with three backbones (ViT-B, ViT-L, ViT-H). We further validate our framework across multiple attention types (Q-K, self-self, and Proxy augmented with MAE, SAM, and DINO). Our approach consistently improves their performance across eight benchmarks.

Figures

Figures reproduced from arXiv: 2508.20265 by Huan Liu, Konstantinos N. Plataniotis, Li Gu, Yanan Wu, Yang Wang, Yang Zhang, Zhixiang Chi, Ziqiang Wang.

Figure 1
Figure 1. Figure 1: Comparison with the existing training-free methods. Top: Prior works refine intermediate attention but face limitations: (1) improvements may not propagate to final segmentation; (2) at￾tention lacks direct class information. Bottom: We introduce a feedback-driven self-adaptive mechanism that reintegrates seman￾tic outputs with visual and textual cues into CLIP. Our approach is orthogonal to existing metho… view at source ↗
Figure 2
Figure 2. Figure 2: Effectiveness of our feedback self-adaptive mecha￾nism on an image segmented into grass and cat. The attention maps correspond to a reference patch (red dot) on the cat face. In (a), the reference patch misaligns with grass in the attention map, causing incorrect segmentation in (b). Our proposed FSA adjusts (a) using output predictions to generate semantically aligned at￾tention, focusing on cat in (c), a… view at source ↗
Figure 3
Figure 3. Figure 3: Semantic coherence retention between attention and final predictions (higher is better) and mIoU improvements av￾eraged on 8 benchmarks. Our self-adaptive attention feeds the patch-wise semantics at output back to modulate the intermediate attention to both improve semantic retention and final mIoU. dimension d, (·) is the matrix multiplication, Attninit ∈ R L×L represents the initial attention maps, which… view at source ↗
Figure 4
Figure 4. Figure 4: Overview of our feedback-driven self-adaptive attention framework. We isolate the contribution of initial attention using a dual branch with uniform attention. Pairwise patch similarity is then calculated by applying KL divergence to the output logits. A confidence-based pruning step follows to eliminate irrelevant patches and amplify the correlation between related patches. The resulting sparse feedback a… view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of segmentation results of ProxyCLIP integrated with our FSA. The attention maps (2 nd and 4 th columns) correspond to the reference patch shown in the 1 st column. ProxyCLIP produces with holes within the same object due to weak attention across regions of the object. In contrast, our FSA effectively aggregates similar patches, enabling the correction of missegmented regions. Input MaskCLIP … view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results. By integrating our feedback self￾adaptive mechanism, we correct missegmented patches, ensuring consistent segmentation within each object. SAM [25], and DINOV2 [39]), each displaying varying lev￾els of spatial correspondence. Despite these differences, our method proves flexible, achieving consistent improve￾ments across all VFMs, benchmarks, and backbones. No￾tably, while MAE shows we… view at source ↗
Figure 7
Figure 7. Figure 7: Sensitivity on p (left) and λ (right) with ViT-L/14. As p increases, mIoU rises then falls with added irrelevant patches. λ adjusts amplification and remains stable between 1 and 2.5. Attention isolation MaskCLIP SCLIP ProxyCLIP w/o FSA (baseline) 13.9 29.0 42.9 ✗ 29.8 (+15.9) 33.2 (+4.2) 43.4 (+0.5) ✓ 32.6 (+18.7) 35.0 (+6.0) 43.6 (+0.7) [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative results on attention isolation (iso.). Attention isolation. Our feedback attention modulates the intermediate attention, making it crucial that output rela￾tionships reflect only the initial attention maps. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SPRITE: From Static Mockups to Engine-Ready Game UI

    cs.HC 2026-03 unverdicted novelty 7.0

    SPRITE converts static game UI screenshots into editable engine-ready assets by using VLMs to parse complex layouts into a YAML intermediate representation.

Reference graph

Works this paper leans on

67 extracted references · 48 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. Layer normalization. arXiv preprint arXiv:1607.06450,

    Pith/arXiv arXiv

  2. [2]

    Grounding everything: Emerging localization properties in vision-language transformers

    Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding everything: Emerging localization properties in vision-language transformers. arXiv preprint arXiv:2312.00878, 2023. 2, 3

    Pith/arXiv arXiv 2023

  3. [3]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 1209–1218, 2018. 5

    work page 2018

  4. [4]

    Cascade r-cnn: Delv- ing into high quality object detection

    Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delv- ing into high quality object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 6154–6162, 2018. 3

    work page 2018

  5. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 2, 5

    work page 2021

  6. [6]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11165–11174, 2023. 1, 5

    work page 2023

  7. [7]

    Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only

    Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Sean Chang Cu- latana, and Mohamed Elhoseiny. Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 699–710, 2023. 1, 2

    work page 2023

  8. [8]

    A simple framework for contrastive learn- ing of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learn- ing of visual representations. In International Conference on Machine Learning (ICML), pages 1597–1607. PMLR, 2020. 3

    work page 2020

  9. [9]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023. 1, 5

    work page 2023

  10. [10]

    All at once: Temporally adaptive multi-frame interpolation with advanced motion modeling

    Zhixiang Chi, Rasoul Mohammadi Nasiri, Zheng Liu, Juwei Lu, Jin Tang, and Konstantinos N Plataniotis. All at once: Temporally adaptive multi-frame interpolation with advanced motion modeling. InEuropean conference on com- puter vision, pages 107–123. Springer, 2020. 3

    work page 2020

  11. [11]

    Test- time fast adaptation for dynamic scene deblurring via meta- auxiliary learning

    Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang. Test- time fast adaptation for dynamic scene deblurring via meta- auxiliary learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9137–9146, 2021. 3

    work page 2021

  12. [12]

    Adapt- ing to distribution shift by visual domain prompt generation

    Zhixiang Chi, Li Gu, Tao Zhong, Huan Liu, YUANHAO YU, Konstantinos N Plataniotis, and Yang Wang. Adapt- ing to distribution shift by visual domain prompt generation. In The Twelfth International Conference on Learning Repre- sentations, 2024. 1

    work page 2024

  13. [13]

    Learning to adapt frozen clip for few-shot test-time domain adaptation

    Zhixiang Chi, Li Gu, Huan Liu, Ziqiang Wang, Yanan Wu, Yang Wang, and Konstantinos N Plataniotis. Learning to adapt frozen clip for few-shot test-time domain adaptation. arXiv preprint arXiv:2506.17307, 2025. 1, 12

    Pith/arXiv arXiv 2025

  14. [14]

    Mmsegmentation: Open- mmlab semantic segmentation toolbox and benchmark,

    MMSegmentation Contributors. Mmsegmentation: Open- mmlab semantic segmentation toolbox and benchmark,

  15. [15]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016. 5

    work page 2016

  16. [16]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3

    Pith/arXiv arXiv 2010

  17. [17]

    The pascal visual object classes challenge 2012 (voc2012) development kit

    Mark Everingham and John Winn. The pascal visual object classes challenge 2012 (voc2012) development kit. Pattern Anal. Stat. Model. Comput. Learn., Tech. Rep, 2007(1-45):5,

    work page 2012

  18. [18]

    Bootstrap your own latent: A new ap- proach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doer- sch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new ap- proach to self-supervised learning. Advances in Neural In- formation Processing Systems (NeurIPS) , 33:21271–21284,

    work page

  19. [19]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1321–1330. PMLR, 2017. 3

    work page 2017

  20. [20]

    How to train the teacher model for effective knowledge distillation

    Shayan Mohajer Hamidi, Xizhen Deng, Renhao Tan, Lin- feng Ye, and Ahmed Hussein Salamah. How to train the teacher model for effective knowledge distillation. In Euro- pean Conference on Computer Vision, pages 1–18. Springer,

    work page

  21. [21]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000– 16009, 2022. 2, 6

    work page 2022

  22. [22]

    Distill- ing the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 2, 3, 12

    Pith/arXiv arXiv 2015

  23. [23]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  24. [24]

    Weakly supervised ground- ing for vqa in vision-language transformers

    Aisha Urooj Khan, Hilde Kuehne, Chuang Gan, Niels Da Vi- toria Lobo, and Mubarak Shah. Weakly supervised ground- ing for vqa in vision-language transformers. In European Conference on Computer Vision , pages 652–670. Springer,

    work page

  25. [25]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. arXiv preprint arXiv:2304.02643, 2023. 2, 7

    Pith/arXiv arXiv 2023

  26. [26]

    Clearclip: Decom- posing clip representations for dense vision-language infer- ence

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decom- posing clip representations for dense vision-language infer- ence. arXiv preprint arXiv:2407.12442 , 2024. 1, 2, 3, 6, 12

    Pith/arXiv arXiv 2024

  27. [27]

    Proxyclip: Proxy attention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. arXiv preprint arXiv:2408.04883, 2024. 1, 2, 3, 5, 6, 12

    Pith/arXiv arXiv 2024

  28. [28]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In In- ternational Conference on Machine Learning, pages 12888– 12900. PMLR, 2022. 1, 2

    work page 2022

  29. [29]

    Clip surgery for better explainability with enhancement in open- vocabulary tasks

    Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open- vocabulary tasks. arXiv preprint arXiv:2304.05653, 2023. 2, 3

    Pith/arXiv arXiv 2023

  30. [30]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 7061–7070, 2023. 2

    work page 2023

  31. [31]

    Self-supervised spa- tiotemporal representation learning by exploiting video con- tinuity

    Hanwen Liang, Niamul Quader, Zhixiang Chi, Lizhe Chen, Peng Dai, Juwei Lu, and Yang Wang. Self-supervised spa- tiotemporal representation learning by exploiting video con- tinuity. In Proceedings of the AAAI conference on artificial intelligence, 2022. 2

    work page 2022

  32. [32]

    Refinenet: Multi-path refinement networks for high- resolution semantic segmentation

    Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for high- resolution semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 1925–1934, 2017. 3

    work page 1925

  33. [33]

    Few-shot class-incremental learning via entropy-regularized data-free replay

    Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang. Few-shot class-incremental learning via entropy-regularized data-free replay. In European Con- ference on Computer Vision, pages 146–162. Springer, 2022. 3

    work page 2022

  34. [34]

    Meta-auxiliary learning for future depth pre- diction in videos

    Huan Liu, Zhixiang Chi, Yuanhao Yu, Yang Wang, Jun Chen, and Jin Tang. Meta-auxiliary learning for future depth pre- diction in videos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023. 2

    work page 2023

  35. [35]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3

    Pith/arXiv arXiv 2023

  36. [36]

    Ttt++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems , 34: 21808–21820, 2021

    Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Ttt++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems , 34: 21808–21820, 2021. 12

    work page 2021

  37. [37]

    The role of context for object detection and semantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 891–898, 2014. 5

    work page 2014

  38. [38]

    Open vocabulary semantic segmentation with patch aligned con- trastive learning

    Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned con- trastive learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 19413–19423, 2023. 1

    work page 2023

  39. [39]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 7

    Pith/arXiv arXiv 2023

  40. [40]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 1, 5

    work page 2021

  41. [41]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- formation Processing Systems, 35:25278–25294, 2022. 1

    work page 2022

  42. [42]

    Test- time prompt tuning for zero-shot generalization in vision- language models

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. Advances in Neural Information Process- ing Systems, 35:14274–14289, 2022. 12

    work page 2022

  43. [43]

    Test-time training with self- supervision for generalization under distribution shifts

    Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self- supervision for generalization under distribution shifts. In International conference on machine learning. PMLR, 2020. 2

    work page 2020

  44. [44]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Ol- shausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726,

    Pith/arXiv arXiv 2006

  45. [45]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A Ol- shausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (ICLR), 2021. 3

    work page 2021

  46. [46]

    Sclip: Rethink- ing self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethink- ing self-attention for dense vision-language inference. arXiv preprint arXiv:2312.01597, 2023. 1, 2, 3, 5, 6

    Pith/arXiv arXiv 2023

  47. [47]

    SAM-CLIP: Merging vision foundation mod- els towards semantic and spatial understanding

    Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. SAM-CLIP: Merging vision foundation mod- els towards semantic and spatial understanding. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023. 1, 2

    work page 2023

  48. [48]

    Distribution align- ment for fully test-time adaptation with dynamic online data streams

    Ziqiang Wang, Zhixiang Chi, Yanan Wu, Li Gu, Zhi Liu, Konstantinos Plataniotis, and Yang Wang. Distribution align- ment for fully test-time adaptation with dynamic online data streams. In European Conference on Computer Vision, pages 332–349. Springer, 2024. 3, 12

    work page 2024

  49. [49]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gon- tijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 7959–7971, 2022. 1, 2

    work page 2022

  50. [50]

    Clipself: Vision trans- former distills itself for open-vocabulary dense prediction

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision trans- former distills itself for open-vocabulary dense prediction. arXiv preprint arXiv:2310.01403, 2023. 2

    Pith/arXiv arXiv 2023

  51. [51]

    Metagcd: Learning to continually learn in generalized cat- egory discovery

    Yanan Wu, Zhixiang Chi, Yang Wang, and Songhe Feng. Metagcd: Learning to continually learn in generalized cat- egory discovery. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , pages 1655–1665,

    work page

  52. [52]

    Test-time domain adaptation by learning domain-aware batch normalization

    Yanan Wu, Zhixiang Chi, Yang Wang, Konstantinos N Pla- taniotis, and Songhe Feng. Test-time domain adaptation by learning domain-aware batch normalization. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pages 15961–15969, 2024. 12

    work page 2024

  53. [53]

    Clip-dinoiser: Teaching clip a few dino tricks.arXiv preprint arXiv:2312.12359, 2023

    Monika Wysocza ´nska, Oriane Sim´eoni, Micha¨el Ramamon- jisoa, Andrei Bursuc, Tomasz Trzci ´nski, and Patrick P ´erez. Clip-dinoiser: Teaching clip a few dino tricks.arXiv preprint arXiv:2312.12359, 2023. 5

    Pith/arXiv arXiv 2023

  54. [54]

    Clip-diy: Clip dense infer- ence yields open-vocabulary semantic segmentation for-free

    Monika Wysocza ´nska, Micha ¨el Ramamonjisoa, Tomasz Trzci´nski, and Oriane Sim ´eoni. Clip-diy: Clip dense infer- ence yields open-vocabulary semantic segmentation for-free. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision , pages 1403–1413, 2024. 1, 2

    work page 2024

  55. [55]

    Demystify- ing clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystify- ing clip data. arXiv preprint arXiv:2309.16671, 2023. 1

    Pith/arXiv arXiv 2023

  56. [56]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18134–18144, 2022. 1

    work page 2022

  57. [57]

    Markov knowledge distil- lation: Make nasty teachers trained by self-undermining knowledge distillation fully distillable

    En-hui Yang and Linfeng Ye. Markov knowledge distil- lation: Make nasty teachers trained by self-undermining knowledge distillation fully distillable. In European Con- ference on Computer Vision, pages 154–171. Springer, 2024. 2

    work page 2024

  58. [58]

    Conditional mutual information con- strained deep learning: Framework and preliminary results

    En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information con- strained deep learning: Framework and preliminary results. In 2024 IEEE International Symposium on Information The- ory (ISIT), pages 569–574. IEEE, 2024. 3

    work page 2024

  59. [59]

    Bayes conditional distribution estimation for knowledge distillation based on conditional mutual informa- tion

    Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, and EN- HUI Y ANG. Bayes conditional distribution estimation for knowledge distillation based on conditional mutual informa- tion. In The Twelfth International Conference on Learning Representations, 2020. 2

    work page 2020

  60. [60]

    Towards undistillable models by minimizing conditional mu- tual information

    Linfeng Ye, Shayan Mohajer Hamidi, and EN-HUI Y ANG. Towards undistillable models by minimizing conditional mu- tual information. Transactions on Machine Learning Re- search, 2025. 3

    work page 2025

  61. [61]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 1, 2

    Pith/arXiv arXiv 2022

  62. [62]

    Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively

    Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open-vocabulary sam: Segment and recognize twenty-thousand classes interactively. arXiv preprint arXiv:2401.02955, 2024. 1, 2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  63. [63]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation

    Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chen- glong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3713–3722, 2019. 3, 12

    work page 2019

  64. [64]

    Self- distillation: Towards efficient and compact neural networks

    Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self- distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 44(8):4388–4403, 2021. 2, 12

    work page 2021

  65. [65]

    Meta-dmoe: Adapting to domain shift by meta- distillation from mixture-of-experts

    Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu, and Jin Tang. Meta-dmoe: Adapting to domain shift by meta- distillation from mixture-of-experts. Advances in Neural In- formation Processing Systems, 2022. 2, 12

    work page 2022

  66. [66]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019. 5

    work page 2019

  67. [67]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Com- puter Vision, pages 696–712. Springer, 2022. 1, 2, 3, 6 A. Summary In this supplementary material, we present the following ad- ditional content to complement the main paper: • Additional qualitative comparisons on various datasets. • We present mo...

    work page 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.