Pith. sign in

REVIEW 3 major objections 7 minor 55 references

This paper claims that CLIP can gain fine-grained region understanding from plain image-text pairs alone, using heat maps from the model's own gradients to self-annotate region-phrase pairs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 04:29 UTC pith:AZRI2CG4

load-bearing objection A useful, reasonably solid CLIP fine-tuning method that uses the model's own XAI heatmaps as region pseudo-labels; the central mechanism is plausible but not externally validated, and the ablation tables have a concrete inconsistency that needs fixing. the 3 major comments →

arxiv 2607.13661 v1 pith:AZRI2CG4 submitted 2026-07-15 cs.CV

Fine-grained CLIP fine-tuning with self-annotated region alignment

Chenyang Zhao , Wei Lin , Antoni B. Chan , Janet H. Hsiao This is my paper
classification cs.CV
keywords CLIP fine-tuningfine-grained representationregion-phrase alignmentself-annotationtext-specific heat mapsexplainable AIopen-vocabulary detectionimage-text retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes SFF-CLIP, a fine-tuning method that improves CLIP's fine-grained region understanding using only image-text pairs, the same data CLIP was pretrained on. Instead of relying on region annotations, predefined categories, or external detectors, it extracts noun phrases from captions, locates them in the image with text-specific heat maps generated by the model's own gradients, and aligns the resulting region features with phrase embeddings. The authors report significant gains on zero-shot region classification, open-vocabulary object detection, and semantic segmentation, while preserving CLIP's image-level retrieval performance. If correct, this removes the main data-preparation bottleneck in adapting CLIP for dense prediction tasks.

Core claim

SFF-CLIP establishes that fine-grained region-language alignment can be self-annotated at runtime during fine-tuning. From each caption, adjective+noun phrases are extracted and encoded as text embeddings; a text-specific heat map, computed with a gradient-based explainability method (Grad-ECLIP), localizes each phrase in the image; the image's dense feature map is weighted by that heat map to produce a region embedding; and a matching loss pulls each region embedding toward its phrase embedding while pushing it away from other phrases. A phrase-matching weight, set to the heat map's maximum value, down-weights phrases that are likely absent from the image. The global contrastive loss is ret

What carries the argument

The central mechanism is the text-specific heat map H_t = sum_c g_c · u_i · v_i, computed as a channel-weighted aggregation of gradient and attention information for each extracted phrase. It converts a phrase embedding into a spatial localization signal, which is then used to aggregate the dense feature map F_d into a region embedding F_rt. The weighted fine-grained matching loss, with per-phrase weight w_t = max(H_t), is what carries the alignment; the momentum-weighted global contrastive loss preserves image-level ability.

Load-bearing premise

The heat maps produced by the explainability method correctly localize each extracted noun phrase in the image, so that the weighted region features used as training targets are trustworthy; the paper itself acknowledges this dependence.

What would settle it

Use a dataset with ground-truth region annotations, fine-tune with SFF-CLIP, and measure the overlap between each extracted phrase's heat-map peak and the corresponding object's ground-truth mask. If a large fraction of high-weight phrases localize outside their true object, the self-annotated pairs would be reinforcing misalignment; one could then test whether artificially corrupting the heat maps degrades the reported region-classification gains.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • CLIP can be adapted for region-level tasks without any region annotations, predefined category pools, or external region proposal models, lowering the cost of scaling fine-grained training.
  • Fine-grained fine-tuned CLIP backbones improve open-vocabulary object detection on novel categories and rare categories, as shown on OV-COCO and OV-LVIS.
  • The method improves open-vocabulary semantic segmentation across multiple benchmarks, exceeding prior region-annotation-based fine-tuning methods.
  • Unlike several previous fine-grained fine-tuning methods, SFF-CLIP preserves or slightly improves CLIP's image-text retrieval performance, so the same backbone can serve both image-level and dense-prediction tasks.
  • The phrase-matching weight is essential: without it, fine-grained alignment training can degrade performance below the original CLIP, indicating that filtering mislocalized phrases is load-bearing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the supervision signal comes from the model's own heat maps, the method's ceiling is tied to the quality of the explainability technique; any future improvement in gradient-based localization for vision-language models should directly translate into stronger fine-grained CLIP fine-tuning.
  • The adjective+noun phrase-extraction rule means captions that describe scenes without concrete noun phrases provide little supervision; pairing the method with a captioner that generates denser, more object-focused descriptions could further boost gains.
  • A testable extension is to apply the same self-annotated alignment to other vision-language encoders or to multimodal models beyond CLIP, where heat maps can be produced from the model's own gradients.
  • One could also use SFF-CLIP as a warm-start before detector-based fine-tuning, potentially combining self-annotated region alignment with explicit region supervision when available.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes SFF-CLIP, a fine-tuning method for CLIP that uses only image-text pairs. For each caption, adjective+noun phrases are extracted with NLTK; Grad-ECLIP heatmaps are generated per phrase; the heatmaps weight the dense visual features to form region embeddings; these are aligned with phrase embeddings using the loss in Eq. (3). A phrase-matching weight wt = max(Ht) down-weights poorly matched phrases. Global CLIP contrastive learning is kept, with a momentum model used to re-weight image-text pairs so that global retrieval performance is maintained. Experiments cover zero-shot region classification on ADE20K and COCO panoptic, open-vocabulary detection on COCO and LVIS, open-vocabulary segmentation with CatSeg, Flickr30k retrieval, and ablations over wt, the momentum model, the number of phrases N, XAI method, image size, and data scale. The paper claims significant gains in fine-grained dense representation with no region annotations or external detectors, while preserving image-level CLIP performance.

Significance. If the claims hold, the contribution is substantial: it removes the need for region proposals, region labels, pre-defined category pools, and external vision models in CLIP fine-grained adaptation, which would lower the barrier for dense-prediction applications. The empirical scope is broad: the method is compared against RegionCLIP, CLIPSelf, FineCLIP, DenseVLM, CLIM, and CLIP-g across several benchmarks, and ablations cover the main components. The paper is also honest in stating in Limitation H that the method inherits the quality of the adopted XAI heatmaps. However, the central supervision signal is self-generated, and the paper does not validate that signal against any external region-grounding criterion. This, together with a numerical inconsistency in the ablation tables and the absence of error bars, leaves the core mechanism not fully established.

major comments (3)
  1. [Sec. 3.2.2, Eq. (2); Limitation H] The entire fine-grained supervision is built on text-specific heatmaps H_t from Grad-ECLIP, and the paper never measures whether these heatmaps localize the extracted phrase accurately. Eq. (2) uses H_t to aggregate the dense features F_d, and Eq. (3) then aligns the resulting region embedding with the phrase embedding. If H_t is confidently wrong (e.g., peaking on a distractor or background), the loss actively reinforces an incorrect region-phrase association; w_t = max(H_t) only down-weights low-confidence heatmaps, not confidently wrong ones. The paper's own Limitation H admits this dependence. Since the central abstract claim is that fine-grained alignment improves without annotations, the core mechanism needs a concrete validation: for example, measure heatmap localization quality against available region annotations on a validation set, compare with an oracle that uses ground-truth
  2. [Table 5 vs Tables 1 and 7] There is a numerical inconsistency in the ablation for the phrase-matching weight. Table 5 reports SFF-CLIP 'w/ wt' as 57.8 boxes / 50.1 masks, while Table 7 reports N=5 (which is the default configuration) as 62.2 / 52.1, and Table 1 reports the same SFF-CLIP ViT-B/16 row as 62.2 / 52.1. Since Table 5 is the only evidence isolating wt, the mismatch makes the claimed effect of wt unreliable. Moreover, all tables appear to report single runs with no error bars; for differences of a few points (e.g., Table 4 retrieval, Table 2 OVD), this is not sufficient to support the comparisons. Please report the exact configuration for each ablation, clarify the discrepancy, and provide mean±std over at least 3 seeds for the main tables.
  3. [Sec. 3.2.3, Eq. (3)] The weighting mechanism w_t = max(H_t) is presented as a measure of phrase-image matching, but it is not calibrated or evaluated. The paper gives an example in Figure 3 but no statistics on the distribution of w_t, no precision/recall for the NLTK 'adjective+noun' extraction, and no check that w_t correlates with actual phrase presence or correct localization. If many extracted phrases are absent from the image, the weighted loss could still train on false positives whenever the heatmap has a high spurious response. Please report extraction statistics (e.g., fraction of phrases that match annotated objects), analyze the distribution of w_t, and add an ablation with a strict threshold or a calibrated filter.
minor comments (7)
  1. [General] The acronym is inconsistently written as 'SFF-CLIP' and 'SSF-CLIP' in multiple places (e.g., Table 6, Appendix C, Appendix D, Appendix G). Please unify.
  2. [Sec. 3.3, after Eq. (6)] Typo: 'by by adding' should be 'by adding'.
  3. [Conclusion] Typo: 'eithetasksnually' should be 'either manually'.
  4. [Table 4] Dataset name is misspelled as 'Flicker30k'; should be 'Flickr30k'.
  5. [Eq. (3)] The negative-pair notation is unclear: it is not specified whether t' runs over phrases within the same caption, within the same batch, or across all captions in the batch. Please clarify the indexing.
  6. [Table 1] DenseVLM appears twice, with very different numbers (18.5 vs 60.1 for boxes). The footnote explains the source, but the presentation is confusing; please label rows more clearly or give the explanation in the caption.
  7. [Fig. 2] The text 'weight momentum model update' in the figure is grammatically unclear; please rephrase.

Circularity Check

0 steps flagged

No significant circularity: the self-annotated training signal is self-referential, but the central claim is tested on external benchmarks and the key XAI choice is ablated.

full rationale

The fine-grained supervision is self-generated: Eq. (2) defines text-specific heatmaps from the model's own gradients/attention and forms region embeddings as F_rt = sum(H_t * F_d), then Eq. (3) trains those region embeddings to match phrase embeddings. This is indeed a self-referential training loop, and the paper's Limitation H explicitly concedes that "the self-annotated region-phrase alignment will be influenced by the performance of heat maps generated by the adopted XAI method." However, that is a correctness/robustness limitation, not a circular derivation: the paper never claims the self-annotated pairs are ground truth, and the central claims (Tables 1-4) are evaluated on external dense-prediction and retrieval benchmarks that do not reuse the same pseudo-labeling. The reliance on Grad-ECLIP, a prior work by the same first author, is not an unverified self-citation chain because Appendix C ablates it against Grad-CAM and MaskCLIP and reports independent gains. The reported improvements are empirical outcomes; whether the heatmap signal is trustworthy is an open validation issue, not an instance of a result reducing to its inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The method's effectiveness rests on the quality of self-generated region-phrase pairs: (1) Grad-ECLIP heatmaps must localize the phrase's concept; (2) NLTK phrase extraction must select visually present concepts; (3) the modified dense feature map must preserve spatial semantics; (4) COCO captions must provide sufficient phrase-region co-occurrence; (5) the momentum model must preserve the global distribution. No external validation of pseudo-label accuracy is provided.

free parameters (4)
  • Maximum phrases per caption N = 5
    Ablated in Table 7; N=3 gives close performance, N=5 chosen as default. This cap controls how many region-phrase pairs are created per image.
  • Momentum rate alpha = 0.005
    Set in Section 3.3 for updating the momentum CLIP model; no sensitivity analysis is provided.
  • Momentum weight gamma = 0.4
    Set in Section 3.3 to control the contribution of momentum-model matching confidence in Eq. (5); no sensitivity analysis is provided.
  • Fine-grained/contrastive loss balance = 1.0 (implicit)
    Final loss L = L_fg + L_contrastive (Section 3.3); equal weighting is adopted without a scaling factor or sensitivity study, despite likely different loss scales.
axioms (5)
  • domain assumption Grad-ECLIP text-specific heatmaps localize the phrase concept in the image well enough to produce correct region features.
    Section 3.2.2 Eq. (2); the region embedding is a heatmap-weighted average of dense features. Limitation H acknowledges the method inherits XAI quality.
  • domain assumption NLTK 'adjective + noun' extraction yields noun phrases that correspond to visually present objects; absent phrases can be detected by low w_t = max(H_t).
    Section 3.2.2; Fig. 3 gives counterexamples ('dinner', 'Tuesday') handled only by the heatmap-max weight.
  • domain assumption The modified last transformer layer (removing self-attention, keeping projection/norm) produces a dense feature map with sufficient spatial detail for region matching.
    Section 3.2.1, following MaskCLIP and CLIPSelf; no new validation for this use beyond cited prior work.
  • domain assumption MS COCO image-caption pairs contain enough phrase-region co-occurrence to supervise dense alignment.
    Training is on COCO train2017 captions; if captions are not grounded in visible objects, the pseudo-labels are noisy.
  • domain assumption The EMA momentum model tracks the original CLIP distribution and yields reliable matching weights for the contrastive loss.
    Section 3.3 Eqs. (5)-(6); the weighting scheme assumes the momentum model is a better reference for preserving global ability.

pith-pipeline@v1.3.0-alltime-deepseek · 18321 in / 15219 out tokens · 662296 ms · 2026-08-02T04:29:05.804754+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Fine-grained CLIP fine-tuning with self-annotated region alignment." pith.science (2026). https://pith.science/paper/AZRI2CG4

@misc{pith2026260713661,
  author       = {Pith},
  title        = {Pith review of: Fine-grained CLIP fine-tuning with self-annotated region alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AZRI2CG4}},
  note         = {Machine review of arXiv:2607.13661}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Contrastive Language-Image Pre-training (CLIP) has been shown to have limitations in its fine-grained dense feature representation, due to its pre-training focusing on matching the whole image to a text description. Considering the large data and computational burden in pre-training a vision-language model from scratch, a series of works aim to enhance the fine-grained ability of CLIP through a fine-tuning scheme. However, existing works suffer from a variety of limitations: additional region annotations are usually required, which limits the semantic diversity due to the predefined categories and leads to a large effort to process the training data; and they usually sacrifice CLIP's original ability for global visual representation. To bypass these limitations, we propose SFF-CLIP (Self-annotated Fine-grained Fine-tuning for CLIP), which only uses image-text pairs as input to boost the fine-grained representation ability in the CLIP fine-tuning, while maintaining the global visual-semantic consistency. Concretely, a run-time region-phrase alignment scheme is designed, which obtains concept phrases from the input sentence, and aligns them with corresponding extracted region-based features using text-specific heat maps. Extensive experiments demonstrate that SFF-CLIP leads to significant performance improvements on fine-grained dense feature representation, as well as maintaining the performance of the original CLIP on image-level tasks. Code will be released later.

Figures

Figures reproduced from arXiv: 2607.13661 by Antoni B. Chan, Chenyang Zhao, Janet H. Hsiao, Wei Lin.

Figure 1
Figure 1. Figure 1: Comparison of different fine-grained alignment methods for CLIP. The required preparations for region [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed SFF-CLIP. Multiple phrases or words representing objects (e.g., “dog”, “black [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An example of phrase alignment weight wt, which is defined as the maximum value on the corresponding text-specific heat map. In this image-text pair, “beer” and “banana” are better matched with the image regions, as indicated by their higher weights wt than “dinner” and “Tuesday”. where τ is the trainable temperature parameter. To better maintain the image-level representation ability of the pre-trained CL… view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of the fine-grained representations using similarity maps between the image dense feature and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study on training data in three different scales. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of cosine similarity maps between text embeddings and the dense feature maps generated by [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 7 linked inside Pith

  1. [1]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  2. [2]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InCVPR, pages 3558–3568, 2021

    2021

  3. [3]

    Domain generalization by mutual-information regularization with pre-trained models

    Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual-information regularization with pre-trained models. InEuropean conference on computer vision, pages 440–457. Springer, 2022

    2022

  4. [4]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.Neurocomputing, 508:293–304, 2022

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.Neurocomputing, 508:293–304, 2022

    2022

  5. [5]

    A simple baseline for zeroshot semantic segmentation with pre-trained vision-language model.ECCV, 2022

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zeroshot semantic segmentation with pre-trained vision-language model.ECCV, 2022

    2022

  6. [6]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. InCVPR, pages 7061–7070, 2023

    2023

  7. [7]

    Open-vocabulary object detection via vision and language knowledge distillation.ICLR, 2022

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation.ICLR, 2022. 14 Fine-grained CLIP fine-tuning with self-annotated region alignment

    2022

  8. [8]

    Learning to prompt for open-vocabulary object detection with vision-language model

    Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. InCVPR, pages 14084–14093, 2022

    2022

  9. [9]

    F-vlm: Open-vocabulary object detection upon frozen vision and language models

    Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vlm: Open-vocabulary object detection upon frozen vision and language models. InICLR, 2023

    2023

  10. [10]

    Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching

    Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. InCVPR, pages 7031–7040, 2023

    2023

  11. [11]

    Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip.NeurIPS, 36, 2024

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip.NeurIPS, 36, 2024

    2024

  12. [12]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Fg-clip: Fine-grained visual and textual alignment

    Chunyu Xie et al. Fg-clip: Fine-grained visual and textual alignment. InICML, 2025

    2025

  14. [14]

    Improving fine-grained understanding in image-text pre-training

    Ioana Bica et al. Improving fine-grained understanding in image-text pre-training. InICML, 2024

    2024

  15. [15]

    Position-guided text prompt for vision-language pre-training

    Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan. Position-guided text prompt for vision-language pre-training. InCVPR, pages 23242–23251, 2023

    2023

  16. [16]

    Regionclip: Region-based language-image pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. InCVPR, pages 16793–16803, 2022

    2022

  17. [17]

    Densevlm: A retrieval and decoupled alignment framework for open-vocabulary dense prediction

    Yunheng Li, Yuxuan Li, Quansheng Zeng, Wenhai Wang, Qibin Hou, and Ming-Ming Cheng. Densevlm: A retrieval and decoupled alignment framework for open-vocabulary dense prediction. InICCV, 2025

    2025

  18. [18]

    Faster r-cnn: Towards real-time object detection with region proposal networks.NeurIPS, 28, 2015

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks.NeurIPS, 28, 2015

    2015

  19. [19]

    Fineclip: Self-distilled region-based clip for better fine-grained understanding.Advances in Neural Information Processing Systems, 37:27896–27918, 2024

    Dong Jing, Xiaolong He, Yutian Luo, Nanyi Fei, Wei Wei, Huiwen Zhao, Zhiwu Lu, et al. Fineclip: Self-distilled region-based clip for better fine-grained understanding.Advances in Neural Information Processing Systems, 37:27896–27918, 2024

    2024

  20. [20]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  21. [21]

    Clipself: Vision transformer distills itself for open-vocabulary dense prediction.arXiv preprint arXiv:2310.01403, 2023

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction.arXiv preprint arXiv:2310.01403, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    Atas: Any-to-any self-distillation for enhanced open-vocabulary dense prediction

    Juan Yeo et al. Atas: Any-to-any self-distillation for enhanced open-vocabulary dense prediction. InICCV, 2025

    2025

  23. [23]

    Clim: Contrastive language-image mosaic for region representation

    Size Wu et al. Clim: Contrastive language-image mosaic for region representation. InAAAI, volume 38, pages 6117–6125, 2024

    2024

  24. [24]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337–2348, 2022

    2022

  25. [25]

    Coca: Contrastive captioners are image-text foundation models.arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv:2205.01917, 2022

    Pith/arXiv arXiv 2022

  26. [26]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InICML, 2022

    2022

  27. [27]

    Region-aware pretraining for open-vocabulary object detection with vision transformers

    Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for open-vocabulary object detection with vision transformers. InCVPR, 2023

    2023

  28. [28]

    Open-vocabulary object detection upon frozen vision and language models

    Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. Open-vocabulary object detection upon frozen vision and language models. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  29. [29]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. InEuropean conference on computer vision, pages 728–755. Springer, 2022

    2022

  30. [30]

    Scaling open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. InEuropean conference on computer vision, pages 540–557. Springer, 2022

    2022

  31. [31]

    Cascade-clip: Cascaded vision-language embeddings alignment for zero-shot semantic segmentation

    Yunheng Li, Zhong-Yu Li, Quan-Sheng Zeng, Qibin Hou, and Ming-Ming Cheng. Cascade-clip: Cascaded vision-language embeddings alignment for zero-shot semantic segmentation. InInternational Conference on Machine Learning, pages 28243–28258. PMLR, 2024. 15 Fine-grained CLIP fine-tuning with self-annotated region alignment

    2024

  32. [32]

    Visualizing and understanding convolutional networks

    Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. InECCV, pages 818–833, 2014

    2014

  33. [33]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. InICCV, pages 618–626, 2017

    2017

  34. [34]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. InACM SIGKDD, 2016

    2016

  35. [35]

    Rise: Randomized input sampling for explanation of black-box models

    Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv:1806.07421, 2018

    Pith/arXiv arXiv 2018

  36. [36]

    Odam: Gradient-based instance-specific visual explanations for object detection

    Chenyang Zhao and Antoni B Chan. Odam: Gradient-based instance-specific visual explanations for object detection. InICLR, 2023

    2023

  37. [37]

    Attcat: Explaining transformers via attentive class activation tokens.NeurIPS, 2022

    Yao Qiang, Deng Pan, Chengyin Li, Xin Li, Rhongho Jang, and Dongxiao Zhu. Attcat: Explaining transformers via attentive class activation tokens.NeurIPS, 2022

    2022

  38. [38]

    Vit-cx: Causal explanation of vision transformers

    Weiyan Xie, Xiao-Hui Li, Caleb Chen Cao, and Nevin L Zhang. Vit-cx: Causal explanation of vision transformers. pages 1569–1577, 2023

    2023

  39. [39]

    X-pruner: explainable pruning for vision transformers

    Lu Yu and Wei Xiang. X-pruner: explainable pruning for vision transformers. InCVPR, pages 24355–24363, 2023

    2023

  40. [40]

    Gradient-based visual explanation for transformer-based clip

    Chenyang Zhao, Kun Wang, Xingyu Zeng, Rui Zhao, and Antoni B Chan. Gradient-based visual explanation for transformer-based clip. InICML, pages 61072–61091. PMLR, 2024

    2024

  41. [41]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. InECCV, pages 696–712. Springer, 2022

    2022

  42. [42]

    O’Reilly Media, Inc., 2009

    Steven Bird, Ewan Klein, and Edward Loper.Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc., 2009

    2009

  43. [43]

    Eva-clip: Improved training techniques for clip at scale.preprint arXiv:2303.15389, 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale.preprint arXiv:2303.15389, 2023

    Pith/arXiv arXiv 2023

  44. [44]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean conference on computer vision, pages 740–755. Springer, 2014

    2014

  45. [45]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. InCVPR, 2017

    2017

  46. [46]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Pith/arXiv arXiv 2015

  47. [47]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019

    2019

  48. [48]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. InCVPR, pages 14393–14402, 2021

    2021

  49. [49]

    Detecting twenty-thousand classes using image-level supervision

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. InECCV, 2022

    2022

  50. [50]

    Learning object-language alignments for open-vocabulary object detection.arXiv preprint arXiv:2211.14843, 2022

    Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection.arXiv preprint arXiv:2211.14843, 2022

    Pith/arXiv arXiv 2022

  51. [51]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- network.org/challenges/VOC/voc2012/workshop/index.html, 2012

    2012

  52. [52]

    The role of context for object detection and semantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. InCVPR, pages 891–898, 2014

    2014

  53. [53]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmentation

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. InCVPR, pages 4113–4123, 2024

    2024

  54. [54]

    Coco-stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. InCVPR, pages 1209–1218, 2018. 16 Fine-grained CLIP fine-tuning with self-annotated region alignment

    2018

  55. [55]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. InICCV, 2015. 17

    2015