REVIEW 3 major objections 5 minor 85 references
FOR: Finetuning for Object Level Open Vocabulary Image Retrieval
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read FOR fine-tunes CLIP's image encoder with a learnable-query head and a pseudo-label loss so that object-centric open-vocabulary retrieval improves by up to 8 mAP@50 points over the previous state of the art, using only 50 embeddings per…
desk verdict Competent, well-ablated fine-tuning method for object-centric open-vocabulary retrieval with genuine gains, but the headline novel-class improvements rest on a pseudo-label threshold tuned without a clear split and noisy pseudo-labels that are never quantified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is SUM-CLIP, a variant of CLIP's last attention layer that produces $N$ representative embeddings per image (typically 50) instead of one global embedding or hundreds of dense patch embeddings. Its learnable queries $\mathbf{Q} \in \mathbb{R}^{N \times C_e}$ are modulated by DETR-style decoder layers conditioned on the image feature map, and then fed into CLIP's original cross-attention with the query, key, value, and output linear layers frozen. This lets gradients flow into a compact set of object summaries while preserving the original CLIP text-image alignment. The second load-bearing mechanism is the two-loss training scheme: the supervised loss uses base-category text embeddings and Hungarian matching (the DETR set-prediction loss), while the pseudo-label loss uses softmax-thresholded Cluster-CLIP similarities to ImageNet-21K text embeddings as targets, extending the vocabulary beyond the dataset's closed set.
What would settle it
Annotate a random sample of the pseudo-labels accepted at the $5 \times 10^{-4}$ threshold and measure their precision; then retrain FOR with those labels removed or corrected. If the novel-category mAP@50 gain over Cluster-CLIP survives the correction, or vanishes when the labels are removed, the paper's proposed mechanism—that pseudo-label knowledge transfer drives the gain—would be supported or refuted.
Extended reading notes
Core claim
The paper's central claim is that open-vocabulary forgetting, the usual failure mode when fine-tuning CLIP on closed-set labels, can be avoided by training a decoder-style CLIP head with two objectives: one that matches a set of learnable query embeddings to base-category text embeddings, and one that matches the same embeddings to pseudo-labels produced by Cluster-CLIP over ImageNet-21K. The SUM-CLIP head replaces CLIP's final pooling attention with a small set of learnable queries modulated by decoder layers, so the model can summarize multiple objects in an image in a fixed number of embeddings while keeping the last attention layer's linear projections frozen to preserve CLIP's vision-language association. During training a frozen Cluster-CLIP head supplies the pseudo-labels; at inference only the SUM-CLIP head is used. The paper reports that the full FOR system surpasses Cluster-CLIP by up to 8 mAP@50 points on novel categories across COCO, LVIS, and nuImages, and that the pseudo-label branch alone already beats Cluster-CLIP on most benchmarks.
Load-bearing premise
The extra training labels are machine-generated guesses about which of thousands of object categories appear in each image; the whole method depends on those guesses being correct often enough to teach the model about categories it was never given human labels for.
Editorial extensions
If this is right
- If FOR's central claim holds, object-centric retrieval can be fine-tuned to a target domain without sacrificing open-vocabulary performance, meaning closed-set labels can improve retrieval for queries that were never annotated.
- Eliminating CPU clustering at inference, with 50 embeddings per image instead of thousands, makes the method compatible with large-scale nearest-neighbor indexes while roughly tripling visual inference speed relative to Cluster-CLIP.
- Because the pseudo-label branch extends beyond dataset categories, the same framework can exploit unlabeled data: at 1-10% labeled data on COCO, novel-category retrieval stays near the fully supervised level.
- The method also transfers to open-vocabulary multi-label classification, where it outperforms CLIP and a prior CLIP-based method on COCO-2014.
- The design suggests that freezing the linear layers of CLIP's last attention module is a generally useful stability-preserving choice when fine-tuning CLIP heads for retrieval.
Reading between the lines
- Editorial extension: The success of the pseudo-label branch suggests that a similar recipe could improve retrieval for other dual-encoder vision-language models, provided their last attention layers can be converted into a learnable-query head with frozen projections.
- Editorial extension: Since the paper acknowledges pseudo-label noise but never quantifies it, a natural next test is measuring how novel-category gains vary with the softmax threshold in Eq. (4); if performance is flat over a wide threshold range, the method is robust to pseudo-label noise, whereas a sharp peak would make threshold selection load-bearing.
- Editorial extension: FOR's 50-embedding representation could serve as a compact region-proposal-free feature for open-vocabulary detection or segmentation, trading some spatial precision for a much smaller index footprint; the paper only gestures at these applications.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FOR, a fine-tuning framework for object-centric open-vocabulary image retrieval (OC-OVIR). FOR modifies CLIP's last attention layer into a SUM-CLIP head with learnable queries and decoder layers, producing a small number of image embeddings without CPU clustering. Training combines a supervised loss on base categories with a pseudo-label loss generated by Cluster-CLIP over ImageNet-21K classes. Experiments on COCO, LVIS, and nuImages report consistent improvements over Cluster-CLIP, especially on novel categories, with 50 embeddings per image and no CPU post-processing. The paper also reports semi-supervised results showing that pseudo-labels alone can maintain most of the novel-category performance under very low labeling rates.
Significance. If the reported results hold, FOR is a practically relevant advance: it shows that fine-tuning for OC-OVIR can improve novel-category retrieval while retaining a compact, dual-encoder representation, and it provides a concrete recipe (SUMs-CLIP head plus two-loss training) that outperforms the existing non-finetuned state of the art. The paper's strengths include systematic comparisons across three datasets, ablations of the head design, freezing strategy, query count, decoder depth, and pseudo-label source, and an explicit semi-supervised evaluation. The central weakness is that the pseudo-label mechanism, which is the sole driver of the novel-category gains, depends on a confidence threshold whose selection protocol is under-specified and whose noise is acknowledged but never quantified. This leaves the headline improvement over Cluster-CLIP not fully established.
major comments (3)
- [Section 4.2 and Section 3.2 (Eq. 4)] The pseudo-label confidence threshold is load-bearing: Table 1 shows that removing the pseudo-label loss collapses COCO novel mAP@50 from 89.17 to 21.51, and Section 4.2 states only that the threshold was "chosen through an hyper-parameter search to be 5e−4" without specifying the split used. If the COCO, LVIS, or nuImages validation sets were used for this search, the reported novel-category gains are tuned to the evaluation protocol. Please specify the validation procedure and provide a sensitivity analysis over the threshold (e.g., 1e−5 to 1e−2) either on a disjoint split or with a single threshold fixed across all three datasets, reporting the resulting mAP@50 for base and novel categories.
- [Section 3.2 and Supplementary Section D] The pseudo-labels are generated by Cluster-CLIP, which is also the primary baseline in Tables 1 and 3. The paper acknowledges in Supplement D that pseudo-labels contain errors, including association errors (an airplane image yielding 'AirFrance' then 'Parisian') and context errors, but it never quantifies the noise rate. Because the pseudo-label loss is the mechanism responsible for the novel-category gains, the reported improvements could in part reflect systematic biases in the teacher rather than genuine open-vocabulary ability. Please report a quantitative evaluation of pseudo-label quality (e.g., precision/recall against a held-out sample of ground-truth annotations, or the fraction of images receiving at least one erroneous label) and, if possible, an experiment with corrupted or randomly replaced pseudo-labels to show how performance degrades.
- [Table 1 and Section 4.3] The claim of "up to 8 mAP@50 points" over state of the art is based on comparisons where FOR is fine-tuned on the target dataset while Cluster-CLIP is not. This is a legitimate comparison for the OC-OVIR setting, but the paper should state explicitly that the improvement is achieved under the assumption that target-dataset base labels are available for fine-tuning. For practitioners, this is an important caveat, and the abstract and introduction should be clearer that the gains apply to the fine-tuning setting, not to zero-shot retrieval from a frozen model.
minor comments (5)
- [Abstract and Section 4.3] The abstract says "up to 8 mAP@50 points," while Section 4.3 says "up to 7.6 mAP@50 points on novel categories and 10.3 on base categories." These numbers should be harmonized.
- [Equation (5)] The notation "NX j=1" is a typographical artifact and should be rendered as a proper summation subscript.
- [Reference [38]] The reference for Cluster-CLIP lists the venue as "34th British Machine Vision Conference 2022" but the publication year is given as 2023; please correct the year or venue label.
- [Supplementary Section D] The sentence "green, orange, and red, indicating labels that exists, might exists, and do not exists in the image" contains grammatical errors and should be rewritten for clarity.
- [Table 4] The row labels "none," "v,o," and "q,k,v,o" are ambiguous because the column header is "frozen"; it would be clearer to label the rows explicitly as "layers frozen: none," "layers frozen: v,o," and "layers frozen: q,k,v,o."
Circularity Check
No significant circularity: pseudo-labels are training targets, not test predictions, and the independent ablation supports the novel-category gains.
full rationale
The derivation is self-contained in the sense relevant to circularity. The pseudo-label branch (Sec. 3.2, Eq. 4) uses Cluster-CLIP only to generate training targets on ImageNet-21K categories; the reported numbers (Tables 1, 3, 5) are mAP@50 against human ground-truth labels on COCO/LVIS/nuImages, not against Cluster-CLIP's own outputs. A student trained by distillation may outperform its teacher, so training on teacher-generated pseudo-labels and then beating the teacher is not a reduction of the prediction to the input. The paper also ablates the main leakage concern: Table 3 shows FOR with pseudo-labels that explicitly exclude ImageNet-21K categories overlapping the novel evaluation classes still improves novel mAP from 82.60 to 87.66 on COCO and from 56.55 to 62.83 on LVIS over Cluster-CLIP, so the novel-category gain is not solely an artifact of copying evaluation-category labels. The self-citation to Cluster-CLIP [38] is used as a baseline and as a teacher, but it is an externally published method with its own benchmark results; this is normal incremental research rather than a load-bearing circular citation. The acknowledged limitations (Supp. D: erroneous pseudo-labels due to associative and context errors; Supp. F: inherited CLIP biases; and the unstated validation split for the 5e-4 confidence threshold) are correctness and robustness concerns, not instances of the paper's predictions being equivalent by construction to its inputs. No equation in the paper is defined in terms of the quantity it claims to predict, and no fitted parameter is renamed as a prediction.
Assumptions & free parameters
free parameters (5)
- Number of learnable queries N =
50 (25 in an ablation)
- Pseudo-label confidence threshold th =
5e-4
- Loss weights gamma_sup and gamma_pse =
1 and 1 for COCO/LVIS; 1 and 10 for nuImages
- Number of decoder layers =
2
- w_phi (empty-class weight) =
0.1 for COCO, 0 for LVIS and nuImages
assumptions (4)
- domain assumption Cluster-CLIP provides pseudo-labels that are accurate enough to teach the model novel categories.
- domain assumption Freezing the linear layers of the CLIP last attention module preserves CLIP's vision-language association while allowing useful fine-tuning.
- domain assumption ImageNet-21K categories provide a sufficient pseudo-label vocabulary for any target dataset.
- standard math The evaluation protocol (mAP@50 with maximum similarity over image embeddings, as defined in [38]) is an appropriate measure of retrieval quality.
Cite this review
Pith. "Pith review of FOR: Finetuning for Object Level Open Vocabulary Image Retrieval." pith.science (2026). https://pith.science/paper/7OHTRCFR
@misc{pith2026241218806,
author = {Pith},
title = {Pith review of: FOR: Finetuning for Object Level Open Vocabulary Image Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/7OHTRCFR}},
note = {Machine review of arXiv:2412.18806}
}
read the original abstract
As working with large datasets becomes standard, the task of accurately retrieving images containing objects of interest by an open set textual query gains practical importance. The current leading approach utilizes a pre-trained CLIP model without any adaptation to the target domain, balancing accuracy and efficiency through additional post-processing. In this work, we propose FOR: Finetuning for Object-centric Open-vocabulary Image Retrieval, which allows finetuning on a target dataset using closed-set labels while keeping the visual-language association crucial for open vocabulary retrieval. FOR is based on two design elements: a specialized decoder variant of the CLIP head customized for the intended task, and its coupling within a multi-objective training framework. Together, these design choices result in a significant increase in accuracy, showcasing improvements of up to 8 mAP@50 points over SoTA across three datasets. Additionally, we demonstrate that FOR is also effective in a semi-supervised setting, achieving impressive results even when only a small portion of the dataset is labeled.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Label-embedding for image classification
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelli- gence, 38(7):1425–1438, 2015. 13, 14
2015
-
[2]
Bottom-up and top-down attention for image captioning and visual question answering
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 6077–
2018
-
[3]
Pseudo-labeling and confirmation bias in deep semi-supervised learning
Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1–8. IEEE, 2020. 3
2020
-
[4]
Bridg- ing the gap between object and image-level representations for open-vocabulary detection
Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shahbaz Khan. Bridg- ing the gap between object and image-level representations for open-vocabulary detection. Advances in Neural Informa- tion Processing Systems, 35:33781–33794, 2022. 3
2022
-
[5]
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chel- lappa, and Ajay Divakaran. Zero-shot object detection. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I, volume 11205 ofLecture Notes in Computer Science...
work page 2018
-
[6]
Mixmatch: A holistic approach to semi-supervised learning
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019. 3
2019
-
[7]
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 11618–11628. Computer V...
work page 2020
-
[8]
X-detr: A versatile architecture for instance-wise vision- language tasks
Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Er- han Bas, Zhuowen Tu, Rahul Bhotika, and Stefano Soatto. X-detr: A versatile architecture for instance-wise vision- language tasks. In European Conference of Computer Vision (ECCV), 2022. 3
work page 2022
Show all 85 references
-
[9]
End-to- end object detection with transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 5, 6
2020
-
[10]
Big self-supervised mod- els are strong semi-supervised learners
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised mod- els are strong semi-supervised learners. Advances in neural information processing systems, 33:22243–22255, 2020. 3
2020
-
[11]
UNITER: universal image-text representation learning
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX , vo...
2020
-
[12]
Proba- bilistic embeddings for cross-modal retrieval
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Proba- bilistic embeddings for cross-modal retrieval. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1, 2, 3, 6, 7
2021
-
[13]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 3, 5
2009
-
[14]
Finding beans in burgers: Deep semantic- visual embedding with localization
Martin Engilberge, Louis Chevallier, Patrick P ´erez, and Matthieu Cord. Finding beans in burgers: Deep semantic- visual embedding with localization. In 2018 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pa...
2018
-
[15]
Fleet, Jamie Ryan Kiros, and Sanja Fidler
Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: improving visual-semantic embeddings with hard negatives. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 12. BMV A Press, 2018. 2
2018
-
[16]
De- vise: A deep visual-semantic embedding model.Advances in neural information processing systems (NeurIPS), 26, 2013
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. De- vise: A deep visual-semantic embedding model.Advances in neural information processing systems (NeurIPS), 26, 2013. 2
2013
-
[17]
Understanding the diffi- culty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio. Understanding the diffi- culty of training deep feedforward neural networks. In Pro- ceedings of the thirteenth international conference on artifi- cial intelligence and statistics, pages 249–256. JMLR Work- shop and Conference Proceedings, 2010. 6
2010
-
[18]
Improving image-sentence embeddings using large weakly annotated photo collections
Yunchao Gong, Liwei Wang, Micah Hodosh, Julia Hocken- maier, and Svetlana Lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. In Computer Vision - ECCV 2014 - 13th European Confer- ence, Zurich, Switzerland, September 6-12, 2014, Proce...
2014
-
[19]
Open-vocabulary object detection via vision and language knowledge distillation
Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations (ICLR), 2022. 2, 3
2022
-
[20]
Girshick
Agrim Gupta, Piotr Doll ´ar, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 5356–5364. Computer Vision Foundation / IEEE, 2...
2019
-
[21]
Generative multi-label zero-shot learning
Akshita Gupta, Sanath Narayan, Salman Khan, Fahad Shah- baz Khan, Ling Shao, and Joost Van De Weijer. Generative multi-label zero-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 13, 14
2023
-
[22]
Mean average precision map@k metric explained code
Nandeshwar Gupta. Mean average precision map@k metric explained code. https://www.kaggle.com/code/ nandeshwar / mean - average - precision - map - k-metric-explained-code/notebook , 2022. 6
2022
-
[23]
Instance-aware im- age and sentence matching with selective multimodal LSTM
Yan Huang, Wei Wang, and Liang Wang. Instance-aware im- age and sentence matching with selective multimodal LSTM. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 7254–7262. IEEE Computer Society, 2017. 2, 3
2017
-
[24]
Pixel-bert: Aligning image pixels with text by deep multi-modal transformers
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. CoRR, abs/2004.00849,
2004 arXiv
-
[25]
A shared multi-attention framework for multi-label zero-shot learning
Dat Huynh and Ehsan Elhamifar. A shared multi-attention framework for multi-label zero-shot learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8776–8786, 2020. 13, 14
2020
-
[26]
Saliency-guided attention network for image-sentence matching
Zhong Ji, Haoran Wang, Jungong Han, and Yanwei Pang. Saliency-guided attention network for image-sentence matching. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 5753–5762. IEEE, 2019. 2, 3
2019
-
[27]
Scaling up visual and vision-language representa- tion learning with noisy text supervision
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–4916. PMLR,
-
[28]
Billion- scale similarity search with GPUs
Jeff Johnson, Matthijs Douze, and Herv ´e J ´egou. Billion- scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019. 1, 8
2019
-
[29]
Deep fragment embeddings for bidirectional image sentence map- ping
Andrej Karpathy, Armand Joulin, and Li Fei-Fei. Deep fragment embeddings for bidirectional image sentence map- ping. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Ad- vances in Neural Information Processing Systems 27: ...
2014
-
[30]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. 6
2015
-
[31]
Unifying visual-semantic embeddings with multimodal neu- ral language models
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neu- ral language models. arXiv preprint arXiv:1411.2539, 2014. 2
2014 arXiv
-
[32]
Shamma, Michael S
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ...
2017
-
[33]
Harold W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly , 2(1–2):83– 97, March 1955. 5
1955
-
[34]
Weicheng Kuo, Yin Cui, Xiuye Gu, A. J. Piergiovanni, and Anelia Angelova. Open-vocabulary object detection upon frozen vision and language models. In ICLR 2023. OpenRe- view.net, 2023. 3
2023
-
[35]
Temporal ensembling for semi-supervised learning
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenRe- view.net, 2017. 3
2017
-
[36]
Pseudo-label: The simple and effi- cient semi-supervised learning method for deep neural net- works
Dong-Hyun Lee et al. Pseudo-label: The simple and effi- cient semi-supervised learning method for deep neural net- works. In Workshop on challenges in representation learn- ing, ICML, volume 3, page 896. Atlanta, 2013. 3
2013
-
[37]
Stacked cross attention for image-text match- ing
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xi- aodong He. Stacked cross attention for image-text match- ing. In Computer Vision - ECCV 2018 - 15th European Con- ference, Munich, Germany, September 8-14, 2018, Proceed- ings, Part IV, volume 11208 of Lecture Notes in Com...
2018
-
[38]
Object- centric open-vocabulary image retrieval with aggregated fea- tures
Hila Levi, Guy Heller, Dan Levi, and Ethan Fetaya. Object- centric open-vocabulary image retrieval with aggregated fea- tures. In 34th British Machine Vision Conference 2022, BMVC 2022, Aberdeen, UK, November 20-24, 2023 , page
2022
-
[39]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, edito...
2023
-
[40]
Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In In- ternational Conference on Machine Learning, pages 12888– 12900. PMLR, 2022. 3
2022
-
[41]
Selvaraju, Akhilesh Gotmare, Shafiq R
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu-Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang,...
2021
-
[42]
Adapting clip for phrase localization without further train- ing
Jiahao Li, Greg Shakhnarovich, and Raymond A Yeh. Adapting clip for phrase localization without further train- ing. arXiv preprint arXiv:2204.03647, 2022. 1
2022 arXiv
-
[43]
Visual semantic reasoning for image-text matching
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In 2019 IEEE/CVF International Conference on Computer Vi- sion, ICCV 2019, Seoul, Korea (South), October 27 - Novem- ber 2, 2019, pages 4653–4661. IEEE, 2019. 1, 2, 3, 6, 7
2019
-
[44]
Oscar: Object-semantics aligned pre-training for vision-language tasks
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision - ECCV 2020 - 16th European Conference, ...
2020
-
[45]
Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision - ECCV 2014 - 13th Eu- ropean Conference, Zurich, Switzerland, September 6-12, 2014, ...
2014
-
[46]
OVIS: open-vocabulary visual instance search via visual-semantic aligned representation learning
Sheng Liu, Kevin Lin, Lijuan Wang, Junsong Yuan, and Zicheng Liu. OVIS: open-vocabulary visual instance search via visual-semantic aligned representation learning. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative App...
2022
-
[47]
Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Informa- tion Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, Ne...
2019
-
[48]
Simple open-vocabulary object detection with vi- sion transformers
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection with vi- si...
2022
-
[49]
Gritsenko, and Neil Houlsby
Matthias Minderer, Alexey A. Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on...
2023
-
[50]
Y . Mori, H. Takahashi, and R. Oka. Image-to-word transfor- mation based on dividing and vector quantizing images with words. In MISRM’99 First International Workshop on Multi- media Intelligent Storage and Retrieval Management, 1999. 2
1999
-
[51]
Zero-shot learning by convex combination of semantic embeddings
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650 ,
-
[52]
Meta pseudo labels
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11557– 11568, 2021. 3
2021
-
[53]
Learning transferable visual models from natural language supervi- sion
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...
2021
-
[54]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...
2021
-
[55]
Denseclip: Language-guided dense prediction with context- aware prompting
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context- aware prompting. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1808...
2022
-
[56]
Regularization with stochastic transformations and perturba- tions for deep semi-supervised learning
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturba- tions for deep semi-supervised learning. Advances in neural information processing systems, 29, 2016. 3
2016
-
[57]
Objects365: A large-scale, high-quality dataset for object detection
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In IEEE/CVF International Conference on Computer Vision, ICCV, pages 8429–8438. IEEE, 2019. 13
2019
-
[58]
Fixmatch: Simplifying semi-supervised learning with consistency and confidence
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:5...
2020
-
[59]
A simple semi-supervised learning framework for object detection
Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister. A simple semi-supervised learning framework for object detection. arXiv preprint arXiv:2005.04757, 2020. 8
2005 arXiv
-
[60]
Dualcoop: Fast adaptation to multi-label recognition with limited annotations
Ximeng et al Sun. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. NeurIPS, 2022. 13, 14
2022
-
[61]
LXMERT: learning cross- modality encoder representations from transformers
Hao Tan and Mohit Bansal. LXMERT: learning cross- modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP- IJC...
2019
-
[62]
GIT: A generative image-to-text transformer for vision and language
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. Trans. Mach. Learn. Res., 2022, 2022. 3
2022
-
[63]
Object-aware dis- tillation pyramid for open-vocabulary object detection
Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware dis- tillation pyramid for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 11186–11196, 2023. 3
2023
-
[64]
Simvlm: Simple visual language model pretraining with weak supervision
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In The Tenth In- ternational Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net,
2022
-
[65]
Aligning bag of regions for open- vocabulary object detection
Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open- vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15254–15264, 2023. 3, 13
2023
-
[66]
CLIPSelf: Vision transformer distills itself for open-vocabulary dense predic- tion
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. CLIPSelf: Vision transformer distills itself for open-vocabulary dense predic- tion. In The Twelfth International Conference on Learning Representations, 2024. 13
2024
-
[67]
CLIM: contrastive language-image mosaic for region representation
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Wentao Liu, and Chen Change Loy. CLIM: contrastive language-image mosaic for region representation. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty- Eighth AAAI Conference on Artificial Intelligence, ...
2024
-
[68]
CORA: adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching
Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. CORA: adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , pages 7031...
2023
-
[69]
Unsupervised data augmentation for consistency training
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33:6256–6268, 2020. 3
2020
-
[70]
Self-training with noisy student improves imagenet classification
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10687– 10698, 2020. 3
2020
-
[71]
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions
Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. Trans. Assoc. Comput. Linguistics, 2:67–78, 2014. 2
2014
-
[72]
Coca: Contrastive captioners are image-text foundation models
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Mach. Learn. Res., 2022, 2022. 1, 3, 6, 7
2022
-
[73]
Florence: A new foundation model for computer vision
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. 1, 3
2021 arXiv
-
[74]
Lit: Zero-shot transfer with locked-image text tuning
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18- 2...
2022
-
[75]
Fast zero- shot image tagging
Yang Zhang, Boqing Gong, and Mubarak Shah. Fast zero- shot image tagging. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5985–5994. IEEE, 2016. 13, 14
2016
-
[76]
Exploiting unlabeled data with vision and language models for object detection
Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, BG Vijay Kumar, Anastasis Stathopoulos, Manmohan Chan- draker, and Dimitris N Metaxas. Exploiting unlabeled data with vision and language models for object detection. In European Conference on Computer Vision, pages 159–17...
2022
-
[77]
Regionclip: Region-based language-image pretraining
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region-based language-image pretraining. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 202...
2022
-
[78]
Extract free dense labels from CLIP
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from CLIP. In Shai Avidan, Gabriel J. Brostow, Moustapha Ciss ´e, Giovanni Maria Farinella, and Tal Hass- ner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2...
2022
-
[79]
Detecting twenty-thousand classes using image-level supervision
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr¨ahenb¨uhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Confer- ence on Computer Vision , pages 350–368. Springer, 2022. 3
2022
-
[80]
Semi-supervised learning literature survey
Xiaojin Zhu. Semi-supervised learning literature survey. Technical Report 1530, Computer Sciences, University of Wisconsin-Madison, 2005. 3
2005
-
[81]
Rethinking pre- training and self-training
Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanx- iao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre- training and self-training. Advances in neural information processing systems, 33:3833–3845, 2020. 3 Supplementary Materials A. More Implementation Details Followin...
2020
-
[85]
by 3-5 mAP points, indicating improved retrieval ac- curacy. The performance gap widens significantly when GZSL ZSL Method F1 (K=3) mAP F1 (K=3) mAP CONSE [51] 19.6 7.7 18.4 13.2 LabelEM [1] 6.7 4.0 10.3 9.6 Fast0tag [75] 33.8 27.9 37.5 43.3 LESA [25] 26.7 17.5 33.6 31.8 Gener...
-
[137]
Springer, 2020. 2, 3
2020
-
[608]
1, 2, 3, 4, 6, 8, 13
BMV A Press, 2023. 1, 2, 3, 4, 6, 8, 13
2023
-
[6086]
Computer Vision Foundation / IEEE Computer Soci- ety, 2018. 2
2018
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.