REVIEW 4 major objections 4 minor 1 cited by
This paper claims that decomposing pseudo-label generation into three visual reasoning steps—object verification, zero-shot naming, and background grounding—lets open-vocabulary detectors surpass single-step CLIP alignment, improving novel-
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 09:30 UTC pith:SXJB7TYX
load-bearing objection A well-ablated CoT-based pseudo-labeling recipe that deserves referee time, but a single ambiguous figure caption about an inference-time CBL++ step could invalidate the headline numbers if it is what it looks like. the 4 major comments →
MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that structured visual chain-of-thought, executed by an MLLM on SAM proposals, produces pseudo-labels good enough to push open-vocabulary detection well past single-step CLIP alignment: +7.7 box AP50 on novel OV-COCO classes and +2.9 mask mAP on rare OV-LVIS classes over the BARON baseline, with even larger relative gains in pseudo-label quality for crowded and occluded objects. The design is a three-step CoT—object existence verification, zero-shot category recognition, and background grounding—whose final step enables contrastive background learning (CBL), using identified background concepts as negative training signals to disentangle objects from background in fe
What carries the argument
The load-bearing machinery is the three-step visual chain-of-thought pipeline (region perception via SAM proposals plus an MLLM's Yes/No/Unsure gate; caption-free zero-shot category naming; grounding of background concepts), together with contrastive background learning (CBL), an InfoNCE-style loss in which MLLM-grounded background embeddings (sky, grass, etc.) act as negatives for the bag-of-regions alignment loss inherited from BARON. This turns the third CoT step into a training signal, not just a filtering step.
Load-bearing premise
The whole pipeline assumes that a sufficiently capable multimodal LLM can reliably answer three questions on SAM-generated crops: Does the crop contain an object? What is it? Is it foreground or background?
What would settle it
On a labeled subset of crowded/occluded images, compute precision/recall of the MLLM's Yes/No/Unsure gate against human annotation; if the gate rejects a large share of true objects (or accepts background), the CoT pipeline's pseudo-labels lose their value. Alternatively, run the pipeline with a deliberately weak or always-'Unsure' MLLM—if AP50 does not fall to near baseline, the central claim about MLLM-dependent reasoning is unsupported.
If this is right
- If correct, pseudo-labeling for OVD no longer needs image captions or predefined novel-class names; the detector learns novel classes from SAM boxes named by an MLLM.
- Crowded and occluded scenes, the known weak point of CLIP-based pseudo-labeling, become the method's relative strength: pseudo-label quality improves by 103.4% (crowded) and 168.4% (occluded) over the best prior pseudo-labeling method.
- Performance scales with teacher MLLM capability: swapping BLIP2 for Qwen2 raises AP50 by 4.1 points, suggesting that future MLLM advances will translate into further OVD gains.
- The contrastive background loss reduces background collapse, where occluded objects are mistakenly learned as background, and cached semantic anchors cut training time by roughly 4x relative to online sampling.
- Cross-dataset transfer improves slightly without fine-tuning, narrowing the gap to fully supervised detectors on COCO and Objects365.
Where Pith is reading between the lines
- A natural extension the authors do not draw: the same three-question CoT could be applied beyond pseudo-labeling, e.g., to open-vocabulary instance segmentation or interactive annotation, where SAM masks already exist and an MLLM can name and ground them.
- The paper's hard gate on 'Unsure' (discarding those boxes) suggests a testable variant: instead of discarding, re-query with different crops or use soft pseudo-labels, which might recover genuinely occluded objects that the current pipeline drops.
- Because the pipeline depends on a strong MLLM, a lightweight student that distills the CoT teacher's yes/no/name/background decisions could be trained on a small annotated subset; measuring whether the student retains the crowded/occluded gains would isolate the value of the CoT structure from the teacher's raw capability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper (whose abstract and title use "MSPL," while the body uses "CoT-PL") proposes a pseudo-labeling pipeline for open-vocabulary object detection built on BARON. SAM generates class-agnostic proposals; an MLLM performs three visual chain-of-thought steps—object-presence verification, zero-shot category recognition, and background/foreground grounding; a frequency-based "semantic anchor" filter retains reliable pseudo-labels; and a contrastive background learning (CBL) loss pulls object features away from background embeddings. On OV-COCO and OV-LVIS the paper reports state-of-the-art novel-class results (+7.7 box AP50 and +2.9 mask APr over BARON), plus improved pseudo-label quality on crowded and occluded subsets.
Significance. If the results hold, the paper makes a useful contribution: it is, to my knowledge, the first to formulate OVD pseudo-labeling as explicit visual chain-of-thought, it ships code and detailed prompts, and the ablations in Table 5 support the incremental contribution of preprocessing, multi-step CoT, and CBL. The MLLM-capability dependence is acknowledged in Appendix A and quantified in Table 8. However, the evaluation is undermined by the unexplained inference-time CBL++ step described in Figure 4, which appears to use ground-truth novel-class information at inference. This concern is load-bearing because the headline numbers—Tables 1 and 3—depend on the validity of open-vocabulary evaluation.
major comments (4)
- [Figure 4 / Appendix C] CBL++ is described only in the Figure 4 caption: "At inference time, we apply CBL++ to mitigate class interference by removing pseudo-labels associated with the ground-truth novel classes." This directly contradicts Appendix C, which states that pseudo-labels are "used exclusively during training and discarded during inference." If CBL++ suppresses or reweights outputs using ground-truth novel-class annotations, then Tables 1 and 3 are not valid open-vocabulary evaluations. If the phrase means something else, that definition and an ablation are missing. Because CBL++ is never defined, pseudo-coded, or ablated, the reported state-of-the-art numbers cannot currently be interpreted.
- [Sec. 3.3, 3.4 / Appendix C] The main text says the third CoT step identifies background concepts C_bg and that these are encoded and used as negatives in CBL. Appendix C, however, states that "CBL background prototypes are built from hand-crafted prompts rather than category names" and lists only five generic prompt types. These are different sources of background cues. Please clarify whether the MLLM grounding output actually determines C_bg or whether CBL uses only the five hand-crafted prompts. If the latter, the claim that the third CoT step "naturally motivates" and "leads to" CBL is weakened, and an ablation separating hand-crafted cues from MLLM-grounded cues is needed.
- [Table 4 / Sec. 4.2] The pseudo-label quality comparison (crowded/occluded) is not controlled: PB-OVD, VL-PLM, and SAS-Det use their original CLIP-based teachers and proposal-generation pipelines, so the 103.4% and 168.4% relative improvements may be due to differences in teacher model (MLLM vs CLIP), proposal generator, or filtering, rather than to visual CoT reasoning per se. To support the paper's claim that CoT reasoning specifically improves complex-scene pseudo-labels, please include an ablation using the same SAM proposals and the same MLLM with and without the three-step CoT structure, and ideally a CLIP-based variant under the same proposal pipeline.
- [Sec. 3.2 / Table 6 / Appendix A] The semantic-anchor threshold is set to the minimum base-class annotation count, and Table 6 tests only ALL versus MIN. Appendix A admits that this threshold "unfairly removes" long-tailed categories. Without a sensitivity analysis (e.g., a few thresholds between ALL and MIN, or per-dataset justification), the paper's claim that this choice is reliable and not a source of uncontrolled bias is not fully supported. This is a reproducibility and robustness concern, though less severe than the CBL++ issue.
minor comments (4)
- [Title / Abstract] The abstract and title refer to "MSPL," while the main text and GitHub use "CoT-PL." The abstract also reports gains (+9.4 AP50, +3.2/+2.2 APr) that differ from the body (+7.7, +2.9). Please reconcile names and numbers.
- [Sec. 3.4 vs Appendix F] The main text says the cached semantic-anchor composition generator "reduces overall training time by 4x," while Appendix F says caching "reduced training time by 25%." These are inconsistent statements and should be clarified.
- [Table 5 / Sec. 4.2] The prose states that CBL "improves performance by 7.1%" when the table shows the difference between CoT(3x) and CoT(3x)+CBL is 1.4 points. If the 7.1 is relative to the no-preprocessing baseline, say so explicitly.
- [Appendix B] All experiments use a single fixed seed. Given that the gains over BARON are on the order of a few points, reporting variance across at least three seeds would strengthen the empirical claims.
Circularity Check
CBL++ inference step is undefined and, read literally, uses ground-truth novel-class information at evaluation time; this is the only load-bearing circular reduction found.
specific steps
-
other
[Figure 4 caption (Section 3.4); contradicted by Appendix C; affects Tables 1 and 3]
"At inference time, we apply CBL++ to mitigate class interference by removing pseudo-labels associated with the ground-truth novel classes. ... The pseudo-labels are used exclusively during training and discarded during inference."
The paper's SOTA claims are novel-class AP50 on OV-COCO and rare-class mask AP on OV-LVIS. The only description of CBL++ says it removes pseudo-labels associated with the ground-truth novel classes at inference. Appendix C states pseudo-labels are training-only and discarded at inference, so this step is either undefined or it injects test-set novel-class information into inference. If the latter reading holds, the evaluated novel-class predictions are partially constructed from the evaluation target, so the headline improvements are not independent predictions. No pseudocode, ablation, or method text defines or isolates CBL++, making the main comparison unverifiable.
full rationale
The three-step CoT pseudo-labeling and contrastive background learning are empirical pipeline components trained from base-split images plus MLLM/SAM outputs; their training-time operation does not reduce to the evaluation target. The semantic-anchor threshold is a base-class-count heuristic, not a parameter fitted to novel-class AP, and Table 6 explicitly ablates it. There is no self-citation chain and no equation-level derivation-equals-input in the loss or pseudo-labeling formulas. The sole load-bearing problem is Figure 4's CBL++: it is the only description of an inference-time step applied to the main results, it is contradicted by Appendix C, and it is never defined or ablated. Read literally, it removes pseudo-labels using ground-truth novel-class knowledge, which would make the reported AP_N/AP_r numbers at least partially constructed from the answer key. Read benignly, it is a missing definition of a step that affects the central comparison. Either way, the main state-of-the-art claim is not independently verifiable as written; the score of 6 reflects this central, though isolated, reduction.
Axiom & Free-Parameter Ledger
free parameters (3)
- semantic_anchor_threshold =
1237 (OV-COCO), 1 (OV-LVIS)
- background_temperature_tau'' =
10.0
- blur_kernel_and_sigma =
31x31, sigma=0
axioms (5)
- domain assumption SAM whole-object masks can be converted to tight boxes that cover objects; MLLM 'Yes' responses correctly identify valid objects.
- domain assumption MLLM zero-shot class recognition on blurred/grayscale proposals is accurate enough for novel classes.
- ad hoc to paper Five hand-crafted background prompts (sky, water surface, vegetation, paved ground, plain wall) adequately represent background for CBL.
- ad hoc to paper Semantic anchor filtering with the minimum base-class annotation threshold removes noise without discarding useful long-tail classes.
- domain assumption BARON's bag-of-regions alignment and CLIP text-image embedding space are valid for training the detector.
read the original abstract
Open-vocabulary object detection (OVD) aims to recognize and localize object categories beyond the training set. Recent approaches leverage vision-language models to generate pseudo-labels using image-text alignment, allowing detectors to generalize to unseen classes without explicit supervision. However, these methods depend heavily on single-step image-text matching, neglecting the intermediate reasoning steps crucial for interpreting semantically complex visual contexts, such as crowding or occlusion. In this paper, we introduce MSPL, a framework that incorporates multi-step visual reasoning into the pseudo-labeling process for OVD. It decomposes complex scene understanding into three interpretable steps-object localization, category recognition, and background grounding-where these intermediate reasoning states serve as rich supervision sources. Extensive experiments on standard OVD evaluation protocols demonstrate that MSPL achieves state-of-the-art performance with superior pseudo-labeling efficiency, outperforming the strong baseline by 9.4 AP50 for novel classes on OV-COCO and improving box and mask APr by 3.2 and 2.2, respectively, on OV-LVIS. Code and models are available at https://github.com/hchoi256/mspl.
Figures
Forward citations
Cited by 1 Pith paper
-
Open-Vocabulary Gaze Object Prediction: Benchmark and Method
A COCO+GazeFollow-derived benchmark (86 categories) plus a Grounding DINO + gaze-selection pipeline with selective tuning improves open-vocabulary gaze object prediction over existing closed-vocabulary methods.
Reference graph
Works this paper leans on
-
[2]
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision- language models? InAdvances in Neural Information Processing Systems 38: Annual Confer- ence on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Can...
2024
-
[4]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In9th International Conference on Learning Representations, ICLR 2021,...
2021
-
[7]
Smith, Wei-Chiu Ma, and Ranjay Krishna
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: multimodal large language models can see but not perceive. InComputer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXIII,
2024
-
[8]
Open vocabulary object detection with pseudo bounding-box labels
10 Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-box labels. InComputer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X,
2022
-
[9]
Open-vocabulary object detection via vi- sion and language knowledge distillation
Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vi- sion and language knowledge distillation. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022,
2022
-
[10]
Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models. InIEEE/CVF Conference on Computer Vision and Pattern Recognit...
2024
-
[11]
Girshick
Agrim Gupta, Piotr Doll ´ar, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019,
2019
-
[13]
Deep residual learning for image recog- nition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June 27-30, 2016,
2016
-
[15]
Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.),Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtu...
2021
-
[16]
Llms meet vlms: Boost open vocabulary object detection with fine-grained descriptors
Sheng Jin, Xueying Jiang, Jiaxing Huang, Lewei Lu, and Shijian Lu. Llms meet vlms: Boost open vocabulary object detection with fine-grained descriptors. InThe Twelfth International Confer- ence on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024,
2024
-
[17]
Contrastive feature masking open-vocabulary vision transformer
11 Dahun Kim, Anelia Angelova, and Weicheng Kuo. Contrastive feature masking open-vocabulary vision transformer. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023,
2023
-
[18]
Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chlo ´e Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B. Girshick. Segment anything. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023,
2023
-
[19]
Learning background prompts to discover implicit knowledge for open vocabulary object detection
Jiaming Li, Jiacheng Zhang, Jichang Li, Ge Li, Si Liu, Liang Lin, and Guanbin Li. Learning background prompts to discover implicit knowledge for open vocabulary object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024,
2024
-
[20]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. InInternational Con- ference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA,
2023
-
[21]
Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. InComputer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V,
2014
-
[23]
Class-agnostic object detection with multi-modal transformer
Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang. Class-agnostic object detection with multi-modal transformer. InComputer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X,
2022
-
[24]
LP-OVOD: open-vocabulary object detection by linear probing
Chau Pham, Truong Vu, and Khoi Nguyen. LP-OVOD: open-vocabulary object detection by linear probing. InIEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2024, Waikoloa, HI, USA, January 3-8, 2024,
2024
-
[25]
Belongie, Alan L
Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge J. Belongie, Alan L. Yuille, Philip H. S. Torr, and Song Bai. Occluded video instance segmentation: A benchmark. Int. J. Comput. Vis., 130:2022–2039,
2022
-
[26]
Langsplat: 3d lan- guage gaussian splatting
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d lan- guage gaussian splatting. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024,
2024
-
[27]
Learning transferable visual models from natural language supervision
12 Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceed- ings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July...
2021
-
[28]
Khan, and Fahad Shahbaz Khan
Hanoona Abdul Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman H. Khan, and Fahad Shahbaz Khan. Bridging the gap between object and image-level representations for open- vocabulary detection. InAdvances in Neural Information Processing Systems 35: Annual Con- ference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, ...
2022
-
[29]
Girshick, and Jian Sun
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. InAdvances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada,
2015
-
[30]
Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hong- sheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. InAdvances in Neural Information Processing Sys- tems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIP...
2024
-
[31]
Objects365: A large-scale, high-quality dataset for object detection
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, 2019a. Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu,...
2019
-
[34]
Chi, Quoc V
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 2...
2022
-
[35]
Visual chatgpt: Talking, drawing and editing with visual foundation models.CoRR, 2023a
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models.CoRR, 2023a. Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. Llmdet: A third party large language models generated text detection tool. InFindings of the Association for Computa- t...
2023
-
[36]
CORA: adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching
Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. CORA: adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching. InIEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, 2023d. Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visu...
2023
-
[37]
Open- vocabulary SAM: segment and recognize twenty-thousand classes interactively
Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open- vocabulary SAM: segment and recognize twenty-thousand classes interactively. InComputer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XLIII,
2024
-
[38]
MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for exp...
2024
-
[39]
Mert Y¨uksekg¨on¨ul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023,
2023
-
[40]
Open-vocabulary DETR with conditional matching
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary DETR with conditional matching. InComputer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX,
2022
-
[41]
Open-vocabulary object detection using captions
14 Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021,
2021
-
[42]
Cyclic contrastive knowledge transfer for open-vocabulary object detection.CoRR, 2025a
Chuhan Zhang, Chaoyang Zhu, Pingcheng Dong, Long Chen, and Dong Zhang. Cyclic contrastive knowledge transfer for open-vocabulary object detection.CoRR, 2025a. Hang Zhang, Kristin J. Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In2018 IEEE Conference on Computer Vision and P...
2018
-
[44]
Cot-vla: Visual chain-of-thought reasoning for vision-language-action models
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, Ankur Handa, Tsung-Yi Lin, Gordon Wetzstein, Ming-Yu Liu, and Donglai Xiang. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Na...
2025
-
[45]
Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, B. G. Vijay Kumar, Anastasis Stathopou- los, Manmohan Chandraker, and Dimitris N. Metaxas. Exploiting unlabeled data with vision and language models for object detection. InComputer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX,
2022
-
[46]
Shiyu Zhao, Samuel Schulter, Long Zhao, Zhixing Zhang, B. G. Vijay Kumar, Yumin Suh, Man- mohan Chandraker, and Dimitris N. Metaxas. Taming self-training for open-vocabulary object detection. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024,
2024
-
[47]
Regionclip: Region-based language- image pretraining
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Lu- owei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region-based language- image pretraining. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022,
2022
-
[48]
Detecting twenty-thousand classes using image-level supervision
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr ¨ahenb¨uhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. InComputer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX,
2022
-
[49]
We adopt the 1×training schedule for OV- COCO (Lin et al.,
and utilizes synchronized Batch Normalization (SyncBN) (Zhang et al., 2018). We adopt the 1×training schedule for OV- COCO (Lin et al.,
2018
-
[50]
Pseudo-label generation process.During our offline pseudo-label generation process, we lever- age SAM (Kirillov et al., 2023; Qin et al.,
and 2×for OV-LVIS (Gupta et al., 2019). Pseudo-label generation process.During our offline pseudo-label generation process, we lever- age SAM (Kirillov et al., 2023; Qin et al.,
2019
-
[51]
a photo of [OBJ]
is used to encode textual prompts, which are constructed using the hand-crafted template “a photo of [OBJ]” following ViLD (Gu et al., 2022). Pseudo-labels are generated using only the training set, without leveraging any image cap- tions. The pseudo-labels are used exclusively during training and discarded during inference. Contrastive Background Learnin...
2022
-
[52]
During training, model checkpoints are saved every 10,000 iter- ations for OV-COCO and every 30,000 iterations for OV-LVIS
and 180,000 iterations on OV-LVIS (Gupta et al., 2019), with a fixed batch size of 16 across all experiments. During training, model checkpoints are saved every 10,000 iter- ations for OV-COCO and every 30,000 iterations for OV-LVIS. The best-performing checkpoint on the validation set is selected for final evaluation. For our proposed modules, we provide...
2019
-
[53]
For semantic anchor construction, we filter out infrequent pseudo-labels using a minimum annotation threshold—set to 1,237 for OV-COCO and 1 for OV-LVIS
experiments. For semantic anchor construction, we filter out infrequent pseudo-labels using a minimum annotation threshold—set to 1,237 for OV-COCO and 1 for OV-LVIS. Additionally, the background contrastive loss temperature 16 parameter is set toτ ′′ = 10.0, which controls the regularization strength of background embeddings relative to foreground embedd...
2014
-
[54]
Due to the severe long-tail distribution, some rare categories contain fewer than five instances; such categories are removed during semantic anchor construction
protocol, we treat 337 rare categories as novel, and the remaining frequent and common categories as base. Due to the severe long-tail distribution, some rare categories contain fewer than five instances; such categories are removed during semantic anchor construction. Evaluation is performed on the standard LVIS validation split. •Objects365(Shao et al.,...
2015
-
[57]
is a general-purpose segmentation model that predicts instance masks given spatial prompts. It enables high-quality, class-agnostic mask generation via zero-shot segmentation, providing fine-grained object candidates valuable for downstream tasks (Yuan et al., 2024; Han et al., 2025). Recently, LangSplat (Qin et al.,
2024
-
[58]
By densely sampling point prompts across the image, SAM generates a diverse set of masks that capture object regions at varying levels of granularity
leveraged SAM to extract hierarchical segmentation masks from images, enabling structured multi- scale object representation. By densely sampling point prompts across the image, SAM generates a diverse set of masks that capture object regions at varying levels of granularity. These masks are filtered and organized into three semantic levels—subpart, part,...
2025
-
[59]
This setup enables efficient vision-language alignment and achieves strong performance on tasks such as image captioning and visual question answer- ing (VQA) with minimal training
adopts a modular architecture comprising a frozen im- age encoder, a trainable QFormer (Zhang et al., 2024b), and a frozen language model such as OPT (Zhang et al., 2022). This setup enables efficient vision-language alignment and achieves strong performance on tasks such as image captioning and visual question answer- ing (VQA) with minimal training. •In...
2022
-
[60]
This design allows the model to follow natural language instructions and generalize across diverse multimodal tasks
and a frozen language model, such as Flan-T5 (Chung et al., 2024). This design allows the model to follow natural language instructions and generalize across diverse multimodal tasks. As shown in Table 9, this MLLM exhibits fair zero-shot performance on academic multimodal benchmarks, with accuracy ranging from 24% to 32% on most tasks. •Qwen2:Qwen2 (Bai et al.,
2024
-
[61]
However, when applied to object-level understanding, recent studies (Zang et al., 2025; Fu et al.,
MLLMs (Li et al., 2023; Dai et al., 2023; Bai et al., 2023; Wang et al., 2023a) exhibit strong zero- shot reasoning across vision-language tasks such as image captioning and retrieval. However, when applied to object-level understanding, recent studies (Zang et al., 2025; Fu et al.,
2023
-
[62]
In our setting, where the MLLM is prompted on individual region proposals, it is essential to emphasize the target object while suppressing ir- relevant background information
have shown that they remain highly sensitive to visual context. In our setting, where the MLLM is prompted on individual region proposals, it is essential to emphasize the target object while suppressing ir- relevant background information. As shown in Table 10, omitting preprocessing slightly degrades performance, yielding an APN 50 of 33.6 relative to t...
2023
-
[63]
In practice, we validate that this strategy improves localization and reasoning in MLLMs (Bai et al., 2023), as shown in Table
suggests that grayscaling and blur- ring regions outside the mask can effectively suppress background noise and enhance model focus. In practice, we validate that this strategy improves localization and reasoning in MLLMs (Bai et al., 2023), as shown in Table
2023
-
[64]
dog,” “knife,
Statistics.Figure 7 illustrates the annotation counts of pseudo-labels in the OV-COCO dataset, revealing a typical long-tail distribution. A small number of frequent categories account for the majority of annotations, reflecting their higher prevalence in the training data. This imbalance natu- rally emerges, as the MLLM tends to predict commonly occurrin...
2024
-
[65]
Methods MS-COCO(Lin et al., 2014)Objects365(Shao et al., 2019b) AP (%) AP 50 (%) AP 75 (%) AP (%) AP 50 (%) AP 75 (%) Supervised (Gu et al.,
and Objects365 (Shao et al., 2019b).We use BARON as the baseline and evaluate all methods without fine-tuning. Methods MS-COCO(Lin et al., 2014)Objects365(Shao et al., 2019b) AP (%) AP 50 (%) AP 75 (%) AP (%) AP 50 (%) AP 75 (%) Supervised (Gu et al.,
2014
-
[66]
and OV-LVIS (Gupta et al., 2019), as shown in Figures 13 and
2019
-
[2008]
Kankanhalli, and Ying Shan
Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan S. Kankanhalli, and Ying Shan. What makes for good visual tokenizers for large language models?CoRR, 2023a. Junjie Wang, Bin Chen, Bin Kang, Yulin Li, Weizhi Xian, Yichi Chen, and Yong Xu. OV-DQUO: open-vocabulary DETR with denoising text query training and open-world unknown objects su- pervision. InAAAI-25, ...
2025
-
[2014]
Hayes, Elisa Ricci, Gabriela Csurka, and Riccardo V olpi
Mingxuan Liu, Tyler L. Hayes, Elisa Ricci, Gabriela Csurka, and Riccardo V olpi. Shine: Semantic hierarchy nexus for open-vocabulary object detection. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, 2024a. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chun...
2024
-
[2015]
a photo of{category}in the scene
for OVD, replacing its classifier with a linear layer that projects region features into the word embedding space. This enables each region to be represented by mul- tiple pseudo-words, capturing the rich semantics of each object. GivenCobject categories, the probability of a region being classified as thec-th category: pc = exp(τ· ⟨T(w), fc⟩)PC−1 i=0 exp...
2015
-
[2016]
Girshick
Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross B. Girshick. Mask R-CNN. InIEEE Inter- national Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017,
2017
-
[2017]
VLMs support novel class recognition in OVD through various techniques, such as pseudo-labeling
in open- vocabulary object detection (OVD) have been largely driven by the emergence of foundation models, including vision-language models (VLMs) (Radford et al., 2021; Jia et al., 2021). VLMs support novel class recognition in OVD through various techniques, such as pseudo-labeling. We build upon Faster R-CNN (Ren et al.,
2021
-
[2018]
Explor- ing region-word alignment in built-in detector for open-vocabulary object detection
Heng Zhang, Qiuyu Zhao, Linyu Zheng, Hao Zeng, Zhiwei Ge, Tianhao Li, and Sulong Xu. Explor- ing region-word alignment in built-in detector for open-vocabulary object detection. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, 2024a. Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Fil...
2024
-
[2019]
Boosting segment anything model towards open-vocabulary learning
Xumeng Han, Longhui Wei, Xuehui Yu, Zhiyang Dou, Xin He, Kuiran Wang, Yingfei Sun, Zhenjun Han, and Qi Tian. Boosting segment anything model towards open-vocabulary learning. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA,
2025
-
[2021]
Learning to prompt for open-vocabulary object detection with vision-language model
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022,
2022
-
[2022]
Instagen: Enhancing object detection by training on synthetic dataset
Chengjian Feng, Yujie Zhong, Zequn Jie, Weidi Xie, and Lin Ma. Instagen: Enhancing object detection by training on synthetic dataset. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024,
2024
-
[2023]
Zero-shot ob- ject detection
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot ob- ject detection. InComputer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I,
2018
-
[2024]
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, ...
2023
-
[2025]
Object-aware distillation pyramid for open-vocabulary object detection
Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware distillation pyramid for open-vocabulary object detection. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 11186–11196, 2023b. Zhao Wang, Aoxue Li, Fengwei Zhou, Zhenguo Li, a...
2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.