REVIEW 4 major objections 5 minor 27 references
Fine-Grained Open-Vocabulary Object Recognition via User-Guided Segmentation
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read FOCUS wraps open-vocabulary detectors with SAM and a VLM, lifting fine-grained detection F1 from 0.465 to 0.835.
desk verdict A sensible three-stage recipe, but Table 1 measures self-consistency with the VLM's own proposals rather than detection accuracy, so the central claim is unsupported as submitted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the three-stage FOCUS pipeline: (1) Region Isolation, where SAM takes the user's bounding box $B_{\text{input}}$ and returns a binary mask $M_{\text{binary}}$, producing the masked image $I_{\text{atten}}$ that blanks out everything outside the target; (2) Proposal Extraction, where a VLM prompted with a free-form task prompt $P_{\text{task}}$ and formatting instructions $P_{\text{add}}$ returns an object list $L_p$; and (3) Entity Identification, where an open-vocabulary detector consumes $I_{\text{atten}}$ and $L_p$ to output bounding boxes $B_{\text{output}}$. The performance is scored by $F_1(L_p, L_d)$, the harmonic mean of recall and precision between the VLM proposal and the detected list. The key mechanism is that masking removes the large-object bias that causes detectors to overlook small nested components.
What would settle it
Take a set of images, have a human annotator list the fine-grained objects they consider present inside the user box, and compute F1 for GroundingDINO and GroundingDINO+FOCUS against that human list. If the FOCUS gain over baseline shrinks to near zero, or if human precision on FOCUS outputs is low because the VLM proposed objects that are absent, then the claimed enhancement is an artifact of self-consistent evaluation rather than a real detection improvement.
Extended reading notes
Core claim
The paper's central claim is that the failure of open-vocabulary detectors on fine-grained parts is not mainly a detector-capacity problem; it is a focusing and proposal problem. By first applying a binary mask from SAM to the user-designated bounding box, the pipeline removes distracting context so the detector attends to the target. A VLM then proposes the candidate object list $L_p$ from this masked image, and an open-vocabulary detector locates those candidates, producing $L_d$. The paper reports that this yields consistent gains across detectors and object types, with the largest improvement on granular human targets: at a 0.2 threshold, GroundingDINO+FOCUS reaches 0.835 F1 versus 0.465 for GroundingDINO. The method is intended to be zero-shot and model-agnostic, with any VLM or open-vocabulary detector substitutable in the pipeline.
Load-bearing premise
The load-bearing premise is that the vision-language model's proposed object list can stand in for ground truth, so the F1 score in Equation (1) is assumed to reflect how well objects are actually detected rather than how well the detector agrees with that same VLM.
Editorial extensions
If this is right
- Users can re-target the same image to different granularities by editing a natural-language prompt, e.g., from physical objects worn by a person to visible body parts.
- The pipeline is model-agnostic: the segmentation module, VLM, and detector can each be swapped, so improvements should transfer to newer foundation models as they appear.
- FOCUS-equipped detectors are more resistant to confidence-threshold changes, meaning detections remain useful even when the cutoff is low.
- The gains hold across both human and non-human targets, with vehicle-detection improvements becoming clearer at higher score thresholds.
- Because the same VLM proposal list is fed to baselines, the comparison isolates the effect of the region-isolation mask.
Reading between the lines
- Because the evaluation treats the VLM's proposal as ground truth, the reported F1 measures how well the detector reproduces the VLM's object list; it would be informative to re-run with human-annotated fine-grained labels to see how much of the gain reflects agreement with human perception.
- Masking everything outside the user box could suppress context that helps identify objects that extend beyond the box or that are semantically defined by surroundings; a targeted study on truncation cases would bound this effect.
- Prompt sensitivity is an implicit free parameter: the same pipeline could produce different object lists for differently worded prompts, so systematic prompt variation is a natural testable extension.
- If VLM hallucination is common in cluttered scenes, errors propagate into the detector; measuring precision against human labels would reveal whether FOCUS trades recall for hallucination-driven false positives.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FOCUS, a zero-shot pipeline that combines SAM, a vision-language model (GPT-4o), and an open-vocabulary detector. The user supplies a bounding box and a natural-language prompt; SAM masks everything outside the box, the VLM proposes an object list, and the detector localizes those objects. The authors report F1 scores on COCO person images and PASCAL VOC vehicle images, claiming that FOCUS improves granular detection over GroundingDINO, OWLv2, and OpenSeeD, and that it is robust to confidence thresholds and image complexity.
Significance. If the reported gains were supported by independent ground truth, FOCUS would be a practically useful, training-free framework for prompt-controlled fine-grained detection, and the modular design (model-agnostic across all three stages) is a genuine strength. The qualitative figures illustrate appealing flexibility, such as switching between instance-level and body-part-level detection by editing a text prompt. However, the central quantitative claim is currently not established because the evaluation is circular: the same VLM that proposes objects inside FOCUS is used to define ground truth in the F1 computation. The paper also provides no localization metric, no error bars, no image counts, and an incomplete ablation (an undefined 'N%'). For these reasons, the significance of the contribution cannot be assessed from the present manuscript.
major comments (4)
- [Section 4.1, Eq. (1)] The evaluation is circular and does not measure detection accuracy. Eq. (1) computes F1 between Lp (the VLM proposal list) and Ld (the detector outputs), and Section 4.1 states that 'the object list proposal made by the VLM' is assumed to be ground truth. Since the same VLM, GPT-4o, is a component of FOCUS (Section 3.2), all F1 scores in Table 1 measure self-consistency between an internal proposal module and the detector, not correctness relative to human labels. For example, the 0.835 vs. 0.465 difference for GroundingDINO at threshold 0.2 may only reflect that the masked image makes it easier for the detector to reproduce the VLM's own list. The authors should evaluate against human-annotated object lists, or at least against a VLM not used inside FOCUS, and report agreement with those independent references.
- [Section 3.3 and Section 4.1] No box-level localization metric is reported. The paper claims that FOCUS 'locates' granular components and that the Entity Identification module outputs final bounding boxes, but Eq. (1) only compares label lists; it never checks whether a detected box actually covers the corresponding object. Without an IoU-based metric (e.g., mAP at IoU=0.5, or at least box-center accuracy), the reported F1 values cannot support the claim that detection capabilities are enhanced, because a detector could produce the right labels with wrong boxes and still score perfectly under Eq. (1).
- [Section 5.4] The ablation study is incomplete: it states that 'the FOCUS framework improves performance on the granular detection task by N% compared to the baseline itself,' where N is never defined. This is not a quantitative result and cannot support the claim that SAM-based masking is responsible for the observed gains. The authors must provide the actual value, the uncertainty estimate, and the evaluation protocol used for the ablation.
- [Section 4.2 and Section 5.3] Essential experimental details are missing, making the results non-reproducible. The manuscript does not report the number of test images per class, the exact prompt templates used for Ptask and Padd, the GPT-4o version or inference settings, or the procedure behind the 'top 10 items with highest score discrepancies' mentioned in Section 5.3. Table 1 and Figure 6 report no error bars or significance tests across threshold values, so the claimed consistency and threshold robustness are not substantiated.
minor comments (5)
- [Abstract] The abstract contains a typo: 'Object Re Cognition' should be 'Object Recognition'.
- [Section 3.3] The term 'spacial relations' should be 'spatial relations'.
- [Figure 5 caption] In the caption, 'Top later shows instance detection; bottom later shows anatomic detection' should read 'Top layer' and 'bottom layer'.
- [References] References [15] and [16] are the same CLIP paper by Radford et al.; one should be removed or merged to avoid a duplicate citation.
- [Section 5.1] The phrase 'humane and inhumane target categories' is awkward; consider replacing with 'human-related and non-human target categories'.
Circularity Check
Evaluation assumes the VLM's own proposal list is ground truth (Sec. 4.1), so Table 1 measures detector/VLM self-consistency, not independent detection accuracy.
-
self definitional
[Section 4.1 (Problem Setup) and Equation (1); coupled with Section 3.2 Proposal Extraction]
"Therefore, we frame our task into an quantitative alignment task between the VLM proposal and the results from the detection models. In other words, the results are evaluated based on the number of correctly detected entities while assuming the object list proposal made by the VLM to be the ground truth."
The F1 score in Eq. (1) is computed between Lp (the object list) and Ld (detector outputs). Lp is generated by GPT-4o inside FOCUS's Proposal Extraction stage (Sec. 3.2), so the metric measures how well the detector agrees with an internal pipeline module, not how accurately objects are detected relative to human labels. Moreover, the Region Isolation module feeds the detector the same masked image the VLM saw, making agreement with the VLM's proposal list easier by construction. The abstract's claim that FOCUS 'effectively enhances the detection capabilities of baseline models' therefore rests on a self-consistency measure; object omissions, hallucinations, and naming mismatches in Lp are never checked against independent ground truth.
full rationale
The central quantitative claim is circular in its evaluation setup: the paper explicitly defines the VLM proposal list as ground truth (Sec. 4.1) while that same VLM is a component of the proposed pipeline (Sec. 3.2). Consequently, Table 1's F1 scores quantify agreement between two FOCUS modules rather than detection accuracy against human-annotated objects. The relative gain from Region Isolation is a controlled comparison (baselines receive the same Lp), so that part is not itself circular, but the absolute 'detection capability enhancement' claim is not independently grounded. No load-bearing self-citation, uniqueness importation, or ansatz-smuggling patterns are present. The paper's qualitative examples and prompt-controllability demonstration have independent content, but the main quantitative support reduces to a self-consistency metric, warranting a partial-circularity score of 7.
Assumptions & free parameters
free parameters (1)
- detection score threshold =
0.1
assumptions (4)
- domain assumption SAM's binary mask from the user-supplied bounding box isolates exactly the intended target and removes only irrelevant context.
- ad hoc to paper The object list produced by GPT-4o is an acceptable ground truth for detection performance.
- domain assumption GroundingDINO and OWLv2 can detect arbitrary open-vocabulary noun phrases from the VLM proposal list.
- domain assumption Images with more people are inherently harder for fine-grained detection.
Cite this review
Pith. "Pith review of Fine-Grained Open-Vocabulary Object Recognition via User-Guided Segmentation." pith.science (2026). https://pith.science/paper/JE4IUCRA
@misc{pith2026241115620,
author = {Pith},
title = {Pith review of: Fine-Grained Open-Vocabulary Object Recognition via User-Guided Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/JE4IUCRA}},
note = {Machine review of arXiv:2411.15620}
}
read the original abstract
Recent advent of vision-based foundation models has enabled efficient and high-quality object detection at ease. Despite the success of previous studies, object detection models face limitations on capturing small components from holistic objects and taking user intention into account. To address these challenges, we propose a novel foundation model-based detection method called FOCUS: Fine-grained Open-Vocabulary Object ReCognition via User-Guided Segmentation. FOCUS merges the capabilities of vision foundation models to automate open-vocabulary object detection at flexible granularity and allow users to directly guide the detection process via natural language. It not only excels at identifying and locating granular constituent elements but also minimizes unnecessary user intervention yet grants them significant control. With FOCUS, users can make explainable requests to actively guide the detection process in the intended direction. Our results show that FOCUS effectively enhances the detection capabilities of baseline models and shows consistent performance across varying object types.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[20]
Grounded sam: Assembling open-world models for diverse visual tasks,
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks,
-
[1]
Yolo-world: Real-time open- vocabulary object detection
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. Yolo-world: Real-time open- vocabulary object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3
work page 2024
-
[2]
Everingham, S
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual ob- ject classes challenge: A retrospective.International Journal of Computer Vision, 111(1):98–136, 2015. 7
2015
-
[3]
Salience detr: Enhancing detection trans- former with hierarchical salience filtering refinement
Xiuquan Hou, Meiqin Liu, Senlin Zhang, Ping Wei, and Badong Chen. Salience detr: Enhancing detection trans- former with hierarchical salience filtering refinement. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17574–17583, 2024. 2
work page 2024
-
[4]
Relation detr: Exploring explicit position relation prior for object detection
Xiuquan Hou, Meiqin Liu, Senlin Zhang, Ping Wei, Badong Chen, and Xuguang Lan. Relation detr: Exploring explicit position relation prior for object detection. In European con- ference on computer vision. Springer, 2024. 2
work page 2024
-
[5]
Scaling up visual and vision-language representation learning with noisy text supervision
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning , pages 4904–4916. PMLR, 2021. 2
work page 2021
-
[6]
Rea- soning grasping via multimodal large language model
Shiyu Jin, Jinxuan Xu, Yutian Lei, and Liangjun Zhang. Rea- soning grasping via multimodal large language model. Con- ference on Robot Learning, 2024. 3
work page 2024
-
[7]
Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023. 2, 4, 5, 8
arXiv 2023
Show all 27 references
-
[8]
Lisa: Reasoning segmentation via large language model
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9579–9589, 2024. 3
2024
-
[9]
Grounded language-image pre-training
Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022. 1
2022
-
[10]
Belongie, Lubomir D
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’a r, and C. Lawrence Zitnick. Mi- crosoft coco: Common objects in context. arXiv preprint arXiv:1405.0312, 2014. 7
2014 arXiv
-
[11]
Grounding dino: Marrying dino with grounded pre-training for open-set object detection
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Con- ference on Computer Vision, 2024. 1, 2, 4, 5, 8
2024
-
[12]
Swin transformer: Hierarchical vision transformer using shifted windows
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 8 Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 5
2021
-
[13]
Scaling open-vocabulary object detection
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. In Conference on Neural Information Processing Systems, 2024. 4, 5
2024
-
[14]
Gpt-4 technical report
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 5
2023 arXiv
-
[15]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...
2021
-
[16]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv...
2021 arXiv
-
[17]
Sam 2: Segment anything in images and videos
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...
2024 arXiv
-
[18]
Yolov3: An incremental improvement
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv, 2018. 2
2018
-
[19]
Faster r-cnn: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Pro- cessing Systems. Curran Associates, Inc., 2015. 2
2015
-
[21]
Grounding language models for visual entity recognition
Zilin Xiao, Ming Gong, Paola Cascante-Bonilla, Xingyao Zhang, Jie Wu, and Vicente Ordonez. Grounding language models for visual entity recognition. In Proceedings of the IEEE/CVF European Conference on Computer Vision (ECCV), 2024. 3
2024
-
[22]
Visa: Reasoning video object segmentation via large language models
Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. In Proceedings of the IEEE/CVF European Con- ference on Computer Vision (ECCV), 2024. 3
2024
-
[23]
Detclipv3: Towards versatile generative open-vocabulary object detec- tion
Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Detclipv3: Towards versatile generative open-vocabulary object detec- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 27391–...
2024
-
[24]
Ferret: Refer and ground anything anywhere at any granularity
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations, 2024. 3
2024
-
[25]
A simple framework for open-vocabulary segmentation and detection
Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In ICCV, 2023. 4, 5
2023
-
[26]
Recognize anything: A strong image tagging model
Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. arXiv preprint arXiv:2306.03514 ,
-
[27]
Detrs with col- laborative hybrid assignments training
Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with col- laborative hybrid assignments training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6748–6758, 2023. 2, 3 9
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.