Pith. sign in

REVIEW 4 major objections 5 minor 43 references

VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VL-SAM-V2 claims that fusing open-set and open-ended query streams yields state-of-the-art zero-shot detection on both modes, with rare-object AP rising from 37.3 to 41.2.

desk verdict A solid, well-scoped engineering extension of VL-SAM and LLMDet with genuinely useful fusion ideas, but the headline SOTA claims are ahead of the controlled evidence. read the letter →

arxiv 2505.18986 v1 pith:AH5ZP4BK submitted 2025-05-25 cs.CV

Zhiwei Lin , Yongtao Wang This is my paper
classification cs.CV
keywords open-worldobjectdetectionopen-setopen-endedqueryfusionvision-languagemodelsLVISraredenoisingqueries
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that an object detector does not have to choose between the high accuracy of open-set models and the open-world recall of open-ended models. Its proposed framework, VL-SAM-V2, takes the specific queries of a strong open-set detector and augments them with general queries generated by a vision-language pipeline that proposes objects with no category list, letting the two query types interact inside the decoder. On LVIS, this fusion raises rare-object AP from 37.3 to 41.2 (Swin-T) and from 45.1 to 47.2 (Swin-L) while leaving frequent-object AP unchanged, and it sets new state-of-the-art numbers in both open-set and open-ended evaluations. A sympathetic reader should care because it suggests a practical route to detectors that discover novel objects on their own without sacrificing performance on known ones.

What carries the argument

The load-bearing mechanism is the general-and-specific query fusion module: a self-attention layer inserted into each transformer decoder stage that concatenates the specific queries $Q_s$ from the open-set detector with the general queries $Q_g$, applies self-attention across the combined set, then routes the separated updates through shared query-to-text and query-to-image cross-attention and two unshared box heads that predict box offsets $\Delta B_g$ and $\Delta B_s$. Two supporting designs make it trainable: ranked learnable queries, which sort point proposals by classification score and match them to a fixed set of $N=900$ learnable queries so that each point gets a consistent role, and the denoising-points strategy, which replaces the expensive VL-SAM points during training with synthetic noisy points sampled around ground-truth boxes under the inequality $|\Delta X_p| < \lambda_1 w/2 < |\Delta X_n| < \lambda_2 w/2$ (and likewise for $Y$), letting the model learn to suppress negative points. The ranked matching converts disorderly point prompts into structured query slots, and the denoising surrogate is what keeps training cost at half of running real VL-SAM inference.

What would settle it

Measure the distribution shift between the synthetic denoising points used in training and VL-SAM's actual inference-time point proposals, for instance by comparing score distributions and coordinate offsets, or run inference with random points of the same count in place of VL-SAM's points. If random points reproduce the 3.9-point rare-object AP gain, or if a model fine-tuned on real VL-SAM points differs materially from the denoising-trained one, then the vision-language prior is not what is driving the improvement.

Watch

Extended reading notes

Core claim

The paper's central claim is that the apparent tradeoff between open-set precision and open-ended recall can be resolved at the level of decoder queries. VL-SAM-V2 observes that LLMDet, the open-set backbone, is strong on frequent classes but weak on rare ones, whereas VL-SAM's attention-derived point prompts, generated by an 8B vision-language model, preferentially surface rare or unusual objects. The method converts those point proposals into ranked learnable queries, matches them one-to-one by score, and then lets them interact with the detector's own queries through a self-attention based fusion module that is the only part fine-tuned alongside the box heads. The result, measured on LVIS minival with fixed AP, is that rare-object AP jumps from 37.3 to 41.2 on Swin-T while frequent-object AP holds at 50.7, and the same pattern appears with Swin-L; in open-ended mode, with the category list emptied, VL-SAM-V2 reports the best AP and rare-object AP among published methods. The paper further shows the fusion transfers to GroundingDINO and to several vision-language models, and that combining with SAM gives open-ended instance segmentation.

Load-bearing premise

The training never uses the real point proposals that VL-SAM produces at inference; it approximates them with random noisy points sampled around ground-truth boxes, so the reported gains depend on those two point distributions being close enough for the learned ranking and fusion to transfer.

Editorial extensions

If this is right

  • VL-SAM-V2 reports state-of-the-art zero-shot open-set detection on LVIS, beating LLMDet by 1.0 AP with Swin-T and 0.8 AP with Swin-L.
  • In open-ended mode it reports AP of 29.5 (Swin-T) and 31.8 (Swin-L), exceeding GenerateU by 2.7 and 3.9 AP and raising rare-object AP well above both GenerateU and VL-SAM.
  • The fusion module is model-agnostic: applied to GroundingDINO it improves rare-object AP by 2.3 to 4.9 depending on the vision-language model.
  • Because the general queries originate as point prompts, the framework also supports user-provided prompts and, with SAM attached, open-ended instance segmentation with mask AP of 28.7 on LVIS minival.
  • Each added component is necessary: fusion alone contributes 1.8 rare-object AP, ranked queries add 1.1, and denoising points add 1.0, so the gain is not attributable to any single design choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fusion generalizes as the ablations suggest, the same query-interaction design could be applied to video object discovery or open-vocabulary instance segmentation, where rare-object recall is likewise the bottleneck; the paper does not test these settings.
  • The strong results with a synthetic denoising surrogate leave open whether the vision-language model's semantic content matters or whether any extra foreground proposal stream would do; a control experiment replacing VL-SAM points with uniform random points inside ground-truth boxes would isolate the source of the gain.
  • The ranked-queries design implies a testable specialisation: the earliest learnable queries should specialise in salient frequent objects and later ones in rarer objects; examining query assignment across categories would tell whether the ranking is doing that work.
  • The paper's own limitation section concedes that the framework inherits the vision-language model's latency and hallucination, which suggests that distilling the point proposer into a small network is a natural next step for real-time deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes VL-SAM-V2, an open-world object detector that fuses 'general' queries derived from VL-SAM point prompts with 'specific' queries from an open-set detector (LLMDet), using a lightweight self-attention fusion module, ranked learnable queries, and a denoising-point training strategy. The model can be evaluated in open-set mode (with a predefined category list) or open-ended mode (empty list). Experiments on LVIS report improvements over LLMDet and prior open-ended methods, especially on rare classes.

Significance. If the claimed improvements are reproducible and attributable to the proposed fusion mechanism, this is a useful step toward closing the gap between open-set and open-ended detection. Strengths include the component-wise ablation in Table 3, the demonstration of generalization across detectors and vision-language models in Table 4, and the use of publicly available components. However, the main claims currently rest on confounded comparisons and an unverified training-inference distribution for point prompts, so the significance of the specific mechanism remains uncertain.

major comments (4)
  1. [4.2, Table 1] The claim in Section 4.2 that 'VL-SAM-V2 beats all previous open-set models and achieves the new state-of-the-art zero-shot open-set object detection results' is not supported on LVIS minival with Swin-L: DetCLIPv3 reports APr 49.9 and APc 49.7, while VL-SAM-V2 reports APr 47.2 and APc 46.7 in the same table. The statement that DetCLIPv3's advantage is due to balanced data and noun concept corpora is a conjecture rather than a controlled comparison; the paper should either qualify the state-of-the-art claim to fixed AP/APf or provide an ablation of DetCLIPv3 augmented with the fusion module.
  2. [4.2, Table 2] The open-ended comparison is confounded: VL-SAM-V2 uses InternVL-2.5-8B and is fine-tuned on GroundingCap-1M within the LLMDet framework, while VL-SAM is training-free with CogVLM-17B and GenerateU uses FlanT5-base. Therefore the reported gains (e.g., APr 30.5 vs. 23.4 for Swin-L/ViT-H) cannot be attributed to the fusion of general and specific queries, because the vision-language model, the training data, and the detector backbone all change jointly. Please report a variant of VL-SAM-V2 without the fusion module but with the same VLM and fine-tuning protocol, or a training-free VL-SAM with InternVL-2.5 as the point source.
  3. [3.2, Eq. (1), Table 3] The denoising-point strategy trains on uniform random points sampled from or around ground-truth boxes, while inference uses VL-SAM's attention-derived point prompts; the paper neither measures the distribution shift between these two types of points nor reports a model trained on actual VL-SAM points. The rows of Table 3 that precede 'Denoising Points' do not specify their point source, and Section 3.2 states that training replaces VL-SAM points with noisy points. Without such a comparison, the APr gain from 37.3 to 41.2 (Swin-T) could reflect extra ground-truth-informed queries rather than the value of real open-ended proposals. Add either a training run with real VL-SAM point prompts or a quantitative comparison of the two point distributions.
  4. [3.2, Eq. (1) and 4.1] With the stated hyperparameters lambda_1 = lambda_2 = 1, Eq. (1) requires |Delta X_n| to satisfy lambda_1 * w/2 < |Delta X_n| < lambda_2 * w/2, i.e., w/2 < |Delta X_n| < w/2, which has no solution; hence no negative points can be sampled under the reported configuration. Please correct the equation or the hyperparameter values and clarify how negative-point denoising is actually implemented.
minor comments (5)
  1. [4.2, Table 1] The caption says 'Grey results denote using additional private data', but no grey shading is visible in the table; please add a visible legend or shading.
  2. [4.2, open-ended mode] Please explain how 'specific queries' are produced when the predefined category list is empty; Section 3.2 mentions combining discovered objects with the empty list but does not specify how LLMDet's text queries are formed in this mode.
  3. [4.3, Table 3] The component gains are small (0.5 to 1.0 AP) and are reported without standard errors or multiple seeds; please report variance or significance information.
  4. [4.3, Table 4] There is a typo in the table: 'LLaV A' should be 'LLaVA'.
  5. [3.2, Eq. (3)] Please define all symbols in Eq. (3) explicitly in the text; in particular, clarify the relationship between the ground-truth boxes \hat B and the general/specific box sets B_g and B_s.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the fusion gain is benchmarked against a baseline without fusion, the self-cited VL-SAM is an input component rather than an unverified warrant, and no predicted quantity is defined by construction from the target.

full rationale

The paper's central claim is that fusing general queries from VL-SAM with specific queries from an open-set detector (LLMDet) improves open-set and open-ended object detection, especially for rare classes. This claim is tested directly against the LLMDet baseline without fusion (Table 1 and Table 3), and the ablation study shows each added component produces an incremental gain. No equation or definition in the paper defines a predicted quantity in terms of the target result; Eq. 1 describes a training-time denoising surrogate for VL-SAM points, but inference uses real VL-SAM point prompts, so any train/test distribution shift is an empirical robustness concern, not a definitional reduction. The authors' prior VL-SAM is used as an input component and is therefore a dependency, but the self-citation is not load-bearing in a circular sense: the fusion module is evaluated with different vision-language models and open-set backbones (Table 4), and the prior VL-SAM is not invoked as an unverified theorem or uniqueness result. No ansatz is smuggled in via citation, and no known empirical pattern is merely renamed as a new contribution. Thus the derivation chain is self-contained against external benchmarks and no specific circular step can be exhibited.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on empirical components rather than derived quantities. The free parameters are hyperparameters of the denoising-points and VLM sampling; they are not fitted to LVIS, but no sensitivity analysis is provided. The key loading assumptions are the reliability of VL-SAM points and the proxy validity of noisy-point training. No new physical entities are introduced.

free parameters (5)
  • lambda_1 (denoising positive-point noise bound) = 1
    Chosen in Eq. 1 to control the uniform sampling range for positive noisy points; no sensitivity analysis is reported.
  • lambda_2 (denoising negative-point noise bound) = 1
    Same as lambda_1 for negative points; set to 1 in Section 4.1 without ablation.
  • N (number of ranked learnable queries) = 900
    Set to 900 in Section 4.1; no study of how this interacts with the typical number of point prompts from VL-SAM.
  • VLM sampling temperature = 0.8
    Set for InternVL-2.5-8B in Section 4.1; affects the point proposals generated by VL-SAM.
  • VLM top-p (nucleus sampling) = 0.8
    Set for InternVL-2.5-8B in Section 4.1; affects the point proposals generated by VL-SAM.
assumptions (4)
  • domain assumption VL-SAM attention-derived point prompts provide useful object hypotheses, especially for rare objects (Section 3.1).
    The general-query branch depends on the vision-language model's attention maps being a reliable proposal source; if the VLM misses objects, the fusion cannot recover them.
  • domain assumption GroundingCap-1M provides a training distribution similar enough to LVIS that the fine-tuned fusion transfers in a zero-shot manner (Section 4.1).
    The model never trains on LVIS, so cross-dataset transfer is assumed rather than demonstrated.
  • ad hoc to paper Ground-truth-box-derived noisy points are a valid proxy for VL-SAM's real point prompts (Eq. 1, Section 3.2).
    This proxy is introduced to cut training cost by 50 percent, but the paper provides no direct comparison to training on true VL-SAM points.
  • domain assumption The open-ended evaluation protocol, using CLIP to match generated category names to LVIS categories, is a fair measure of open-ended detection (Section 4.2).
    Reported open-ended AP depends on CLIP matching quality; this protocol is inherited from prior work but not validated for this model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion." pith.science (2026). https://pith.science/paper/AH5ZP4BK

@misc{pith2026250518986,
  author       = {Pith},
  title        = {Pith review of: VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AH5ZP4BK}},
  note         = {Machine review of arXiv:2505.18986}
}
read the original abstract

Current perception models have achieved remarkable success by leveraging large-scale labeled datasets, but still face challenges in open-world environments with novel objects. To address this limitation, researchers introduce open-set perception models to detect or segment arbitrary test-time user-input categories. However, open-set models rely on human involvement to provide predefined object categories as input during inference. More recently, researchers have framed a more realistic and challenging task known as open-ended perception that aims to discover unseen objects without requiring any category-level input from humans at inference time. Nevertheless, open-ended models suffer from low performance compared to open-set models. In this paper, we present VL-SAM-V2, an open-world object detection framework that is capable of discovering unseen objects while achieving favorable performance. To achieve this, we combine queries from open-set and open-ended models and propose a general and specific query fusion module to allow different queries to interact. By adjusting queries from open-set models, we enable VL-SAM-V2 to be evaluated in the open-set or open-ended mode. In addition, to learn more diverse queries, we introduce ranked learnable queries to match queries with proposals from open-ended models by sorting. Moreover, we design a denoising point training strategy to facilitate the training process. Experimental results on LVIS show that our method surpasses the previous open-set and open-ended methods, especially on rare objects.

Figures

Figures reproduced from arXiv: 2505.18986 by the authors.

Figure 1
Figure 1. Illustration of VL-SAM-V2. VL-SAM-V2 combines the general queries from VL-SAM and the specific queries of an open-set model with a query fusion module. detect drones or wild animals in the input images. Therefore, they need humans to intervene by providing a more comprehensive list including these objects. Additionally, although open-set models perform well on frequent objects, they often struggle with rare objects … view at source ↗
Figure 2
Figure 2. The overall pipeline of VL-SAM-V2. VL-SAM-V2 utilizes a vision-language model to generate general queries and a standard open-set detection model to generate specific queries. Then, the two distinct queries are sent to the general and specific query fusion module for interaction. Finally, a box head and an optional SAM are applied to predict the perception results. During the training, we only fine-tune the general … view at source ↗
Figure 3
Figure 3. Illustration of general and specific query fusion module. General and specific queries interact with a self-attention mechanism. Then, the shared query-to-text and query-to-image cross￾attention are applied for the two queries independently. Finally, the unshared box heads predict the offset of corresponding bounding boxes. During the training, we only update the parameters in the self-attention and box heads. featu… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization results VL-SAM-V2 combining with SAM on CODA [22]. We show input images and detection and segmentation prediction results in the open-ended mode. VL-SAM-V2 can discover various uncommon objects. Best viewed by zooming in. 6 Conclusion In this work, we int…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 32 canonical work pages

  1. [1]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    work page 2020

  2. [2]

    Mmdetection: Open mmlab detection toolbox and benchmark

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019

    arXiv 1906

  3. [3]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020

    2020

  4. [4]

    Pix2seq: A language modeling framework for object detection

    Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10852, 2021

    arXiv 2021

  5. [5]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  6. [6]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  7. [7]

    Yolo-world: Real- time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time open-vocabulary object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    work page 2024

  8. [8]

    Evaluating large- vocabulary object detectors: The devil is in the details

    Achal Dave, Piotr Dollár, Deva Ramanan, Alexander Kirillov, and Ross Girshick. Evaluating large- vocabulary object detectors: The devil is in the details. arXiv preprint arXiv:2102.01066, 2021

    arXiv 2021

Show all 43 references
  1. [9]

    Reducing network agnostophobia

    Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. Neural Information Processing Systems (NeurIPS), 2018

    2018

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2020

  3. [11]

    Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models

    Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, and Wei-Shi Zheng. Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models. arXiv preprint arXiv:2501.18954, 2025

    2025 arXiv

  4. [12]

    Recent advances in open set recognition: A survey

    Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Recognition and Machine Intelligence (PAMI), 2020

    2020

  5. [13]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  6. [14]

    Ow- detr: Open-world detection transformer

    Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow- detr: Open-world detection transformer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  7. [15]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  8. [16]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEEE International Conference on Computer Vision (ICCV), 2017

    2017

  9. [17]

    T-rex2: Towards generic object detection via text-visual prompt synergy

    Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. In European Conference on Computer Vision (ECCV), 2024

    2024

  10. [18]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In IEEE International Conference on Computer Vision (ICCV), 2023. 10

    2023

  11. [19]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  12. [20]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  13. [21]

    Dn-detr: Accelerate detr training by introducing query denoising

    Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  14. [22]

    Coda: A real-world road corner case dataset for object detection in autonomous driving

    Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. In European Conference on Computer Vision (ECCV), 2022

    2022

  15. [23]

    Desco: Learning object recognition with rich language descriptions

    Liunian Li, Zi-Yi Dou, Nanyun Peng, and Kai-Wei Chang. Desco: Learning object recognition with rich language descriptions. Neural Information Processing Systems (NeurIPS), 2023

    2023

  16. [24]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  17. [25]

    Generative region-language pretraining for open-ended object detection

    Chuang Lin, Yi Jiang, Lizhen Qu, Zehuan Yuan, and Jianfei Cai. Generative region-language pretraining for open-ended object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  18. [26]

    Training-free open-ended object detection and segmentation via attention as prompts

    Zhiwei Lin, Yongtao Wang, and Zhi Tang. Training-free open-ended object detection and segmentation via attention as prompts. In Neural Information Processing Systems (NeurIPS), 2024

    2024

  19. [27]

    Visual instruction tuning.Neural Information Processing Systems (NeurIPS), 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Neural Information Processing Systems (NeurIPS), 2023

    2023

  20. [28]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision (ECCV), 2024

    2024

  21. [29]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE International Conference on Computer Vision (ICCV), 2021

    2021

  22. [30]

    Capdet: Unifying dense captioning and open-world detection pretraining

    Yanxin Long, Youpeng Wen, Jianhua Han, Hang Xu, Pengzhen Ren, Wei Zhang, Shen Zhao, and Xiaodan Liang. Capdet: Unifying dense captioning and open-world detection pretraining. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  23. [31]

    Scaling open-vocabulary object detection.Neural Information Processing Systems (NeurIPS), 2023

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection.Neural Information Processing Systems (NeurIPS), 2023

    2023

  24. [32]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (IC...

    2021

  25. [33]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Neural Information Processing Systems (NeurIPS), 2015

    2015

  26. [34]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024

    2024 arXiv

  27. [35]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang,...

    2020

  28. [36]

    Ov-dino: Unified open-vocabulary detection with language-aware selective fusion

    Hao Wang, Pengzhen Ren, Zequn Jie, Xiao Dong, Chengjian Feng, Yinlong Qian, Lin Ma, Dongmei Jiang, Yaowei Wang, Xiangyuan Lan, et al. Ov-dino: Unified open-vocabulary detection with language-aware selective fusion. arXiv preprint arXiv:2407.07844, 2024. 11

    2024 arXiv

  29. [37]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Neural Information Processing Systems (NeurIPS), 2024

    2024

  30. [38]

    Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment

    Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  31. [39]

    Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection

    Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. Neural Information Processing Systems (NeurIPS), 2022

    2022

  32. [40]

    Detclipv3: Towards versatile generative open-vocabulary object detection

    Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Detclipv3: Towards versatile generative open-vocabulary object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  33. [41]

    Ni, and Heung-Yeung Shum

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2023

    2023

  34. [42]

    Llava-grounding: Grounded visual chat with large multimodal models

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Leizhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision (ECCV), 2024

    2024

  35. [43]

    Glipv2: Unifying localization and vision-language understanding

    Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Neural Information Processing Systems (NeurIPS), 2022. 12

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.