Pith. sign in

REVIEW 17 cited by

Simple Open-Vocabulary Object Detection with Vision Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.06230 v2 pith:IGY5WSB3 submitted 2022-05-12 cs.CV

Matthias Minderer , Alexey Gritsenko , Austin Stone , Maxim Neumann , Dirk Weissenborn , Alexey Dosovitskiy , Aravindh Mahendran , Anurag Arnab
show 6 more authors
Mostafa Dehghani Zhuoran Shen Xiao Wang Xiaohua Zhai Thomas Kipf Neil Houlsby
This is my paper · ORCID
classification cs.CV
keywords detectionobjectpre-trainingopen-vocabularyimage-textimprovementsmodelsscaling
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Combining simple architectures with large-scale pre-training has led to massive improvements in image classification. For object detection, pre-training and scaling approaches are less well established, especially in the long-tailed and open-vocabulary setting, where training data is relatively scarce. In this paper, we propose a strong recipe for transferring image-text models to open-vocabulary object detection. We use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. Our analysis of the scaling properties of this setup shows that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. We provide the adaptation strategies and regularizations needed to attain very strong performance on zero-shot text-conditioned and one-shot image-conditioned object detection. Code and models are available on GitHub.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3D

    cs.CV 2025-04 conditional novelty 7.0 of 10

    LOCATE 3D localizes objects named in referring expressions directly from posed RGB-D frames, achieving 61.7% top-1 accuracy at IoU 25 on SR3D/NR3D/ScanRefer and 63.7% when trained with a new 130K-annotation dataset.

  2. SAMIC: Segment Anything with In-Context Spatial Prompt Engineering

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A 2.6-million-parameter learned prompt generator for SAM achieves state-of-the-art or competitive one-shot segmentation on multiple benchmarks using only 20% of the training data.

  3. RegionDet: A Benchmark for Region Detection Beyond Object Instances

    cs.CV 2026-08 conditional novelty 6.0 of 10

    RegionDet shows supervised detectors can partially learn to localize activity and context regions, while open-vocabulary detectors nearly fail, indicating current vision-language models are object-centric.

  4. TransForSeg: A Multitask Stereo ViT for Joint Stereo Segmentation and 3D Force Estimation in Catheterization

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A shared-weight stereo ViT with cross-attention fusion segments the catheter in two views and regresses 3D tip forces, claiming state-of-the-art results on synthetic X-ray datasets.

  5. On Evaluating the Adversarial Robustness of Foundation Models for Multimodal Entity Linking

    cs.IR 2025-08 unverdicted novelty 6.0 of 10

    Multimodal entity linking models are vulnerable to visual adversarial perturbations, and the proposed retrieval-augmented LLM method (LLM-RetLink) reportedly improves accuracy by 0.4% to 35.7%.

  6. Text-guided Generation of Efficient Personalized Inspection Plans

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A training-free pipeline uses a vision-language model and segmentation to convert text instructions into smooth, order-respecting drone inspection trajectories in known 3D maps.

  7. Instance Segmentation of Scene Sketches Using Natural Image Priors

    cs.CV 2025-02 conditional novelty 6.0 of 10

    InkLayer combines class-agnostic fine-tuning of Grounding DINO, SAM masks, and depth-based foreground refinement to achieve state-of-the-art instance segmentation on scene sketches across styles.

  8. A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewards

    cs.RO 2025-02 conditional novelty 6.0 of 10

    IKER uses VLM-generated keypoint rewards to train manipulation policies in simulation that transfer to a real robot, enabling multi-step tasks and replanning.

  9. ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration

    cs.CV 2024-11 conditional novelty 6.0 of 10

    ZoomEye uses tree-based zooming, guided by an MLLM's own confidence scores, to improve high-resolution visual question answering without retraining the model.

  10. Decompose and Leverage Preferences from Expert Models for Improving Trustworthiness of MLLMs

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Decomposing MLLM responses into atomic verification tasks and checking them with an ensemble of open-source expert models yields preference data that reduces hallucination in LLaVA and Qwen-VL-Chat.

  11. Neuro-Symbolic Frameworks: Conceptual Characterization and Empirical Comparative Analysis

    cs.AI 2025-09 conditional novelty 5.0 of 10

    A facet-based comparison of DeepProbLog, Scallop, and DomiKnowS with efficiency measurements on four toy tasks, identifying challenges for future neurosymbolic frameworks.

  12. Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    Textual Inversion is applied to open-vocabulary object detectors to learn a few new tokens while keeping the VLM frozen and preserving zero-shot abilities.

  13. (Almost) Free Modality Stitching of Foundation Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A hypernetwork that generates connector weights for all image-text model pairs can rank pairs like grid search at about 10x lower training cost, but the best connector lags grid search by a few points.

  14. Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A probabilistic overlap measure for object positions yields a human-aligned spatial relationship metric and a training-free generation guidance method for text-to-image models.

  15. Relational Programming with Foundation Models

    cs.AI 2024-12 conditional novelty 5.0 of 10

    Vieira extends the Scallop relational engine with a foreign interface that lets foundation models act as probabilistic relations, enabling neuro-symbolic programs across nine tasks.

  16. Street Gaussians without 3D Object Tracker

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Replacing 3D object trackers with a 2D foundation model plus LiDAR and a motion-learning correction produces state-of-the-art street-scene reconstructions without ground-truth object poses.

  17. Towards Wearable Interfaces for Robotic Caregiving

    cs.RO 2025-02 conditional novelty 4.0 of 10

    This paper reports lessons from prior HAT teleoperation studies, preliminary shared-control gains, and a passive-control concept for robot-assisted feeding that is not yet evaluated.

Pith tools