Pith. sign in

REVIEW 3 major objections 4 minor 30 references

Test-time Vocabulary Adaptation for Language-driven Object Detection

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-image class filters lift open-vocabulary detectors by up to 3.3 AP.

desk verdict Per-image vocabulary filtering gives consistent small gains, but without a non-semantic shrinkage control the paper can't claim the captioner is what makes it work. read the letter →

arxiv 2506.00333 v1 pith:J22SISAY submitted 2025-05-31 cs.CV

Mingxuan Liu , Tyler L. Hayes , Massimiliano Mancini , Elisa Ricci , Riccardo Volpi , Gabriela Csurka This is my paper
classification cs.CV
keywords open-vocabularyobjectdetectionvocabularyadaptationtest-timevision-languagemodelslargelanguageimagecaptioningzero-shotplug-and-play
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that open-vocabulary object detectors suffer when users give them overly broad class vocabularies, because irrelevant categories act as distractors. It proposes VocAda, a training-free module that adapts the vocabulary to each individual image at test time: a captioner describes the visible objects, a parser extracts noun phrases, and a selector keeps only vocabulary entries relevant to the image. Applied to three off-the-shelf detectors on two benchmarks, VocAda consistently improves average precision, with the LLM-based variant gaining up to +3.3 points on COCO and +2.2 points on Objects365. An oracle experiment, which uses ground-truth classes, shows headroom of up to +6.9 and +19.3 points, implying the idea is sound even if the current selector is imperfect.

What carries the argument

VocAda is a three-stage, training-free pipeline: (i) an Image Captioner, a vision-language model (LLaVA-Next-7B) prompted to list primary and secondary objects in the image; (ii) a Noun Extractor, spaCy dependency parsing that pulls out noun phrases; and (iii) a Class Selector, with two variants, a CLIP ViT-L/14 text-similarity top-k matcher or an LLM (Llama3-8B) given the caption, extracted nouns, and vocabulary enriched with synonyms. The selector's output is the adapted vocabulary used by the frozen detector's classifier.

What would settle it

Construct a benchmark dominated by small or background objects drawn from the long tail of Objects365 and measure per-class recall of the filtered vocabulary; if VocAda's recall for those classes falls below the detector's baseline recall, those classes will be lost and the method will show a per-class AP drop instead of a gain. Alternatively, feed the captioner ground-truth class names as a natural-language description: if the selector still fails to match them to the vocabulary, the bottleneck is the selector rather than the captioner.

Watch

Extended reading notes

Core claim

The central discovery is that test-time vocabulary adaptation improves off-the-shelf open-vocabulary object detection, and that a fully training-free pipeline using a vision-language captioner, a noun parser, and either CLIP similarity or LLM reasoning can implement this adaptation. The paper shows that discarding irrelevant classes from the user-defined vocabulary for a given image reduces confusions (e.g., a curling stone mistaken for a teapot) and yields consistent AP gains across Detic, VLDet, and CoDet on COCO and Objects365. It also quantifies the adaptation quality: the LLM selector achieves higher precision (73.7% vs. 19.5%) while the CLIP selector achieves higher recall (94.8% vs. 86.1%).

Load-bearing premise

The approach assumes the image captioner and class selector will notice every object category that actually appears in the image; if a class is missed or mis-mapped, the detector can never propose it, and the gains vanish.

Editorial extensions

If this is right

  • If VocAda is correct, any frozen open-vocabulary detector can be improved without retraining simply by filtering its input vocabulary per image.
  • The gap between baseline and oracle (up to +6.9 on COCO and +19.3 on Objects365) indicates that vocabulary quality is a major bottleneck in current open-vocabulary detection pipelines.
  • Combining VocAda with prompt-engineering methods like Descriptor further improves results, while combining it with SHiNe gives only minor additional gains.
  • The LLM-based selector is more precise but less recallful; improving recall while maintaining precision is a clear direction for further gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • In deployed settings where a vision-language model is already running (for example, autonomous driving), VocAda's marginal cost would be small, making per-image vocabulary filtering a natural default pre-processing step rather than an optional add-on.
  • The same vocabulary-adaptation principle could transfer to other open-vocabulary tasks such as open-vocabulary segmentation or retrieval, where the user-specified class or query list is the main input to be filtered.
  • A testable extension: using a region-aware captioner that describes localized objects rather than the whole image might attack the 86.1% recall bottleneck and recover small-object classes that whole-image captions tend to miss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces VocAda, a training-free, plug-and-play module that adapts a user-specified class vocabulary per image at test time. For each image, VocAda generates a caption with LLaVA-Next-7B, extracts noun phrases with spaCy, and selects a subset of the original vocabulary using either a CLIP-based top-k text-similarity matching or an LLM-based (Llama3-8B) selection. The detector then classifies proposals only within this per-image subset (Eq. 2). Experiments on OVE-COCO and CDTE-Objects365 with Detic, VLDet, and CoDet report consistent AP50 gains over the full-vocabulary baseline (up to +3.3 and +2.2 points, respectively) and outperform the RAM++ tagger. An oracle variant using ground-truth classes shows much larger gains (+6.9 and +19.3 points), validating that vocabulary refinement has large potential.

Significance. The idea of adapting the test-time vocabulary to the image content using a VLM captioner is simple, novel in the OvOD context, and potentially useful as a complementary plug-in to any open-vocabulary detector. The paper is clearly written, includes detailed prompts, reports results across three detectors and two benchmarks, and provides a computational cost analysis. The method is not circular: the adapted vocabulary is derived from an external captioner and text matching, not from the detector outputs or test labels. However, the central empirical claim is currently confounded by the absence of a non-semantic control for label-set shrinkage; if that control is added and the gains remain, the contribution would be solid and likely of interest to the detection community.

major comments (3)
  1. [§4.1, Tables 1-2] The reported gains over the Baseline do not control for the trivial benefit of reducing the candidate class set. Because Eq. 2 restricts the argmax to a per-image subset C̃_I and AP is computed per class, any subset that removes classes absent from an image can improve AP by eliminating false-positive opportunities for those classes, even if the selection is not semantically grounded. The comparisons to RAM++ and prompt-engineering methods do not include a non-semantic control matched in cardinality (e.g., a random per-image subset of C with the same size as C̃_I, or a subset chosen by training-set class frequency). Without such a control, the experiments do not establish that the captioner and class selector are the cause of the observed improvements. Please add a matched-cardinality control experiment on at least one benchmark (ideally both) to support the central claim.
  2. [§4.2, Table 3 and Fig. 6] The top-k hyperparameter in the CLIP-based selector and the choice among CLIP, SBert, and LLM selectors are evaluated on the same test benchmarks used to select the final configuration. The paper does not describe a validation-based selection procedure. Given the small magnitude of the gains (1–3 AP50), this creates a risk of overfitting to the test set. Please specify how k and the selector variant were chosen (e.g., using a validation split), or report results for all variants on both training and test splits.
  3. [§4.2, Fig. 5 and Tables 1-2] The reported improvements are not accompanied by any measure of variance, such as error bars across detector training seeds, captioner/LLM temperature samples, or significance tests. With gains of +2–3 AP50, it is unclear whether the differences are statistically meaningful. I request either significance tests or repeated-run variance reporting for at least one detector to support the claim that the improvements are consistent and not due to a single favorable run.
minor comments (4)
  1. [Abstract and §1] The abstract states "The code is open source," while the contributions section says "The code will be released upon publication." Please make these statements consistent.
  2. [Eq. (2)] The notation C̃_I is used in Eq. (2) before it is formally defined in the following paragraph. Move the definition before or immediately after the equation.
  3. [Supp. C.2, Fig. 10] The label "InsertedCustomerPrompt" appears to be a typo for "Inserted User Prompt." Please correct it.
  4. [Supp. Table 6] The main text (Sec. 1) states that VocAda "minimizes the computational overhead introduced by large VLMs," but Table 6 shows a 50–90x slowdown over the baseline detector (5.6–10.7 s/img vs. 0.115 s/img). The supplement's mitigation paragraph is helpful, but the main text should be more precise about the actual overhead.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the adapted vocabulary comes from an external captioner and text-based selector, not from evaluation labels or detector outputs.

full rationale

The paper's derivation chain is not circular. The method uses an external image captioner (LLaVA-Next), a noun extractor (spaCy), and a class selector (CLIP text similarity or an LLM) to produce a per-image subset of the user-defined vocabulary, which is then used in the detector's argmax in Eq. 2. Nothing in this pipeline is fitted to the COCO or Objects365 evaluation labels, and the reported gains are measured against fixed-vocabulary baselines on external benchmarks. The Oracle row in Tables 1 and 2 is explicitly labeled as an upper bound using ground-truth classes, and the paper presents it as motivation rather than as a method prediction; observing that an oracle is better than the baseline is a sanity check, not a circular derivation. The only self-citations are to the authors' SHiNe, which appears as a related-work contrast and as a comparison baseline in Fig. 6; it is not load-bearing evidence for the correctness of VocAda. The paper also discloses a genuine limitation, namely the 86.1% recall of VocAda-LLM in Fig. 5, which confirms that the captioner can miss relevant classes and that the improvements are empirical rather than guaranteed by construction. The skeptical concern that label-set shrinkage alone may explain part of the gain is a legitimate experimental-control issue, but it is not circularity: the paper does not define the method's output in terms of the AP metric, and no equation equates the reported gain to the act of shrinking the vocabulary.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. Its load-bearing assumptions are about the reliability of off-the-shelf VLMs and LLMs, plus the choice of the top-k hyperparameter.

free parameters (1)
  • top-k in CLIP-based selector = 1
    The number of vocabulary classes kept per extracted noun is set to k=1 after analyzing k=1,2,3 on the same OVE-COCO benchmark (Tab. 3), which is a form of test-set model selection.
assumptions (3)
  • domain assumption The image captioner (LLaVA-Next) lists the primary and secondary objects comprehensively enough for vocabulary selection.
    Section 3.1 and 4.2: the recall of the adapted vocabulary is only 86.1% for the LLM variant, showing the assumption holds imperfectly. The method depends on this to avoid missing true classes.
  • domain assumption The optimal per-image vocabulary is the set of classes that actually appear in the image.
    Section 4.1 Oracle validation: the oracle uses ground-truth classes to define the upper bound. The entire motivation assumes that removing absent classes is beneficial and that this target is achievable.
  • domain assumption Noun phrases extracted with spaCy can be mapped to user-defined class names via text similarity or LLM reasoning.
    Section 3.1 Class Selector: both the CLIP-based and LLM-based selectors rely on this mapping, and the synonym enrichment in Supp. A shows that direct matching is error-prone.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Test-time Vocabulary Adaptation for Language-driven Object Detection." pith.science (2026). https://pith.science/paper/J22SISAY

@misc{pith2026250600333,
  author       = {Pith},
  title        = {Pith review of: Test-time Vocabulary Adaptation for Language-driven Object Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J22SISAY}},
  note         = {Machine review of arXiv:2506.00333}
}
read the original abstract

Open-vocabulary object detection models allow users to freely specify a class vocabulary in natural language at test time, guiding the detection of desired objects. However, vocabularies can be overly broad or even mis-specified, hampering the overall performance of the detector. In this work, we propose a plug-and-play Vocabulary Adapter (VocAda) to refine the user-defined vocabulary, automatically tailoring it to categories that are relevant for a given image. VocAda does not require any training, it operates at inference time in three steps: i) it uses an image captionner to describe visible objects, ii) it parses nouns from those captions, and iii) it selects relevant classes from the user-defined vocabulary, discarding irrelevant ones. Experiments on COCO and Objects365 with three state-of-the-art detectors show that VocAda consistently improves performance, proving its versatility. The code is open source.

Figures

Figures reproduced from arXiv: 2506.00333 by the authors.

Figure 5
Figure 5. Vocabulary adaptation quality on OVE-COCO, measured via average precision and recall. 4.2. Analysis Quality of the adapted vocabulary. Next, we evaluate our two best methods in terms of vocabulary adaptation quality in [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 7
Figure 7. As observed, the baseline detector using the full vocabulary [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 26 canonical work pages

  1. [1]

    Test-time Vocabulary Adaptation for Language-driven Object Detection

    INTRODUCTION The goal of object detection is answering the question“What objects are present, and where?”, by locating and classify- ing objects in images. Such detection is vital for applications like autonomous driving and embodied AI [ 1]. Traditional detectors [2, 3] only recognize classes seen in training, requir- ing finetuning for new classes. To o...

    work page Pith review arXiv 2025

  2. [2]

    RELATED WORK Open-vocabulary object detection (OvOD) [11] aims to map predicted region features to a frozen vision-language embed- ding space, typically from contrastive models like CLIP [4]. OvOD detectors usually train on box-labeled data [9, 12] with limited categories due to high annotation costs, and supplement these with datasets annotated at image ...

    work page

  3. [3]

    a {Class Name}

    METHOD In this work, we aim to improve off-the-shelf OvOD detectors viavocabulary adaptation. First, we briefly introduce modern OvOD models. Then, we present V ocAda, detailing its key components and their integration in a detection pipeline. Background.Modern OvOD detectors [ 11] follow a two- stage process:Stage 1:A region proposal network takes an im-...

    work page

  4. [4]

    Storage Box

    EXPERIMENTS Benchmarks.We evaluate V ocAda on COCO [ 9] and Ob- jects365 [10]. For COCO, we follow the open-vocabulary evaluation (OVE) protocol [11], splitting the 80 classes into 48 base (seen) and 17 novel (unseen) while excluding 15 classes lacking WordNet [20] synsets. We train OvOD detectors on box-labeled base classes and evaluate them on 5 k test ...

    work page

  5. [5]

    Inspired by our Ora- cle results—where removing distracting classes yields large gains—we introduce V ocAda, aplug-and-playmodule foroff- the-shelfOvOD detectors

    CONCLUSION Open-vocabulary object detection (OvOD) allows users to freely define the vocabulary for a detector , making vocab- ulary quality pivotal to performance. Inspired by our Ora- cle results—where removing distracting classes yields large gains—we introduce V ocAda, aplug-and-playmodule foroff- the-shelfOvOD detectors. Our experiments show that V o...

    work page

  6. [6]

    Seeing the unseen: Visual common sense for semantic placement,

    Ram Ramrakhya, Aniruddha Kembhavi, Dhruv Batra, Zsolt Kira, Kuo-Hao Zeng, and Luca Weihs, “Seeing the unseen: Visual common sense for semantic placement,” inCVPR, 2024

    work page 2024

  7. [7]

    Mask r-cnn,

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, “Mask r-cnn,” inICCV, 2017

    work page 2017

  8. [8]

    Yolov3: An incremental improvement,

    Joseph Redmon and Ali Farhadi, “Yolov3: An incremental improvement,”arXiv:1804.02767, 2018

    arXiv 2018

Show all 30 references
  1. [9]

    Learning Transferable Visual Models From Natural Language Supervision,

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” inICML, 2021

    2021

  2. [10]

    Detecting Twenty-thousand Classes using Image-level Supervision,

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähen- bühl, and Ishan Misra, “Detecting Twenty-thousand Classes using Image-level Supervision,” inECCV, 2022

    2022

  3. [11]

    LLaV A-NeXT: Improved reasoning, OCR, and world knowledge,

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee, “LLaV A-NeXT: Improved reasoning, OCR, and world knowledge,” 2024

    2024

  4. [12]

    Learning object-language alignments for open-vocabulary object detec- tion,

    Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gho- lamreza Haffari, Zehuan Yuan, and Jianfei Cai, “Learning object-language alignments for open-vocabulary object detec- tion,” inICLR, 2023

    2023

  5. [13]

    CoDet: Co-Occurrence Guided Region-Word Alignment for Open-V ocabulary Object Detection,

    Chuofan Ma, Yi Jiang, Xin Wen, Zehuan Yuan, and Xiaojuan Qi, “CoDet: Co-Occurrence Guided Region-Word Alignment for Open-V ocabulary Object Detection,” inNeurIPS, 2023

    2023

  6. [14]

    Microsoft COCO: Common Objects in Context,

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick, “Microsoft COCO: Common Objects in Context,” in ECCV, 2014

    2014

  7. [15]

    Objects365: A large-scale, high-quality dataset for object detection,

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun, “Objects365: A large-scale, high-quality dataset for object detection,” inICCV, 2019

    2019

  8. [16]

    A Survey on Open-V ocabulary Detection and Segmentation: Past, Present, and Future,

    Chaoyang Zhu and Long Chen, “A Survey on Open-V ocabulary Detection and Segmentation: Past, Present, and Future,” arXiv:2307.09220, 2023

    2023 arXiv

  9. [17]

    LVIS: A Dataset for Large V ocabulary Instance Segmentation,

    Agrim Gupta, Piotr Dollar, and Ross Girshick, “LVIS: A Dataset for Large V ocabulary Instance Segmentation,” inCVPR, 2019

    2019

  10. [18]

    ImageNet: a Large-Scale Hierarchical Image Database,

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “ImageNet: a Large-Scale Hierarchical Image Database,” inCVPR, 2009

    2009

  11. [19]

    RegionCLIP: Region- based Language-Image Pretraining,

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao, “RegionCLIP: Region- based Language-Image Pretraining,” inCVPR, 2022

    2022

  12. [20]

    SHiNe: Semantic hierarchy nexus for open-vocabulary object detection,

    Mingxuan Liu, Tyler L. Hayes, Elisa Ricci, Gabriela Csurka, and Riccardo V olpi, “SHiNe: Semantic hierarchy nexus for open-vocabulary object detection,” inCVPR, 2024

    2024

  13. [21]

    mplug-owl2: Revolutionizing multi-modal large language model with modal- ity collaboration,

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang, “mplug-owl2: Revolutionizing multi-modal large language model with modal- ity collaboration,” inCVPR, 2024

    2024

  14. [22]

    spaCy: Industrial-strength Natural Language Processing in Python,

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd, “spaCy: Industrial-strength Natural Language Processing in Python,” 2020

    2020

  15. [23]

    Introducing meta llama 3: The most capable openly available llm to date,

    AI Meta, “Introducing meta llama 3: The most capable openly available llm to date,”Meta AI, 2024

    2024

  16. [24]

    Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo, “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,” in ICCV, 2021

    2021

  17. [25]

    Christiane Fellbaum,WordNet: an Electronic Lexical Database, MIT Press, 1998

    1998

  18. [26]

    Open-set image tagging with multi-grained text supervision,

    Xinyu Huang, Yi-Jie Huang, Youcai Zhang, Weiwei Tian, Rui Feng, Yuejie Zhang, Yanchun Xie, Yaqian Li, and Lei Zhang, “Open-set image tagging with multi-grained text supervision,” arXiv:2310.15200, 2023

    2023 arXiv

  19. [27]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    Nils Reimers and Iryna Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” inEMNLP, 2019

    2019

  20. [28]

    Mistral 7b,

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al., “Mistral 7b,”arXiv:2310.06825, 2023

    arXiv

  21. [29]

    ChatGPT: A Large-Scale GPT-3.5-Based Model,

    OpenAI, “ChatGPT: A Large-Scale GPT-3.5-Based Model,” https://openai.com/blog/chatgpt, 2022

    2022

  22. [30]

    Visual Classification via Description from Large Language Models,

    Sachit Menon and Carl V ondrick, “Visual Classification via Description from Large Language Models,” inICLR, 2023. Test-time Vocabulary Adaptation for Language-driven Object Detection Supplementary Material In this supplementary material, we first presents further experi- ment...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.