Pith. sign in

REVIEW 3 major objections 6 minor 67 references

C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes

T0 review · 3 major / 6 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Iteratively rewriting image captions alone can raise minority-class detection scores on frozen open-vocabulary detectors by tens of percent, without labels or retraining.

desk verdict Solid empirical engineering: real minority AP gains on frozen OVDs via composite captions + AP-triaged LLM refinement, but the annotation-free/online framing overreaches because triage needs labeled AP. read the letter →

arxiv 2607.09008 v1 pith:LW52AXUJ submitted 2026-07-10 cs.CV

Francis Fernandez , Arash Jahangiri , Salimeh Sekeh This is my paper
classification cs.CV
keywords open-vocabularydetectionclassimbalancepromptrefinementminority-classAPfrozendetectorscaptionengineeringsafety-criticalperception
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Safety-critical cameras must still catch rare objects (cyclists, buses, trucks) even when most frames are dominated by common ones. Long-tailed training methods need many labeled minority examples and large label spaces; they do not fit small fixed label sets with severe imbalance. Open-vocabulary detectors already accept free-form text at inference, so the authors treat the text prompt itself as the control knob. They first build a composite caption that concatenates a scene description with an explicit class-count statement, then let a vision-language model rewrite each image’s caption, keeping only those rewrites that raise minority-class average precision at IoU 0.5. Detector weights never change. On four detectors and three datasets the procedure lifts minority AP@0.5 in ten of twelve settings, recovering detections from zero-recall baselines and producing an 81 percent relative gain on the COCO bus class for one backbone. Both stages matter: starting refinement from scene-only or count-only captions yields smaller gains.

What carries the argument

C-GAP’s three-bucket triage: each candidate caption set is accepted, held as tentative, or regenerated according to whether its minority-class AP@0.5 lies above, inside, or below a dynamic margin around the composite-caption baseline; refinement can stop early once the keep threshold is crossed.

What would settle it

Run the identical frozen detectors and composite starting captions on the same validation images, but replace the AP@0.5 triage with random or length-matched caption rewrites; if minority AP@0.5 no longer improves (or improves equally under random rewrites), the claim that detector-measured AP feedback is what drives the gains is falsified.

Watch

Extended reading notes

Core claim

Caption quality is a first-class, training-free lever for minority-class detection under low-cardinality imbalance. A two-phase procedure—composite (scene + quantity) initialization followed by detector-AP-guided iterative rewriting—raises minority-class AP@0.5 on frozen open-vocabulary detectors, with gains largest precisely where static prompts fail.

Load-bearing premise

The method needs ground-truth boxes on a validation or working set so that minority-class AP@0.5 can be measured and used as the triage signal; without those labels the accept/regenerate decisions cannot be formed.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes C-GAP, a two-phase, detector-agnostic pipeline that improves minority-class detection for frozen open-vocabulary detectors by refining per-image language prompts. Phase I builds a composite caption (scene description + class-quantity) offline with a VLM; Phase II has an LLM iteratively rewrite each image’s caption, triaging trials into regenerate / tentative / keep buckets using minority-class AP@0.5 against dynamic thresholds derived from the composite baseline (Algorithm 1, Eq. 1–3). No detector weights are updated. Across four OVDs (Grounding DINO, OmDet-Turbo, OWLv2, YOLO-World) and three datasets (COCO bus, Cityscapes truck, Chula Vista Bike), C-GAP improves minority AP@0.5 in 10 of 12 configurations, with large relative gains (e.g., YOLO-World COCO 17.69→32.09) and recovery of nonzero detections from zero-recall baselines. Ablations cover T0 initialization, Phase-II generator, trial budget K, and a minority-only working set; overall mAP@0.5 is reported as a guardrail.

Significance. If the results hold under a correctly scoped claim, the work is a useful, practical contribution for safety-critical perception with small fixed label spaces: it shows that prompt content alone can move minority-class AP substantially on frozen OVDs, without retraining or architectural change. Strengths include multi-backbone / multi-dataset evaluation, explicit disclosure of mAP trade-offs, systematic caption-type study (Table 1), and ablations that support both phases (Tables 4–7, minority-only probe). The three-bucket triage also supplies a label-free diagnostic of backbone–prompt compatibility. The main limitation is that the reported protocol and the “annotation-free / online” framing are in tension: every triage and best-over-budget selection uses labeled minority AP@0.5. With that scope clarified, the empirical finding remains of interest for offline prompt search and for systems that already have a small labeled validation set.

major comments (3)
  1. Abstract, §1, §3.3, Algorithm 1, and Conclusion repeatedly call C-GAP “annotation-free” and sketch an online field loop, yet Phase II forms τ_low / τ_high from AP_CC, buckets every trial by AP50(cm; Tk), early-stops on B3, and reports best-over-budget including k=0. All of that requires ground-truth boxes for the minority class on the evaluation (or stratified working) set. This does not invalidate the offline numbers, but it is load-bearing for the claim as framed. The manuscript should either (i) restrict the claim to offline / validation-set prompt search and remove or heavily qualify the annotation-free and online production language, or (ii) provide a label-free surrogate (e.g., confidence, pseudo-label consistency) and re-evaluate triage under that signal.
  2. §4–5 and Table 2: On COCO, iterative refinement uses a 100-image stratified working set (at least 5% minority images), while Cityscapes and Chula Vista use full validation splits. Best-over-budget selection is then averaged over three seeds. Absolute AP numbers and relative gains (including the headline ~81% on YOLO-World) are therefore not directly comparable across datasets, and the COCO protocol is closer to a small labeled search set than to full-set evaluation. Report full-validation COCO numbers (or a clear sensitivity study of working-set size) and state the selection protocol in the main tables so that the central claim is not overstated.
  3. §5.2, Table 2 and Figure 5: Overall mAP@0.5 declines in several configurations (e.g., OmDet-Turbo COCO −2.12 pp, Cityscapes −2.45 pp; Grounding DINO Cityscapes −1.17 pp). The paper correctly reports this as a guardrail, but the abstract and contributions emphasize minority gains without quantifying the majority-class cost. A short per-class breakdown or explicit statement of when the trade-off is acceptable for safety-critical use would strengthen the central claim that minority improvement is the dominant effect.
minor comments (6)
  1. Abstract: “C-GAP Caption-Guided…” is missing the opening parenthesis; also “Our experiments shows” → “show”.
  2. §1 Contributions bullet 1: “datase” → “datasets”; several other small typos (e.g., “a small label space” / “spaces”).
  3. Table 1 vs. Table 2: CC minority AP for the same backbone–dataset sometimes differs (e.g., Grounding DINO COCO 26.61 in Table 1 vs. 15.30 in Table 2). Clarify that Table 2 uses the stratified working set / different seeds so readers do not treat them as contradictory.
  4. Figure 2 and Algorithm 1: the feedback conditioning of G_ϕ on a_{k−1} and β_{k−1} is clear in the algorithm but only sketched in the figure; a one-line note that the LLM receives the previous aggregate AP and bucket would help.
  5. §5.3 / Table 4: SD as T0 yields a higher absolute C-GAP score than CC on Grounding DINO/COCO; the text still prefers CC as “backbone-agnostic.” A sentence on when SD might be preferred would avoid the appearance of cherry-picking the primary initialization.
  6. Supplementary prompt templates are valuable; consider moving a short example of a B1 vs. B3 refinement into the main paper (near Figure 4) so the mechanism is self-contained.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild best-over-budget selection on the same validation set used for triage makes reported gains non-negative by construction; no definitional or self-citation circularity in the method.

  1. fitted input called prediction [Sec. 4 Evaluation Metrics; Sec. 5.2; Algorithm 1]
    "The reported score is the best found across all trials, including the initial CC baseline as trial k=0. So C-GAP results represent best-over-budget prompt selection averaged over three seeds. ... C-GAP selects the best minority AP@0.5 across all K trials including k=0 (the CC baseline), so it cannot score below CC by construction."

    Prompt sets are triaged and the final reported AP is chosen by maximizing the identical minority AP@0.5 metric on the same validation/working images used for feedback. The comparison to the composite baseline is therefore non-negative by the selection protocol itself, and absolute numbers are optimistically fitted to the evaluation set rather than predicted on held-out data.

full rationale

C-GAP is an empirical prompt-refinement pipeline, not a first-principles derivation. The sole mild circularity is that the reported C-GAP score is defined as the max minority AP@0.5 over K trials including the CC baseline (k=0) on the identical validation/working set used for bucket triage and early stopping; gains are therefore guaranteed >=0 and optimistically selected on the evaluation data itself. This is ordinary best-of-search reporting, not a fitted constant renamed as an independent prediction, nor a self-definitional equation, nor a load-bearing self-citation of a uniqueness theorem. Composite captions, LLM refinements, and frozen-detector evaluations remain independent of one another; no ansatz is smuggled via prior author work, and the paper is self-contained against external OVD benchmarks. The annotation-free framing is a separate overclaim (labels are required for AP feedback) but is not circularity under the stated patterns. Score 2 reflects only this minor selection bias.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The central empirical claim rests on a small set of free design choices (margin δ, trial budget K, working-set construction) and standard domain assumptions about open-vocabulary detectors and AP evaluation. No new physical or mathematical entities are postulated; the invented constructs are purely algorithmic (composite caption, three-bucket triage).

free parameters (3)
  • bucket margin δ = 0.05
    Sets τ_high = AP_CC + δ and τ_low = AP_CC − δ; fixed at 0.05 without sensitivity sweep beyond the reported K ablations.
  • max refinement trials K = 15
    Default budget for Phase II; recommended as 15 after ablation on Chula Vista.
  • COCO stratified working-set size = 100 images
    100-image subset with ≥5 % minority-class images used for tractability; absolute AP numbers are not full-val.
assumptions (3)
  • domain assumption Open-vocabulary detectors accept free-form text queries at inference and their detection quality is sensitive to prompt wording.
    Stated in Introduction and Related Work; underpins the entire approach of never updating θ.
  • domain assumption Minority-class AP@0.5 on a labeled set is a valid scalar feedback signal for caption triage.
    Equation (1) and Algorithm 1; without GT boxes the method cannot run.
  • ad hoc to paper Composite caption (scene description + class quantity) is a sufficiently rich initialization for subsequent LLM refinement.
    Phase I design choice; ablation Table 4 shows other initializations also improve but CC is preferred as backbone-agnostic.
invented entities (2)
  • Composite Caption (CC)
    purpose: Per-image concatenation of scene description and class-quantity captions used as T0.
    Algorithmic construct; no independent physical existence.
  • Three-bucket triage (B1 regenerate / B2 tentative / B3 keep)
    purpose: Maps minority AP relative to dynamic thresholds into accept / continue / discard decisions with early stopping.
    Algorithmic control structure introduced in Phase II.

how reviews work

0 comments
Cite this review

Pith. "Pith review of C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes." pith.science (2026). https://pith.science/paper/LW52AXUJ

@misc{pith2026260709008,
  author       = {Pith},
  title        = {Pith review of: C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LW52AXUJ}},
  note         = {Machine review of arXiv:2607.09008}
}
read the original abstract

Safety-critical perception systems must reliably detect rare object classes within small label spaces, a setting that long-tailed detection methods, designed for hundreds of classes with dense annotation, fundamentally do not address. Open-vocabulary detectors offer a promising alternative, as they use natural language queries at inference time, making prompt quality a first-class lever for detection performance. We exploit this property to address class imbalance: rather than retraining models or collecting additional annotations, we ask whether iteratively refining the language prompts, fed to frozen detectors, can improve minority class detection. We introduce C-GAP Caption-Guided Augmentation and Prompting), a detector-agnostic, annotation-free framework that operates in two phases. First, we establish a composite caption baseline combining per-image scene descriptions with class-quantity context, which we show outperforms scene-description only or class-quantity-only prompts across multiple open-vocabulary architectures and benchmarks. Second, an LLM iteratively refines each image's caption individually, with trials triaged into accept, tentative, or regenerate buckets based on minority-class AP@0.5 against a dynamic threshold derived from the composite baseline. Refinement terminates early once sufficient AP@0.5 gain is achieved. No detector weights are updated at any stage. Our experiments shows that C-GAP improves minority-class average precision up to 53% over the baselines. On COCO, C-GAP improves minority-class AP@0.5 by ~81% relative over the composite baseline (17.69 -> 32.09). Experiments confirm that composite captions provide the critical foundation for effective refinement: using scene-description-only or class-quantity-only prompts as the refinement starting point yields diminishing returns, supporting both stages of C-GAP as necessary contributions.

Figures

Figures reproduced from arXiv: 2607.09008 by the authors.

Figure 1
Figure 1. Chula Vista intersection: cars dominate ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of C-GAP. Phase I (top): per-image Scene Description (t SD i ) and Class Quantity (t CQ i ) captions are generated offline and concatenated into a Composite Caption ti,0 = concat(t SD i , tCQ i ), forming the initial caption set T0. Phase II (bottom): a VLM refines T0 over K trials. Each candidate set Tk is evaluated by the frozen detector fθ, yielding minority-class AP50(cm; Tk). Trials are triaged into B1… view at source ↗
Figure 3
Figure 3. Paired CC→C-GAP minority AP@0.5 per backbone and dataset. Each line is one frozen detector; upward slopes indicate minority-class gains from caption refinement alone [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Minority AP@0.5 gain vs. overall mAP@0.5 change [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison of CC baseline (top) and C-GAP (bottom). C-GAP recovers minority-class detections missed by [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: T0 initialization: CC baseline and C-GAP best mi￾nority AP@0.5 for SD, CQ, and CC starting captions (Ground￾ing DINO, COCO/bus, llava-phi3, K = 15, mean/3 seeds) [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Class-instance distribution for the three evaluation datasets. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: C-GAP minority AP@0.5 improvement (∆ pp) over the CC baseline across all backbone–dataset pairs (llava-phi3, K=15). Green cells indicate positive gains. The single zero cell (OWLv2 and YOLO-World on Chula Vista) indicates C￾GAP equals CC by construction. All improvemen…
Figure 10
Figure 10. Figure 10: LLaVA Phase II bucket distribution per backbone– [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Aggregate bucket distribution by Phase II genera [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Refined caption length vs. minority AP@0.5 across [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 9 linked inside Pith

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhi- tao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Ne- matzadeh, Sahand Sharifzadeh, Mikołaj ...

    2022

  2. [2]

    Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. In arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  3. [3]

    Khan, and Fahad Shahbaz Khan

    Monika Bangalath, Muhammad Maaz, Muham- mad Uzair Khattak, Salman H. Khan, and Fahad Shahbaz Khan. Bridging the gap between object and image-level representations for open-vocabulary detection. In Advances in Neural Information Processing Systems (NeurIPS), pages 33781–33794, 2022

    2022

  4. [4]

    YOLOv4: Optimal speed and accuracy of object detection

    Alexey Bochkovskiy, Chien-Yao Wang, and Hong- Yuan Mark Liao. YOLOv4: Optimal speed and accuracy of object detection. InarXiv preprint arXiv:2004.10934, 2020

    arXiv 2004

  5. [5]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pages 9630–9640, 2021

    2021

  6. [6]

    Lawrence Zitnick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. InarXiv preprint arXiv:1504.00325, 2015

    arXiv 2015

  7. [7]

    PaLI: A jointly-scaled multilingual language- image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergio- vanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Chris- tian Szegedy, David Wipf, Mario Lucic, and Xiaohua Zhai. PaLI: A jointly-scaled multilingual lan...

    2023

  8. [8]

    YOLO-World: Real- time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. YOLO-World: Real- time open-vocabulary object detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16901–16911, 2024

    2024

Show all 67 references
  1. [9]

    Open-vocabulary object detection using pseudo caption labels.arXiv preprint arXiv:2303.13040, 2023

    Han-Cheol Cho, Won Young Jhoo, Wooyoung Kang, and Byungseok Roh. Open-vocabulary object detection using pseudo caption labels.arXiv preprint arXiv:2303.13040, 2023

    2023 arXiv

  2. [10]

    The cityscapes dataset for semantic urban scene understand- ing

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. InProceedings of the IEEE Conference on Computer Vision and Pattern Rec...

    2016

  3. [11]

    Class-balanced loss based on effective number of samples

    Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9268–9277, 2019

    2019

  4. [12]

    BERT: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics: Human...

    2019

  5. [13]

    Learning to prompt for open- vocabulary object detection with vision-language model

    Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 14084–14093, 2022

    2022

  6. [14]

    Cut, paste and learn: Surprisingly easy synthesis for instance detection

    Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance detection. InProceedings of the IEEE Inter- national Conference on Computer Vision (ICCV), pages 1310–1319, 2017

    2017

  7. [15]

    PromptDet: Towards open-vocabulary detection using uncurated images

    Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. PromptDet: Towards open-vocabulary detection using uncurated images. InProceedings of the European Con- ference on Computer Vision (ECCV), pages 701–717, 2022

    2022

  8. [16]

    Simple copy-paste is a strong data augmentation method for instance segmentation

    Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin Dogus Cubuk, Quoc Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (...

    2021

  9. [17]

    Open-vocabulary object detection via vision and lan- 9 guage knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and lan- 9 guage knowledge distillation. InInternational Confer- ence on Learning Representations (ICLR), 2022

    2022

  10. [18]

    LVIS: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5356– 5364, 2019

    2019

  11. [19]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, 2022

    2022

  12. [20]

    Disen- tangling label distribution for long-tailed visual recog- nition

    Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disen- tangling label distribution for long-tailed visual recog- nition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6626–6636, 2021

    2021

  13. [21]

    Unsupervised prompt learning for vision-language models

    Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. InarXiv preprint arXiv:2204.03649, 2022

    2022 arXiv

  14. [22]

    MDETR – modulated detection for end-to-end multi-modal under- standing

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR – modulated detection for end-to-end multi-modal under- standing. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1780– 1790, 2021

    2021

  15. [23]

    MaPLe: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muham- mad Maaz, Salman Khan, and Fahad Shahbaz Khan. MaPLe: Multi-modal prompt learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19113–19122, 2023

    2023

  16. [24]

    Grounded language-image pre- training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre- training. InProceedings of the IEEE/CVF Conference on Computer Vision and P...

    2022

  17. [25]

    A comprehensive survey on test-time adaptation under distribution shifts

    Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303.15361, 2023

    2023 arXiv

  18. [26]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO: Common ob- jects in context. InEuropean Conference on Computer Vision (ECCV), pages 740–755, 2014

    2014

  19. [27]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll ´ar. Focal loss for dense object detection. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2980–2988, 2017

    2017

  20. [28]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural In- formation Processing Systems (NeurIPS), pages 34892– 34916, 2023

    2023

  21. [29]

    Grounding DINO: Marrying DINO with grounded pre-training for open-set object de- tection.arXiv preprint arXiv:2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object de- tection.arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  22. [30]

    Long-tail learning via logit ad- justment

    Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit ad- justment. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  23. [31]

    Simple open- vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open- vocabulary object detection. InProce...

    2022

  24. [32]

    Scaling open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  25. [33]

    Imbalance problems in object detection: A re- view.IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, pages 1–1, 2020

    Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance problems in object detection: A re- view.IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, pages 1–1, 2020

    2020

  26. [34]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. Technical report, Ope- nAI, 2023

    2023

  27. [35]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceed- ings of t...

    2021

  28. [36]

    Aligning and prompting everything all at once for universal visual perception

    Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for universal visual perception. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recogniti...

    2024

  29. [37]

    Test-time prompt tuning for zero-shot generalization in vision-language models

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. InAdvances in Neural Informa- tion Processing Systems (NeurIPS), pages 14274–14289, 2022

    2022

  30. [38]

    Equal- ization loss for long-tailed object recognition

    Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equal- ization loss for long-tailed object recognition. InPro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 11662– 11671, 2020

    2020

  31. [39]

    Equalization loss v2: A new gradient bal- ance approach for long-tailed object detection

    Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li. Equalization loss v2: A new gradient bal- ance approach for long-tailed object detection. InPro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 1685–1694, 2021

    2021

  32. [40]

    Moondream: A small vision language model

    vikhyatk. Moondream: A small vision language model. https://github.com/vikhyat/moondream, 2024. 10

    2024

  33. [41]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. InInternational Conference on Learning Representations (ICLR), 2021

    2021

  34. [42]

    Seesaw loss for long-tailed instance segmentation

    Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed instance segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page...

    2021

  35. [43]

    GroupViT: Semantic segmentation emerges from text su- pervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. GroupViT: Semantic segmentation emerges from text su- pervision. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18134–18144, 2022

    2022

  36. [44]

    End-to-end semi-supervised object detection with soft teacher

    Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Li- juan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end semi-supervised object detection with soft teacher. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3060– 3069, 2021

    2021

  37. [45]

    Improving pseudo labels for open-vocabulary object detection

    Shiyu Yang, Zhifeng Wu, Lianming Shi, Xin Wei, Yi Xu, and Bingbing Liu. Improving pseudo labels for open-vocabulary object detection. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6514–6524, 2023

    2023

  38. [46]

    UniTab: Uni- fying tabular learning at scale

    Zhen Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yu- mao Lu, Zicheng Liu, and Lijuan Wang. UniTab: Uni- fying tabular learning at scale. InProceedings of the Eu- ropean Conference on Computer Vision (ECCV), pages 272–290, 2022

    2022

  39. [47]

    Visual- language prompt tuning with knowledge-guided context optimization

    Hantao Yao, Rui Zhang, and Changsheng Xu. Visual- language prompt tuning with knowledge-guided context optimization. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 6757–6767, 2023

    2023

  40. [48]

    CutMix: Regularization strategy to train strong classi- fiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classi- fiers with localizable features. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6023–6032, 2019

    2019

  41. [49]

    Open-vocabulary DETR with condi- tional matching

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary DETR with condi- tional matching. InProceedings of the European Confer- ence on Computer Vision (ECCV), pages 106–122, 2022

    2022

  42. [50]

    Unified vision and language prompt learning

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. InarXiv preprint arXiv:2210.07225, 2022

    2022 arXiv

  43. [51]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14393–14402, 2021

    2021

  44. [52]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Ciss ´e, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk mini- mization. InInternational Conference on Learning Rep- resentations (ICLR), 2018

    2018

  45. [53]

    GLIPv2: Unifying localization and vision-language un- derstanding

    Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen- Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. GLIPv2: Unifying localization and vision-language un- derstanding. InAdvances in Neural Information Process- ing Systems (NeurIPS...

    2022

  46. [54]

    Real-time transformer-based open- vocabulary detection with efficient fusion head.arXiv preprint arXiv:2403.06892, 2024

    Tiancheng Zhao, Peng Liu, Xuan He, Lu Zhang, and Kyusong Lee. Real-time transformer-based open- vocabulary detection with efficient fusion head.arXiv preprint arXiv:2403.06892, 2024

    2024 arXiv

  47. [55]

    Re- gionCLIP: Region-based language-image pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chun- yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Re- gionCLIP: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2022

  48. [56]

    Conditional prompt learning for vision- language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 16816–16825, 2022

    2022

  49. [57]

    Learning to prompt for vision-language mod- els.International Journal of Computer Vision, 130: 2337–2348, 2022

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language mod- els.International Journal of Computer Vision, 130: 2337–2348, 2022

    2022

  50. [58]

    Detecting twenty- thousand classes using image-level supervision

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr¨ahenb¨uhl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. InPro- ceedings of the European Conference on Computer Vi- sion (ECCV), pages 350–368, 2022

    2022

  51. [59]

    MiniGPT-4: Enhancing vision- language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision- language understanding with advanced large language models. InInternational Conference on Learning Rep- resentations (ICLR), 2024. 11 Supplementary Material This supplementary pro...

    2024

  52. [60]

    Data augmentation remedies include copy-paste [16], Cut- Paste [14], MixUp [52], CutMix [48], and mosaic tiling [4, 8]

    Extended Related Work Class Imbalance in Object Detection.Class imbal- ance in detection has been comprehensively surveyed by [33], who taxonomize remedies into foreground– background imbalance (focal loss [27], repeat factor sampling [18]) and foreground–foreground imbalance ...

  53. [61]

    Per-class instance counts in each validation split

    Dataset Statistics Table 8. Per-class instance counts in each validation split. Class COCO (2948 img) Cityscapes (500 img) Chula Vista (288 img) Person / Pedestrian 10,777 1,080 472Bicycle / Bike 1,294 376 170Car 4,504 3,478 1,975Bus 283 152 226Truck 615 204 303Van — — 316 Tab...

  54. [62]

    Detector Inference Settings All detector weights are frozen throughout and no fine- tuning is performed at any stage

    Implementation Details 10.1. Detector Inference Settings All detector weights are frozen throughout and no fine- tuning is performed at any stage. Each backbone uses its published default thresholds, which were not tuned on our datasets. All models run with batch size 8. 12 Fi...

  55. [63]

    Minority-Class-Only Working Set The stratified working set used in the main experiments guarantees minority-class images are present but mirrors the natural class distribution of the full validation set, 13 meaning roughly 95% of the 100 sampled images con- tain no bus ground ...

  56. [64]

    We additionally ran CGAP with LLaV A (7B) as an ex- ploratory Phase II refiner for all completed backbone– dataset configurations

    LLaV A as Phase II Generator The main paper (Table 5) compares llava-phi3 and moondream as the two primary Phase II generators. We additionally ran CGAP with LLaV A (7B) as an ex- ploratory Phase II refiner for all completed backbone– dataset configurations. Phase I captions a...

  57. [65]

    moondream (K=15,T 0=CC, strati- fied, three runs)

    Full Generator Comparison: Bucket Statistics Table 13 extends Table 5 of the main paper with the complete per-backbone, per-dataset bucket breakdown for llava-phi3 vs. moondream (K=15,T 0=CC, strati- fied, three runs). The table covers the three backbones for which both genera...

  58. [66]

    Detection Quality Figure 12 plots refined caption length (words) against minority-class AP@0.5 across all 1,104 CGAP trials, colored by bucket assignment

    Caption Length vs. Detection Quality Figure 12 plots refined caption length (words) against minority-class AP@0.5 across all 1,104 CGAP trials, colored by bucket assignment. The Pearson correlation isr=0.07, indicating negligible linear dependence be- tween caption length and ...

  59. [67]

    The primary metric is AP@0.5 for the designated mi- nority class (bus/COCO, truck/Cityscapes, Bike/Chula Vista)

    Evaluation Protocol All detection outputs are stored in COCO-format JSON. The primary metric is AP@0.5 for the designated mi- nority class (bus/COCO, truck/Cityscapes, Bike/Chula Vista). Secondary metrics are per-class AP@0.5 for all classes and overall mAP@0.5 across the full...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.