Pith. sign in

REVIEW 2 major objections 5 minor 71 references

Few-shot object detection with multi-modal LLMs works better when visual examples optimize a text-only class prompt than when those same images are shown in context.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 19:36 UTC pith:R5SI7GGL

load-bearing objection Solid black-box recipe that turns few-shot FP/FN feedback into better text prompts; gains are real and multi-model, white-box still wins. the 2 major comments →

arxiv 2603.23455 v2 pith:R5SI7GGL submitted 2026-03-24 cs.CV

DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection

Gautam Rajendrakumar Gare , Neehar Peri , Matvei Popov , Shruti Jain , John Galeotti , Deva Ramanan This is my paper
classification cs.CV
keywords few-shot object detectionin-context learningprompt optimizationmulti-modal LLMsvision-language modelsvisual groundingconfidence calibrationblack-box adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multi-modal large language models detect everyday objects well, but they still fail on rare classes, specialized tasks, and imaging modalities that never appeared in pre-training. The usual remedy—showing a few labeled images as in-context examples—often lowers detection accuracy relative to simply naming the class and giving a short instruction. This paper argues that the few-shot images are still useful: not as live visual context at test time, but as a training signal for rewriting a text-only class description. The proposed method, Detection Prompt Optimization, runs the frozen model on the few-shot set, reads its true positives, false positives, and false negatives, and iteratively revises the prompt so it excludes the mistakes and includes the misses, then calibrates per-box confidence. On multi-domain few-shot benchmarks the resulting prompts raise mean average precision by large margins, can outrun specialist open-vocabulary detectors, and work without any weight updates—so they apply even to closed APIs.

Core claim

Current multi-modal LLMs cannot yet exploit multi-modal in-context examples for object detection, but the same few-shot visual examples can be distilled into a better text-only class definition by iterative contrastive prompt refinement driven by the model’s own detection errors; that black-box procedure, plus confidence calibration, yields consistent large gains in detection accuracy across generalist models.

What carries the argument

Detection Prompt Optimization (DetPO): a per-class, gradient-free loop that bootstraps a class description from positive and contrastive negative ground-truth boxes, then repeatedly rewrites that description to exclude the worst false positives and include the worst false negatives on the few-shot train set, with optional yes/no visual-question re-scoring of boxes.

Load-bearing premise

The method assumes that fixing the model’s mistakes on a handful of training images by rewriting the class text will still help on new test images rather than only fitting training quirks.

What would settle it

Run DetPO on the ten-shot training split of a held-out multi-domain few-shot detection set and measure test mAP for a strong generalist model: if the optimized text prompt does not beat the class-name-plus-instruction baseline, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Users with only API access to frontier multi-modal models can adapt them to novel classes and modalities without fine-tuning weights.
  • Text-only prompts refined from few-shot errors can transfer across model sizes and families as a practical stand-in for multi-modal in-context examples.
  • Specialist open-vocabulary detectors that cannot usefully consume few-shot images remain limited relative to well-prompted generalists on out-of-distribution domains.
  • Self-reported box confidences and visual-question re-scoring are needed companions to prompt refinement for ranking-based metrics such as mean average precision.
  • Detection-specific error-driven prompt search outperforms general-purpose black-box prompt optimizers on this task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • As next-generation closed models improve spatial reasoning, the same error-driven text rewrite may close more of the remaining gap to white-box fine-tuning of specialist detectors.
  • The finding that live multi-modal in-context examples hurt detection suggests post-training locked models into rigid prompt formats; alignment that rewards visual few-shot box reasoning could reduce the need for offline prompt search.
  • Error-driven contrastive rewriting of class text may transfer to other dense prediction tasks—segmentation, keypoints—where false positives and misses are similarly informative.
  • Per-class sequential optimization will need parallelization before the method scales cleanly to benchmarks with hundreds of classes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that current multi-modal LLMs (MLLMs) do not reliably benefit from multi-modal in-context examples for object detection on out-of-distribution domains, and that black-box, text-only prompt optimization is a practical alternative when fine-tuning is unavailable or expensive. It introduces Detection Prompt Optimization (DetPO): per-class iterative refinement of natural-language class definitions using the MLLM’s own true-positive, false-positive, and false-negative detections on a few-shot training split (contrastive prompt refinement), plus optional confidence calibration via self-reported scores or VQA Score. On Roboflow20-VL and an LVIS Rare-50 10-shot subset, DetPO improves Qwen and Gemini families over class-name/instruction baselines and over GEPA/MIPROv2, with best reported numbers of 21.6 mAP (Qwen3-VL 30B-A3B + VQA) and 26.3–26.4 mAP (Gemini 3 Pro + DetPO), exceeding zero-shot specialists such as LLMDet and GroundingDINO while remaining below white-box fine-tuned GroundingDINO.

Significance. If the gains hold under broader evaluation, DetPO is a useful, reproducible recipe for adapting closed-source and large open-weight MLLMs to few-shot detection without gradients. Strengths include multi-model transfer (Table 3), ablations on confidence scoring and shot count (Tables 4, 7), TIDE and confusion-matrix diagnostics (Figs. 4–5, 7), comparison to GEPA/MIPROv2 and specialists (Tables 2, 5–6), and public code plus optimized prompts. The work is empirical rather than theoretical, but it cleanly frames a practical black-box setting and documents that naive multi-modal ICL can hurt detection (Table 1). White-box fine-tuning still leads (Table 5), which the authors acknowledge; that does not erase the black-box contribution.

major comments (2)
  1. Section 3 / Algorithm 1 (Stage 2–3): The central claim that train-set TP/FP/FN-driven rewrites generalize rests on early stopping and best-of-candidates selection on a held-out validation split. The manuscript should state, for RF20-VL and LVIS Rare-50, how the few-shot pool is partitioned into optimization vs. validation, whether validation images are disjoint from the test set, and report sensitivity of final mAP to that split (or multi-seed re-runs). Without this, the 9.7-point gains (Table 2) remain hard to separate from mild overfitting to the small support set.
  2. Tables 2–3 and Appendix F: DetPO is optimized class-wise and incurs non-trivial API/token cost; GEPA/MIPROv2 are multi-class. The paper should report wall-clock and token cost for a full RF20-VL run (all classes), not only aerial-airport, and clarify whether optimized prompts transfer across MLLMs without re-optimization. Cost and transferability are load-bearing for the “practical alternative to fine-tuning” claim.
minor comments (5)
  1. Abstract vs. Table 2: abstract says “up to 9.7 mAP” and “26.4”; body uses 26.3/26.4 and mixes “%” with mAP points—standardize units and the headline number.
  2. Table 1 and several figure captions use placeholder/garbled tokens (e.g., black squares, “������”); clean for camera-ready.
  3. Appendix C Algorithm 1 and Eqs. (1)–(11) are useful but notation is dense; a short worked example of one FP/FN rewrite would help reproducibility.
  4. Gemini VQA re-scoring uses Qwen3-VL (30B-A3B) because token probs are unavailable—state this limitation more prominently when claiming Gemini + DetPO SOTA.
  5. Related work on discrete prompt search is adequate; a brief note on why soft prompt tuning is out of scope for API models would tighten Section 2.

Circularity Check

0 steps flagged

No circularity: DetPO is empirical black-box prompt search scored on held-out few-shot data, not a derivation that reduces predictions to fitted inputs by construction.

full rationale

The paper's central claim is an empirical performance gain from gradient-free iterative refinement of text-only class descriptions, driven by the target MLLM's own TP/FP/FN errors on a few-shot training split (Section 3; Algorithm 1 / Appendix C Stages 1–2), with final candidate selection on a held-out validation split (Stage 3) and reported mAP on the test split of RF20-VL and LVIS Rare-50. There are no equations equating a claimed prediction to a fitted constant; self-reported confidences and optional VQA Score re-ranking are post-hoc calibrations, not circular derivations. Citations to overlapping-author work (e.g., Roboflow100-VL [45]) define the evaluation benchmark and are not load-bearing uniqueness theorems or smuggled ansatzes. The method is self-contained against external baselines (specialist detectors, GEPA, MIPROv2) and multi-model transfer tables; standard train/val/test prompt search does not constitute circularity under the stated criteria.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The work is an empirical systems paper. Load-bearing choices are the few-shot training split used both for optimization and early stopping, the hand-chosen iteration budget and error-ranking heuristics, and the assumption that text-only prompts can encode the visual distinctions the model fails to use in multi-modal ICL. No new physical entities or free constants are fitted to produce a theoretical prediction.

free parameters (3)
  • max refinement iterations T_max and early-stopping rule = typically ~6–10 (paper shows plateau)
    Controls how long the contrastive loop runs; performance plateaus around iteration 6 in Fig. 3 but is not derived.
  • number of worst FPs/FNs sampled per iteration (top-k)
    Selects which errors drive the next rewrite; chosen by the authors.
  • shot count used for optimization (3/5/10) = 10-shot (main)
    Ablated in Table 7; main results use the benchmark's 10-shot split.
axioms (4)
  • domain assumption Current MLLMs cannot effectively exploit multi-modal in-context visual examples for detection under standard post-training prompt formats.
    Stated from Table 1 and used to motivate text-only optimization (Introduction, Section 1).
  • domain assumption Detection accuracy (mAP / F1) on the few-shot training images is a valid surrogate for selecting prompts that will generalize to the test distribution.
    Core of Stage 2 and Algorithm 1; validation selection is the only guard against overfitting.
  • ad hoc to paper An MLLM can act as a reliable prompt critic that rewrites class definitions to exclude a shown false positive or include a shown false negative when given boxed images.
    Assumed throughout the contrastive refinement loop (Section 3, Fig. 2).
  • domain assumption Self-reported per-box confidences or VQA-score likelihoods of the 'Yes' token are sufficiently calibrated for ranking detections.
    Used for the optional re-ranking step and for Gemini post-processing with Qwen (Section 3, Table 4).
invented entities (1)
  • Detection Prompt Optimization (DetPO) procedure no independent evidence
    purpose: Gradient-free, class-wise iterative refinement of natural-language class descriptions from ranked detection errors plus confidence calibration.
    The named method is the paper's central contribution; it is defined operationally in Algorithm 1 and has no independent existence outside the experiments.

pith-pipeline@v1.1.0-grok45 · 29442 in / 3047 out tokens · 29528 ms · 2026-07-13T19:36:48.350971+00:00 · methodology

0 comments
read the original abstract

Multi-Modal LLMs (MLLMs) demonstrate strong visual grounding capabilities on popular object detection benchmarks like OdinW-13 and RefCOCO. However, state-of-the-art models still struggle to generalize to out-of-distribution classes, tasks and imaging modalities not typically found in their pre-training. While in-context prompting is a common strategy to improve performance across diverse tasks, we find that it often yields lower detection accuracy than prompting with class names alone. This suggests that current MLLMs cannot yet effectively leverage few-shot visual examples and rich textual descriptions for object detection. Since frontier MLLMs are typically only accessible via APIs, and state-of-the-art open-weights models are prohibitively expensive to fine-tune on consumer-grade hardware, we instead explore black-box prompt optimization for few-shot object detection. To this end, we propose Detection Prompt Optimization (DetPO), a gradient-free test-time optimization approach that refines text-only prompts by maximizing detection accuracy on few-shot visual training examples while calibrating prediction confidence. Our proposed approach yields consistent improvements across generalist MLLMs on Roboflow20-VL and LVIS, outperforming prior black-box approaches by up to 9.7 mAP. Our code and optimized prompts are available at https://ggare-cmu.github.io/DetPO/

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 21 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023) 1

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) 1

    Pith/arXiv arXiv 2023

  2. [2]

    arXiv preprint arXiv:2507.19457 (2025) 5, 7, 8, 21, 31

    Agrawal, L.A., Tan, S., Soylu, D., Ziems, N., Khare, R., Opsahl-Ong, K., Singhvi, A., Shandilya, H., Ryan, M.J., Jiang, M., et al.: Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457 (2025) 5, 7, 8, 21, 31

    Pith/arXiv arXiv 2025

  3. [3]

    Advances in neural information processing systems35, 23716– 23736 (2022) 5

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems35, 23716– 23736 (2022) 5

    2022

  4. [4]

    arXiv preprint arXiv:2511.21631 (2025) 1, 3, 4, 7, 8, 9, 12, 24, 31, 34

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, ...

    Pith/arXiv arXiv 2025

  5. [5]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) 1, 3, 4, 7, 9

    Pith/arXiv arXiv 2025

  6. [6]

    Transactions of the Association for Computational Linguistics10, 414–433 (2022) 5

    Ben-David, E., Oved, N., Reichart, R.: Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains. Transactions of the Association for Computational Linguistics10, 414–433 (2022) 5

    2022

  7. [7]

    In: European Conference on Computer Vision

    Bolya, D., Foley, S., Hays, J., Hoffman, J.: Tide: A general toolbox for identifying object detection errors. In: European Conference on Computer Vision. pp. 558–573. Springer (2020) 12, 13, 32

    2020

  8. [8]

    Advances in neural information processing systems33, 1877–1901 (2020) 2, 5

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020) 2, 5

    1901

  9. [9]

    arXiv preprint arXiv:2511.16719 (2025) 7, 8, 31

    Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Alwala, K.V., Khedr, H., Huang, A., et al.: Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719 (2025) 7, 8, 31

    Pith/arXiv arXiv 2025

  10. [10]

    arXiv preprint arXiv:1906.07155 (2019) 12

    Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., Zhang, Z., Cheng, D., Zhu, C., Cheng, T., Zhao, Q., Li, B., Lu, X., Zhu, R., Wu, Y., Dai, J., Wang, J., Shi, J., Ouyang, W., Loy, C.C., Lin, D.: MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019) 12

    Pith/arXiv arXiv 1906

  11. [11]

    arXiv preprint arXiv:2507.06261 (2025) 1

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025) 1

    Pith/arXiv arXiv 2025

  12. [12]

    DeepMind, G.: Introducing gemini 2.0: our new ai model for the agentic era (Dec 2024), ���������������������������������������������������������������� ���������������������4 16 Gare and Peri et. al

    2024

  13. [13]

    https://storage.googleapis.com/deepmind- media/Model-Cards/Gemini-3-Pro-Model-Card.pdf (2025) 3, 4, 7, 8, 12

    DeepMind, G.: Gemini 3 pro model card. https://storage.googleapis.com/deepmind- media/Model-Cards/Gemini-3-Pro-Model-Card.pdf (2025) 3, 4, 7, 8, 12

    2025

  14. [14]

    In: Proceedings of the 2024 conference on empirical methods in natural language processing

    Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al.: A survey on in-context learning. In: Proceedings of the 2024 conference on empirical methods in natural language processing. pp. 1107–1128 (2024) 2, 5

    2024

  15. [15]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Du, Y., Wei, F., Zhang, Z., Shi, M., Gao, Y., Li, G.: Learning to prompt for open-vocabulary object detection with vision-language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14084–14093 (2022) 4

    2022

  16. [16]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Fan, Q., Zhuo, W., Tang, C.K., Tai, Y.W.: Few-shot object detection with attention- rpn and multi-relation detector. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4013–4022 (2020) 4

    2020

  17. [17]

    arXiv preprint arXiv:1909.00505 (2019) 5

    Feldman, J., Davison, J., Rush, A.M.: Commonsense knowledge mining from pretrained models. arXiv preprint arXiv:1909.00505 (2019) 5

    Pith/arXiv arXiv 1909

  18. [18]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Fu, S., Yang, Q., Mo, Q., Yan, J., Wei, X., Meng, J., Xie, X., Zheng, W.S.: Llmdet: Learning strong open-vocabulary object detectors under the supervision of large language models. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 14987–14997 (2025) 7, 8, 31

    2025

  19. [20]

    arXiv preprint arXiv:2104.13921 (2021) 4

    Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 (2021) 4

    Pith/arXiv arXiv 2021

  20. [21]

    Jiang, Z., Xu, F.F., Araki, J., Neubig, G.: How can we know what language models know? Transactions of the Association for Computational Linguistics8, 423–438 (2020) 5

    2020

  21. [22]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kang, B., Liu, Z., Wang, X., Yu, F., Feng, J., Darrell, T.: Few-shot object detection via feature reweighting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8420–8429 (2019) 4

    2019

  22. [23]

    arXiv preprint arXiv:2310.03714 (2023) 7, 20

    Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T.T., Moazam, H., et al.: Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714 (2023) 7, 20

    Pith/arXiv arXiv 2023

  23. [24]

    arXiv preprint arXiv:2406.10115 (2024) 4

    Khurana, M., Peri, N., Ramanan, D., Hays, J.: Shelf-supervised multi-modal pre- training for 3d object detection. arXiv preprint arXiv:2406.10115 (2024) 4

    Pith/arXiv arXiv 2024

  24. [25]

    arXiv preprint arXiv:2112.11699 (2021) 4

    Köhler, M., Eisenbach, M., Gross, H.M.: Few-shot object detection: A comprehensive survey. arXiv preprint arXiv:2112.11699 (2021) 4

    Pith/arXiv arXiv 2021

  25. [26]

    arXiv preprint arXiv:2209.15639 (2022) 4

    Kuo, W., Cui, Y., Gu, X., Piergiovanni, A., Angelova, A.: F-vlm: Open- vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209.15639 (2022) 4

    Pith/arXiv arXiv 2022

  26. [27]

    In: Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (2023) 13

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J.E., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with pagedattention. In: Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (2023) 13

    2023

  27. [28]

    arXiv preprint arXiv:2104.08691 (2021) 5

    Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021) 5

    Pith/arXiv arXiv 2021

  28. [29]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024) 12

    Li, B., Lin, Z., Peng, W., et al.: Naturalbench: Evaluating vision-language models on natural adversarial samples. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024) 12

    2024

  29. [30]

    Neural Information Processing Systems (2022) 1, 4 In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection 17

    Li, C., Liu, H., Li, L.H., Zhang, P., Aneja, J., Yang, J., Jin, P., Hu, H., Liu, Z., Lee, Y.J., Gao, J.: Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Neural Information Processing Systems (2022) 1, 4 In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection 17

    2022

  30. [31]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li,L.H.,Zhang,P.,Zhang,H.,Yang,J.,Li,C.,Zhong,Y.,Wang,L.,Yuan,L.,Zhang, L., Hwang, J.N., et al.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10965–10975 (2022) 4

    2022

  31. [32]

    In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13. pp. 740–755. Springer (2014) 7

    2014

  32. [33]

    In: European Conference on Computer Vision

    Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., Ramanan, D.: Evaluating text-to-visual generation with image-to-text generation. In: European Conference on Computer Vision. pp. 366–384. Springer (2024) 3, 7, 8, 9, 34

    2024

  33. [34]

    In: European conference on computer vision

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In: European conference on computer vision. pp. 38–55. Springer (2024) 1, 7, 8, 12, 31

    2024

  34. [35]

    arXiv preprint arXiv:2303.05499 (2023) 4, 20, 31

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023) 4, 20, 31

    Pith/arXiv arXiv 2023

  35. [36]

    AI Open5, 208–215 (2024) 5

    Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., Tang, J.: Gpt understands, too. AI Open5, 208–215 (2024) 5

    2024

  36. [37]

    arXiv preprint arXiv:2312.10986 (2023) 4

    Ma, Y., Peri, N., Wei, S., Hua, W., Ramanan, D., Li, Y., Kong, S.: Long-tailed 3d detection via 2d late fusion. arXiv preprint arXiv:2312.10986 (2023) 4

    arXiv 2023

  37. [38]

    Advances in Neural Information Processing Systems 37, 19547–19560 (2024) 2

    Madan, A., Peri, N., Kong, S., Ramanan, D.: Revisiting few-shot object detection with vision-language models. Advances in Neural Information Processing Systems 37, 19547–19560 (2024) 2

    2024

  38. [39]

    arXiv preprint arXiv:2306.09683 (2023) 4

    Minderer, M., Gritsenko, A., Houlsby, N.: Scaling open-vocabulary object detection. arXiv preprint arXiv:2306.09683 (2023) 4

    Pith/arXiv arXiv 2023

  39. [40]

    In: European Conference on Computer Vision

    Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Doso- vitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al.: Simple open-vocabulary object detection. In: European Conference on Computer Vision. pp. 728–755. Springer (2022) 4

    2022

  40. [41]

    arXiv preprint arXiv:2406.11695 (2024) 5, 7, 8, 21, 31

    Opsahl-Ong, K., Ryan, M.J., Purtell, J., Broman, D., Potts, C., Zaharia, M., Khattab, O.: Optimizing instructions and demonstrations for multi-stage language model programs. arXiv preprint arXiv:2406.11695 (2024) 5, 7, 8, 21, 31

    Pith/arXiv arXiv 2024

  41. [42]

    In: ECCV (2024) 4

    Osep, A., Meinhardt, T., Ferroni, F., Peri, N., Ramanan, D., Leal-Taixe, L.: Better call sal: Towards learning to segment anything in lidar. In: ECCV (2024) 4

    2024

  42. [43]

    Peri, N., Dave, A., Ramanan, D., Kong, S.: Towards long-tailed 3d detection (2023) 4

    2023

  43. [44]

    In: Conference on Robot Learning

    Peri, N., Dave, A., Ramanan, D., Kong, S.: Towards long-tailed 3d detection. In: Conference on Robot Learning. pp. 1904–1915. PMLR (2023) 9

    1904

  44. [45]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2025) 2, 7, 12

    Popov, M., Robicheaux, P., Madan, A., Robinson, I., Nelson, J., Ramanan, D., Peri, N.: Roboflow100-vl: A multi-domain object detection benchmark for vision-language models. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2025) 2, 7, 12

    2025

  45. [46]

    arXiv preprint arXiv:2104.06599 (2021) 5

    Qin, G., Eisner, J.: Learning how to ask: Querying lms with mixtures of soft prompts. arXiv preprint arXiv:2104.06599 (2021) 5

    Pith/arXiv arXiv 2021

  46. [47]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 4 18 Gare and Peri et. al

    2021

  47. [48]

    arXiv preprint arXiv:2502.16923 (2025) 5

    Ramnath, K., Zhou, K., Guan, S., Mishra, S.S., Qi, X., Shen, Z., Wang, S., Woo, S., Jeoung, S., Wang, Y., et al.: A systematic survey of automatic prompt optimization techniques. arXiv preprint arXiv:2502.16923 (2025) 5

    arXiv 2025

  48. [49]

    In: Extended abstracts of the 2021 CHI conference on human factors in computing systems

    Reynolds, L., McDonell, K.: Prompt programming for large language models: Beyond the few-shot paradigm. In: Extended abstracts of the 2021 CHI conference on human factors in computing systems. pp. 1–7 (2021) 5

    2021

  49. [50]

    International journal of computer vision115, 211–252 (2015) 4

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recog- nition challenge. International journal of computer vision115, 211–252 (2015) 4

    2015

  50. [51]

    arXiv preprint arXiv:2110.08207 (2021) 5

    Sanh, V., Webson, A., Raffel, C., Bach, S.H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T.L., Raja, A., et al.: Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207 (2021) 5

    Pith/arXiv arXiv 2021

  51. [52]

    In: Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume

    Schick, T., Schütze, H.: Exploiting cloze-questions for few-shot text classification and natural language inference. In: Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume. pp. 255–269 (2021) 5

    2021

  52. [53]

    Advances in neural information processing systems35, 25278–25294 (2022) 4

    Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems35, 25278–25294 (2022) 4

    2022

  53. [54]

    arXiv preprint arXiv:2010.15980 (2020) 5

    Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020) 5

    Pith/arXiv arXiv 2010

  54. [55]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., Wang, X.: Generative multimodal models are in-context learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14398–14409 (2024) 5

    2024

  55. [56]

    In: International Conference on Machine Learning (ICML) (2025) 4

    Takmaz, A., Saltori, C., Peri, N., Meinhardt, T., de Lutio, R., Leal-Taixe, L., Osep, A.: Towards Learning to Complete Anything in Lidar. In: International Conference on Machine Learning (ICML) (2025) 4

    2025

  56. [57]

    arXiv preprint arXiv:2502.14786 (2025) 9

    Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al.: Siglip 2: Multilingual vision-language encoders with improved semantic understanding, lo- calization, and dense features. arXiv preprint arXiv:2502.14786 (2025) 9

    Pith/arXiv arXiv 2025

  57. [58]

    Wang, A., Liu, L., Chen, H., Lin, Z., Han, J., Ding, G.: Yoloe: Real-time seeing anything (2025),��������������������������������7, 8

    2025

  58. [59]

    In: International Conference on Machine Learning (ICML) (2020) 4

    Wang, X., Huang, T.E., Darrell, T., Gonzalez, J.E., Yu, F.: Frustratingly simple few-shot object detection. In: International Conference on Machine Learning (ICML) (2020) 4

    2020

  59. [60]

    Webson, A., Pavlick, E.: Do prompt-based models really understand the meaning of their prompts? In: Proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language technologies. pp. 2300–2344 (2022) 5

    2022

  60. [61]

    Advances in neural information processing systems35, 24824–24837 (2022) 5

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, 24824–24837 (2022) 5

    2022

  61. [62]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wu, A., Han, Y., Zhu, L., Yang, Y.: Universal-prototype enhancing for few-shot object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9567–9576 (2021) 4 In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection 19

    2021

  62. [63]

    IEEE Transactions on Pattern Analysis and Machine Intelligence45(3), 3090–3106 (2022) 4

    Xiao,Y.,Lepetit,V.,Marlet,R.:Few-shotobjectdetectionandviewpointestimation for objects in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence45(3), 3090–3106 (2022) 4

    2022

  63. [64]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xu, J., Le, H., Samaras, D.: Generating features with increased crop-related diversity for few-shot object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19713–19722 (2023) 4

    2023

  64. [65]

    Advances in Neural Information Processing Systems 36, 4452–4469 (2023) 7, 8

    Xu, Y., Zhang, M., Fu, C., Chen, P., Yang, X., Li, K., Xu, C.: Multi-modal queried object detection in the wild. Advances in Neural Information Processing Systems 36, 4452–4469 (2023) 7, 8

    2023

  65. [66]

    Advances in neural information processing systems36, 11809–11822 (2023) 5

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., Narasimhan, K.: Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems36, 11809–11822 (2023) 5

    2023

  66. [67]

    In: European conference on computer vision

    Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: European conference on computer vision. pp. 69–85. Springer (2016) 1, 4

    2016

  67. [68]

    Advances in neural information processing systems34, 27263–27277 (2021) 5

    Yuan, W., Neubig, G., Liu, P.: Bartscore: Evaluating generated text as text genera- tion. Advances in neural information processing systems34, 27263–27277 (2021) 5

    2021

  68. [69]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)

    Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., Ma, Y.: Llamafactory: Unified efficient fine-tuning of 100+ language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Association for Computational Linguistics, Bangkok, Thailand (2024),�������������������������������12

    2024

  69. [70]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L.H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al.: Regionclip: Region-based language-image pretraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16793–16803 (2022) 4

    2022

  70. [71]

    a brown dog

    Zhou, X., Girdhar, R., Joulin, A., Krähenbühl, P., Misra, I.: Detecting twenty- thousand classes using image-level supervision. In: European Conference on Com- puter Vision. pp. 350–368. Springer (2022) 4 20 Gare and Peri et. al. A Baseline Implementation Details We present additional implementation details to reproduce our baseline experi- ments below. �...

    2022

  71. [72]

    white-box

    with pretrained weights from mmdetection (MM-GroundingDINO-L*). We prompt the model with all the class names combined into a single prompt. For “white-box” experiments, we fine-tune GroundingDINO on each few-shot dataset for 1000 iterations with a batch size of 4 and a learning rate of 3e-4. We resize all images to (640, 1333) and don’t use any additional...