Pith. sign in

REVIEW 3 major objections 6 minor 61 references

Teaching VLMs to Localize Specific Objects from In-context Examples

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VLMs can learn to localize a specific object from a few examples.

desk verdict Useful, well-ablated recipe for teaching VLMs in-context localization, with real gains; the main gap is that no ablation proves the model actually uses the support bounding boxes, so the 'specific object' claim is partially unverified. read the letter →

arxiv 2411.13317 v2 pith:2USS25BZ submitted 2024-11-20 cs.CV

Sivan Doveh , Nimrod Shabtay , Wei Lin , Eli Schwartz , Hilde Kuehne , Raja Giryes , Rogerio Feris , Leonid Karlinsky
show 4 more authors
James Glass Assaf Arbelle Shimon Ullman M. Jehanzeb Mirza
This is my paper · ORCID
classification cs.CV
keywords vision-languagemodelsin-contextlearningpersonalizedlocalizationfew-shotinstructiontuningvideoobjecttrackingpseudo-nameregularizationLoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether today's vision-language models can learn to find a specific object in a new image when shown just a few annotated examples of that object. It argues that current VLMs, including a proprietary model like GPT-4o, largely cannot do this, and it introduces a data-only remedy: instruction-tuning conversations mined from video object tracking, where the same object reappears across frames. The conversations present each support image with its category name and raw bounding-box coordinates, and the model is trained to output coordinates for the query. A pseudo-name regularization step replaces real category names with invented names, forcing the model to rely on the visual examples rather than on pre-trained knowledge. On three evaluation sets, the fine-tuned models raise average mIoU from 31.77 to 43.31 for the 7B model and from 34.21 to 55.79 for the 72B model while retaining general benchmark performance within about 1%.

What carries the argument

The central mechanism is the IPLoc instruction-tuning format: each user turn is an image prefixed by `<image><ref>Category</ref>` followed by raw bounding-box coordinates `((xmin,ymin),(xmax,ymax))`, and the assistant turn must output the coordinates for the same category in the query image. Because each conversation comes from frames of one tracked object, the model sees the same instance under pose, lighting, and background changes, which is what makes 'personalized' localization learnable. The pseudo-name regularization replaces real class names with arbitrary names so the model cannot lean on pre-trained object knowledge; LoRA constrains the parameter update to preserve generalization. The combination turns a standard next-token-prediction objective into a context-driven localization learner.

What would settle it

Show a trained model the same support and query conversations but replace all support bounding-box coordinates with random or permuted numbers; if localization accuracy on the query stays the same, the model never learned the coordinate-to-image mapping and the claimed mechanism is falsified. Equivalently, re-run training with support images that have visible drawn boxes instead of raw coordinates; if performance does not improve (or drops), the representation assumption is doing the work, not genuine context learning.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that few-shot personalized localization is a missing capability in current VLMs and that the gap can be closed by data alone. The central result is that fine-tuning Qwen2-VL with LoRA on dialogues built from TAO, LaSOT, and GOT video tracking sequences produces a model that can take one to eight support frames, each annotated only by text and numeric coordinates, and localize the same object instance in a query image. The pseudo-name trick is the load-bearing regularization: renaming categories to arbitrary names during training lifts the 7B model's average by 12.33 mIoU over training with real names only and the 72B model by 7.81, and a test that swaps in pseudo-names confirms the model looks at the in-context images rather than the label vocabulary. The paper also reports that the same fine-tuning extends to LLaVA-OV with about 7 mIoU improvement, and that the learned ability beats classic few-shot object detectors and an open-vocabulary detector on the ICL-LASOT benchmark.

Load-bearing premise

The whole method assumes the model actually learns to connect the raw coordinate numbers in each example to the right part of the example image, even though no box is drawn; without that link, the gains could just come from the conversation format.

Editorial extensions

If this is right

  • If the claim holds, an off-the-shelf open VLM can be turned into a personalized localizer with tracking-video data alone, no new architecture or box-drawing supervision.
  • The pseudo-name regularization implies that the model is learning to match visual identity across frames, so the same recipe should transfer to any setting where category names are unreliable or uninformative, such as novel or rare objects.
  • The 72B variant reaches 55.79 average mIoU, so larger VLMs benefit more from the data recipe, suggesting scaling laws for in-context localization similar to those in language-only in-context learning.
  • LoRA keeps general VQA and hallucination benchmarks within about 1% of the base model, so the capability can be added without a specialization trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never ablates alternative grounding formats, so a natural extension would compare raw numeric coordinates against drawn boxes or highlighted regions to see how much of the gain depends on the exact coordinate-to-image mapping.
  • Because pseudo-names make category labels uninformative, the same training recipe could be tested on object categories never seen in pre-training, or even on synthetic 'alien' objects, to measure true instance-level generalization.
  • The method is currently limited to single-object localization per query, and the paper states this explicitly; extending the dialogues to multi-object outputs would likely require a different decoding format and may be a natural next step.
  • The improvement on ICL-LASOT over a specialized open-vocabulary detector suggests that VLM in-context localization could eventually challenge dedicated detection models on tasks where the target is defined by a few examples rather than a name.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes IPLoc, a data-centric fine-tuning recipe that aims to give VLMs few-shot personalized localization ability: given a small number of in-context support images, each annotated with a category name and a bounding box, the model must localize the same instance in a query image. The authors construct instruction-tuning conversations from video object tracking datasets (TAO, LaSOT, GOT-10k) by sampling frames of the same tracked object, and they introduce pseudo-name regularization that replaces true category labels with arbitrary names to force reliance on visual context. They fine-tune Qwen2-VL 7B/72B and LLaVA-OV with LoRA and evaluate on PDM, PerSeg, and a newly constructed ICL-LASOT benchmark. The main results show large mIoU gains over the base models and over strong proprietary and open VLMs such as GPT-4o and InternVL2, while a small evaluation on GQA/SEED/POPE indicates that generalization is largely retained.

Significance. If the reported results hold, this is a valuable contribution: it identifies a concrete weakness of modern VLMs, proposes a simple data-oriented remedy with a crisp regularization idea, and benchmarks a task that is likely to become more important. The paper is transparent in several respects: it evaluates across model families and sizes, includes a generalization-retention check, performs multiple ablations (data sources, semantic coherence, pseudo names, cross-model fine-tuning), and states that code and data will be released. At the same time, the central claim of instance-specific (personalized) localization is not yet fully verified because the paper does not demonstrate that the model actually uses the support bounding-box coordinates, and the data recipe is not specified in enough detail to be reproduced from the text. The large reported gains make the approach promising, but the manuscript needs additional experiments and reporting changes before the central claim is established.

major comments (3)
  1. [Section 3.1, Table 2, Table 6a] The central claim is that IPLoc performs personalized, instance-specific localization. In the conversation format, the only signal that identifies which instance to localize is the raw bounding-box coordinate text in each support turn, since the pseudo-name is arbitrary and the support image may contain multiple objects (PDM has 4.2 objects per image on average). However, the paper provides no experiment that varies or removes the support coordinates: there is no ablation with shuffled, random, or omitted support boxes, and no comparison with drawn/highlighted boxes. The pseudo-name ablation in Table 6a only shows that the model does not rely on the category name; it does not show that the model uses the coordinates to select the correct instance. Without such evidence, the reported gains could be driven by improved category-level localization or output-format compliance rather than by true in-context instance grounding. Please add ablations that manipulate the support bounding boxes (e.g., randomize, shuffle, or drop them) and, if possible, a distractor setting where the support and query images contain multiple objects of the same category, to verify the 'specific object' component of the method.
  2. [Section 3.2, Section 3.3] The paper is explicitly data-centric, but it does not specify the composition of the fine-tuning data mix: it reports only that TAO, LaSOT, and GOT-10k are used, without giving the relative proportions of these datasets, the total number of conversations, the frame-sampling interval strategy, the distribution of shot counts (1-8), or the pseudo-name replacement rate. The LoRA hyperparameters are described only as 'the default parameters' from LlamaFactory, with no rank, alpha, learning rate, batch size, or number of epochs. Since the method's central contribution is the data recipe, these omissions prevent replication and make it impossible for readers to judge how sensitive the results are to these choices. Please provide a detailed table of the data mix, including per-dataset conversation counts, sampling parameters, pseudo-name rates, and all training hyperparameters.
  3. [Table 5 and Section 4.2] There are internal numerical inconsistencies that need to be reconciled. First, in Table 5 the 'IPLoc' column reports PDM=50.05, PerSeg=27.55, and ICL-LASOT=39.13, but none of these values appears in Table 2 for either IPLoc (7B) (Real) or IPLoc (7B) (Real+Pseudo); moreover, PDM=50.05 exceeds even the best 72B result (46.60 on PDM 2-shot in Table 2). The caption and text do not specify which model, data mix, or shot settings are used, so the reader cannot verify the semantic-coherence ablation. Second, in Section 4.2 the text states 'our fine-tuned model also outperforms the base model for the PerSeg and LASOT test sets with 12.58% average improvement.' This number does not match Table 2: the Real-only model is actually worse than the base model on ICL-LASOT on average, and the Real+Pseudo model yields an average improvement of roughly 12.1% over base across PerSeg and ICL-LASOT (not 12.58%). Please correct the text and clarify which data-mix variant is being discussed.
minor comments (6)
  1. [Section 3.1] The assistant-response format is ambiguous: the text says the assistant 'provides a category label based on the context' and 'the bounding box coordinates,' but the shown template only contains coordinates after 'Assistant:'. Please clarify whether the prediction output includes the category label, the coordinates, or both.
  2. [Table 2] The 'Average' column in Table 2 treats all ten shot-configurations equally, even though the numbers of shots differ across datasets (2 for PDM, 4 for PerSeg, 4 for ICL-LASOT). Please state this averaging convention explicitly in the caption or text.
  3. [Section 4.2] The verbal summaries of Table 2 switch between 'our best model (trained only with real category names)' and 'our fine-tuned model' without specifying which data mix is meant. Use consistent labels ('Real' vs. 'Real+Pseudo') when describing the improvements.
  4. [Appendix D.1] There is a typo, 'Suprisingly', in the first paragraph of Appendix D.1.
  5. [Section 5 (Limitations)] The limitation paragraph states that 'Qwen2-VL scores 5.3% mIOU, when evaluated for the multi-object localization task in MS-COCO,' but no source or evaluation protocol is provided; please add a citation or a brief description of this experiment.
  6. [Section 4.1] The paper says 'To encourage reproducibility, our entire codebase is provided as part of the appendix and will be released upon acceptance.' The appendix as submitted contains no code; please rephrase to state that the code will be released upon acceptance, and ideally provide an anonymous link for review.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are measured on external benchmarks or on a held-out category split, and no fitted parameter is renamed as a prediction.

full rationale

The central claim is that LoRA fine-tuning on tracking-derived conversations improves few-shot personalized localization. The supporting evidence (Table 2) compares IPLoc against base VLMs on PDM, PerSeg, and ICL-LASOT. PDM and PerSeg are external benchmarks not used in training, and ICL-LASOT uses a held-out 50% category split of LaSOT, so the training conversations and the test conversations are category-disjoint. The pseudo-name regularization is trained on pseudo-named conversations and is probed in Table 6a as an ablation, while the main results use real category names on the test sets; hence the headline numbers are not produced by test-time adaptation. Comparisons against vision-only few-shot detectors and open-vocabulary detectors (Tables 7-8) further anchor the evaluation outside the fine-tuned VLM's own training distribution. The self-citations to LLaVA-ICL [14] and MTV [21] are used to motivate semantically coherent instruction data, but the method's effectiveness is demonstrated against base models and external detectors rather than reduced to those citations. The reviewer concern that no ablation verifies the model uses the support bounding-box coordinates is a verification or correctness gap, not circularity: nothing in the data recipe, objective, or benchmark construction makes the reported mIoU equal to an input by definition. No fitted parameter is renamed as a prediction, and the derivation chain is therefore self-contained for the claims actually made.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical, so the ledger contains no invented entities; the main unstated dependencies are the exact data mixture, the pseudo-name replacement ratio, and the assumption that text coordinates provide adequate spatial grounding.

free parameters (4)
  • data mix composition (TAO/LaSOT/GOT-10k proportions) = not specified
    Section 3.2 describes mixing all three datasets but never states the relative sampling ratios; Table 3 shows the mix matters (best with all three) but the exact mixture is a hand-chosen element the results depend on.
  • pseudo-name replacement rate = not specified
    Section 3.2 says data is 'generated by replacing the true category names with pseudo category names' but does not specify what fraction of conversations uses pseudo names; the Real+Pseudo mixture in Table 2 depends on this unstated ratio.
  • shot count range = 1-8 shots per conversation
    Section 3.2 fixes conversations to 1-8 shots; the performance curves in Figure 3 vary with shot count, so this design choice affects results.
  • LoRA hyperparameters = Llama-Factory defaults
    Section 4.1 defers to default parameters in the Llama-Factory codebase; rank, alpha and dropout are not reported, making exact reproduction dependent on an external codebase version.
assumptions (3)
  • domain assumption The VLM can map plain-text bounding-box coordinates to the corresponding spatial region in a support image.
    Section 3.1 defines the conversation format with coordinates only as text; no ablation tests drawn boxes or token-based grounding, so the method implicitly assumes this mapping is learnable.
  • domain assumption Same-instance tracking annotations across frames in TAO, LaSOT and GOT-10k are correct and define a consistent object identity.
    Section 3.2 builds all training conversations on the premise that a tracked box refers to the same physical object; tracking drift or annotation noise would inject false instance supervision.
  • domain assumption Converting segmentation masks in PDM and PerSeg to bounding boxes preserves enough spatial information to measure localization quality.
    Appendix B describes the mask-to-box conversion; tight boxes discard boundary detail but the paper does not validate the converted benchmark against a native box benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Teaching VLMs to Localize Specific Objects from In-context Examples." pith.science (2026). https://pith.science/paper/2USS25BZ

@misc{pith2026241113317,
  author       = {Pith},
  title        = {Pith review of: Teaching VLMs to Localize Specific Objects from In-context Examples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2USS25BZ}},
  note         = {Machine review of arXiv:2411.13317}
}
read the original abstract

Vision-Language Models (VLMs) have shown remarkable capabilities across diverse visual tasks, including image recognition, video understanding, and Visual Question Answering (VQA) when explicitly trained for these tasks. Despite these advances, we find that present-day VLMs (including the proprietary GPT-4o) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context. In this work, we focus on the task of few-shot personalized localization, where a model is given a small set of annotated images (in-context examples) -- each with a category label and bounding box -- and is tasked with localizing the same object type in a query image. Personalized localization can be particularly important in cases of ambiguity of several related objects that can respond to a text or an object that is hard to describe with words. To provoke personalized localization abilities in models, we present a data-centric solution that fine-tunes them using carefully curated data from video object tracking datasets. By leveraging sequences of frames tracking the same object across multiple shots, we simulate instruction-tuning dialogues that promote context awareness. To reinforce this, we introduce a novel regularization technique that replaces object labels with pseudo-names, ensuring the model relies on visual context rather than prior knowledge. Our method significantly enhances the few-shot localization performance of recent VLMs ranging from 7B to 72B in size, without sacrificing generalization, as demonstrated on several benchmarks tailored towards evaluating personalized localization abilities. This work is the first to explore and benchmark personalized few-shot localization for VLMs -- exposing critical weaknesses in present-day VLMs, and laying a foundation for future research in context-driven vision-language applications.

Figures

Figures reproduced from arXiv: 2411.13317 by the authors.

Figure 1
Figure 1. In-context personalized localization involves localizing object instances present in a scene (or query image) similar to the object presented as an in-context example. In this setting, the input to the model is a category name, in-context image, bounding box coordinates (not shown in this figure), and a query image. The model is tasked with localizing the same category of interest (presented as an in-context example… view at source ↗
Figure 2
Figure 2. Overview of data creation and conversation format. To instill few-shot personalized localization abilities in VLMs, our IPLoc creates multi-modal conversations by harnessing data from multiple video object tracking datasets. For semantic coherence, focus on personalization and stronger contextual awareness, we create these conversations by sampling frames from the same video, tracking a particular object of interest… view at source ↗
Figure 3
Figure 3. Effect of increasing number of shots. We report mIOU (%) on the LASOT [16] test split. IPLoc refers to the Qwen2- VL [50] fine-tuned on the proposed data mix in this paper. PDM PerSeg ICL-LASOT Mean OwlViT [39] 48.47 40.71 50.18 46.45 IPLoc 45.15 51.62 65.30 54.02 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (8 more)
Figure 7
Figure 7. Figure 7: Qwen2-VL / InternVL2 Prompt 3 “Please provide the bounding box of the element element” [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 9
Figure 9. Figure 9: GPT-4o Prompt 2 D.1.2. InternVL2-76B Examples: In Tables 17, 18, 19 and 20 we provide additional and detailed examples of the variants of the prompts we used throughout our experiments. D.2. GPT-4o Prompt Variants Following the successful experimental approaches imple￾…
Figure 5
Figure 5. Figure 5: Qwen2-VL / InternVL2 Prompt 1 Prompt 2: “Task: Locate the element in the image. Provide its bounding box coordinates in the format [xmin, ymin, xmax, ymax]” [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Qwen2-VL / InternVL2 Prompt 2 D.1.1. Qwen2-VL-72B Examples: In Tables 13, 14, 15 and 16, we provide additional and detailed examples of the variants of the prompts we used throughout our experiments. Prompt 3: “Please analyze this image and locate the exact element. Re…
Figure 10
Figure 10. Figure 10: One-shot Localization Results. Representative examples of object localization using a single support frame in the PerSeg dataset. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Two-shot Localization Results. Performance visualization with two support frames on the LaSOT and PDM datasets. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Four-shot Localization Examples. Visualization of our method’s localization results when provided with four support frames on PerSeg and LaSOT datasets [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 14
Figure 14. Figure 14: Challenging Cases in One-shot Setting. Examples where the model identifies semantically similar objects (incorrect airplane and boat) but fails to distinguish the specific target instance, highlighting the complexity of instance-level discrimination. 24 [PITH_FULL_IM…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 36 canonical work pages

  1. [1]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024. 6

    arXiv

  2. [2]

    Flamingo: a Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Se- bastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Bink...

    work page 2022

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 2, 3

    arXiv 2023

  4. [4]

    Decimamba: Exploring the length extrapolation potential of mamba

    Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Na- dav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba. arXiv preprint arXiv:2406.14528, 2024. 8

    arXiv 2024

  5. [5]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 3

    1901

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz...

    work page 2020

  7. [7]

    MiniGPT-v2: Large Language Model as a Unified Interface for Vision-Language Multi-task Learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. MiniGPT-v2: Large Language Model as a Unified Interface for Vision-Language Multi-task Learning. In Proc. ICLR,

    work page

  8. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 6

    arXiv 2024

Show all 61 references
  1. [9]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, 2023. 3

    2023

  2. [10]

    InstructBLIP: Towards General-purpose Vision- Language Models with Instruction Tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision- Language Models with Instruction Tuning. In NeurIPS,

  3. [11]

    Tao: A large-scale bench- mark for tracking any object

    Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: A large-scale bench- mark for tracking any object. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part V 16, pages 436–454. Springer,

    2020

  4. [12]

    Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Amit Alfassy, Roei Herzig, Donghyun Kim, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, et al. Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models. In NeurIPS, 2023. 2

    2023

  5. [13]

    Teaching structured vision & language concepts to vision & language models

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Rameswar Panda, Roei Herzig, Eli Schwartz, Donghyun Kim, Raja Giryes, Rogerio Feris, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In Proc. CVPR, 2023. 2

    2023

  6. [14]

    Towards multimodal in-context learning for vision & language models

    Sivan Doveh, Shaked Perek, M Jehanzeb Mirza, Amit Al- fassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlin- sky. Towards multimodal in-context learning for vision & language models. arXiv preprint arXiv:2403.12736 , 2024. 2, 3, 4

    2024 arXiv

  7. [15]

    The Llama 3 Herd of Models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783,

    arXiv

  8. [16]

    Lasot: A high-quality benchmark for large-scale single ob- ject tracking

    Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single ob- ject tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, page...

  9. [17]

    SEED: Self-supervised Dis- 9 tillation for Visual Representation

    Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED: Self-supervised Dis- 9 tillation for Visual Representation. In Proc. ICLR, 2021. 6, 8

    2021

  10. [18]

    Cross-domain few-shot object detection via enhanced open-set object detector

    Yuqian Fu, Yu Wang, Yixuan Pan, Lian Huai, Xingyu Qiu, Zeyu Shangguan, Tong Liu, Yanwei Fu, Luc Van Gool, and Xingqun Jiang. Cross-domain few-shot object detection via enhanced open-set object detector. In European Conference on Computer Vision, pages 247–264. Springer, 2025. 8

    2025

  11. [19]

    Task vectors are cross- modal

    Amir Bar Grace Luo, Trevor Darrell. Task vectors are cross- modal. arXiv preprint arXiv:2410.22330, 2024. 2

    2024 arXiv

  12. [20]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5

    2021 arXiv

  13. [21]

    Multimodal task vectors enable many-shot multimodal in-context learn- ing

    Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig. Multimodal task vectors enable many-shot multimodal in-context learn- ing. arXiv preprint arXiv:2406.15334, 2024. 2, 3

    2024 arXiv

  14. [22]

    ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs

    Irene Huang, Wei Lin, M Jehanzeb Mirza, Jacob A Hansen, Sivan Doveh, Victor Ion Butoi, Roei Herzig, Assaf Arbelle, Hilde Kuhene, Trevor Darrel, et al. ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs. arXiv preprint arXiv:2406.08164, 2024. 6

    2024 arXiv

  15. [23]

    Got-10k: A large high-diversity benchmark for generic object tracking in the wild

    Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE transactions on pattern analysis and machine intelligence, 43(5):1562–1577, 2019. 2, 5, 6, 7

    2019

  16. [24]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 6, 8

    2019

  17. [25]

    Le, Yunhsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representa- tion Learning With Noisy Text Supervision. In Proc. ICML,

  18. [26]

    Im- proving Zero-Shot Models with Label Distribution Priors

    Jonathan Kahana, Niv Cohen, and Yedid Hoshen. Im- proving Zero-Shot Models with Label Distribution Priors. arXiv:2212.00784, 2022. 2

    2022 arXiv

  19. [27]

    Building and better understanding vision- language models: insights and future directions., 2024

    Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L ´eo Tronchon. Building and better understanding vision- language models: insights and future directions., 2024. 2, 3, 6

    2024

  20. [28]

    Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web- scale filtered dataset of interleaved image-text documents,

  21. [29]

    LLaV A-OneVision: Easy Visual Task Trans- fer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaV A-OneVision: Easy Visual Task Trans- fer. arXiv preprint arXiv:2408.03326, 2024. 3, 6, 8

    2024 arXiv

  22. [30]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In Proc. ICML, 2023. 3

    2023

  23. [31]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 6, 8

    2023 arXiv

  24. [32]

    Video-LLaV A: Learning united visual repre- sentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning united visual repre- sentation by alignment before projection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5971–5984, 2024. 2

    2024

  25. [33]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  26. [34]

    MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge

    Wei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Posseg- ger, Mateusz Kozinski, Rameswar Panda, Rogerio Feris, Hilde Kuehne, and Horst Bischof. MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge. In Proc. ICCV ,

  27. [35]

    LLaV A-NeXT: Improved reasoning, OCR, and world knowl- edge, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. LLaV A-NeXT: Improved reasoning, OCR, and world knowl- edge, 2023. 2

    2023

  28. [36]

    Improved Baselines with Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. InProc. CVPR, 2024. 2

    2024

  29. [37]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In NeurIPS, 2023. 2, 3, 4

    2023

  30. [38]

    MetaICL: Learning to Learn In Context

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICL: Learning to Learn In Context. In Proc. NAACL, 2022. 2, 3

    2022

  31. [39]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European conference on computer vision , pages 728–755. ...

    2022

  32. [40]

    Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, , Jakub Micorek, Mateusz Kozinski, Hilde Kuhene, and Horst Possegger

    M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, , Jakub Micorek, Mateusz Kozinski, Hilde Kuhene, and Horst Possegger. Meta-Prompting for Automating Zero- shot Visual Recognition with LLMs. In Proc. ECCV, 2024. 2

    2024

  33. [41]

    Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Pos- segger, Rogerio Feris, and Horst Bischof

    M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Pos- segger, Rogerio Feris, and Horst Bischof. TAP: Targeted Prompting for Task Adaptive Generation of Textual Train- ing Instances for Visual Classification. arXiv preprint arXiv:2309.06809, 2023

    2023 arXiv

  34. [42]

    LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections

    Muhammad Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Possegger, Mateusz Kozinski, Rogerio Feris, and Horst Bischof. LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections. In NeurIPS, 2023

    2023

  35. [43]

    Glov: Guided large language models as implicit optimizers for vision language models

    M Jehanzeb Mirza, Mengjie Zhao, Zhuoyuan Mao, Sivan Doveh, Wei Lin, Paul Gavrikov, Michael Dorkenwald, Shiqi Yang, Saurav Jha, Hiromi Wakaki, et al. Glov: Guided large language models as implicit optimizers for vision language models. arXiv preprint arXiv:2410.06154, 2024. 2

    arXiv

  36. [44]

    GPT-4 Technical Report

    OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023. 2, 6

    2023 arXiv

  37. [45]

    Train short, 10 test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, 10 test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. 8

    2021 arXiv

  38. [46]

    Learning Transferable Visual Models from Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models from Natural Language Supervision. InProc. ICML,

  39. [47]

    Where’s waldo: Diffusion features for person- alized segmentation and retrieval

    Dvir Samuel, Rami Ben-Ari, Matan Levy, Nir Darshan, and Gal Chechik. Where’s waldo: Diffusion features for person- alized segmentation and retrieval. NeurIPS, 2024. 6, 12

    2024

  40. [48]

    LAION-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev....

    2022

  41. [49]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13286, 2023. 3

    2023 arXiv

  42. [50]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 3, 6, 7, 8, 12

    2024 arXiv

  43. [51]

    Frustratingly simple few-shot object detection

    Xin Wang, Thomas E Huang, Trevor Darrell, Joseph E Gon- zalez, and Fisher Yu. Frustratingly simple few-shot object detection. arXiv preprint arXiv:2003.06957, 2020. 8

    2003 arXiv

  44. [52]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS, 2022. 2, 3

    2022

  45. [53]

    Larger language models do in-context learning differently

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846, 2023. 7

    2023 arXiv

  46. [54]

    Demystify- ing CLIP Data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystify- ing CLIP Data. In Proc. ICLR, 2023. 2

    2023

  47. [55]

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In NeurIPS, 2023. 2, 3

    2023

  48. [56]

    Sigmoid Loss for Language Image Pre- training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre- training. In Proc. ICCV, 2023. 2

    2023

  49. [57]

    Personalize segment anything model with one shot

    Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junt- ing Pan, Xianzheng Ma, Hao Dong, Peng Gao, and Hong- sheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023. 6, 12

    2023 arXiv

  50. [58]

    Mmicl: Empowering vision-language model with multi-modal in-context learning

    Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915, 2023. 3

    2023 arXiv

  51. [59]

    Llamafac- tory: Unified efficient fine-tuning of 100+ language mod- els

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafac- tory: Unified efficient fine-tuning of 100+ language mod- els. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 3: S...

    2024

  52. [60]

    MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. In Proc. ICLR, 2024. 3 11 Appendix In the following, we provide additional experiments and fur- ther explanations that...

    2024

  53. [61]

    <ref>category</ref>

    We find that our IPLoc consistently improves the base model on all the few-shot splits we test on. These results provide insights regarding the generaliza- tion of our fine-tuning methodology across different vision language models (VLMs). Note that in the main manuscript (Tab...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.