{"id":"4d50466e-e246-4fd4-8d14-1e606c6fdfc6","arxiv_id":"2411.13317","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning VLMs on video-tracking conversations with made-up object names teaches them to localize a specific object in a new image from only a few in-context examples.","lead":"This paper shows that current vision-language models, including GPT-4o, are weak at a task called few-shot personalized localization: given a couple of photos with boxes around a specific object, they often cannot find that same object in a new photo. The authors fix this with a training recipe built from video tracking data and made-up object names, improving localization accuracy by about 12 to 14 points on average.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed gain may not require support bounding-box coordinates; no ablation shows the model actually grounds the numeric boxes, so the 'specific object' component is unverified.","rationale":"The paper makes a credible empirical contribution: large gains on three benchmarks, a pseudo-name ablation that demonstrates dependence on visual context rather than category semantics, retention of generic VLM capabilities, and cross-model generalization. The reader's weakest assumption is exactly the load-bearing gap I find: the model must map the raw text coordinates in each support turn to the correct visual region, and the paper never tests this mapping. The concern is concrete and falsifiable: shuffled or omitted support coordinates would either confirm that the model uses the grounded supervision or reveal that the gains are format-level or category-level artifacts. Independent support for the method (parameter-efficient tuning, ablations of data sources and coherence, pseudo-name robustness) is real, so this is a conditional-accept situation rather than a rejection. The reader's CONDITIONAL verdict already captures the needed condition; adding the coordinate-grounding test would settle it, so I recommend no change to the verdict.","tokens_in":24470,"tokens_out":7776,"duration_ms":90712,"concrete_test":"Evaluate the released IPLoc models on PDM, PerSeg, and ICL-LASOT in three conditions: (a) correct support coordinates as in Table 2; (b) support coordinates replaced by random boxes (uniform in the image, or the box of a different object in the same support image); (c) no coordinates after the <ref> token. If mIoU in (b) or (c) does not fall substantially below (a), the model is not learning the text-coordinate-to-region mapping and the personalized-instance claim fails. A complementary check is to draw the support box as a visual overlay and measure whether performance changes, which also tests whether the chosen representation is the bottleneck.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the fine-tuned VLM performs few-shot personalized localization, identifying the same object instance in the query from the in-context support images. In the format of Section 3.1, the support object's location is provided only as raw text coordinates ((xmin,ymin),(xmax,ymax)) following the '<image><ref>Category</ref>' token, and the model must associate these coordinate tokens with the correct visual region in the support image. The pseudo-name ablation in Table 6a rules out reliance on category semantics, but no experiment tests whether the numeric coordinates contribute anything: there is no ablation with shuffled, random, or omitted support boxes, and no comparison to drawn/highlighted boxes. Because PDM contains 4.2 objects per image on average, the support coordinates are the only signal selecting the right instance; without evidence that the model uses them, the 'specific object' component of the headline result is unsupported, and the gains could instead reflect improved category-level localization or output-format compliance rather than true instance-level in-context grounding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IPLoc, a data-centric fine-tuning recipe that aims to give VLMs few-shot personalized localization ability: given a small number of in-context support images, each annotated with a category name and a bounding box, the model must localize the same instance in a query image. The authors construct instruction-tuning conversations from video object tracking datasets (TAO, LaSOT, GOT-10k) by sampling frames of the same tracked object, and they introduce pseudo-name regularization that replaces true category labels with arbitrary names to force reliance on visual context. They fine-tune Qwen2-VL 7B/72B and LLaVA-OV with LoRA and evaluate on PDM, PerSeg, and a newly constructed ICL-LASOT benchmark. The main results show large mIoU gains over the base models and over strong proprietary and open VLMs such as GPT-4o and InternVL2, while a small evaluation on GQA/SEED/POPE indicates that generalization is largely retained.","tokens_in":24674,"tokens_out":11177,"duration_ms":101567,"significance":"If the reported results hold, this is a valuable contribution: it identifies a concrete weakness of modern VLMs, proposes a simple data-oriented remedy with a crisp regularization idea, and benchmarks a task that is likely to become more important. The paper is transparent in several respects: it evaluates across model families and sizes, includes a generalization-retention check, performs multiple ablations (data sources, semantic coherence, pseudo names, cross-model fine-tuning), and states that code and data will be released. At the same time, the central claim of instance-specific (personalized) localization is not yet fully verified because the paper does not demonstrate that the model actually uses the support bounding-box coordinates, and the data recipe is not specified in enough detail to be reproduced from the text. The large reported gains make the approach promising, but the manuscript needs additional experiments and reporting changes before the central claim is established.","major_comments":[{"comment":"The central claim is that IPLoc performs personalized, instance-specific localization. In the conversation format, the only signal that identifies which instance to localize is the raw bounding-box coordinate text in each support turn, since the pseudo-name is arbitrary and the support image may contain multiple objects (PDM has 4.2 objects per image on average). However, the paper provides no experiment that varies or removes the support coordinates: there is no ablation with shuffled, random, or omitted support boxes, and no comparison with drawn/highlighted boxes. The pseudo-name ablation in Table 6a only shows that the model does not rely on the category name; it does not show that the model uses the coordinates to select the correct instance. Without such evidence, the reported gains could be driven by improved category-level localization or output-format compliance rather than by true in-context instance grounding. Please add ablations that manipulate the support bounding boxes (e.g., randomize, shuffle, or drop them) and, if possible, a distractor setting where the support and query images contain multiple objects of the same category, to verify the 'specific object' component of the method.","section":"Section 3.1, Table 2, Table 6a"},{"comment":"The paper is explicitly data-centric, but it does not specify the composition of the fine-tuning data mix: it reports only that TAO, LaSOT, and GOT-10k are used, without giving the relative proportions of these datasets, the total number of conversations, the frame-sampling interval strategy, the distribution of shot counts (1-8), or the pseudo-name replacement rate. The LoRA hyperparameters are described only as 'the default parameters' from LlamaFactory, with no rank, alpha, learning rate, batch size, or number of epochs. Since the method's central contribution is the data recipe, these omissions prevent replication and make it impossible for readers to judge how sensitive the results are to these choices. Please provide a detailed table of the data mix, including per-dataset conversation counts, sampling parameters, pseudo-name rates, and all training hyperparameters.","section":"Section 3.2, Section 3.3"},{"comment":"There are internal numerical inconsistencies that need to be reconciled. First, in Table 5 the 'IPLoc' column reports PDM=50.05, PerSeg=27.55, and ICL-LASOT=39.13, but none of these values appears in Table 2 for either IPLoc (7B) (Real) or IPLoc (7B) (Real+Pseudo); moreover, PDM=50.05 exceeds even the best 72B result (46.60 on PDM 2-shot in Table 2). The caption and text do not specify which model, data mix, or shot settings are used, so the reader cannot verify the semantic-coherence ablation. Second, in Section 4.2 the text states 'our fine-tuned model also outperforms the base model for the PerSeg and LASOT test sets with 12.58% average improvement.' This number does not match Table 2: the Real-only model is actually worse than the base model on ICL-LASOT on average, and the Real+Pseudo model yields an average improvement of roughly 12.1% over base across PerSeg and ICL-LASOT (not 12.58%). Please correct the text and clarify which data-mix variant is being discussed.","section":"Table 5 and Section 4.2"}],"minor_comments":[{"comment":"The assistant-response format is ambiguous: the text says the assistant 'provides a category label based on the context' and 'the bounding box coordinates,' but the shown template only contains coordinates after 'Assistant:'. Please clarify whether the prediction output includes the category label, the coordinates, or both.","section":"Section 3.1"},{"comment":"The 'Average' column in Table 2 treats all ten shot-configurations equally, even though the numbers of shots differ across datasets (2 for PDM, 4 for PerSeg, 4 for ICL-LASOT). Please state this averaging convention explicitly in the caption or text.","section":"Table 2"},{"comment":"The verbal summaries of Table 2 switch between 'our best model (trained only with real category names)' and 'our fine-tuned model' without specifying which data mix is meant. Use consistent labels ('Real' vs. 'Real+Pseudo') when describing the improvements.","section":"Section 4.2"},{"comment":"There is a typo, 'Suprisingly', in the first paragraph of Appendix D.1.","section":"Appendix D.1"},{"comment":"The limitation paragraph states that 'Qwen2-VL scores 5.3% mIOU, when evaluated for the multi-object localization task in MS-COCO,' but no source or evaluation protocol is provided; please add a citation or a brief description of this experiment.","section":"Section 5 (Limitations)"},{"comment":"The paper says 'To encourage reproducibility, our entire codebase is provided as part of the appendix and will be released upon acceptance.' The appendix as submitted contains no code; please rephrase to state that the code will be released upon acceptance, and ideally provide an anonymous link for review.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is on an interesting and timely topic, and the main idea is simple and potentially useful. My main concerns are the missing ablation for the grounding of the support bounding-box coordinates, the underspecified data recipe, and the numerical inconsistencies between Table 5 and Table 2 and in the text. These are fixable with additional experiments and a careful revision. I would not recommend acceptance in the current form, but the direction is promising and the requested changes are within the scope of a major revision. I also note that the manuscript leans on several self-citations (e.g., [14], [41]-[43]) but this does not affect the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper delivers a genuinely useful data recipe for teaching VLMs in-context personalized localization, with large, consistent gains across model sizes. The main weakness is a missing ablation: nothing shows the model actually uses the support bounding-box coordinates, so the instance-specific part of the claim is less solid than the rest.\n\nWhat's new and good: it's the first to frame few-shot personalized localization as a VLM ICL task, and the recipe is clearly described—instruction conversations built from tracking videos, frames sampled from the same object, pseudo-name regularization. The gains are real and consistent: roughly +11.5 mIoU average for Qwen2-VL-7B and +21.6 for the 72B variant, with LLaVA-OV also improving ~7 points. The ablations are well-targeted: same-video vs random sampling matters, pseudo-names help beyond just renaming, and LoRA preserves generalization within about 1% on GQA/SEED/POPE. Comparing against classic few-shot detectors and OwlViT gives useful context.\n\nThe stress-test concern is legitimate. Support boxes are given only as raw text coordinates, and there's no ablation with shuffled, omitted, or drawn boxes. With PDM averaging 4.2 objects per image, those coordinates are the only signal telling the model which instance is meant. Without an ablation, 'personalized' is plausible but not proven—the gains could partly reflect better category-level localization or format compliance. This is fixable with one well-designed experiment. I'd also want error bars or significance tests; PerSeg has only 154 test conversations, so some differences could be noise, though the pattern across three datasets and two model sizes makes that less worrying. The novelty overlap with detector-side in-context detection (FIND, VIP) should be acknowledged; the contribution is the VLM setting and the pseudo-name regularization, not the general idea. The 'fundamental cognitive ability' framing is overbroad, and code/data are promised but not yet public.\n\nWho it's for: VLM instruction-tuning and ICL researchers, and anyone building personalization for assistants or search. It deserves a serious referee; I'd send it to review with a request for the grounding ablation, error bars, and artifact release. The core recipe is likely to hold, but the instance-specific framing needs the missing experiment to be fully convincing.","headline":"Useful, well-ablated recipe for teaching VLMs in-context localization, with real gains; the main gap is that no ablation proves the model actually uses the support bounding boxes, so the 'specific object' claim is partially unverified.","tokens_in":25258,"tokens_out":2774,"would_cite":true,"duration_ms":31676,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VLMs can learn to localize a specific object from a few examples.","keywords":["vision-language models","in-context learning","personalized localization","few-shot localization","instruction tuning","video object tracking","pseudo-name regularization","LoRA"],"falsifier":"Show a trained model the same support and query conversations but replace all support bounding-box coordinates with random or permuted numbers; if localization accuracy on the query stays the same, the model never learned the coordinate-to-image mapping and the claimed mechanism is falsified. Equivalently, re-run training with support images that have visible drawn boxes instead of raw coordinates; if performance does not improve (or drops), the representation assumption is doing the work, not genuine context learning.","tokens_in":24267,"feed_emoji":"🎯","tokens_out":5923,"duration_ms":57218,"temperature":0.7,"pith_summary":"This paper asks whether today's vision-language models can learn to find a specific object in a new image when shown just a few annotated examples of that object. It argues that current VLMs, including a proprietary model like GPT-4o, largely cannot do this, and it introduces a data-only remedy: instruction-tuning conversations mined from video object tracking, where the same object reappears across frames. The conversations present each support image with its category name and raw bounding-box coordinates, and the model is trained to output coordinates for the query. A pseudo-name regularization step replaces real category names with invented names, forcing the model to rely on the visual examples rather than on pre-trained knowledge. On three evaluation sets, the fine-tuned models raise average mIoU from 31.77 to 43.31 for the 7B model and from 34.21 to 55.79 for the 72B model while retaining general benchmark performance within about 1%.","feed_headline":"Data recipe gives VLMs few-shot object localization","feed_subtitle":"Fine-tuning on tracking-video dialogues lifts Qwen2-VL-7B from 31.8 to 43.3 mIoU and 72B from 34.2 to 55.8.","key_machinery":"The central mechanism is the IPLoc instruction-tuning format: each user turn is an image prefixed by `<image><ref>Category</ref>` followed by raw bounding-box coordinates `((xmin,ymin),(xmax,ymax))`, and the assistant turn must output the coordinates for the same category in the query image. Because each conversation comes from frames of one tracked object, the model sees the same instance under pose, lighting, and background changes, which is what makes 'personalized' localization learnable. The pseudo-name regularization replaces real class names with arbitrary names so the model cannot lean on pre-trained object knowledge; LoRA constrains the parameter update to preserve generalization. The combination turns a standard next-token-prediction objective into a context-driven localization learner.","core_discovery":"On its own terms, the paper claims that few-shot personalized localization is a missing capability in current VLMs and that the gap can be closed by data alone. The central result is that fine-tuning Qwen2-VL with LoRA on dialogues built from TAO, LaSOT, and GOT video tracking sequences produces a model that can take one to eight support frames, each annotated only by text and numeric coordinates, and localize the same object instance in a query image. The pseudo-name trick is the load-bearing regularization: renaming categories to arbitrary names during training lifts the 7B model's average by 12.33 mIoU over training with real names only and the 72B model by 7.81, and a test that swaps in pseudo-names confirms the model looks at the in-context images rather than the label vocabulary. The paper also reports that the same fine-tuning extends to LLaVA-OV with about 7 mIoU improvement, and that the learned ability beats classic few-shot object detectors and an open-vocabulary detector on the ICL-LASOT benchmark.","pith_inferences":["The paper never ablates alternative grounding formats, so a natural extension would compare raw numeric coordinates against drawn boxes or highlighted regions to see how much of the gain depends on the exact coordinate-to-image mapping.","Because pseudo-names make category labels uninformative, the same training recipe could be tested on object categories never seen in pre-training, or even on synthetic 'alien' objects, to measure true instance-level generalization.","The method is currently limited to single-object localization per query, and the paper states this explicitly; extending the dialogues to multi-object outputs would likely require a different decoding format and may be a natural next step.","The improvement on ICL-LASOT over a specialized open-vocabulary detector suggests that VLM in-context localization could eventually challenge dedicated detection models on tasks where the target is defined by a few examples rather than a name."],"forward_implications":["If the claim holds, an off-the-shelf open VLM can be turned into a personalized localizer with tracking-video data alone, no new architecture or box-drawing supervision.","The pseudo-name regularization implies that the model is learning to match visual identity across frames, so the same recipe should transfer to any setting where category names are unreliable or uninformative, such as novel or rare objects.","The 72B variant reaches 55.79 average mIoU, so larger VLMs benefit more from the data recipe, suggesting scaling laws for in-context localization similar to those in language-only in-context learning.","LoRA keeps general VQA and hallucination benchmarks within about 1% of the base model, so the capability can be added without a specialization trade-off."],"supporting_citations":[{"why":"Supplies the TAO multi-object tracking videos from which the paper mines instruction-tuning conversations.","marker":"[11]"},{"why":"Supplies the LaSOT single-object tracking videos used for training conversations and the ICL-LASOT test split.","marker":"[16]"},{"why":"Supplies the GOT-10k generic object tracking videos used as a third training data source.","marker":"[23]"},{"why":"Provides the Qwen2-VL base models that the paper fine-tunes for its main results.","marker":"[50]"},{"why":"Provides the LoRA parameter-efficient fine-tuning method that preserves generalization.","marker":"[20]"},{"why":"Provides the PDM benchmark used to evaluate personalized localization performance.","marker":"[47]"},{"why":"Provides the PerSeg benchmark used to evaluate personalized localization performance.","marker":"[57]"},{"why":"Motivates the data-centric post-hoc fine-tuning approach for improving VLM in-context learning.","marker":"[14]"},{"why":"Serves as the proprietary GPT-4o baseline that illustrates the weakness of current VLMs on this task.","marker":"[44]"},{"why":"Provides the LLaVA-OV model used to demonstrate that the fine-tuning recipe transfers across different VLM architectures.","marker":"[29]"}],"fun_headline_variants":["Pseudo-names teach VLMs to localize from context","Tracking-video dialogues unlock few-shot localization in VLMs","Data-only fine-tuning gives VLMs in-context object localization","Renaming objects in training sharpens VLM few-shot localization","VLMs learn to find specific objects from a few annotated frames"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method assumes the model actually learns to connect the raw coordinate numbers in each example to the right part of the example image, even though no box is drawn; without that link, the gains could just come from the conversation format.","fun_headline_variants_meta":{"raw":{"variants":["Pseudo-names teach VLMs to localize from context","Tracking-video dialogues unlock few-shot localization in VLMs","Data-only fine-tuning gives VLMs in-context object localization","Renaming objects in training sharpens VLM few-shot localization","VLMs learn to find specific objects from a few annotated frames"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3250,"prompt_tokens":1090,"completion_tokens":2160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":706,"completion_tokens_details":{"reasoning_tokens":2075}},"tokens_in":706,"tokens_out":2160,"duration_ms":17535,"temperature":1.0,"reasoning_tokens":2075,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:34:13.343279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show a trained model the same support and query conversations but replace all support bounding-box coordinates with random or permuted numbers; if localization accuracy on the query stays the same, the model never learned the coordinate-to-image mapping and the claimed mechanism is falsified. Equivalently, re-run training with support images that have visible drawn boxes instead of raw coordinates; if performance does not improve (or drops), the representation assumption is doing the work, not genuine context learning.","supporting_citations":[{"cited_title":"Tao: A large-scale bench- mark for tracking any object","cited_arxiv_id":null,"evidence_quote":"Supplies the TAO multi-object tracking videos from which the paper mines instruction-tuning conversations."},{"cited_title":"Got-10k: A large high-diversity benchmark for generic object tracking in the wild","cited_arxiv_id":null,"evidence_quote":"Supplies the GOT-10k generic object tracking videos used as a third training data source."},{"cited_title":"Where’s waldo: Diffusion features for person- alized segmentation and retrieval","cited_arxiv_id":null,"evidence_quote":"Provides the PDM benchmark used to evaluate personalized localization performance."}],"review_version":1}