{"id":"84bf2fa9-2d22-433f-b617-a9bd9e872c39","arxiv_id":"2412.06720","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Visual prompts on images replace textual mention words in a new multimodal entity linking task, with a new dataset (VPWiki) and a framework (IIER) that achieves the best reported results on it.","lead":"This paper introduces VP-MEL, a new entity linking task that uses visual prompts (boxes drawn on images) instead of textual mention words to link objects to knowledge base entities. It also introduces VPWiki, a new benchmark dataset, and IIER, a framework that combines visual prompt features with a vision-language model to outperform prior entity linking methods on this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The printed Detective-VLM prompt may leak the gold entity name/type at inference; the exact test-time prompt is never specified, so the claimed VP-MEL gain is not reproducible or ruleable-out.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing risk: the Detective-VLM prompt printed in §5.2 includes '{Entity Name}' and '{Entity Type}' in the question and as the answer, and the paper does not specify how these slots are filled at test time. This concern is the single most load-bearing one because the paper's primary experimental claim — a 5-point Hit@1 advantage over the best VLM baseline on VP-MEL — is attributed largely to Detective-VLM (Table 4 and the ablation in Table 6). If the gold label is present in the prompt at inference, mwj is not a model-generated inference but a copy of the answer, and the final scoring pipeline in §5.3–§5.4 rewards string overlap with that answer. If the gold label is not present, the exact prompt is undisclosed, so no independent reproduction can verify the result. The proposed test settles the question by comparing gold-filled, blanked, and generic-slot prompt variants and measuring both retrieval metrics and the generated text's overlap with gold entity names. The concern is consistent with the reader's conditional verdict: the task and framework are plausible, but current evidence is insufficient. I do not see a stronger objection than this; other issues (dataset filtering to correlated pairs, absent code/data release, baseline reimplementation) are secondary and would further support CONDITIONAL rather than full acceptance. Since the reader already returned CONDITIONAL and this stress-test does not change that assessment, the appropriate verdict remains UNCHANGED.","tokens_in":15472,"tokens_out":2696,"duration_ms":30924,"concrete_test":"Reproduce IIER on the VPWiki test set with three Detective-VLM inference prompts, holding all other components fixed: (1) the prompt printed in §5.2 with '{Entity Name}' and '{Entity Type}' replaced by a generic phrase such as 'the marked object'; (2) the same prompt with those slots left blank; (3) the same prompt with the gold entity name and type filled in. Compute Hit@1/Hit@3/Hit@5 with five seeds as in the paper. Also record the generated mwj strings and compute the exact-match rate between mwj and the gold entity name for each variant. If variant (3) alone recovers 48.36 Hit@1 and variants (1)/(2) drop substantially, the result is leakage-driven; if variants (1)/(2) match (3), the concern is resolved. Either way, the published appendix should specify the exact inference-time prompt string.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that IIER outperforms all VP-MEL baselines (Hit@1 48.36 vs 43.20 for LLaVA-1.5-7b, Table 2a), and the main source of this gain is the Detective-VLM module (Table 4, Table 6). The prompt template in §5.2 is: 'Question: Based on the text ... tell me briefly what is the {Entity Type} and {Entity Name} in the red box of the {Image}? Answer: {Entity Name} {Entity Type}.' The only statement about inference is 'During the inference process, {Entity Name} and {Entity Type} need to be generated by VLM.' This is ambiguous in a load-bearing way: if the question fields {Entity Name} and {Entity Type} are filled with the gold labels at test time, then the auxiliary text mwj contains the answer, and the final similarity score in §5.3–§5.4 partly measures whether the model recognizes the string it was handed. If the fields are removed or blanked at test time, that exact prompt is never shown, so the experiment cannot be reproduced and the measured 48.36 Hit@1 could be an artifact of gold-label leakage. Because the central advantage over baselines depends on this module, the ambiguity is not cosmetic; it determines whether the paper's headline result is a valid model or a prompt-leakage artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines a new task, VP-MEL, in which a marked image region (visual prompt) is linked to a KB entity, and introduces the VPWiki dataset built from WikiDiverse and WikiMEL. It proposes the IIER framework, which combines CLIP-based global/local visual features guided by visual prompts with a fine-tuned Detective-VLM that generates auxiliary textual information, followed by multi-grained multimodal interaction and contrastive learning. Experiments report that IIER outperforms all baselines on VP-MEL (Hit@1 48.36 vs. 43.20 for the best VLM baseline) and is competitive on standard MEL datasets (Hit@1 69.47 on WikiDiverse).","tokens_in":15794,"tokens_out":3109,"duration_ms":34827,"significance":"If the results hold, VP-MEL is a useful extension of entity linking to settings where textual mention words are absent, and the VPWiki dataset with its annotation pipeline and inter-annotator agreement is a valuable resource. The IIER architecture is well motivated, and the paper includes multiple ablations and cross-dataset evaluations. However, the main claimed advantage depends critically on the Detective-VLM module, and the prompt specification in §5.2 leaves open a direct gold-label leakage path. Because the headline result is not reproducible or ruleable-out without an exact test-time prompt, the contribution is currently conditional on closing that gap.","major_comments":[{"comment":"The Detective-VLM prompt template printed in §5.2 places '{Entity Name}' and '{Entity Type}' inside the question itself, immediately before the answer field. The only inference-time statement is 'During the inference process, {Entity Name} and {Entity Type} need to be generated by VLM,' which is ambiguous in a load-bearing way. If those fields are filled with gold labels at test time, then the generated mwj is a copy of the answer, and the textual and cross-modal similarity scores in §5.3–§5.5 partly measure string matching against the label the model was handed. If the fields are blanked or removed at test time, the exact inference prompt is never specified, so the experiment cannot be reproduced and the 48.36 Hit@1 gain in Table 2a could be an artifact of leakage. Please specify the exact test-time prompt, report the Detective-VLM outputs for a few examples, and re-run the core comparison with a prompt that does not contain the answer fields.","section":"§5.2, Table 2a, Table 4, Table 6"},{"comment":"Even during instruction fine-tuning, the training template contains the gold entity name and type in the question as well as in the answer. The loss in §5.2 is a standard next-token prediction loss, so the model can minimize it by copying the answer fields from the input rather than by learning to ground the visual prompt in the image. This makes the claim that Detective-VLM acts as an 'implicit knowledge base' unsubstantiated: the learned behavior may be degenerate copying, which would break when the fields are removed at inference. Please either train with a prompt that omits the answer fields from the question, or provide an analysis (e.g., attention or perturbation experiments) showing that the generated mwj actually depends on the image and visual prompt rather than on the input placeholders.","section":"§5.2, fine-tuning objective"},{"comment":"The evaluation protocol for the VLM baselines on the VP-MEL task is underspecified. It is not stated whether LLaVA, BLIP-2, MiniGPT-4, and mPLUG-Owl3 receive the same red-box visual prompt, what instruction or question template is used, whether they are fine-tuned on VPWiki, or how their outputs are converted to a KB entity ranking. Since the headline improvement over the best VLM baseline is 5.16 Hit@1 points, the comparison is only meaningful if the baselines are given the same visual prompt and task instruction. Please provide the exact input construction, decoding settings, and candidate entity ranking procedure for each baseline.","section":"§6.1, Appendix A.2, Table 2a"},{"comment":"The text states that 'even without Detective-VLM, visual prompts, or mention words, IIER can still achieve the best metrics,' but Table 3 only reports IIER with all components on WikiDiverse without mention words. No ablation row is shown for IIER without Detective-VLM or without visual prompts on this setting, so the claim is not directly supported by the displayed results. Please add the corresponding ablation rows or qualify the statement.","section":"§6.3, Table 3"}],"minor_comments":[{"comment":"There is a typo in 'additionlLy' near the description of the CLIP visual encoder.","section":"§1"},{"comment":"In the final sentence of §5.3, 'we obtain the text features T G Ei and T G Ei' should read 'T G Ei and T L Ei'.","section":"§5.3"},{"comment":"The spacing in 'LLaV A' is inconsistent (e.g., 'LLaV A-1.5' vs. 'LLaVA' in prose); please use a single consistent notation.","section":"Table 2"},{"comment":"The phrase 'It is noted that within these two datasets, certain metrics of IIER exhibit values that are comparable to those of MELOV' is repetitive and could be tightened.","section":"Appendix A.9"}],"recommendation":"major_revision","confidential_remarks":"The core concern is the Detective-VLM prompt leakage, which affects the headline VP-MEL result. I would encourage the editor to require the authors to specify the exact test-time prompt and, if necessary, retrain Detective-VLM without answer fields in the input before the paper can be accepted. The paper would also benefit from releasing code and data to make the evaluation reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper introduces VP-MEL, a sensible extension of entity linking where the mention is a marked image region rather than a span of text, and contributes VPWiki, a dataset of about 10k pairs. That is the real novelty. The IIER framework itself is a composition of off-the-shelf parts—CLIP layers, a fine-tuned mplug-owl2, BERT, and MIMIC-style interaction units—but that is not a problem; the task formulation and dataset are new, and the authors are honest that the MEL gains are incremental.\n\nThe soft spot is the Detective-VLM module, and it is load-bearing. The printed fine-tuning template in §5.2 puts '{Entity Name}' and '{Entity Type}' inside the question and also in the answer. The only statement about inference is that those fields 'need to be generated by VLM.' If at test time the question fields are blanked or replaced with something like 'the entity in the red box,' the experiment is reproducible only by guessing that unshown prompt. If instead the gold name and type are filled into the question, then the auxiliary text mwj is largely copied from the input, and the final score partly measures string matching against the answer the model was handed. Either way the paper does not close the gap, and since the main advantage over baselines comes from this module (Table 4 and Table 6), the ambiguity is not cosmetic. This needs to be fixed before the headline number can be trusted.\n\nTwo smaller issues: no code or data release, so the new dataset and pipeline cannot be reused or checked; and the VP-MEL evaluation is on a single dataset with means only, no variance, despite the paper saying runs were repeated. Those are minor relative to the prompt question.\n\nOn balance the core idea is worth pursuing and the dataset, if released, could be useful to the MEL community. The paper deserves a serious referee, but the referee should require the authors to specify the exact inference prompt for Detective-VLM and, ideally, release the VPWiki annotations.","headline":"The VP-MEL task and VPWiki dataset are a genuine contribution, but the unspecified inference-time prompt for Detective-VLM could allow gold-label leakage and must be resolved before the headline result is credible.","tokens_in":16304,"tokens_out":2071,"would_cite":false,"duration_ms":20627,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces VP-MEL, where a marked image region replaces a mention word, and claims its IIER framework links such regions to knowledge-base entities at 48.36% Hit@1 on the new VPWiki dataset, beating all baselines.","keywords":["multimodal entity linking","visual prompts","vision-language model","entity disambiguation","visual grounding","knowledge base","VP-MEL","contrastive learning"],"falsifier":"Run the VPWiki test set with the Detective-VLM prompt rewritten so the entity-name and entity-type fields are blank, generated only by the model, and compare Hit@1 against the reported 48.36. If the score falls toward the no-VLM ablation at 35.65 or toward LLaVA-1.5 at 43.20, the reported gain comes from the gold label appearing in the prompt rather than from latent knowledge.","tokens_in":1709,"feed_emoji":"🖼️","tokens_out":8960,"duration_ms":153314,"temperature":0.7,"pith_summary":"Multimodal entity linking usually starts from a mention word in the text and retrieves a knowledge-base entity. This paper argues that the mention can be replaced by a visual prompt, a marked region in the image, and that a vision-language model fine-tuned on the new VPWiki dataset can generate enough auxiliary text to make the link work even when no mention word is written. The proposed IIER framework combines prompt-guided image features, deep and shallow CLIP layers, and the generated text, and the experiments claim a Hit@1 of 48.36 on VP-MEL, about five points above the strongest vision-language baseline, while remaining competitive on standard MEL datasets. If the claim holds, entity linking no longer depends on text density, and any image region a user marks can be resolved to a knowledge-base entity.","feed_headline":"Image-box entity linking hits 48.36% Hit@1, beating all baselines","feed_subtitle":"IIER links the boxed object in an image to a knowledge-base entity without a written mention word.","key_machinery":"The load-bearing object is the visual prompt: a manually or automatically drawn red box that tells the model which region to link. Around it, IIER stacks four components: a CLIP image encoder whose shallow and deep layers are both pooled, with shallow layers preserving the texture and geometry of the prompt and deep layers preserving semantics; Detective-VLM, an instruction-tuned vision-language model that turns the boxed region and the sentence into a short generated phrase $\\text{mw}_j$; a BERT text encoder that reads the sentence with $\\text{mw}_j$ prepended; and three interaction units, VFI, TFI, and CMFI, that compute visual, textual, and cross-modal similarities averaged into one contrastive score. The visual prompt does the work that mention words used to do: it localizes the referent, and the generated auxiliary text adapts the language side to that localization.","core_discovery":"The central claim is that visual prompts alone can carry the mention signal in entity linking. Given an image with a red-box region, IIER extracts global and local features with a CLIP image encoder, uses an instruction-tuned Detective-VLM to read the box and produce a short auxiliary phrase identifying the object and its type, concatenates that phrase with the sentence, and scores candidate entities by averaged visual, textual, and cross-modal similarity under contrastive training. On the VPWiki test set the paper reports Hit@1 48.36, Hit@3 67.51, and Hit@5 77.50, outperforming every compared MEL and vision-language method, and on WikiDiverse it reports Hit@1 69.47, ahead of the previous best baseline. The paper interprets these results as evidence that a region-level prompt plus latent vision-language knowledge can substitute for explicit mention words.","pith_inferences":["Not tested in the paper: VP-MEL could be pushed to image-only linking, where the sentence is removed and the generated auxiliary text is the only language signal; the reported no-mention results on WikiDiverse suggest the gap would be modest.","Not tested in the paper: the box prompt is rectangular, so pairing the method with segmentation masks or point prompts would test whether the claimed gain comes from precise localization rather than from the prompt's visual appearance.","Not tested in the paper: because VPWiki is built from news-photo datasets, an adversarial test on user-generated social images with unrelated captions would show whether the image-text correlation requirement limits the task."],"forward_implications":["MEL systems no longer need annotated mention words: any marked region can be linked, which makes image-first and text-poor multimodal content usable.","The instruction-tuned VLM's generated auxiliary phrase carries enough signal to lift Hit@1 by more than 12 points over the same model without the VLM on VP-MEL.","On WikiDiverse without mention words, IIER reaches Hit@1 23.87, far above the 17.66 of the best MEL baseline, showing that the framework degrades gracefully when text cues are missing.","The VPWiki dataset, with 12,720 samples and visual-prompt annotations, provides a benchmark for future region-to-entity linking.","VP-MEL still needs image-text correlation and box-shaped prompts, so irregular user marks and unrelated image-text pairs remain open challenges."],"supporting_citations":[{"why":"Supplies WikiDiverse, the main source of VPWiki image-text pairs and the standard MEL benchmark used for comparison.","marker":"Wang et al. (2022c)"},{"why":"Supplies WikiMEL and RichpediaMEL data, the merged knowledge base, and the GHMFC baseline.","marker":"Wang et al. (2022a)"},{"why":"Defines the multi-grained interaction units (VFI, TFI, CMFI) that IIER adapts, and provides the MIMIC baseline.","marker":"Luo et al. (2023)"},{"why":"Inspires the automated visual-prompt annotation pipeline using visual entailment and grounding modules.","marker":"Li et al. (2024)"},{"why":"Shows that visual prompts overlaid on the image canvas can guide vision-language models, supporting the red-box prompt design.","marker":"Cai et al. (2024)"},{"why":"Provides evidence that CLIP can interpret red-circle visual markers, motivating the prompt-guided visual features.","marker":"Shtedritski et al. (2023)"},{"why":"Provides mPLUG-Owl2, the pretrained model that Detective-VLM is instruction-tuned from.","marker":"Ye et al. (2024)"},{"why":"Defines the LLaVA-1.5 baselines and the visual-instruction-tuning paradigm used for comparison.","marker":"Liu et al. (2024)"},{"why":"Introduces the MELOV method, the strongest MEL baseline IIER is compared against.","marker":"Song et al. (2024)"},{"why":"Supplies the reimplemented baseline numbers marked with asterisks in the MEL comparison tables.","marker":"Sui et al. (2024)"}],"fun_headline_variants":["Visual prompts alone drive entity linking to new SOTA","Box a region, link it: IIER beats baselines without mention words","VP-MEL: link boxed visuals to entities, no mention words needed","IIER uses red-box prompts to beat all entity linking baselines"],"cache_read_input_tokens":18432,"weakest_assumption_plain":"The weakest load-bearing premise is that Detective-VLM really produces the entity name and type at test time. The printed prompt template places '{Entity Name}' and '{Entity Type}' inside the question and the expected answer, so if those fields are present during inference the auxiliary text is taken from the gold label, and if they are absent the paper never specifies the exact test-time prompt.","fun_headline_variants_meta":{"raw":{"variants":["Visual prompts alone drive entity linking to new SOTA","Box a region, link it: IIER beats baselines without mention words","VP-MEL: link boxed visuals to entities, no mention words needed","IIER uses red-box prompts to beat all entity linking baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000415,"raw_usage":{"total_tokens":2137,"prompt_tokens":935,"completion_tokens":1202,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1126}},"tokens_in":551,"tokens_out":1202,"duration_ms":9764,"temperature":1.0,"reasoning_tokens":1126,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:18:43.248641+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the VPWiki test set with the Detective-VLM prompt rewritten so the entity-name and entity-type fields are blank, generated only by the model, and compare Hit@1 against the reported 48.36. If the score falls toward the no-VLM ablation at 35.65 or toward LLaVA-1.5 at 43.20, the reported gain comes from the gold label appearing in the prompt rather than from latent knowledge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that CLIP can interpret red-circle visual markers, motivating the prompt-guided visual features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reimplemented baseline numbers marked with asterisks in the MEL comparison tables."}],"review_version":1}