{"id":"361a2dc5-59bc-44d6-bb22-0781d56eda76","arxiv_id":"2504.21559","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Object-based visual prompting can mitigate object hallucination in LVLMs, and a learned router that picks the best prompt per image outperforms fixed or random prompts on POPE and CHAIR.","lead":"Large vision-language models often imagine objects that are not in the picture. This paper shows that overlaying simple visual markers, like boxes or circles, can reduce such hallucinations, and trains a router to pick the best marker for each image, without modifying the model itself.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Router's six-question training proxy may not transfer to captioning; CHAIR gains need per-image validation.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the per-image optimal VP is defined from only six binary presence questions, and the paper must assume this proxy transfers to open-ended captioning for the CHAIR results to support the central claim. I agree with this assessment. The paper's own Limitations (3) and (8) explicitly flag that the router ignores question context and that the VP preference signal is subtle and unstable, which is exactly where the proxy-transfer concern bites. A competing concern such as missing error bars affects significance but not the internal logic of the claimed mechanism; the six-question proxy is more load-bearing because if it fails, the CHAIR numbers have no mechanistic explanation. The proposed check is feasible because it reuses the same black-box querying already performed in training and only requires per-image caption scoring on a modest sample. If the agreement test passes, the central claim is substantially supported; if it fails, the paper should be revised to restrict claims to POPE or to demonstrate a caption-level training signal. This does not change the reader's CONDITIONAL verdict, which already requires additional evidence before the claims are accepted at face value.","tokens_in":12386,"tokens_out":7631,"duration_ms":86735,"concrete_test":"On a held-out set of at least 500 COCO val images, compute per-image the best VP according to the six-question accuracy (replicating Eq. 2) and the best VP according to CHAIR by generating captions with each candidate VP and measuring CHS/CHI. Measure agreement between the two rankings, and between the router's predicted VP and the CHAIR-best VP, using Cohen's kappa. If agreement is at chance (kappa below 0.2), the six-question training signal does not transfer to captioning and the Table 2 improvements cannot be attributed to the router's learned proxy; if agreement is substantial, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"BBVPE's central CHAIR claim rests on the assumption that the optimal VP defined by Eq. 2, maximizing S over exactly six object-presence questions per image (Appendix A), is also optimal for open-ended captioning. The router is trained by cross-entropy against this six-question label and at inference selects a VP from image features alone (Eq. 7), with no question context (Limitation 3). Nothing in the derivation connects Eq. 1's accuracy proxy to CHAIR/CHS, and the authors themselves report a 'subtle learning signal' from LVLM preferences and convergence instability (Limitation 8). If the six-question VP ranking does not correlate per image with the CHAIR VP ranking, the CHAIR improvement in Table 2 is not explained by the router learning generalizable visual-prompt preferences, and the central claim that BBVPE reduces hallucination in descriptions is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether overlaying simple object-based visual prompts (e.g., bounding boxes, circles, arrows) on images can reduce object hallucination in large vision-language models (LVLMs). It proposes Black-Box Visual Prompt Engineering (BBVPE), which trains a lightweight router (a frozen CLIP vision encoder plus a trainable MLP) on input-output pairs from a target LVLM to select, for each image, the visual prompt that maximizes the LVLM's accuracy on six binary object-presence questions. At inference, the router selects a visual prompt from image features alone, and the prompted image is fed to the LVLM. Experiments on POPE, CHAIR, and a GPT-4o-based description quality evaluation across two open-source and two proprietary LVLMs show that BBVPE outperforms no prompting, random prompting, and a fixed best prompting baseline, while remaining below an Oracle upper bound. A cross-dataset experiment on GQA and a latency/TFLOPs comparison are also reported.","tokens_in":12471,"tokens_out":7473,"duration_ms":77251,"significance":"If the findings hold, BBVPE is a novel and practically relevant contribution: it offers a model-agnostic, black-box approach to mitigating object hallucination, which is directly applicable to proprietary LVLMs where white-box methods cannot be used. The paper's strengths include a simple and interpretable idea, consistent improvements across multiple benchmarks and models, substantial gains on CHAIR, and a cross-dataset generalization experiment. The framework is reproducible from the provided implementation details, and the authors are transparent about several limitations. The main weaknesses are the absence of statistical significance testing, the small absolute gains on POPE, and an unexamined gap between the router's training objective and the captioning benchmark.","major_comments":[{"comment":"The POPE results report only point estimates with no error bars, confidence intervals, or significance tests. The absolute gains are small (e.g., Random setup for LLaVA-1.5: 89.60 to 91.37; Popular: 86.20 to 87.23; Adversarial: 79.73 to 81.33), yet the abstract and Section 1 claim that BBVPE 'significantly reduces' object hallucination. To support this claim, the authors should report variability across the POPE question subsets or provide bootstrap confidence intervals, and ideally paired significance tests for the main comparisons against the baseline and the best-fixed-VP baseline.","section":"§4.1, Table 1"},{"comment":"The router is trained to predict the visual prompt that maximizes accuracy on exactly six object-presence questions per image (three positive, three negative), and at inference it selects a prompt from image features alone (Eq. 7; Limitation 3). The central CHAIR result in Table 2 depends on the unexamined assumption that this six-question proxy correlates with per-image visual-prompt effectiveness for open-ended captioning. The paper provides no per-image analysis linking the router's selections to CHAIR-optimal prompts, and the authors' own Limitation 8 notes a 'subtle learning signal' and convergence instability. I request either a per-image comparison of router-chosen prompts against per-image CHAIR-optimal prompts on a subset of images, or a sensitivity analysis varying the number and type of training questions to show that the CHAIR gains are robust to the choice of proxy. Additionally, Eq. 3 excludes images where multiple prompts tie; the authors should report how many images are discarded, as this may bias the training distribution.","section":"§3 (Eq. 2), Appendix A, Table 2"},{"comment":"The 'best VP' baseline is described as 'consistently using a fixed best VP that delivers the highest overall performance for the model.' If this VP is chosen using test-set performance (as the footnotes in Tables 1 and 2 suggest), then it is an oracle-style baseline, and the comparison is not a fair reflection of a realistically selectable fixed prompt. I ask the authors to clarify the selection procedure and, if test labels are used, to instead select the best VP on a validation split or explicitly label the baseline as an oracle upper bound for fixed prompting. The same clarification is needed for the cross-dataset experiment in Table 5.","section":"§4, Tables 1-3"},{"comment":"Table 4 compares latency and computational cost against existing hallucination mitigation methods (VCD, OPERA, M3ID, etc.), but the paper does not compare BBVPE to these methods on hallucination metrics such as POPE or CHAIR. Without such a comparison, it is difficult to assess the practical significance of the proposed approach: if white-box methods achieve substantially larger hallucination reductions, the small POPE gains and the black-box advantage may not be sufficient to justify the approach. The authors should add a performance comparison on open-source LVLMs (even if only for a subset of methods), or explicitly argue why such a comparison is not meaningful for their black-box setting.","section":"§4.3, Table 4"}],"minor_comments":[{"comment":"The citation for VCD is incorrect: Table 4 cites 'VCD (Liu et al., 2023a)', but the paper's reference list associates Liu et al. (2023a) with 'Mitigating hallucination in large multi-modal models via robust instruction tuning.' VCD refers to Leng et al. (2023). Please correct this citation.","section":"Table 4"},{"comment":"The GPT-4o-based description evaluation is a model-judge metric, but the paper does not report how many images or descriptions were evaluated, nor any measure of judge reliability. Please provide these details and discuss the potential bias of using an LVLM judge to evaluate hallucination.","section":"Table 3 and Appendix C"},{"comment":"The qualitative examples in Figures 1 and 3 are informative but rely on a few cherry-picked images. Consider adding a quantitative breakdown (e.g., the distribution of per-image optimal prompts) to support the claim that prompt effectiveness varies across images.","section":"§1 and Figure 1"},{"comment":"The statement 'this work is the first to extend APE to visual inputs' is too strong: prior work on visual prompt engineering, including Set-of-Mark prompting (Yang et al., 2023a) and CPT (Yao et al., 2024), also optimizes or selects visual prompts. The novelty claim should be qualified to emphasize the black-box, hallucination-focused, per-image selection aspect.","section":"§2"},{"comment":"The training set size and the number of unique optimal-prompt examples after applying Eq. 3 are not reported. Please provide these statistics, as they are important for assessing the representativeness of the router's training distribution.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practically important problem, and the empirical results suggest that the core idea has value. My main concerns are about statistical rigor, the transferability of the router's training objective to the captioning benchmark, and the fairness of the 'best VP' baseline. These are addressable with additional analyses and clarifications, so I recommend major revision rather than rejection. No concerns about novelty disclosure or citation integrity beyond the VCD citation error noted above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThis is a well-written, honest empirical paper with one genuinely useful finding: overlaying simple object-based visual cues (bounding boxes, circles, arrows) on an image can reduce object hallucination in LVLMs, and a small router trained on black-box input-output pairs can pick a per-image prompt that beats the best fixed prompt. The CHAIR results are notable—CHS drops from 62.8 to 46.3 on LLaVA-1.5, and similar on proprietary models. That alone is worth knowing.\n\nThe router idea is the new bit, but it's incremental. The paper overstates novelty by calling itself the first to extend APE to visual inputs; prior visual prompting work exists, and the router over a fixed pool is a modest extension. What the authors do well is treat the LVLM as a black box and keep the pipeline lightweight and cheap, which is practical for deployed APIs.\n\nNow the soft spots. The load-bearing concern is the gap between the router's training signal and the captioning evaluation. The router is trained on six binary object-presence questions per image, then selects a VP from image features alone at inference. CHAIR measures open-ended descriptions. Nothing in the derivation shows that the optimal VP for those six yes/no questions is also optimal for full captions, and the authors admit the learning signal is subtle and training is unstable (Limitation 8). The cross-dataset POPE-GQA result helps show the router generalizes across images, but it's still the same question format. I'd like to see per-image correlation between the six-question VP ranking and the CHAIR VP ranking; that would directly address the transfer issue.\n\nOther issues are smaller. No error bars or significance tests, and POPE gains are only 1–2 points. The GPT-4o evaluation is a model judge, useful but not ground truth. The Oracle uses test labels, which is fine as an upper bound but should not be read as achievable. Finally, the paper cites Woodpecker and VOLCANO but never compares against them; even a rough comparison would contextualize the black-box advantage.\n\nWho this is for: researchers and practitioners working on hallucination mitigation for proprietary LVLMs. It deserves a serious referee. My recommendation would be accept with major revision—add variance-aware evaluation, compare against a post-hoc correction baseline, and either validate or temper the CHAIR claim.","headline":"Solid empirical study: visual prompting reduces hallucination and a learned router helps, but the six-question training signal may not fully explain the captioning gains.","tokens_in":13077,"tokens_out":2153,"would_cite":true,"duration_ms":21033,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A black-box router that picks a per-image visual prompt reduces object hallucination in large vision-language models.","keywords":["visual prompt engineering","object hallucination","large vision language models","black-box router","POPE benchmark","CHAIR benchmark","bounding box prompting","hallucination mitigation"],"falsifier":"On the COCO validation images, compute the per-image optimal prompt under the paper's six-question score and under CHAIR's caption-level hallucinated-object ratio; if the two rankings agree at chance level, or if a router trained on the six-question labels fails to beat a fixed prompt on attribute- or relation-based questions, the transfer claim is refuted.","tokens_in":12115,"feed_emoji":"🖼️","tokens_out":9921,"duration_ms":90589,"temperature":0.7,"pith_summary":"Large vision-language models (LVLMs) routinely describe objects that are not in an image. This paper tries to establish that a simple, model-agnostic intervention reduces that failure mode: overlay a visual cue—a bounding box, circle, arrow, blur, crop, or center point—on detected objects before the model sees the image. The same cue works well on some images and poorly on others, so the paper trains a lightweight router, using only the target model's answers to six yes/no \"is this object present?\" questions per image, to select the most effective cue for each image. The claim, backed by POPE and CHAIR evaluations on two open-source and two proprietary LVLMs, is that this per-image routing beats no prompting, random prompting, and the best fixed prompt, while remaining far from the oracle upper bound. If correct, it makes hallucination mitigation available for closed models whose weights, logits, and attention are inaccessible.","feed_headline":"Best visual prompt per image cuts hallucination in vision-language AI","feed_subtitle":"The router needs no weights or logits, so closed models can use it; it beats fixed prompts on POPE and CHAIR.","key_machinery":"The router model is the central object: a frozen CLIP-ViT-L/14@336px vision encoder feeding a trainable MLP head that assigns a score to each candidate visual prompt and outputs the argmax as the chosen prompt. Training labels come from a discrete optimization problem defined by the paper's scoring function S, which counts correct answers to object-presence questions; each image's optimal prompt is the one maximizing S, with ties excluded. The machinery also includes the candidate pool (box, circle, arrow, center point, crop, reverse blur, and no prompt) and the object localizer (SAM 2) whose detected boxes define what the prompts highlight. The router's job is to imitate the oracle's selection rule using only the image, so at inference no LVLM query is needed to choose the prompt.","core_discovery":"On its own terms, the discovery is twofold. First, object-based visual prompts are not uniformly helpful: for a given LVLM and image, different prompts elicit different levels of hallucination, and the best prompt varies by image, model, and metric; selecting it per image (the oracle) produces large gains. Second, the per-image optimum can be approximated by a black-box router without any access to model internals. The router is trained on labels generated by the target LVLM itself: for each image, three positive and three negative presence questions are asked for each candidate prompt, the prompt with the highest answer accuracy is declared optimal, and a frozen CLIP encoder plus a trainable MLP learns to predict that prompt from the image alone. The paper demonstrates on POPE, CHAIR, and a GPT-4o-based description-quality evaluation that the routed prompting reduces hallucination relative to no prompting, random prompting, and the best fixed prompt.","pith_inferences":["A natural test of whether the six-question signal is really about hallucination rather than prompt-answer bias: if the router's selected prompts correlate with the LVLM's tendency to say 'yes' under a given prompt, the method may be partly re-ranking answer styles rather than suppressing hallucinated objects.","Because the optimal prompt labels come from the target model's own answers, the framework doubles as a model-characterization tool: the per-image prompt preferences encode what visual emphasis a model needs, which could inform fine-tuning or decoding corrections.","The paper's own limitation notes—router-training instability and sensitivity to object-localizer quality—imply that near-term gains are as likely to come from better localization and more stable routing as from a larger prompt pool."],"forward_implications":["Adding BBVPE as a pre-processing layer requires no training of the LVLM, so it applies to proprietary APIs and to future models without re-training.","The per-model router means deployment can tailor prompting to a model's own preferences, since the paper finds no single prompt works across models or metrics.","The COCO-trained router transfers to GQA-style images, so cross-dataset deployment on similar natural images is plausible without per-dataset router retraining.","The gap to the oracle quantifies remaining headroom, and the paper's preliminary results indicate that better object localization directly raises performance.","Because the router selects without querying the LVLM, it adds only localizer plus router cost, which the paper measures as lower latency than existing decoding-based methods."],"supporting_citations":[{"why":"Defines the POPE yes/no presence-question protocol used both to label optimal prompts and to evaluate them.","marker":"(Li et al., 2023)"},{"why":"Provides CHAIR, the caption-level hallucination metric used to show transfer to open-ended descriptions.","marker":"(Rohrbach et al., 2018)"},{"why":"Supplies SAM 2, the object localizer whose detected boxes determine which objects the visual prompts highlight.","marker":"(Ravi et al., 2024)"},{"why":"Provides the frozen CLIP vision encoder used as the backbone of the VP router.","marker":"(Radford et al., 2021)"},{"why":"Supplies the COCO train/val split used for router training and both benchmarks.","marker":"(Lin et al., 2014)"},{"why":"Established visual prompting with marked objects, the paradigm the candidate prompt pool extends.","marker":"(Yang et al., 2023a)"},{"why":"Supplies GQA for the cross-dataset transfer evaluation of the router.","marker":"(Hudson and Manning, 2019)"}],"fun_headline_variants":["Router picks per-image visual prompts to slash LVLM hallucinations","Black-box prompt routing reduces object hallucination in vision AI","Per-image visual prompts, chosen by router, cut hallucination","No model access needed: router curbs visual chat hallucination","Visual prompt router beats fixed prompts on hallucination benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The idea rests on the assumption that the best visual prompt for an image, as judged by six yes/no questions about object presence, is also the best prompt for describing that image in full sentences and for answering other questions.","fun_headline_variants_meta":{"raw":{"variants":["Router picks per-image visual prompts to slash LVLM hallucinations","Black-box prompt routing reduces object hallucination in vision AI","Per-image visual prompts, chosen by router, cut hallucination","No model access needed: router curbs visual chat hallucination","Visual prompt router beats fixed prompts on hallucination benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2632,"prompt_tokens":891,"completion_tokens":1741,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1659}},"tokens_in":507,"tokens_out":1741,"duration_ms":12623,"temperature":1.0,"reasoning_tokens":1659,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:00:20.487337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the COCO validation images, compute the per-image optimal prompt under the paper's six-question score and under CHAIR's caption-level hallucinated-object ratio; if the two rankings agree at chance level, or if a router trained on the six-question labels fails to beat a fixed prompt on attribute- or relation-based questions, the transfer claim is refuted.","supporting_citations":[],"review_version":1}