{"id":"815b72f6-d065-4caa-86d9-b0a64c632c8c","arxiv_id":"2411.13909","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Panther improves multimodal LLMs by converting the user's text question into visual prompts that steer a frozen image encoder toward instruction-relevant regions, gaining about 2 to 3 points on several VQA benchmarks and up to 5.3 points on MMVP.","lead":"This paper introduces Panther, a multimodal language model that turns the user's text question into a visual prompt inserted into the image encoder, helping the model focus on instruction-relevant image details. The authors report gains of roughly 1 to 3 points on general VQA benchmarks and larger gains on the vision-centric MMVP test, though no code or data is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 isolates the instruction-alignment effect to MMVP alone: on VQAv2/GQA/VQA-T the alignment-free BGE text encoder matches CLIP, so the headline gains are explainable by added prompt capacity; the only supporting evidence sits on a 150-question single-run benchmark.","rationale":"The reader's weakest assumption correctly names the risk — CLIP alignment may not transfer through the prompt-token interface, leaving extra parameters as the real driver — but the paper's own Table 3 sharpens the concern beyond what the reader stated. The BGE rows do not merely confirm alignment is load-bearing; on VQAv2, GQA, and VQA-T they empirically support the reader's feared alternative: an encoder with no image-text alignment matches CLIP-derived prompts within at most 1.0 point and with inconsistent sign, so per-sample instruction content is not what drives those gains. The alignment signal is confined to MMVP, where CLIP-IP beats BGE-IP by 2.0 points (Vicuna) and 7.4 points (Llama-3B), but MMVP is a 150-question benchmark with single-run reports and no error bars; the Vicuna gap (about 3 questions) is within one standard error of a proportion, and even the headline Vicuna MMVP gain of 5.3 points is only about one standard error of a difference. The missing control — per-sample prompts with random content at matched parameter count — is what would separate 'instruction information' from 'prompt capacity'; its absence matters because the contribution's novelty over existing visual prompt tuning is precisely the text-instruction conditioning. I partly agree with the reader: same core concern, but the BGE comparison undercuts the mechanism on the benchmarks that carry the headline numbers, not merely in the abstract sense of 'could be capacity.' I do not move the verdict: the empirical improvements are directionally consistent across three vision encoders and four LLMs (Tables 4-5), FT+IP improves over FT alone (Table 7), supporting orthogonality, and the token-pruning tradeoff is honestly reported (Table 6). Acceptance conditions should, however, include the capacity-matched control, repeated-seed MMVP numbers, and a mechanism claim scoped to benchmarks where alignment evidence holds. This is an internal-evidence concern, not a disagreement with external consensus, and the authors' BGE ablation is exactly the right experiment to have run; the issue is the interpretation the paper draws from it across benchmarks.","tokens_in":19877,"tokens_out":21364,"duration_ms":182799,"concrete_test":"Run the missing capacity-matched ablation: replace the frozen CLIP text encoder's question embeddings with per-sample random Gaussian vectors (same 77-token count, same trained MLP, same data, learning rate, and seeds), keeping SP, connector, and LLM updates identical to the 'w/ SP + IP (CLIP)' row; evaluate VQAv2/GQA/VQA-T/MMVP. If this control matches CLIP-IP within ~1 point outside MMVP (as BGE already does), the instruction-conditioning mechanism is not operative on the headline benchmarks and the reported gains are capacity effects. To settle MMVP itself, run three seeds for baseline / w/ SP / w/ SP+IP(CLIP) / w/ SP+IP(BGE): with 150 questions and accuracy near 0.3 the standard error is ~3.7 points, so the Vicuna CLIP-versus-BGE gap of 2.0 points must be shown to exceed seed spread, leaving the 7.4-point Llama-3B gap as the only clearly non-noise alignment signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Decomposing Table 3 (Panther-VE ablation) isolates where instruction alignment actually matters. For Vicuna-7B, shared prompts (SP) alone add +1.3/+1.4/+1.5/+2.3 (VQAv2/GQA/VQA-T/MMVP). CLIP-derived IP adds +1.0/+1.8/-0.1/+3.0; IP from the alignment-free BGE text encoder adds +0.9/+2.2/-0.9/+1.0. For Llama-3B, CLIP-IP adds +1.4/+2.4/+2.4/+9.4; BGE-IP adds +1.0/+2.0/+3.4/+2.0. On three of four benchmarks, the CLIP-versus-BGE difference is at most 1.0 point and inconsistent in sign (BGE beats CLIP on GQA with Vicuna and on VQA-T with Llama-3B): the instruction content of the IP tokens contributes at noise level there, and the gains are fully explained by added tunable capacity (24 SP tokens, 77 per-sample IP tokens, and a trained MLP), which the 'w/ SP' row is not matched to control for. The alignment mechanism survives only on MMVP (CLIP beats BGE by +2.0 with Vicuna and +7.4 with Llama-3B), but MMVP has 150 questions, the numbers are single-run, and a 2.0-point gap is about 3 questions — within one standard error. The paper also omits the decisive control: 'w/ SP + per-sample random prompts' with identical token count and MLP. Consequently, the paper's own ablation supports the capacity explanation for the headline results (VQAv2 80.8, GQA 65.2) and supports instruction injection only on MMVP; since the claimed novelty over ordinary visual prompt tuning is precisely the text-instruction conditioning, this asymmetry is load-bearing for the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Panther, an MLLM architecture whose central novelty is Panther-VE: user instructions are encoded by a frozen CLIP text encoder, projected by a small MLP into per-sample instruction-aware visual prompts (IP), and prepended to the patch tokens of a frozen ViT together with shared prompts (SP). A Panther-Bridge prunes redundant visual tokens across multi-turn training, and a Panther-Decoder uses interleaved autoregressive training. The authors evaluate on general VQA, instruction-following, and vision-centric benchmarks, reporting gains over LLaVA-1.5 on VQAv2, GQA, MMVP, and CV-Bench with both Vicuna-7B and Llama 3-8B backbones, and they ablate the SP/IP components, vision encoders, LLMs, and the pruning threshold.","tokens_in":20313,"tokens_out":4701,"duration_ms":46152,"significance":"If the instruction-conditioning mechanism is real, the paper makes a useful and largely orthogonal contribution: it offers a way to inject textual intent into a frozen visual encoder without multiple encoders or high-resolution inputs, and it does so with the same training data and base model as LLaVA-1.5. The experimental program is thorough in coverage: the method is tested across three vision encoders and four LLMs, and the controlled LLaVA-1.5 baseline makes the headline comparisons interpretable. The significance of the central claim, however, rests on whether the IP prompts actually carry instruction information into visual features rather than merely adding tunable capacity, and the paper's own ablation does not yet settle that question.","major_comments":[{"comment":"The text states 'Our Panther surpasses Cambrian-dev in all the 4 benchmarks,' but Table 2 shows a tie on MMVP for the Vicuna-7B configuration: Panther achieves 30.0 and Cambrian-dev achieves 30.0. The sentence should be corrected to 'matches or exceeds' or should explicitly say Panther exceeds Cambrian-dev on three of the four benchmarks.","section":"Section 4.3 / Table 2"},{"comment":"The ablation does not isolate instruction alignment from added capacity. On VQAv2, GQA, and VQA-T, the alignment-free BGE-b text encoder matches the CLIP text encoder within 1.0 point in both LLM settings (e.g., Vicuna-7B: 80.7 vs 80.8, 65.6 vs 65.2, 58.8 vs 59.6; Llama-3B: 79.0 vs 79.4, 63.8 vs 64.2, 52.2 vs 51.2). Since the IP branch adds a trained two-layer MLP and 77 per-sample tokens, the headline gains on VQAv2 and GQA are compatible with a pure-capacity explanation. The decisive missing control is 'w/ SP + per-sample random prompts' with matched token count and a trained projector; without it, the paper's own data support instruction injection only on MMVP, not on the larger benchmarks.","section":"Section 4.4 / Table 3"},{"comment":"The only benchmark on which CLIP-based IP clearly beats BGE-based IP is MMVP, which contains 150 questions and is reported as a single run without variance. For Vicuna-7B the CLIP advantage over BGE is 2.0 points (30.0 vs 28.0), about 3 questions, which is within one binomial standard error; for Llama-3B the 7.4-point gap (22.7 vs 15.3) is roughly two standard errors. To make this result load-bearing, the authors should report multiple seeds or confidence intervals, and ideally also the random-prompt control requested above.","section":"Section 4.4 / Table 3, MMVP"}],"minor_comments":[{"comment":"The manuscript does not specify at which ViT layers the SP and IP tokens are inserted; the equation only describes prepending to the visual embedding, while Figure 3(b) may be read as prompt insertion at every Transformer layer. Please state the insertion depth explicitly.","section":"Section 3.2.1 / Eq. (4)"},{"comment":"In the pseudocode for `prune_tokens`, the line `ref_token = cur_T['tensor'][j]` should be `ref_T['tensor'][j]`; as written, the reference token is taken from the current turn's tensor rather than from the reference turn.","section":"Section 3.2.2 / Algorithm 1"},{"comment":"The method text says the lightweight text encoder is 'for instance, BERT [11]', but the implementation uses CLIP's text encoder; please make this consistent.","section":"Section 3.2.1 / Section 4.1"},{"comment":"The text refers to 'Table 3.2.2' when presenting the pruning analysis; this should be Table 6.","section":"Section 4.4 / Token-Pruning Analysis"},{"comment":"Reference [81] is cited for BGE-base, but the given entry describes C-Pack rather than the BGE text embedding model; please cite the correct source.","section":"References"},{"comment":"There are several typos and grammatical errors, including 'we does not make pruning' in Section 3.3, 'the remained texts' in Section 3.2.2, and 'pervious models' in Section 4.3; a language edit would improve readability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central claim is currently under-supported by the published ablation due to the missing random-prompt control and the single-run, small-size MMVP evidence, but these gaps are addressable with additional experiments rather than being intrinsic to the approach. The Table 2 contradiction about surpassing Cambrian-dev should also be corrected. I would be willing to accept after the authors add the matched-capacity control, report variance or multiple seeds, and qualify the Cambrian-dev comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest read: this is a plausible, well-ablated empirical paper. The core idea—generate per-instruction prompt tokens from a frozen CLIP text encoder and prepend them inside a frozen ViT—is a clean, new combination relative to earlier visual-prompt MLLMs, and the main effect is real: Panther beats LLaVA-1.5 by 2–3 points on VQAv2, GQA, MMVP, and CV-Bench, with consistent gains across two LLMs and three vision encoders. The token-pruning bridge is a practical contribution for multi-turn training cost, and the pruning-threshold study is fine.\n\nThe soft spots are in the evidence for the central mechanism, not in the main result. First, a straightforward overclaim: the text says Panther surpasses Cambrian-dev on all four vision-centric benchmarks, but Table 2 shows a tie on MMVP (30.0 vs 30.0). Second, the ablation story is thinner than the prose suggests. The BGE text encoder, which lacks CLIP's image-text alignment, matches CLIP-based IP on VQAv2, GQA, and VQA-T—and even beats it on GQA with Vicuna and VQA-T with Llama-3B. The only benchmark where CLIP clearly wins is MMVP, with a 2-point gap on Vicuna (about three questions on a 150-question benchmark, within one standard error) and a 7.4-point gap on Llama-3B. The paper never runs the decisive control: shared prompts plus per-sample random prompts with the same token count and MLP budget. Without that, the IP gains on the larger benchmarks are fully explainable by extra tunable capacity, not by instruction conditioning. Given that the claimed novelty over ordinary visual prompt tuning is exactly the instruction conditioning, this is a load-bearing gap. The authors also report single-run numbers, no error bars, and no code or weights, which makes the MMVP result hard to judge.\n\nThat said, the paper is not fatally flawed. The improvement over LLaVA-1.5 is consistent and the ablations are directionally sensible. If the authors add the random-prompt control, run repeated seeds, and correct the MMVP sentence, the central claim—that instruction-conditioned visual prompts help—could well hold up. As it stands, the paper is a credible incremental contribution to MLLM visual encoding, worth serious referee time but not ready to be taken at face value.","headline":"Sensible, mostly solid empirical paper; the instruction-conditioning mechanism is under-supported because the decisive random-prompt control is missing, and one table-text mismatch needs fixing.","tokens_in":20887,"tokens_out":4574,"would_cite":false,"duration_ms":42858,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Turning the user's question into visual prompts that are injected early into a frozen vision encoder improves multimodal LLM performance, especially on vision-centric benchmarks.","keywords":["multimodal large language models","instruction-guided visual prompting","visual prompt tuning","vision-centric benchmarks","multi-turn visual token pruning","Amblyopia","frozen vision encoder"],"falsifier":"Run the same training recipe with the instruction prompt replaced by a fixed, question-independent prompt of the same length and parameter budget; if VQAv2, GQA, and MMVP gains match the full model, the content of the question is not doing the work, while if the gains collapse, cross-modal prompt steering is what carries the result.","tokens_in":19648,"feed_emoji":"👁️","tokens_out":8564,"duration_ms":76970,"temperature":0.7,"pith_summary":"Most multimodal LLMs run the image through a frozen vision encoder and only let the user's question influence the language decoder, so the visual features can lose exactly the details the question asks about. This paper proposes Panther, which instead turns the question into visual prompts that are prepended to image tokens inside the vision encoder, so the question shapes what the encoder attends to from early layers. On the standard LLaVA-1.5 setup with Vicuna-7B, the method raises VQAv2 from 78.5 to 80.8, GQA from 62.0 to 65.2, and MMVP from 24.7 to 30.0; with Llama 3-8B it raises CV-Bench 3D from 61.9 to 68.9. The paper argues that instruction-guided visual prompting is an effective and largely orthogonal improvement that works across different vision encoders and language decoders without retraining the visual backbone.","feed_headline":"Questions as visual prompts sharpen multimodal LLM sight","feed_subtitle":"Injecting the user's question into the frozen vision encoder lifts MMVP from 24.7 to 30.0 and VQAv2 to 80.8.","key_machinery":"The load-bearing object is the instruction-aware visual prompt, written as $\\Gamma_{T\\to I}[F_T(q_k)]$: the current question $q_k$ is passed through a frozen CLIP text encoder and a two-layer MLP projector, then the resulting vectors are prepended to the visual tokens entering the frozen vision Transformer, alongside 24 shared learnable prompts. The shared prompts give the model a stable training slot, while the projected instruction prompts carry question content into the visual self-attention that computes the features the LLM later reads. To keep multi-turn training within the 2048-token budget, Panther-Bridge prunes per-turn visual tokens whose cosine similarity to the previous turn's tokens exceeds a threshold of $\\tau = 0.95$.","core_discovery":"The paper's central claim is that the Amblyopia problem of encoder-decoder MLLMs, where visual details relevant to the user's instruction are filtered out before text and image meet, can be mitigated by injecting instruction information into the vision encoder itself. The concrete mechanism is an instruction-aware visual prompt: a frozen text encoder encodes the current question, a small MLP projects those embeddings into the visual token space, and the resulting prompt tokens are prepended to the image patch tokens inside a frozen ViT. Because the prompt is regenerated for every turn, the same image yields different visual features for different questions. The paper reports that this change, combined with a token-pruning bridge and interleaved decoder training, consistently beats the base model on general question answering and by larger margins on vision-centric benchmarks.","pith_inferences":["Beyond the paper's experiments, earlier instruction conditioning may also help Q-former-style connectors that currently cross-attend to visual features only at the final encoder layer, since the paper's result suggests that late conditioning is the failure point.","A testable extension would be to regenerate prompts from multi-turn conversational context rather than only the current question, since follow-up questions such as 'no, the other one' depend on earlier answers.","The token-pruning idea suggests a cheap way to decide when a follow-up question needs new visual information: turns whose prompted visual tokens are nearly identical to the previous turn's could skip re-encoding entirely, which the paper does not explore."],"forward_implications":["Vision-centric error patterns caused by overly diffuse attention can be reduced by moving instruction fusion earlier, without changing image resolution or unfreezing the vision backbone.","The same image can legitimately produce different visual tokens for different questions, and those differences matter for answering correctly in multi-turn conversations.","Cosine-similarity pruning of per-turn visual tokens can cut multi-turn training cost while retaining most of the benefit of instruction-aware encoding.","The improvement transfers across vision encoders such as CLIP, DINOv2, and SigLIP and across LLMs such as OpenLlama-3B, Vicuna-7B, Mistral-7B, and Llama 3-8B, so the method is orthogonal to those architectural choices."],"supporting_citations":[{"why":"Supplies the LLaVA-1.5 baseline model, the two-stage training recipe, and the 558K/665K datasets on which Panther is evaluated.","marker":"[45]"},{"why":"Provides the frozen CLIP ViT-L/14 at 336 pixels that serves as the vision backbone receiving the instruction prompts.","marker":"[61]"},{"why":"Provides the CLIP text encoder and the cross-modal alignment that the instruction-to-visual prompt projection relies on.","marker":"[62]"},{"why":"Supplies the MMVP benchmark that exposes CLIP's visual failures and gives the largest reported gains for Panther.","marker":"[71]"},{"why":"Supplies the CV-Bench 2D/3D benchmark and the Cambrian-dev multi-encoder baseline that Panther is compared against on vision-centric tasks.","marker":"[70]"},{"why":"Supplies the closest visual-prompting MLLM baseline, which uses visual prompts but not text-conditioned ones, making it the key contrast.","marker":"[43]"},{"why":"Establishes the deep visual prompt tuning mechanism that Panther extends from task-specific prompt vectors to instruction-conditioned prompts.","marker":"[24]"},{"why":"Serves as the alternative text encoder in the ablation, and its weaker MMVP result supports the claim that CLIP alignment is what makes the instruction prompts work.","marker":"[11]"}],"fun_headline_variants":["Panther injects user questions into the vision encoder","Ask the vision encoder: Panther's instruction prompts","Panther: question-aware visual tokens lift vision benchmarks","MLLM Amblyopia cured by instruction-cond visual prompts","Turn user text into vision prompts for sharper MLLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the assumption that a CLIP text embedding projected into visual space and prepended to a frozen ViT actually steers the visual features the LLM sees, rather than merely adding trainable capacity that would help regardless of the question text.","fun_headline_variants_meta":{"raw":{"variants":["Panther injects user questions into the vision encoder","Ask the vision encoder: Panther's instruction prompts","Panther: question-aware visual tokens lift vision benchmarks","MLLM Amblyopia cured by instruction-cond visual prompts","Turn user text into vision prompts for sharper MLLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2661,"prompt_tokens":933,"completion_tokens":1728,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1649}},"tokens_in":549,"tokens_out":1728,"duration_ms":12398,"temperature":1.0,"reasoning_tokens":1649,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:45:19.030728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same training recipe with the instruction prompt replaced by a fixed, question-independent prompt of the same length and parameter budget; if VQAv2, GQA, and MMVP gains match the full model, the content of the question is not doing the work, while if the gains collapse, cross-modal prompt steering is what carries the result.","supporting_citations":[{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA-1.5 baseline model, the two-stage training recipe, and the 558K/665K datasets on which Panther is evaluated."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Provides the frozen CLIP ViT-L/14 at 336 pixels that serves as the vision backbone receiving the instruction prompts."},{"cited_title":"Eyes wide shut? exploring the visual shortcomings of multimodal llms","cited_arxiv_id":null,"evidence_quote":"Supplies the MMVP benchmark that exposes CLIP's visual failures and gives the largest reported gains for Panther."}],"review_version":1}