{"id":"0674b2a1-6dc0-4471-8f16-639f85920245","arxiv_id":"2501.08443","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An instruction-guided aggregator that dynamically weights visual features from different encoder layers improves LLaVA-1.5 on most tested multimodal benchmarks.","lead":"This paper shows that different layers of a vision encoder carry different task-relevant information in large vision-language models, and proposes a module that uses the text instruction to weigh those layers adaptively. The method improves a standard 7B model on several multimodal benchmarks, suggesting a cheap way to get more from a single frozen image encoder.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed instruction-guided advantage is confounded: the Table 6 ablation removes the entire aggregator, so added capacity or visual-only conditioning could explain the gain; no control isolates the instruction input.","rationale":"The reader's weakest assumption identified the MPNet instruction embedding as potentially non-discriminative, which is related but not identical to my concern. My concern is more specific and more directly tied to the experimental design: the ablation in Table 6 does not isolate the instruction input from the allocator's parameters and visual conditioning. Even if the MPNet embedding is informative, the paper provides no evidence that the model actually uses it; the weight allocator could learn a task-agnostic or visually-conditioned weighting scheme that happens to improve performance. This is a load-bearing gap because the paper's central contribution is the instruction-guided mechanism, not merely multi-layer fusion or an extra module. The 'w/o aggregator' row in Table 6 is the key comparison, but it conflates two factors. A reader cannot tell from the current data whether the claimed 'instruction-guided fusion' works as advertised. The RealWorldQA anomaly further weakens confidence in the reported numbers. I do not think this rises to REJECT, because the paper is otherwise well-structured and the proposed approach is plausible; however, the authors should either provide the missing control or soften the central claim. Since the reader's CONDITIONAL verdict already requires additional evidence, my assessment does not change that verdict, so I recommend UNCHANGED.","tokens_in":16541,"tokens_out":5244,"duration_ms":50934,"concrete_test":"Run a controlled ablation on the 10 benchmarks in Table 4: keep the same training pipeline and the same weight allocator architecture, but replace the MPNet sentence embedding with a fixed constant vector (or a single learnable vector shared across all samples), so the allocator cannot use the instruction. If this no-instruction variant stays within, say, 0.5 average points of the full method, the instruction signal is not load-bearing. Also, re-run the RealWorldQA evaluation for the full model; if the score is 48.0, investigate and explain the discrepancy with the 56.0 in the 'w/o aggregator' row and 55.8 baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that textual instructions causally drive the dynamic weights in the instruction-guided vision aggregator. The only direct evidence is Table 6, where 'w/o aggregator' replaces the full module with simple average pooling. This comparison differs in two ways at once: (1) the instruction embedding is removed, and (2) the weight allocator's cross-attention parameters are removed. The gain could therefore come entirely from the extra capacity of the allocator, or from conditioning on the groupwise CLS features (keys/values) rather than on the text. The paper reports average per-benchmark weights in Table 7, but these are not per-instance and could be driven by image content, not by the instruction. The authors' own conclusion admits reliance on 'clear and high-quality textual instructions,' but this is a limitation of the input, not a test of whether the instruction is used. Additionally, Table 6 lists RealWorldQA 48.0 for the full method versus 56.0 for 'w/o aggregator' and 55.8 for the baseline; if this is not a typographical error, it directly contradicts the claim of consistent gains and needs explanation. Without an instruction-ablated control, the headline conclusion 'validating the effectiveness of instruction-guided fusion' is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the use of hierarchical visual features from a CLIP-ViT encoder in the LLaVA-1.5 framework. It first presents a systematic analysis across 18 benchmarks in 6 task categories, training models with features from different single layers or pairwise layer combinations. The findings indicate task-dependent complementarity and suboptimality of static fusion. The authors then propose an instruction-guided vision aggregator that divides the 24 encoder layers into four groups, average-pools within each group, computes a weighted sum using weights produced by a cross-attention allocator conditioned on an MPNet sentence embedding of the text instruction, and concatenates the result with the penultimate-layer features. They evaluate the integrated model on 10 benchmarks against LLaVA-1.5, DenseConnector, MMFuser, and several 7B-scale LVLMs, reporting best scores on 8 of 10 benchmarks, and analyze the learned weight distributions per task category.","tokens_in":16803,"tokens_out":8895,"duration_ms":73559,"significance":"If the findings hold, the paper provides a useful layer-wise analysis for LVLMs and a lightweight, token-efficient way to adapt visual feature fusion to task instructions. The study is comprehensive in its benchmark coverage, and the method is clearly described and reproducible in principle. However, the current evidence does not yet establish the central claim that the instruction signal is causally responsible for the gains: the key ablation removes the entire aggregator rather than just the instruction conditioning, and several reported numbers are internally inconsistent. These issues need to be resolved before the contribution can be fully assessed.","major_comments":[{"comment":"The 'w/o aggregator' ablation replaces the entire instruction-guided vision aggregator with average pooling, simultaneously removing the sentence embedding and the weight allocator's parameters. This comparison cannot attribute the gain to the text instruction; the improvement could come purely from the allocator's added capacity or from conditioning on the visual group CLS features. Please add a control that keeps the aggregator architecture but removes the instruction information (e.g., a constant or random embedding) and a visual-only variant to isolate the role of the instruction.","section":"Section 5.5, Table 6"},{"comment":"The tables contain unexplained anomalous values. In Table 3, the 'Decrease-then-increase' row reports MME-p = 472.1, while all other configurations are around 1470; this is likely a typo that needs correction. In Table 6, the full method scores 48.0 on RealWorldQA, whereas the three ablated variants score 56.4, 56.0, and 55.8; the text does not mention this large drop, and it contradicts the narrative of consistent improvements. Please correct or explain these values.","section":"Section 3.3, Table 3; Section 5.5, Table 6"},{"comment":"The architecture and hyperparameters (number of groups, group boundaries, concatenation with the penultimate layer, entropy coefficient lambda=0.02) were selected using the same 18-benchmark evaluation set that substantially overlaps with the 10 benchmarks in the final comparison. This selection-on-evaluation protocol means the reported gains are inflated relative to a fair held-out evaluation. Please evaluate the method on a separate set of benchmarks or provide a clear protocol that avoids using the test benchmarks for model selection.","section":"Sections 3.1-3.3 and 5.2-5.3"},{"comment":"All results are reported as single runs without variance or significance information. Many differences are small (e.g., GQA 63.1 vs 63.8, MMB-en 66.9 vs 67.5), so the claim of being best on 8 out of 10 benchmarks is not statistically supported. Please report multiple runs or at least specify the number of seeds and provide error bars.","section":"Tables 4-6"}],"minor_comments":[{"comment":"There are typos: 'Insturction' in the keywords, 'thier' and 'ecoder' in Section 1, and inconsistent spacing in 'L VLMs' throughout. Please proofread.","section":"Keywords, Section 1"},{"comment":"The entropy regularization term uses sum w_k log w_k, which is the negative entropy. The sign convention should be stated explicitly so that readers understand that minimizing this term with positive lambda encourages balanced weights.","section":"Eq. (8)"},{"comment":"Many entries are marked with '-'. Please clarify whether these are not evaluated, not reported, or not applicable, and add footnote definitions.","section":"Table 5"},{"comment":"The claim that the low-level group receives a 'notable' weight increase in Fine-Grained tasks is supported by only a small numerical difference (0.20 overall vs 0.17-0.18 in other categories). Consider toning down the language or reporting per-benchmark variability.","section":"Section 5.6, Table 7"},{"comment":"The conclusion states the method relies on clear and high-quality instructions, but no experiment tests sensitivity to instruction quality. Adding a small study with corrupted or ambiguous instructions would strengthen the discussion.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of Pattern Recognition and the paper's idea is interesting. However, the evaluation protocol and several numerical inconsistencies need careful attention. I recommend asking the authors to provide corrected tables, an instruction-ablated control, and a clarification of the benchmark selection protocol. It would also help to see code or checkpoints to verify the reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on arXiv 2501.08443. The paper has two parts: a systematic study of how different CLIP-ViT layers affect LLaVA-1.5 performance across 18 benchmarks, and a lightweight 'instruction-guided vision aggregator' that weights four visual groups based on the text instruction. The first part is genuinely useful. They train separate models for layers 6, 12, 18, 23, 24, show that different layers win on different tasks, that combinations beat any single layer, and that static weight vectors are suboptimal. That's a solid, reproducible empirical contribution. The second part is plausible but under-supported in one specific way.\n\nThe aggregator itself is simple and well-motivated: encode the instruction with MPNet, use cross-attention to compute group weights, weighted-sum the patch features, concatenate with the penultimate layer. No extra visual tokens, which is nice. On 10 benchmarks it beats LLaVA-1.5 and the two static fusion baselines on 8 of them, and the comparisons with other 7B models are fine as context.\n\nWhat worries me is the ablation. In Table 6, 'w/o aggregator' replaces the whole module with average pooling. That removes the instruction embedding and the allocator's cross-attention parameters at the same time. So the gain could come from extra capacity, or from conditioning on the groupwise CLS features (the keys/values in the cross-attention), rather than from the text instruction itself. There's no control where the allocator is trained with a fixed or shuffled instruction embedding. Without that, the headline claim that instruction-guided fusion is what helps is not established. The weight allocation analysis in Table 7 shows different average weights per task, which is suggestive, but those are averages across many instances and could be driven by image content.\n\nThere are also a couple of table problems that need fixing: Table 3 lists MME-p 472.1 for the 'Decrease-then-increase' row, which is obviously a typo (should be ~1472). And in Table 6, the full method scores 48.0 on RealWorldQA while 'w/o aggregator' scores 56.0 and the baseline is 55.8. That's an 8-point drop that contradicts the pattern everywhere else, and the text doesn't mention it. Either a typo or a real anomaly, but it needs explanation. No error bars anywhere, and the design choices (group count, λ, penultimate concatenation) were made on these same benchmarks, so a held-out evaluation would be stronger.\n\nOverall: the empirical layer analysis is worth citing, and the module idea is reasonable. But the specific claim that textual instructions drive the improvement needs a proper control. I'd send it to peer review—it has enough substance—but the authors should fix the tables and add that experiment (or at least acknowledge the confound).","headline":"Solid empirical layer-wise analysis plus a plausible fusion module, but the instruction-guidance claim is not isolated by the ablation.","tokens_in":17355,"tokens_out":6602,"would_cite":true,"duration_ms":53292,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that dynamically weighting vision-encoder layers by the text instruction improves large vision-language models, beating a 7B baseline and task-agnostic fusion on 8 of 10 benchmarks without adding visual tokens.","keywords":["large vision-language models","instruction-guided fusion","hierarchical visual features","multi-layer feature fusion","visual instruction tuning","fine-grained perception","weight allocation","CLIP-ViT"],"falsifier":"A reader could settle the claim by taking the trained weight allocator and swapping the instruction embedding for a constant, randomly shuffled, or zero vector while freezing everything else; if benchmark scores barely move, the guidance from text is not carrying the effect, and the gains are attributable to the extra parameters and multi-layer inputs. A complementary check would compare the learned per-task weight patterns against fixed task-specific weights learned from the training set.","tokens_in":16341,"feed_emoji":"🎯","tokens_out":5472,"duration_ms":49824,"temperature":0.7,"pith_summary":"This paper argues that large vision-language models leave performance on the table by using only the final layers of the vision encoder, and that the usual fix of mixing all layer features uniformly is also wrong because different tasks need different visual levels. It claims that a lightweight module can read the text instruction and use it to assign fusion weights to groups of encoder layers, emphasizing low-level details for fine-grained perception and mid-to-high-level details for semantic tasks. Integrated into a standard 7B vision-language model, this instruction-guided aggregator is reported to improve accuracy on most of ten benchmarks compared with the baseline and with existing task-agnostic fusion methods, without increasing the number of visual tokens. The paper also presents a systematic study of 18 benchmarks showing that layer contributions are complementary and task-dependent.","feed_headline":"Instruction-aware vision layer fusion wins 8 of 10 benchmarks","feed_subtitle":"A lightweight module reads the question and reweights shallow and deep visual features, lifting accuracy with no extra visual tokens.","key_machinery":"The instruction-guided vision aggregator is the load-bearing module. It divides the 24 CLIP-ViT layers into four groups (low, low-to-mid, mid-to-high, high), averages the class and patch tokens inside each group, embeds the text instruction with a frozen sentence-embedding model, and feeds the embedding as a query through four cross-attention transformer blocks whose softmax head outputs one weight per group. The weights recombine the pooled patch features by weighted sum, and the result is concatenated with the penultimate layer's features so no visual-token count is added. An entropy-based auxiliary loss with coefficient 0.02 keeps the weight allocator from collapsing onto a single group.","core_discovery":"The central claim is that task-guided weighting of hierarchical visual features, rather than any fixed recipe, is what unlocks the complementary information in a vision encoder. The authors show that single-layer choices each excel in different task categories, that pairing the penultimate layer with another layer helps, and that static weight patterns underperform a dynamic scheme. They then construct the instruction-guided vision aggregator, which pools the encoder's 24 layers into four visual groups, embeds the user instruction with a frozen sentence-embedding model, and uses a cross-attention weight allocator to produce normalized group weights; the group-pooled patch features are summed with those weights and concatenated with the penultimate layer features. On 10 benchmarks the method scores best on 8 of them against the LLaVA-1.5 baseline and two task-agnostic fusion methods, and an analysis of the learned weights shows mid-to-high groups carrying semantic tasks while low groups gain weight in fine-grained benchmarks.","pith_inferences":["A direct test the paper does not run is whether the aggregator's weights are more predictive of the task than the instruction's exact wording; using the same question paraphrased differently should produce similar weight patterns if the module is capturing task semantics.","The weight-balancing loss suggests the raw training signal pushes the allocator toward one group; without it the model may rely on mid-to-high features even when low-level detail is needed, so the entropy term is arguably doing part of the task adaptation.","The method could be plugged into high-resolution or multi-scale vision pipelines, where layer-group pooling plus instruction routing may reduce the cost of processing many patches.","The reliance on clear instructions implies the approach may degrade on vague or multi-intent queries, and a robust version would likely need to fold visual content into the routing decision as well."],"forward_implications":["A vision-language model can improve fine-grained and reasoning performance without paying the cost of extra visual tokens, since the aggregator recombines existing layer features.","Task categories carry recognizable weight signatures, such as mid-to-high dominance for Chart and OCR and a higher low-group weight for fine-grained vision, which could inform which encoder layers to keep or fine-tune.","The same instruction-guided aggregation principle should transfer to other vision encoders and larger language models, making the module a drop-in addition to existing pipelines.","Frozen instruction embeddings suffice for routing, so the added trainable machinery is small relative to the language model."],"supporting_citations":[{"why":"Provides the LLaVA-1.5 architecture, training pipeline, and baseline numbers that the method is built on and compared with.","marker":"[1]"},{"why":"Supplies the CLIP-ViT vision encoder whose 24 layers are grouped and fused.","marker":"[8]"},{"why":"Task-agnostic multi-layer fusion baseline that concatenates features from multiple encoder layers and is compared against.","marker":"[14]"},{"why":"Task-agnostic fusion baseline that uses deep features as queries to extract shallow details, compared against.","marker":"[15]"},{"why":"Supplies the 15-benchmark evaluation set across task categories that the study extends.","marker":"[18]"},{"why":"Provides the frozen sentence-embedding model used to encode instructions for the weight allocator.","marker":"[36]"},{"why":"Adds V*-bench as a fine-grained evaluation benchmark requiring detailed visual search.","marker":"[19]"},{"why":"Adds HR-bench as a high-resolution fine-grained perception benchmark.","marker":"[20]"},{"why":"Adds POPE as an object hallucination evaluation benchmark.","marker":"[21]"}],"fun_headline_variants":["Instruction-guided visual layer fusion wins 8 of 10","Task-aware weighting of visual layers boosts LVLM benchmarks","Ask the text: visual layer weights adapt per instruction","No extra visual tokens: dynamic fusion outperforms static"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the frozen sentence-embedding model produces an instruction representation informative enough for the weight allocator to choose genuinely task-appropriate layer weights; if that signal were weak, the reported gains could come from the extra parameters and multi-layer inputs rather than from instruction guidance.","fun_headline_variants_meta":{"raw":{"variants":["Instruction-guided visual layer fusion wins 8 of 10","Task-aware weighting of visual layers boosts LVLM benchmarks","Ask the text: visual layer weights adapt per instruction","No extra visual tokens: dynamic fusion outperforms static"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1381,"prompt_tokens":948,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":564,"tokens_out":433,"duration_ms":5913,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:34.700484+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the claim by taking the trained weight allocator and swapping the instruction embedding for a constant, randomly shuffled, or zero vector while freezing everything else; if benchmark scores barely move, the guidance from text is not carrying the effect, and the gains are attributable to the extra parameters and multi-layer inputs. A complementary check would compare the learned per-task weight patterns against fixed task-specific weights learned from the training set.","supporting_citations":[{"cited_title":"Mpnet: Masked 34 and permuted pre-training for language understanding","cited_arxiv_id":null,"evidence_quote":"Provides the frozen sentence-embedding model used to encode instructions for the weight allocator."},{"cited_title":"V?: Guided visual search as a core mechanism in multimodal llms","cited_arxiv_id":null,"evidence_quote":"Adds V*-bench as a fine-grained evaluation benchmark requiring detailed visual search."}],"review_version":1}