{"id":"13e0d6a7-8636-49ee-87e0-2ad9cfdc62ea","arxiv_id":"2608.01635","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"SSVAL uses visual anchor prompts plus spatial and frequency-domain alignment losses to reduce visual information loss in MLLMs and improve fine-grained perception benchmarks.","lead":"This paper introduces SSVAL, a training method that adds small learnable anchor prompts to a multimodal language model so visual details are retained as image tokens pass through the model's layers. The authors report large gains on fine-grained spatial perception benchmarks, notably MMVP and CV-Bench 2D, at a small inference cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central evidence for the anchor mechanism is the Fig. 1(b) cosine-similarity proxy, which is never validated against task-relevant visual content; deeper-layer re-representation and the Eq. 1 prompt-residual shortcut can both inflate this metric without preserving usable visual information.","rationale":"The reader's CONDITIONAL verdict identifies the unvalidated cosine-similarity metric as the weakest assumption, and I agree that this is the most load-bearing point. The paper proposes a novel and well-ablated method, and the benchmark gains, especially on perception-heavy tasks, are real and non-trivial even if the mechanism were not as described. However, the central explanatory claim—that prompts serve as 'visual anchors' that 'preserve visual information'—is supported almost exclusively by the similarity curves and by qualitative visualisations. The similarity proxy has a clear alternative explanation: LLMs re-represent tokens in deeper layers, and the VAPI residual of Eq. 1 can make representations more similar to the input without improving task-relevant content. The VAPI+SpeRA collapse in Table 4 further indicates that the anchor mechanism is not robust, which is hard to reconcile with the claim that prompts are inherently stable references. A probe-based test would settle whether the metric tracks information that matters; if it fails, the paper reduces to an empirical method whose gains may come from added capacity or feature injection. This does not change the reader's verdict, because the reader already made the verdict CONDITIONAL on exactly this kind of validation. I do not see a reason to move to ACCEPT or REJECT; the paper has independent support in its systematic ablations and low inference overhead, and the concerns are addressable with an additional experiment.","tokens_in":18016,"tokens_out":4070,"duration_ms":41320,"concrete_test":"Train lightweight linear probes on the visual-token representations of Baseline, VIRAL, and SSVAL at layers 0, 7, 14, 21, and 28 to predict task-relevant attributes (e.g., MMVP pair attributes, object presence, spatial relations). Compare probe accuracy with the cosine-similarity curves from Fig. 1(b): if SSVAL's higher deep-layer cosine similarity is accompanied by higher probe accuracy, the proxy is validated; if the two diverge, the 'preserves visual information' claim is unsupported. As a control, inject a fixed random or input-mean vector into all visual tokens at each layer (a trivial 'anchor'), and measure both cosine similarity and benchmark performance; if this control raises cosine similarity without improving benchmarks, Fig. 1(b) is confounded by the residual shortcut.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's mechanistic claim—that VAPI prompts act as stable visual anchors that preserve visual information across LLM layers—rests almost entirely on the cosine-similarity-to-input curves in Fig. 1(b) and the qualitative PCA/attention figures. This proxy is not validated: a deep transformer legitimately re-represents tokens, so a drop in cosine similarity to the input embedding does not establish information loss. Conversely, SSVAL's Eq. 1 adds a gated residual from fixed prompts to the input visual tokens; this can mechanically keep representations closer to the input direction, raising cosine similarity without preserving task-relevant content. The severe fragility in Table 4 (VAPI+SpeRA: CV-Bench2D 47.71%, MMVP 9.33%, far below Baseline) shows the anchor mechanism is not inherently stabilising, and the paper provides no explanation for this collapse. Additionally, SSVAL adds 5.25M inference parameters over VIRAL (Table 7), and no parameter-matched control is given, so the benchmark gains are compatible with a generic capacity or feature-injection effect rather than the anchor mechanism. The external benchmark results are valuable regardless, but the central 'visual degradation mitigation' claim is not established until the similarity metric is shown to track information that actually matters for downstream perception.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses visual information degradation in multimodal large language models (MLLMs). It first measures a phenomenon it calls visual representation deviation: the cosine similarity between input visual token representations and their intermediate LLM-layer representations drops sharply with depth. It then proposes SSVAL, which combines Visual Anchor Prompt Injection (VAPI) with spatial and spectral representation alignment losses (SpaRA and SpeRA). During training, VAPI prompts absorb features from a frozen vision foundation model (VFM) through gated cross-attention; during inference, the prompts are kept fixed and are claimed to act as stable visual anchors that preserve visual information across layers. The method is evaluated on six multimodal benchmarks with two vision encoders and two LLM scales, and ablations study prompt count, VFM choice, component contribution, multiscale configurations, and frequency bands.","tokens_in":18364,"tokens_out":3493,"duration_ms":31395,"significance":"If the mechanism claim were established, this would be a valuable contribution: the paper identifies a practically relevant phenomenon (representation deviation), shows a counter-intuitive limitation of direct VFM alignment, and reports sizeable gains on perception-heavy benchmarks such as MMVP (+7.86) and CV-Bench 2D (+6.47) over a strong baseline. The external benchmark evaluation is not circular, the ablations are fairly thorough, and the authors state that code will be released. However, the central mechanistic claim rests on a cosine-similarity proxy that is never validated against task-relevant visual information, and the method introduces additional inference parameters without a parameter-matched control. The benchmark results are interesting regardless, but the paper's headline explanation for those results is not yet supported.","major_comments":[{"comment":"The paper's central evidence for both the phenomenon of visual representation deviation and the mitigating effect of SSVAL is the cosine similarity between input visual token representations and deeper-layer representations. This metric is not validated against task-relevant visual information: deep transformer layers legitimately re-represent tokens, and a drop in cosine similarity to the input embedding does not by itself imply information loss. Conversely, the gated residual in Eq. (1) adds a prompt-derived term to every visual token and can mechanically keep representations closer to the input direction, inflating this similarity without preserving usable visual content. To support the mechanism story, the authors should validate the proxy with task-level probes (e.g., decoding object attributes or spatial relations from intermediate layers) and include a control where the same gated residual uses random or non-VFM prompts.","section":"Table 4"},{"comment":"The configuration VAPI+SpeRA collapses to 47.71% on CV-Bench 2D and 9.33% on MMVP, far below the Baseline (58.97% and 33.47%). This result is acknowledged in passing but never explained. It directly undercuts the claim that visual anchor prompts are an inherently stabilising mechanism, since adding the spectral loss to VAPI makes the anchors catastrophically worse. The paper should analyse the failure mode (e.g., what happens to the prompt attention maps or representation similarity in this configuration) and clarify why the interaction with SpaRA is necessary. Without this, the full method appears to depend on a delicate and unexplained balance between the losses.","section":"Table 4"},{"comment":"SSVAL adds 5.25M parameters and 3.65 GFLOPs at inference relative to Baseline, while VIRAL adds zero parameters. The comparison against VIRAL is therefore not parameter-matched. The benchmark gains could stem from the additional capacity or a generic feature-injection effect rather than from the proposed anchor mechanism. A parameter-matched control is needed, for example a model with the same gated cross-attention module but with prompts that are randomly initialised and not trained with VFM knowledge, or a residual connection of matched size. Table 4's +VAPI row is a useful start, but it does not separate the VFM-knowledge component from the extra capacity component.","section":"Section 4.5, Tables 7-8"},{"comment":"Several hyperparameters in the proposed losses are either unreported or selected on the same benchmarks used for final evaluation. Specifically, alpha and beta in Eq. (3), the numerical values of gamma_LF and gamma_HF in Eq. (8), the scale weights w_k, the frequency-mask radius threshold r, and the intermediate alignment layers are not given in the main text. Table 3 selects the teacher VFM (DINOv2-B) by comparing results on the same six benchmarks, so the reported gains risk selection on the test set. The authors should report all hyperparameter values and either use a validation split for model selection or show that the results are stable across a range of choices.","section":"Section 3.3 and Section 4.1"},{"comment":"No multiple-seed runs or error bars are reported. Many of the claimed improvements are small (e.g., MMStar +1.73, POPE +0.22, MM-Bench +1.11 in the CLIP+Qwen2.5-7B configuration), and instruction-tuning pipelines can have non-trivial run-to-run variance. The headline perception benchmarks show larger deltas, but the ablation conclusions (e.g., prompt quantity, component contributions) frequently rely on differences of one to three points. Reporting mean and standard deviation over at least three seeds, or at least for the main comparison and key ablations, is necessary to assess whether the differences are meaningful.","section":"All experiments"}],"minor_comments":[{"comment":"The sentence 'Code are available on our project page' has a subject-verb agreement error; it should be 'Code is available'.","section":"Abstract"},{"comment":"The MME value for the +SpaRA row is given as '1828.21%', which includes a stray percent sign; the other MME entries are unadorned numbers.","section":"Table 4"},{"comment":"There are several typographical issues: 'SSV AL' appears with an unintended space in Figures 1(a) and 7, and in Figure 7 the sub-captions for VIRAL and SSVAL both use the label '(b)', which is confusing.","section":"Figures 1 and 7"},{"comment":"The main text states 'We use 4 visual anchor prompts' without explaining how this number was chosen; Table 2 shows that 4 prompts are best on CV-Bench 2D but 8 prompts are best on MMVP (42.67% vs 41.33%). A sentence justifying the default choice beyond 'considering both accuracy and efficiency' would improve reproducibility.","section":"Section 4.1"},{"comment":"The limitations section is helpful, but it does not mention the fragility shown in Table 4 (VAPI+SpeRA collapse) or the unvalidated similarity metric, which are the two issues most relevant to the paper's central claim.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and addresses an interesting problem, but the central mechanism claim is not yet supported by the evidence. The benchmark results are valuable and should be preserved, but the authors need to validate the cosine-similarity proxy, add parameter-matched and random-prompt controls, explain the VAPI+SpeRA collapse, report all loss hyperparameters, and provide error bars. I would also flag the teacher-VFM selection on the test benchmarks as a potential overfitting concern that should be addressed with a validation split or sensitivity analysis. The scope of the required additions is substantial but feasible, so major revision seems appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a well-ablated methods paper with real benchmark gains, and the anchor-prompt idea is new, but the mechanism claim rests on a cosine-similarity proxy that is never validated against task-relevant content. If you are reading it for the benchmark numbers, it largely holds up. If you are reading it for the story about 'visual degradation,' treat that story as unproven.\n\nWhat is actually new: the paper shows that directly aligning intermediate LLM features with a VFM (VIRAL-style) improves visual semantics but does not stop the layer-wise drift, and it proposes a different fix—learnable prompt vectors fused into the visual tokens via gated cross-attention, plus spatial and frequency-domain alignment losses. That combination is not in the prior work I know. The experiments are consistent: SSVAL beats VIRAL and the LLaVA baseline on almost every setting, with the biggest gains exactly where you would want them, on CV-Bench2D and MMVP. The ablations are reasonably thorough (prompt count, teacher VFM, multi-scale, frequency bands), and the efficiency analysis is honest about the +5.25M inference parameters and +0.04% FLOPs.\n\nThe soft spots are real but fixable. The main one is the metric. The paper operationalizes 'visual information degradation' as a drop in cosine similarity between the input visual tokens and deeper-layer representations. A deep transformer legitimately re-represents tokens, so a similarity drop by itself does not mean the task-relevant information is gone. And the gated residual in Eq. 1 can mechanically keep the representations near the input direction, inflating the similarity without preserving content. The paper never shows that this cosine similarity tracks anything that matters for the downstream tasks. That is a load-bearing gap for the 'anchor' story. Second, no multiple-seed runs and no error bars, and the teacher VFM and hyperparameters are selected on the same benchmarks. Third, there is no parameter-matched baseline: SSVAL adds 5.25M parameters and 3.65 GFLOPs, so part of the gain could be generic capacity. Fourth, the VAPI+SpeRA ablation collapses (MMVP 9.33%), which is a red flag that the anchor mechanism is not inherently stable. The paper does not explain that collapse.\n\nNone of this kills the paper. The benchmark results are external and, taken at face value, they are solid. The honest limitations section helps. I would send this to review, and in the review I would ask the authors to validate the cosine-similarity metric, add seeds, provide a parameter-matched control, and explain the VAPI+SpeRA failure. Then the mechanism story would be credible.","headline":"A solid, well-ablated methods paper with real benchmark gains; the anchor-prompt idea is new, but the central mechanism story rests on an unvalidated cosine-similarity proxy and lacks a parameter-matched control.","tokens_in":18883,"tokens_out":3402,"would_cite":true,"duration_ms":28566,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Visual anchor prompts can stop multimodal LLMs from losing visual information as it passes through deep layers.","keywords":["multimodal large language models","visual representation deviation","visual anchor prompts","spatial-spectral alignment","frequency-domain supervision","vision foundation models","fine-grained visual perception"],"falsifier":"Measure task-relevant visual information at each LLM layer of a baseline MLLM with a linear probe (or an information-theoretic estimate) trained to predict object presence, position, and count from visual-token representations; if probe accuracy stays high in deep layers even as cosine similarity to the input drops, the paper's diagnosis of information degradation is not supported.","tokens_in":17789,"feed_emoji":"👁️","tokens_out":7445,"duration_ms":60188,"temperature":0.7,"pith_summary":"The paper argues that after visual instruction tuning, the internal representations of visual tokens in a multimodal large language model drift away from their initial states as they pass through deeper layers, and this drift is a major cause of poor visual perception. It claims that aligning intermediate representations to an external vision foundation model improves what the features encode but does not stop the drift. To fix this, it introduces Visual Anchor Prompt Injection (VAPI), prompts trained to absorb VFM knowledge and then re-inject stable references at inference, plus spatial and frequency-domain alignment losses during training. If correct, the method gives a lightweight way to preserve visual information through the LLM, with large gains on perception-heavy benchmarks such as +7.86 on MMVP and +6.47 on CV-Bench 2D.","feed_headline":"Prompt anchors stop MLLMs from losing visual details in deep layers","feed_subtitle":"Multimodal models keep image information through deep layers, gaining up to 7.9 points on visual perception benchmarks.","key_machinery":"The load-bearing object is the visual anchor prompt set $P \\in \\mathbb{R}^{N_p \\times D}$: during training the prompts are summed with stride-sampled, projection-mapped features from a frozen vision foundation model, then fused into the visual tokens $Z_v$ by multi-head cross-attention with a gated residual $Z'_v = Z_v + \\mathrm{Gate}(Z_v) \\odot \\mathrm{CrossAttn}(Z_v, P)$. At inference the VFM is removed and the learned prompts alone remain. Two auxiliary losses supervise intermediate layers: SpaRA maximizes cosine similarity between student and teacher features at pooling scales $1\\times1$, $2\\times2$, and $4\\times4$ and matches neighbor-pair similarities, while SpeRA applies a 2D FFT, separates low- and high-frequency bands with binary masks, and matches the inverse-FFT reconstructions. This architecture separates knowledge absorption during training from anchoring during inference, which is what lets the paper claim negligible inference cost.","core_discovery":"Inside a visually instruction-tuned LLM, the representations of image tokens drift away from their incoming states layer by layer, and the paper treats this drift as the mechanism behind poor visual perception. The paper's discovery is that the drift is not cured by giving intermediate layers stronger vision features: direct alignment with a vision foundation model raises the semantic quality of the features but leaves the drift rate almost unchanged. The fix is a set of learnable prompt anchors that absorb VFM knowledge during training and then, at inference, act as stable references via gated cross-attention, keeping visual tokens closer to their input semantics in deeper layers. Auxiliary spatial and spectral alignment losses add extra visual supervision during training and disappear at inference.","pith_inferences":["The paper's drift metric is a cosine-similarity drop, which can also arise from legitimate re-coding in deep layers; a direct test with layer-wise linear probes or mutual-information estimates would show whether task-relevant visual information is truly lost.","The near-orthogonality and specialization of the prompts (one suppresses background, others track foreground objectness) suggests VAPI works like a small set of visual experts; one could probe whether more prompts, or per-head prompts, extend the gains.","Because the VFM is only needed during training, the same anchor mechanism could be tested on post-training adaptation of mature MLLMs, an application the paper itself flags as open.","The high-frequency supervision alone hurts, so a schedule that anneals its weight in later training might let SSVAL capture edge detail without the noise penalty the paper observes."],"forward_implications":["Visual instruction tuning can preserve more of the original image content without changing the vision encoder, projector, or LLM weights at inference.","The external VFM and both alignment losses are training-only, so a deployed model pays only 5.25M extra parameters and about 0.04 percent extra FLOPs.","The largest gains land on perception-heavy benchmarks, which implies that reducing internal drift translates directly into better spatial reasoning and fine-grained matching.","The gains persist when the vision encoder is upgraded to a stronger one, so the anchor mechanism is complementary to encoder quality rather than a replacement.","The ablation attributes most of the improvement to the anchor prompts themselves, with the spatial and spectral losses contributing smaller, complementary gains."],"supporting_citations":[{"why":"Supplies evidence that vision-language models overlook their internal visual representations, motivating the deviation phenomenon.","marker":"[9]"},{"why":"Defines the visual instruction tuning paradigm and the baseline MLLM that SSVAL modifies.","marker":"[28]"},{"why":"Provides the CLIP vision encoder whose image features are enhanced and anchored in the main configuration.","marker":"[34]"},{"why":"The direct VFM alignment baseline (VIRAL) that the paper shows improves semantics but fails to stop drift.","marker":"[50]"},{"why":"DINOv2, the frozen vision foundation model that supplies anchor features and serves as alignment teacher.","marker":"[31]"},{"why":"Prior work treating prompts as knowledge distillers, which VAPI extends from homogeneous to heterogeneous VFM knowledge.","marker":"[24]"},{"why":"MMVP benchmark used to measure fine-grained visual matching, where SSVAL shows its largest gain.","marker":"[41]"},{"why":"CV-Bench 2D benchmark used to measure spatial understanding and fine-grained perception.","marker":"[40]"}],"fun_headline_variants":["Spatial-spectral anchors stop MLLM visual drift","Prompt anchors keep MLLM visual details intact","Visual anchor learning halts MLLM degradation","MLLMs hold visual info with anchor prompts","Anchor prompts stabilize deep-layer visual features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire diagnosis rests on treating a drop in cosine similarity between input visual tokens and deeper LLM-layer representations as loss of task-relevant visual information, but the paper never validates that this proxy tracks what the model can still use.","fun_headline_variants_meta":{"raw":{"variants":["Spatial-spectral anchors stop MLLM visual drift","Prompt anchors keep MLLM visual details intact","Visual anchor learning halts MLLM degradation","MLLMs hold visual info with anchor prompts","Anchor prompts stabilize deep-layer visual features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1303,"prompt_tokens":865,"completion_tokens":438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":481,"tokens_out":438,"duration_ms":3916,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:06:25.297736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure task-relevant visual information at each LLM layer of a baseline MLLM with a linear probe (or an information-theoretic estimate) trained to predict object presence, position, and count from visual-token representations; if probe accuracy stays high in deep layers even as cosine similarity to the input drops, the paper's diagnosis of information degradation is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CLIP vision encoder whose image features are enhanced and anchored in the main configuration."}],"review_version":2}