Pith. sign in

REVIEW 5 major objections 5 minor 54 references

Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Visual anchor prompts can stop multimodal LLMs from losing visual information as it passes through deep layers.

desk verdict A solid, well-ablated methods paper with real benchmark gains; the anchor-prompt idea is new, but the central mechanism story rests on an unvalidated cosine-similarity proxy and lacks a parameter-matched control. read the letter →

arxiv 2608.01635 v1 pith:UEKKQDMF submitted 2026-08-03 cs.CV

Qianlong Yang , Bowen Ye , Xianda Guo , Yanlun Peng , Wenke Huang , Hongyuan Zhang , Yulei Jia This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelsvisualrepresentationdeviationanchorpromptsspatial-spectralalignmentfrequency-domainsupervisionvisionfoundationfine-grainedperception
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that after visual instruction tuning, the internal representations of visual tokens in a multimodal large language model drift away from their initial states as they pass through deeper layers, and this drift is a major cause of poor visual perception. It claims that aligning intermediate representations to an external vision foundation model improves what the features encode but does not stop the drift. To fix this, it introduces Visual Anchor Prompt Injection (VAPI), prompts trained to absorb VFM knowledge and then re-inject stable references at inference, plus spatial and frequency-domain alignment losses during training. If correct, the method gives a lightweight way to preserve visual information through the LLM, with large gains on perception-heavy benchmarks such as +7.86 on MMVP and +6.47 on CV-Bench 2D.

What carries the argument

The load-bearing object is the visual anchor prompt set $P \in \mathbb{R}^{N_p \times D}$: during training the prompts are summed with stride-sampled, projection-mapped features from a frozen vision foundation model, then fused into the visual tokens $Z_v$ by multi-head cross-attention with a gated residual $Z'_v = Z_v + \mathrm{Gate}(Z_v) \odot \mathrm{CrossAttn}(Z_v, P)$. At inference the VFM is removed and the learned prompts alone remain. Two auxiliary losses supervise intermediate layers: SpaRA maximizes cosine similarity between student and teacher features at pooling scales $1\times1$, $2\times2$, and $4\times4$ and matches neighbor-pair similarities, while SpeRA applies a 2D FFT, separates low- and high-frequency bands with binary masks, and matches the inverse-FFT reconstructions. This architecture separates knowledge absorption during training from anchoring during inference, which is what lets the paper claim negligible inference cost.

What would settle it

Measure task-relevant visual information at each LLM layer of a baseline MLLM with a linear probe (or an information-theoretic estimate) trained to predict object presence, position, and count from visual-token representations; if probe accuracy stays high in deep layers even as cosine similarity to the input drops, the paper's diagnosis of information degradation is not supported.

Watch

Extended reading notes

Core claim

Inside a visually instruction-tuned LLM, the representations of image tokens drift away from their incoming states layer by layer, and the paper treats this drift as the mechanism behind poor visual perception. The paper's discovery is that the drift is not cured by giving intermediate layers stronger vision features: direct alignment with a vision foundation model raises the semantic quality of the features but leaves the drift rate almost unchanged. The fix is a set of learnable prompt anchors that absorb VFM knowledge during training and then, at inference, act as stable references via gated cross-attention, keeping visual tokens closer to their input semantics in deeper layers. Auxiliary spatial and spectral alignment losses add extra visual supervision during training and disappear at inference.

Load-bearing premise

The entire diagnosis rests on treating a drop in cosine similarity between input visual tokens and deeper LLM-layer representations as loss of task-relevant visual information, but the paper never validates that this proxy tracks what the model can still use.

Editorial extensions

If this is right

  • Visual instruction tuning can preserve more of the original image content without changing the vision encoder, projector, or LLM weights at inference.
  • The external VFM and both alignment losses are training-only, so a deployed model pays only 5.25M extra parameters and about 0.04 percent extra FLOPs.
  • The largest gains land on perception-heavy benchmarks, which implies that reducing internal drift translates directly into better spatial reasoning and fine-grained matching.
  • The gains persist when the vision encoder is upgraded to a stronger one, so the anchor mechanism is complementary to encoder quality rather than a replacement.
  • The ablation attributes most of the improvement to the anchor prompts themselves, with the spatial and spectral losses contributing smaller, complementary gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's drift metric is a cosine-similarity drop, which can also arise from legitimate re-coding in deep layers; a direct test with layer-wise linear probes or mutual-information estimates would show whether task-relevant visual information is truly lost.
  • The near-orthogonality and specialization of the prompts (one suppresses background, others track foreground objectness) suggests VAPI works like a small set of visual experts; one could probe whether more prompts, or per-head prompts, extend the gains.
  • Because the VFM is only needed during training, the same anchor mechanism could be tested on post-training adaptation of mature MLLMs, an application the paper itself flags as open.
  • The high-frequency supervision alone hurts, so a schedule that anneals its weight in later training might let SSVAL capture edge detail without the noise penalty the paper observes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper addresses visual information degradation in multimodal large language models (MLLMs). It first measures a phenomenon it calls visual representation deviation: the cosine similarity between input visual token representations and their intermediate LLM-layer representations drops sharply with depth. It then proposes SSVAL, which combines Visual Anchor Prompt Injection (VAPI) with spatial and spectral representation alignment losses (SpaRA and SpeRA). During training, VAPI prompts absorb features from a frozen vision foundation model (VFM) through gated cross-attention; during inference, the prompts are kept fixed and are claimed to act as stable visual anchors that preserve visual information across layers. The method is evaluated on six multimodal benchmarks with two vision encoders and two LLM scales, and ablations study prompt count, VFM choice, component contribution, multiscale configurations, and frequency bands.

Significance. If the mechanism claim were established, this would be a valuable contribution: the paper identifies a practically relevant phenomenon (representation deviation), shows a counter-intuitive limitation of direct VFM alignment, and reports sizeable gains on perception-heavy benchmarks such as MMVP (+7.86) and CV-Bench 2D (+6.47) over a strong baseline. The external benchmark evaluation is not circular, the ablations are fairly thorough, and the authors state that code will be released. However, the central mechanistic claim rests on a cosine-similarity proxy that is never validated against task-relevant visual information, and the method introduces additional inference parameters without a parameter-matched control. The benchmark results are interesting regardless, but the paper's headline explanation for those results is not yet supported.

major comments (5)
  1. [Table 4] The paper's central evidence for both the phenomenon of visual representation deviation and the mitigating effect of SSVAL is the cosine similarity between input visual token representations and deeper-layer representations. This metric is not validated against task-relevant visual information: deep transformer layers legitimately re-represent tokens, and a drop in cosine similarity to the input embedding does not by itself imply information loss. Conversely, the gated residual in Eq. (1) adds a prompt-derived term to every visual token and can mechanically keep representations closer to the input direction, inflating this similarity without preserving usable visual content. To support the mechanism story, the authors should validate the proxy with task-level probes (e.g., decoding object attributes or spatial relations from intermediate layers) and include a control where the same gated residual uses random or non-VFM prompts.
  2. [Table 4] The configuration VAPI+SpeRA collapses to 47.71% on CV-Bench 2D and 9.33% on MMVP, far below the Baseline (58.97% and 33.47%). This result is acknowledged in passing but never explained. It directly undercuts the claim that visual anchor prompts are an inherently stabilising mechanism, since adding the spectral loss to VAPI makes the anchors catastrophically worse. The paper should analyse the failure mode (e.g., what happens to the prompt attention maps or representation similarity in this configuration) and clarify why the interaction with SpaRA is necessary. Without this, the full method appears to depend on a delicate and unexplained balance between the losses.
  3. [Section 4.5, Tables 7-8] SSVAL adds 5.25M parameters and 3.65 GFLOPs at inference relative to Baseline, while VIRAL adds zero parameters. The comparison against VIRAL is therefore not parameter-matched. The benchmark gains could stem from the additional capacity or a generic feature-injection effect rather than from the proposed anchor mechanism. A parameter-matched control is needed, for example a model with the same gated cross-attention module but with prompts that are randomly initialised and not trained with VFM knowledge, or a residual connection of matched size. Table 4's +VAPI row is a useful start, but it does not separate the VFM-knowledge component from the extra capacity component.
  4. [Section 3.3 and Section 4.1] Several hyperparameters in the proposed losses are either unreported or selected on the same benchmarks used for final evaluation. Specifically, alpha and beta in Eq. (3), the numerical values of gamma_LF and gamma_HF in Eq. (8), the scale weights w_k, the frequency-mask radius threshold r, and the intermediate alignment layers are not given in the main text. Table 3 selects the teacher VFM (DINOv2-B) by comparing results on the same six benchmarks, so the reported gains risk selection on the test set. The authors should report all hyperparameter values and either use a validation split for model selection or show that the results are stable across a range of choices.
  5. [All experiments] No multiple-seed runs or error bars are reported. Many of the claimed improvements are small (e.g., MMStar +1.73, POPE +0.22, MM-Bench +1.11 in the CLIP+Qwen2.5-7B configuration), and instruction-tuning pipelines can have non-trivial run-to-run variance. The headline perception benchmarks show larger deltas, but the ablation conclusions (e.g., prompt quantity, component contributions) frequently rely on differences of one to three points. Reporting mean and standard deviation over at least three seeds, or at least for the main comparison and key ablations, is necessary to assess whether the differences are meaningful.
minor comments (5)
  1. [Abstract] The sentence 'Code are available on our project page' has a subject-verb agreement error; it should be 'Code is available'.
  2. [Table 4] The MME value for the +SpaRA row is given as '1828.21%', which includes a stray percent sign; the other MME entries are unadorned numbers.
  3. [Figures 1 and 7] There are several typographical issues: 'SSV AL' appears with an unintended space in Figures 1(a) and 7, and in Figure 7 the sub-captions for VIRAL and SSVAL both use the label '(b)', which is confusing.
  4. [Section 4.1] The main text states 'We use 4 visual anchor prompts' without explaining how this number was chosen; Table 2 shows that 4 prompts are best on CV-Bench 2D but 8 prompts are best on MMVP (42.67% vs 41.33%). A sentence justifying the default choice beyond 'considering both accuracy and efficiency' would improve reproducibility.
  5. [Appendix D] The limitations section is helpful, but it does not mention the fragility shown in Table 4 (VAPI+SpeRA collapse) or the unvalidated similarity metric, which are the two issues most relevant to the paper's central claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: externally evaluated benchmarks carry the central claim, and the mechanism figures are interpretive rather than definitionally tied to the result.

full rationale

The paper's central empirical result—SSVAL's benchmark gains, e.g., +6.47 on CV-Bench 2D and +7.86 on MMVP under CLIP+Qwen2.5-7B—is measured on external benchmarks that are not used in the LLaVA-665K training set or in the construction of the VAPI, SpaRA, or SpeRA objectives. The cosine-similarity-to-input curves in Fig. 1(b) are presented as a mechanistic illustration of visual representation deviation; the proxy is not independently validated, but an unvalidated proxy is a validity concern rather than a circular derivation. The training-time addition of projected VFM features to prompts (Algorithm 1, line 3) makes the later observation that prompts 'inherit DINOv2's objectness awareness' partially self-confirming as a mechanism story, yet this does not reduce the externally measured benchmark results to the paper's inputs. Self-citations appear in related-work enumerations and are not load-bearing for the derivation; no uniqueness theorem or ansatz is imported from the authors' prior work, and no fitted parameter is renamed as a prediction. Hyperparameters are tuned on the evaluation suite, but that is a generalizability risk, not circularity. The derivation chain is therefore not circular.

Assumptions & free parameters 9 free parameters · 5 assumptions · 1 invented entities

The paper relies on a small set of model choices and assumptions: the cosine-similarity proxy for visual loss, the value of DINOv2 features as an alignment target, the frequency mask split, and several hyperparameters (loss weights, prompt count, teacher selection) that were ablated on the evaluation benchmarks. No new physics-like entities are introduced; the visual anchor prompts are trainable parameters whose independent evidence is limited to the paper's own experiments.

free parameters (9)
  • Loss weight for SpaRA (lambda_SpaRA) = 0.3
    Set in Section 4.1; no sensitivity analysis or search range reported.
  • Loss weight for SpeRA (lambda_SpeRA) = 0.15
    Set in Section 4.1; no sensitivity analysis or search range reported.
  • Prompt count N_p = 4
    Chosen by ablation on CV-Bench 2D, MMVP, MMStar, and MME (Table 2); configurations 8, 12, 16, 20, and 24 trade off differently.
  • Teacher VFM = DINOv2-B
    Selected by ablation (Table 3); DINOv2-L gives better MMVP, MMStar, and MME, while CLIP-L performs worst.
  • Weight coefficients alpha and beta for SpaRA feature and relation terms
    Used in Eq. (3) but never reported; without these values, Eq. (3) is not reproducible.
  • Frequency band weights gamma_LF and gamma_HF = 1:1
    Appendix and Section 4.1 state equal weights; no sensitivity analysis is provided.
  • SpaRA scale weights w_k = 1:1:1 for scales 1, 2, 4
    Set in Section 4.1; the scale set itself is chosen by ablation (Table 5).
  • Frequency mask radius threshold r = 0.5
    Fixed in Eq. (5); no justification or sensitivity analysis is given.
  • Intermediate alignment layers = 16 and 28 of Qwen2.5-7B
    Selected in Appendix A.3 without ablation or theoretical justification.
assumptions (5)
  • ad hoc to paper The cosine similarity between input visual token projections and deeper-layer visual-token representations measures visual information preservation.
    Section 1 and Fig. 1(b) use this metric to define visual representation deviation and to claim SSVAL mitigates it; the metric is not validated.
  • domain assumption Pretrained VFM features (DINOv2, DINOv3, CLIP) provide complementary knowledge that, when injected as prompts, improves MLLM perception without disrupting language abilities.
    Section 3.2 uses VFM features as anchor sources; Table 3 varies the teacher but does not test a non-VFM control.
  • domain assumption A 2D FFT with fixed low/high frequency masks at normalized radius 0.5 separates global structure from edge and texture detail for alignment purposes.
    Section 3.3, Eqs. (5)-(7) assume this split is meaningful for visual tokens; no empirical validation is provided.
  • domain assumption Gated multi-head cross-attention between visual tokens and prompts preserves original visual structure while adding anchor information.
    Section 3.2, Eq. (1) relies on a learned gate; no analysis of gate behavior is given when the VFM residual is removed at inference.
  • ad hoc to paper Intermediate layers 16 and 28 are representative locations where visual information is still recoverable.
    Appendix A.3 selects these layers without ablation or theoretical justification.
invented entities (1)
  • Visual anchor prompts (learnable prompt vectors of dimension [N_p, D])
    purpose: Provide stable visual references through gated cross-attention during inference to reduce representation deviation.
    The paper presents attention maps and prompt-similarity matrices as evidence, but these are computed from the trained model itself; no external falsifiable prediction about the prompts is given beyond the reported benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning." pith.science (2026). https://pith.science/paper/UEKKQDMF

@misc{pith2026260801635,
  author       = {Pith},
  title        = {Pith review of: Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UEKKQDMF}},
  note         = {Machine review of arXiv:2608.01635}
}
read the original abstract

Despite the progress of multimodal large language models (MLLMs), they continue to exhibit deficiencies in visual perception. Following visual instruction tuning, internal MLLM representations rapidly deviate from their original semantic states during inference, causing severe information degradation. While existing methods attempt to leverage external vision foundation models (VFMs) to align internal representations, we find that direct alignment with VFMs enhances visual semantics but fails to mitigate representation deviation. To address this, we propose Spatial-Spectral Visual Anchor Learning (SSVAL). The core of SSVAL is Visual Anchor Prompt Injection (VAPI), which introduces prompts that absorb rich knowledge from external VFMs during training, enabling them to serve as stable visual anchors that mitigate representation deviation during inference. Additionally, we incorporate auxiliary spatial and frequency-domain representation alignment losses to provide complementary vision-specific supervision at intermediate LLM layers. Extensive experiments demonstrate that SSVAL significantly outperforms existing methods. Code are available on our \href{https://msls38.github.io/SSVAL/}{project page}.

Figures

Figures reproduced from arXiv: 2608.01635 by the authors.

Figure 1
Figure 1. Analysis of SSVAL effectiveness and performance comparison. (a) SSVAL demonstrates superior comprehensive [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of MLLM architectures. (a) Visual Instruction Tuning [ [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of SSVAL framework. Visual prompts are combined with stride-sampled VFM features to inject visual prior [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison among Baseline, VIRAL, and SSVAL(Ours). (a) PCA Visualizations reveal that SSVAL maintains [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Attention map visualization showing prompt [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Cosine similarity matrices of learned visual anchor prompts. (a) 4 prompts. (b) 8 prompts. The near-zero off-diagonal [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison among Baseline, VIRAL, and SSVAL(Ours). [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 11 canonical work pages

  1. [1]

    2024.Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2024.Gpt-4 technical report. arXiv:2303.08774 https: //arxiv.org/abs/2303.08774

    arXiv 2024

  2. [2]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  3. [3]

    2025.Qwen2

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025.Qwen2. 5-vl technical report. arXiv:2502.13923 https://arxiv.org/abs/2502.13923

    arXiv 2025

  4. [5]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al . 2024. Are we on the right way for evaluating large vision-language models?Advances in Neural Information Processing Systems37 (2024), 27056–27087

    2024

  5. [6]

    2024.InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024.InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. arXiv:2312.14238 [cs.CV] https: //arxiv.org/abs/2312.14238

    arXiv 2024

  6. [7]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems36 (2023), 49250–49267

    2023

  7. [8]

    2025.MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. 2025.MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv:2306.13394 https://arxiv.org/abs/2306. 13394

    arXiv 2025

  8. [9]

    2025.Hidden in plain sight: VLMs overlook their visual representations

    Stephanie Fu, Tyler Bonnen, Devin Guillory, and Trevor Darrell. 2025.Hidden in plain sight: VLMs overlook their visual representations. arXiv:2506.08008 [cs.CV] https://arxiv.org/abs/2506.08008

    arXiv 2025

Show all 54 references
  1. [10]

    Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Dujun Nie, Wenke Huang, Chenming Zhang, Shuai Liu, Hao Zhao, and Long Chen. 2026. SURDS: Bench- marking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models. InAdvances in Neural Information Proce...

    2026

  2. [11]

    2023.Eˆ2VPT: An Effective and Efficient Approach for Visual Prompt Tuning

    Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. 2023.Eˆ2VPT: An Effective and Efficient Approach for Visual Prompt Tuning. arXiv:2307.13770 https://arxiv.org/abs/2307.13770

    2023 arXiv

  3. [12]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.ICLR1, 2 (2022), 3

    2022

  4. [13]

    2022.Visual Prompt Tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022.Visual Prompt Tuning. arXiv:2203.12119 https://arxiv.org/abs/2203.12119

    2022 arXiv

  5. [14]

    2024.From clip to dino: Visual encoders shout in multi-modal large language models

    Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin’e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong. 2024.From clip to dino: Visual encoders shout in multi-modal large language models. arXiv:2310.08825 https: //arxiv.org/abs/2310.08825

    2024 arXiv

  6. [15]

    2025.Multimodal Continual Learning with MLLMs from Multi-scenario Perspectives

    Kai Jiang, Siqi Huang, Xiangyu Chen, Jiawei Shao, Hongyuan Zhang, and Xue- long Li. 2025.Multimodal Continual Learning with MLLMs from Multi-scenario Perspectives

    2025

  7. [16]

    2026.Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache

    Yuqiu Jiang, Xiaozhen Qiao, Tianyu Mei, Haojian Huang, Yifan Chen, Ye Zheng, and Zhe Sun. 2026.Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache. arXiv:2511.18811 https://arxiv.org/abs/2511.18811

    2026

  8. [17]

    Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. 2024. BRAVE: Broadening the Visual Encoding of Vision-Language Models. InComputer Vision - ECCV 2024 - 18th European Conference (Lecture Notes in Computer Science), Ales Le...

    2024 doi

  9. [18]

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. 2024. Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models. InForty-first International Confer- ence on Machine Learning (Proceedings of Machin...

    2024

  10. [19]

    Khan, and Fahad Shahbaz Khan

    Muhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. 2023. MaPLe: Multi-modal Prompt Learning. InIEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Vancouver, BC, Canada, 19113–19122. doi:10.1109/CVPR52729.2023.01832

    2023

  11. [20]

    Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. 2023. Self-regulating Prompts: Foundational Model Adaptation without Forgetting. InIEEE/CVF International Conference on Computer Vision. IEEE, Paris, France, 15144–...

    2023

  12. [21]

    2024.Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024.Llava-onevision: Easy visual task transfer. arXiv:2408.03326 https://arxiv.org/abs/2408.03326

    2024 arXiv

  13. [22]

    2024.Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024.Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv:2407.07895 https://arxiv.org/abs/2407.07895

    2024 arXiv

  14. [23]

    2023.Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023.Evaluating object hallucination in large vision-language models. arXiv:2305.10355 https://arxiv.org/abs/2305.10355

    2023 arXiv

  15. [25]

    2025.Depth anything 3: Recovering the visual space from any views

    Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. 2025.Depth anything 3: Recovering the visual space from any views. arXiv:2511.10647 https://arxiv.org/abs/2511.10647

    2025 arXiv

  16. [26]

    2023.Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. 2023.Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv:2311.07575 https://arxiv.org/abs/2311.07575

    2023 arXiv

  17. [28]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning.Advances in neural information processing systems36 (2023), 34892–34916

    2023

  18. [29]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin

  19. [30]

    2024.Deepseek-vl: towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. 2024.Deepseek-vl: towards real-world vision-language understanding. arXiv:2403.05525 https://arxiv.org/ abs/2403.05525

    2024 arXiv

  20. [31]

    2024.Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. 2024.Dinov2: Learning robust visual features without supervision. arXiv:2304.07193 https://arxiv.org/abs/2304.07193

    2024 arXiv

  21. [32]

    2025.Bidirectional prototype-reward co-evolution for test-time adaptation of vision-language models

    Xiaozhen Qiao, Peng Huang, Jiakang Yuan, Xianda Guo, Bowen Ye, Chaocan Xue, Ye Zheng, Zhe Sun, and Xuelong Li. 2025.Bidirectional prototype-reward co-evolution for test-time adaptation of vision-language models. arXiv:2503.09394 https://arxiv.org/abs/2503.09394

    2025 arXiv

  22. [33]

    2025.Class-A ware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models

    Xiaozhen Qiao, Jingkai Zhao, Yuqiu Jiang, Xianda Guo, Zhe Sun, Hongyuan Zhang, and Xuelong Li. 2025.Class-A ware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models. arXiv:2510.19802 https://arxiv.org/abs/2510.19802

    2025

  23. [34]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. InProceedings ...

    2021

  24. [35]

    2024.Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. 2024.Sam 2: Segment anything in images and videos. arXiv:2408.00714 https://arxiv.org/abs/2408.00714

    2024 arXiv

  25. [36]

    2025.Ea- gle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, et al. 2025.Ea- gle: Exploring the design space for multimodal llms with mixture of encoders. arXiv:2408.15998 https://arxiv.org/abs/2408.15998

    2025 arXiv

  26. [37]

    2025.Dinov3

    Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Rama- monjisoa, et al. 2025.Dinov3. arXiv:2508.10104 https://arxiv.org/abs/2508.10104

    2025 arXiv

  27. [38]

    2023.Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023.Eva-clip: Improved training techniques for clip at scale. arXiv:2303.15389 https://arxiv.org/ abs/2303.15389 9 Qianlong Yang et al

    2023 arXiv

  28. [39]

    2026.Kimi k2: Open agentic intelligence

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al . 2026.Kimi k2: Open agentic intelligence. arXiv:2507.20534 https://arxiv.org/abs/2507.20534

    2026 arXiv

  29. [40]

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Veda- giri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. 2024. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information P...

    2024

  30. [41]

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. InIEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Seattle, WA, USA, 9568–9578. doi:10.1109/CVPR52733...

    2024

  31. [42]

    2023.Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al . 2023.Llama: Open and efficient foundation language models. arXiv:2302.13971 https://arxiv.org/abs/2302.13971

    2023 arXiv

  32. [43]

    2025.Siglip 2: Multilingual vision-language encoders with im- proved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. 2025.Siglip 2: Multilingual vision-language encoders with im- proved semantic understanding, localizatio...

    2025 arXiv

  33. [45]

    2024.Reconstructive visual instruction tuning

    Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, and Zhaoxiang Zhang. 2024.Reconstructive visual instruction tuning. arXiv:2410.09575 https://arxiv.org/abs/2410.09575

    2024 arXiv

  34. [46]

    2025.Stop looking for important tokens in multimodal language models: Duplication matters more

    Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. 2025.Stop looking for important tokens in multimodal language models: Duplication matters more. arXiv:2502.11494 https://arxiv.org/abs/2502.11494

    2025 arXiv

  35. [47]

    2025.Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025.Qwen3 technical report. arXiv:2505.09388 https://arxiv.org/abs/2505.09388

    2025 arXiv

  36. [48]

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2025. VisionZip: Longer is Better but Not Necessary in Vision Language Models. InIEEE/CVF Conference on Computer Vision and Pattern Recog- nition. Computer Vision Foundation / IEEE, Nash...

    2025

  37. [49]

    2026.Holo-World: Unified Camera, Object and Weather Control for Video World Model

    Xiangchen Yin, Wenzhang Sun, Jiahui Yuan, Zijie Liu, Yinda Chen, Wei Li, Dachun Kai, Chunfeng Wang, and Xiaoyan Sun. 2026.Holo-World: Unified Camera, Object and Weather Control for Video World Model

    2026

  38. [50]

    2025.Visual Representation Alignment for Multimodal Large Language Models

    Heeji Yoon, Jaewoo Jung, Junwan Kim, Hyungyu Choi, Heeseong Shin, Sang- beom Lim, Honggyu An, Chaehyun Kim, Jisang Han, Donghyun Kim, et al . 2025.Visual Representation Alignment for Multimodal Large Language Models. arXiv:2509.07979 https://arxiv.org/abs/2509.07979

    2025

  39. [51]

    2025.Introducing visual perception token into multimodal large language model

    Runpeng Yu, Xinyin Ma, and Xinchao Wang. 2025.Introducing visual perception token into multimodal large language model. arXiv:2502.17425 https://arxiv.org/ abs/2502.17425

    2025 arXiv

  40. [52]

    2024.Representation alignment for generation: Training diffusion transformers is easier than you think

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. 2024.Representation alignment for generation: Training diffusion transformers is easier than you think

    2024

  41. [53]

    2023.When and why vision-language models behave like bags-of-words, and what to do about it?arXiv:2210.01936 https://arxiv.org/abs/2210.01936

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023.When and why vision-language models behave like bags-of-words, and what to do about it?arXiv:2210.01936 https://arxiv.org/abs/2210.01936

    2023 arXiv

  42. [54]

    2024.Instruct Large Language Models to Drive like Humans

    Ruijun Zhang, Xianda Guo, Wenzhao Zheng, Chenming Zhang, Kurt Keutzer, and Long Chen. 2024.Instruct Large Language Models to Drive like Humans

    2024

  43. [55]

    Ruishu Zhu, Sida Huang, Ziheng Jiao, and Hongyuan Zhang. 2026. Explore How to Inject Beneficial Noise in MLLMs. InFortieth AAAI Conference on Artificial Intelligence, Thirty-Eighth Conference on Innovative Applications of Artificial In- telligence, Sixteenth Symposium on Educa...

    2026 doi

  44. [56]

    2026.ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Models

    Ruishu Zhu, Zhihao Huang, Jiacheng Sun, Ping Luo, Hongyuan Zhang, and Xuelong Li. 2026.ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Models. arXiv:2512.14099 https://arxiv.org/abs/ 2512.14099 10 Mitigating Visual Degradation in MLLMs...

    2026 arXiv

  45. [2022]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems35 (2022), 23716–23736

    2022

  46. [2024]

    MMBench: Is Your Multi-modal Model an All-Around Player?. InComputer Vision - ECCV 2024 - 18th European Conference (Lecture Notes in Computer Science), Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (Eds.). Springer, Milan, Italy, 21...

    2024 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.