Pith. sign in

REVIEW 3 major objections 2 minor 164 references

ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety

T0 review · 3 major / 2 minor · reviewed 2026-05-10 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Projectors in multimodal LLMs encode backdoors in low-rank subspaces with activation scaling linearly by input norm

desk verdict The paper finds low-rank backdoor encoding and norm-linear semantic shifts in MLLM projectors, but the evidence stays observational without tests for causality. read the letter →

arxiv 2604.19083 v1 submitted 2026-04-21 cs.CR cs.AI

Kun Wang , Cheng Qian , Miao Yu , Lilan Peng , Liang Lin , Jiaming Zhang , Tianyu Zhang , Yu Cheng
show 1 more author
Yang Wang
This is my paper · ORCID
classification cs.CRcs.AI
keywords backdoorattacksmultimodallargelanguagemodelsprojectormodulesmodelinterpretabilitysafetyvulnerabilitieslow-ranksubspacessemanticshifts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ProjLens to examine how backdoor attacks succeed in multimodal large language models through projector modules. It establishes that standard fine-tuning for downstream tasks creates exploitable vulnerabilities whose activation differs from text-only LLMs. Experiments across four backdoor variants show that overall updates remain full-rank without dedicated trigger neurons, yet the critical backdoor parameters concentrate in a low-rank subspace of the projector. Both clean and poisoned embeddings shift toward a shared semantic direction aligned with the attack target, but the shift magnitude grows linearly with input norm, selectively triggering the backdoor only on poisoned inputs. This account matters because it identifies concrete structural and dynamic features that could guide targeted defenses.

What carries the argument

The low-rank subspace of the projector module, which isolates backdoor-critical parameters and enforces linear scaling of semantic shifts with input norm to produce selective activation.

What would settle it

An experiment that removes or orthogonalizes the low-rank subspace of the projector and then measures whether backdoor success rate drops to near zero while clean accuracy remains intact.

Watch

Extended reading notes

Core claim

ProjLens shows that backdoor injection updates appear full-rank overall and lack dedicated trigger neurons, but the backdoor-critical parameters are encoded within a low-rank subspace of the projector. Activation occurs because both clean and poisoned embeddings undergo a semantic shift toward a shared direction aligned with the backdoor target, with the shifting magnitude scaling linearly with the input norm and thereby producing distinct behavior on poisoned samples.

Load-bearing premise

The observed low-rank encoding and linear norm scaling are causal drivers of backdoor behavior rather than side effects of the specific training setups and variants tested.

Editorial extensions

If this is right

  • Downstream projector fine-tuning alone introduces backdoor vulnerability in MLLMs.
  • Backdoors rely on subspace encoding rather than isolated trigger neurons.
  • Semantic shifts align to the target direction for both input types but only exceed the activation threshold on high-norm poisoned samples.
  • These projector mechanisms differ from backdoor activation patterns reported in text-only LLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Targeting the low-rank subspace during fine-tuning or inference could suppress backdoors with limited impact on clean performance.
  • Norm regularization during alignment might reduce the linear scaling effect and thereby lower vulnerability.
  • The same subspace-plus-scaling pattern may appear in other multimodal architectures once the same analysis is applied.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper proposes ProjLens, an interpretability framework to analyze backdoor mechanisms in Multimodal Large Language Models (MLLMs), with emphasis on the projector module. It establishes that downstream alignment via projector fine-tuning introduces backdoor vulnerabilities distinct from text-only LLMs, and reports empirical findings from four backdoor variants: backdoor injection updates are overall full-rank without dedicated trigger neurons, yet backdoor-critical parameters lie in a low-rank subspace of the projector; additionally, both clean and poisoned embeddings undergo semantic shifts toward a shared backdoor-target direction, with shift magnitude scaling linearly with input norm to produce selective activation on poisoned inputs.

Significance. If the reported patterns hold under causal scrutiny, the work supplies useful empirical observations on projector subspaces and norm-dependent embedding trajectories that differentiate multimodal backdoors from unimodal cases. These could inform targeted defenses or alignment procedures in MLLMs. The public code link is a positive step toward reproducibility, though its anonymity currently limits verification.

major comments (3)
  1. [Experiments section] Experiments section (four backdoor variants): The low-rank subspace claim rests on post-hoc SVD of weight differences, yet no ablation is described that masks or removes the identified low-rank components and re-measures attack success rate or activation; without such intervention, it is impossible to distinguish whether the subspace is mechanistically necessary or merely correlated with the poisoning objective.
  2. [Activation mechanism analysis] Activation mechanism analysis: The statement that linear norm scaling 'results in' distinct backdoor activation is presented as explanatory, but the manuscript reports only observational trajectories of embedding shifts; no controlled test (e.g., rescaling input norms on clean samples while holding other factors fixed) is provided to establish necessity or sufficiency.
  3. [Methods and results reporting] Methods and results reporting: The abstract and experimental description claim 'extensive experiments' and low-rank quantification, but supply no details on statistical tests, baseline comparisons, variance across runs, or how low-rank structure was thresholded; this absence undermines assessment of whether the patterns are robust or specific to the four variants and architectures tested.
minor comments (2)
  1. [Abstract] Abstract: The phrase 'lack dedicated trigger neurons' is introduced without a precise operational definition or comparison to prior LLM backdoor literature; a brief parenthetical clarification would improve readability.
  2. [Code availability] Code availability: The anonymous link is noted, but the paper would benefit from including a short pseudocode outline of the ProjLens analysis pipeline in the main text or appendix to aid readers before code access is granted.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on ProjLens. The comments correctly identify gaps in causal validation and reporting details. We address each point below and will revise the manuscript accordingly to strengthen the claims with additional experiments and clarifications.

read point-by-point responses

  1. Referee: [Experiments section] Experiments section (four backdoor variants): The low-rank subspace claim rests on post-hoc SVD of weight differences, yet no ablation is described that masks or removes the identified low-rank components and re-measures attack success rate or activation; without such intervention, it is impossible to distinguish whether the subspace is mechanistically necessary or merely correlated with the poisoning objective.

    Authors: We agree that the current SVD-based analysis demonstrates correlation between backdoor updates and a low-rank subspace but does not establish necessity. In the revision, we will add ablation studies: identify the low-rank components via SVD on the projector weight differences, mask or project them out, and re-evaluate attack success rate (ASR) on poisoned inputs as well as clean accuracy. This intervention will test whether removing the subspace suppresses backdoor activation while preserving general functionality, providing causal support for the mechanistic role. revision: yes

  2. Referee: [Activation mechanism analysis] Activation mechanism analysis: The statement that linear norm scaling 'results in' distinct backdoor activation is presented as explanatory, but the manuscript reports only observational trajectories of embedding shifts; no controlled test (e.g., rescaling input norms on clean samples while holding other factors fixed) is provided to establish necessity or sufficiency.

    Authors: We acknowledge that the manuscript currently reports observational linear scaling of embedding shifts with input norm, which correlates with selective activation on poisoned samples. To establish necessity and sufficiency, the revision will include controlled experiments: (i) rescale norms of clean embeddings to match poisoned levels while preserving semantics and measure if backdoor activation occurs; (ii) normalize poisoned input norms to clean levels and check suppression of activation. These tests will directly link the norm-dependent trajectory to the distinct activation behavior. revision: yes

  3. Referee: [Methods and results reporting] Methods and results reporting: The abstract and experimental description claim 'extensive experiments' and low-rank quantification, but supply no details on statistical tests, baseline comparisons, variance across runs, or how low-rank structure was thresholded; this absence undermines assessment of whether the patterns are robust or specific to the four variants and architectures tested.

    Authors: We agree that the current reporting lacks sufficient methodological transparency. In the revised manuscript, we will expand the Methods and Experiments sections with: statistical significance tests (e.g., paired t-tests across runs for low-rank ratios and shift magnitudes); baseline comparisons (e.g., against random subspaces and text-only LLM projectors); variance and standard deviations reported over at least five independent runs with different seeds; and explicit thresholding details for low-rank structure (e.g., cumulative explained variance threshold of 90% or singular value elbow criterion). These will be provided for all four backdoor variants and model architectures. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical observations from backdoor experiments show no circularity

full rationale

The paper reports post-hoc empirical findings from training and analyzing four backdoor variants in MLLMs: low-rank encoding of critical parameters in the projector (via SVD on weight updates) and linear norm-scaling of semantic shifts in embeddings. These are direct measurements of trained model states, not predictions derived from fitted parameters defined on the same data, nor self-definitional, nor reliant on load-bearing self-citations for uniqueness. No equations or claims reduce the reported structures or mechanisms to inputs by construction; the results remain falsifiable observations independent of the analysis pipeline itself.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The work is primarily empirical and interpretability-driven; it does not introduce new mathematical axioms, free parameters, or postulated entities beyond standard deep-learning assumptions about parameter updates reflecting learned behavior.

assumptions (1)
  • domain assumption Parameter updates during fine-tuning reflect the injection of backdoor behavior in a manner that can be isolated by rank analysis and embedding inspection.
    Invoked when interpreting projector updates as containing backdoor-critical parameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety." pith.science (2026). https://pith.science/paper/2604.19083

@misc{pith2026260419083,
  author       = {Pith},
  title        = {Pith review of: ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2604.19083}},
  note         = {Machine review of arXiv:2604.19083}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success in cross-modal understanding and generation, yet their deployment is threatened by critical safety vulnerabilities. While prior works have demonstrated the feasibility of backdoors in MLLMs via fine-tuning data poisoning to manipulate inference, the underlying mechanisms of backdoor attacks remain opaque, complicating the understanding and mitigation. To bridge this gap, we propose ProjLens, an interpretability framework designed to demystify MLLMs backdoors. We first establish that normal downstream task alignment--even when restricted to projector fine--tuning--introduces vulnerability to backdoor injection, whose activation mechanism is different from that observed in text-only LLMs. Through extensive experiments across four backdoor variants, we uncover:(1) Low-Rank Structure: Backdoor injection updates appear overall full-rank and lack dedicated ``trigger neurons'', but the backdoor-critical parameters are encoded within a low-rank subspace of the projector;(2) Activation Mechanism: Both clean and poisoned embedding undergoes a semantic shift toward a shared direction aligned with the backdoor target, but the shifting magnitude scales linearly with the input norm, resulting in the distinct backdoor activation on poisoned samples. Our code is available at: https://anonymous.4open.science/r/ProjLens-8FD7

Figures

Figures reproduced from arXiv: 2604.19083 by the authors.

Figure 1
Figure 1. Introduction to multimodal backdoor attacks in MLLMs (Left). Overview and findings of our ProjLens framework (Right). ity framework designed to deconstruct the lifecycle of mul￾timodal backdoors within the projector. ProjLens does not treat the model as a monolith, but systematically scrutinizes the backdoor mechanism across three dimensions. First, in the feature space, we propose a learnable Visual Trigger Probe (… view at source ↗
Figure 2
Figure 2. SVD decomposition of the projector’s weight difference (∆W2) before and after backdoor injection fine-tuning. (from the backdoored projector) to derive the output features. Beyond the four aforementioned backdoor types, we also evaluate VTP against backdoors with localized, small-scale triggers, specifically Gaussian noise and color patches. All classification results are presented in [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figure 4
Figure 4. Distribution of different neuron metrics across clean and poisoned samples for the Target Refusal backdoor [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (7 more)
Figure 3
Figure 3. Figure 3: Performance of adding SVD rank-k approximation of the clean projector’s weights to inject the refusal backdoor. 5.2. The Projector Weight Paradox The projector is typically instantiated as a two-layer MLP with an activation function σ, serving to align visual features …
Figure 5
Figure 5. Figure 5: SVD decomposition of the clean and poisoned embed￾ding difference for each image token. #1 #2 #3 #4 Top-k Candidates (#k) R-1 R-2 R-3 R-4 R-5 R-6 R-7 R-8 Logit Rank given (40%) emb (17%) m (16%) step (10%) I (32%) emb (19%) step (15%) m (10%) m (32%) I (18%) rome (12%)…
Figure 6
Figure 6. Figure 6: LogitLens results of v0 for the Target Refusal backdoor. 6. Unveiling Multimodal Backdoors The paradox in Section 5 suggests that the projector’s param￾eters do not harbor discernible, backdoor-specific weights. To further elucidate how projector fine-tuning induces ba…
Figure 7
Figure 7. Figure 7: Visualization of the correlation between the u0 and mag￾nitude of image feature for each image token. ∆E(ximg) = UΣVT = Rank X i=1 (σi · ui) · v T i , (9) where ui ∈ R Nv and vi ∈ R dl are the right and left singular vectors corresponding to the i-th singular value σi …
Figure 11
Figure 11. Figure 11: * (c) SVD analysis for ∆W1 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 13
Figure 13. Figure 13: * (a) Targeted Refusal (Global Gaussian Noise) [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 15
Figure 15. Figure 15: * (c) Perceptual Hijack (Specific Icon Trigger) [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

164 extracted references · 164 canonical work pages

  1. [1]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

    work page

  2. [5]

    Advances in neural information processing systems , volume=

    Visual instruction tuning , author=. Advances in neural information processing systems , volume=

    work page

  3. [7]

    MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

    Minigpt-4: Enhancing vision-language understanding with advanced large language models , author=. arXiv preprint arXiv:2304.10592 , year=

    work page Pith review arXiv

  4. [8]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    work page 2021

  5. [10]

    Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages=

    Vl-interpret: An interactive visualization tool for interpreting vision-language transformers , author=. Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages=

    work page

  6. [11]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    V-seam: Visual semantic editing and attention modulating for causal interpretability of vision-language models , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2025

  7. [13]

    Advances in neural information processing systems , volume=

    Flamingo: a visual language model for few-shot learning , author=. Advances in neural information processing systems , volume=

    work page

  8. [14]

    International conference on machine learning , pages=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    work page 2023

Show all 164 references
  1. [16]

    Advances in Neural Information Processing Systems , volume=

    Vhelm: A holistic evaluation of vision language models , author=. Advances in Neural Information Processing Systems , volume=

  2. [18]

    European Conference on Computer Vision , pages=

    Trojvlm: Backdoor attack against vision language models , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  3. [19]

    Advances in Neural Information Processing Systems , volume=

    Shadowcast: Stealthy data poisoning attacks against vision-language models , author=. Advances in Neural Information Processing Systems , volume=

  4. [20]

    International Journal of Computer Vision , pages=

    Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models , author=. International Journal of Computer Vision , pages=. 2025 , publisher=

    2025

  5. [29]

    Trust the Process? Backdoor Attack against Vision--Language Models with Chain-of-Thought Reasoning , author=

  6. [30]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  7. [31]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  8. [34]

    Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , pages=

    Analyzing and editing inner mechanisms of backdoored language models , author=. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , pages=

    2024

  9. [35]

    arXiv e-prints , pages=

    Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models , author=. arXiv e-prints , pages=

  10. [36]

    arXiv preprint arXiv:2304.03277 , year=

    Instruction tuning with gpt-4 , author=. arXiv preprint arXiv:2304.03277 , year=

  11. [37]

    arXiv preprint arXiv:2407.16216 , year=

    A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more , author=. arXiv preprint arXiv:2407.16216 , year=

    arXiv

  12. [41]

    Information Systems , pages=

    GPT-5 and open-weight large language models: Advances in reasoning, transparency, and control , author=. Information Systems , pages=. 2025 , publisher=

    2025

  13. [42]

    Gemini 3 Pro , howpublished =

  14. [44]

    arXiv preprint arXiv:2504.00557 , year=

    Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features , author=. arXiv preprint arXiv:2504.00557 , year=

  15. [45]

    International Conference on Intelligent Computing , pages=

    Agro-LLaVA-Next: A Large Multimodal Model for Plant Diseases Recognization , author=. International Conference on Intelligent Computing , pages=. 2025 , organization=

    2025

  16. [46]

    International Journal of Computer Vision , volume=

    Safebench: A safety evaluation framework for multimodal large language models , author=. International Journal of Computer Vision , volume=. 2026 , publisher=

    2026

  17. [48]

    Anonymous , booktitle=. SaFeR-. 2025 , url=

    2025

  18. [50]

    Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLM s

    Liu, Xiaoyuan and Wang, Wenxuan and Yuan, Youliang and Huang, Jen-tse and Liu, Qiuzhi and He, Pinjia and Tu, Zhaopeng. Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLM s. Proceedings of the 63rd Annual Meeting of the Association for Computational ...

    2025 doi

  19. [51]

    F aithful RAG : Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation

    Zhang, Qinggang and Xiang, Zhishang and Xiao, Yilin and Wang, Le and Li, Junhui and Wang, Xinrun and Su, Jinsong. F aithful RAG : Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation. Proceedings of the 63rd Annual Meeting of the Association for Com...

    2025 doi

  20. [52]

    The Thirteenth International Conference on Learning Representations , year =

    To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts , author =. The Thirteenth International Conference on Learning Representations , year =

  21. [53]

    and Kailin Jiang and Zhi Gao and Chenrui Shi and Zilong Zheng and Siyuan Qi and Qing Li , booktitle =

    Yuntao Du. and Kailin Jiang and Zhi Gao and Chenrui Shi and Zilong Zheng and Siyuan Qi and Qing Li , booktitle =. 2025 , url =

    2025

  22. [54]

    The Thirteenth International Conference on Learning Representations , year =

    Knowledge Graph Finetuning Enhances Knowledge Manipulation in Large Language Models , author =. The Thirteenth International Conference on Learning Representations , year =

  23. [55]

    The Twelfth International Conference on Learning Representations , year =

    Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts , author =. The Twelfth International Conference on Learning Representations , year =

  24. [56]

    The Twelfth International Conference on Learning Representations , year =

    Massive Editing for Large Language Models via Meta Learning , author =. The Twelfth International Conference on Learning Representations , year =

  25. [57]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year =

    Boosting Knowledge Utilization in Multimodal Large Language Models via Adaptive Logits Fusion and Attention Reallocation , author =. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year =

  26. [58]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year =

    Conflict-Aware Knowledge Editing in the Wild: Semantic-Augmented Graph Representation for Unstructured Text , author =. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year =

  27. [59]

    Micro-Act: Mitigate Knowledge Conflict in Question Answering via Actionable Self-Reasoning

    Huo, Nan and Li, Jinyang and Qin, Bowen and Qu, Ge and Li, Xiaolong and Li, Xiaodong and Ma, Chenhao and Cheng, Reynold. Micro-Act: Mitigate Knowledge Conflict in Question Answering via Actionable Self-Reasoning. Proceedings of the 63rd Annual Meeting of the Association for Co...

    2025 doi

  28. [60]

    IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neurons , url =

    Shi, Dan and Jin, Renren and Shen, Tianhao and Dong, Weilong and Wu, Xinwei and Xiong, Deyi , booktitle =. IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neurons , url =. doi:10.52202/079017-0162 , editor =

    doi

  29. [61]

    ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in

    Zhaochen Su and Jun Zhang and Xiaoye Qu and Tong Zhu and Yanshu Li and Jiashuo Sun and Juntao Li and Min Zhang and Yu Cheng , booktitle =. ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in. 2024 , url =

    2024

  30. [62]

    Forty-second International Conference on Machine Learning , year =

    Taming Knowledge Conflicts in Language Models , author =. Forty-second International Conference on Machine Learning , year =

  31. [63]

    Forty-first International Conference on Machine Learning , year =

    Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning , author =. Forty-first International Conference on Machine Learning , year =

  32. [64]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    KnowPO: Knowledge-Aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Models , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2025 , pages =

    2025

  33. [65]

    Proceedings of the AAAI Conference on Artificial Intelligence , year =

    Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answering , author =. Proceedings of the AAAI Conference on Artificial Intelligence , year =

  34. [66]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Editing Language Model-Based Knowledge Graph Embeddings , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2024 , pages =

    2024

  35. [67]

    2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    Words or Vision: Do Vision-Language Models Have Blind Faith in Text? , author =. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    2025

  36. [68]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year =

    A Comprehensive Survey of Continual Learning: Theory, Method and Application , author =. IEEE Transactions on Pattern Analysis and Machine Intelligence , year =

  37. [69]

    The Thirteenth International Conference on Learning Representations , year =

    Deep Linear Probe Generators for Weight Space Learning , author =. The Thirteenth International Conference on Learning Representations , year =

  38. [70]

    The Thirteenth International Conference on Learning Representations , year =

    Monitoring Latent World States in Language Models with Propositional Probes , author =. The Thirteenth International Conference on Learning Representations , year =

  39. [71]

    The Thirteenth International Conference on Learning Representations , year =

    Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models , author =. The Thirteenth International Conference on Learning Representations , year =

  40. [72]

    Yu and Xuming Hu , booktitle =

    Aiwei Liu and Sheng Guan and Yiming Liu and Leyi Pan and Yifei Zhang and Liancheng Fang and Lijie Wen and Philip S. Yu and Xuming Hu , booktitle =. Can Watermarked. 2025 , url =

    2025

  41. [73]

    Forty-first International Conference on Machine Learning , year =

    Q-Probe: A Lightweight Approach to Reward Maximization for Language Models , author =. Forty-first International Conference on Machine Learning , year =

  42. [74]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year =

    Probing Neural Combinatorial Optimization Models , author =. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year =

  43. [75]

    arXiv preprint arXiv:2406.12718 , year =

    AGLA: Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention , author =. arXiv preprint arXiv:2406.12718 , year =

  44. [76]

    Steering Knowledge Selection Behaviours in

    Zhao, Yu and Devoto, Alessio and Hong, Giwon and Du, Xiaotang and Gema, Aryo Pradipta and Wang, Hongru and He, Xuanli and Wong, Kam-Fai and Minervini, Pasquale , booktitle =. Steering Knowledge Selection Behaviours in. 2025 , address =. doi:10.18653/v1/2025.naacl-long.264 , pages =

    2025 doi

  45. [77]

    2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding , author =. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    2024

  46. [78]

    Akari Asai and Zeqiu Wu and Yizhong Wang and Avirup Sil and Hannaneh Hajishirzi , booktitle =. Self-. 2024 , url =

    2024

  47. [79]

    GuardReasoner-

    Yue Liu and Shengfang Zhai and Mingzhe Du and Yulin Chen and Tri Cao and Hongcheng Gao and Cheng Wang and Xinfeng Li and Kun Wang and Junfeng Fang and Jiaheng Zhang and Bryan Hooi , booktitle =. GuardReasoner-. 2025 , url =

    2025

  48. [80]

    ICLR 2026 Conference Submission , year =

    Conflict-Aware Representation Editing for Robust Retrieval-Augmented Generation , author =. ICLR 2026 Conference Submission , year =

    2026

  49. [81]

    Proceedings of the AAAI Conference on Artificial Intelligence , year =

    Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning , author =. Proceedings of the AAAI Conference on Artificial Intelligence , year =

  50. [82]

    AAAI , crossref=

    Shengqiong Wu and Hao Fei and Liangming Pan and William Yang Wang and Shuicheng Yan and Tat-Seng Chua , title=. AAAI , crossref=. 2025 , cdate=

    2025

  51. [83]

    R ank C o T : Refining Knowledge for Retrieval-Augmented Generation through Ranking Chain-of-Thoughts

    Wu, Mingyan and Liu, Zhenghao and Yan, Yukun and Li, Xinze and Yu, Shi and Zeng, Zheni and Gu, Yu and Yu, Ge. R ank C o T : Refining Knowledge for Retrieval-Augmented Generation through Ranking Chain-of-Thoughts. Proceedings of the 63rd Annual Meeting of the Association for Co...

    2025 doi

  52. [84]

    Lingfeng Ming and Yadong Li and Song Chen and Jianhua Xu and Zenan Zhou and Weipeng Chen , title =

  53. [85]

    arXiv preprint arXiv:2503.10615 , year =

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization , author =. arXiv preprint arXiv:2503.10615 , year =

  54. [86]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =

    Xu, Guowei and Jin, Peng and Wu, Ziang and Li, Hao and Song, Yibing and Sun, Lichao and Yuan, Li , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =. 2025 , pages =

    2025

  55. [88]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    TruthfulQA: Measuring How Models Mimic Human Falsehoods , author =. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2022 , doi =

    2022

  56. [89]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

    PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year =

  57. [90]

    International Conference on Learning Representations (ICLR) , year =

    Sparse Autoencoders Find Highly Interpretable Features in Language Models , author =. International Conference on Learning Representations (ICLR) , year =

  58. [91]

    International Conference on Learning Representations (ICLR) , year =

    Scaling and Evaluating Sparse Autoencoders , author =. International Conference on Learning Representations (ICLR) , year =

  59. [92]

    International Conference on Learning Representations (ICLR) , year =

    Sparse Autoencoders Do Not Find Canonical Units of Analysis , author =. International Conference on Learning Representations (ICLR) , year =

  60. [93]

    NeurIPS 2024 Workshop: Safe Generative AI (SafeGenAI) , year =

    Applying Sparse Autoencoders to Unlearn Knowledge in Language Models , author =. NeurIPS 2024 Workshop: Safe Generative AI (SafeGenAI) , year =

    2024

  61. [94]

    International Conference on Learning Representations (ICLR) , year =

    Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching , author =. International Conference on Learning Representations (ICLR) , year =

  62. [95]

    International Conference on Learning Representations (ICLR) , year =

    Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking , author =. International Conference on Learning Representations (ICLR) , year =

  63. [97]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Metamorph: Multimodal understanding and generation via instruction tuning , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  64. [98]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Videodpo: Omni-preference alignment for video diffusion generation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  65. [99]

    Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in vlms , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa...

    2025

  66. [100]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  67. [101]

    Transactions of the association for computational linguistics , volume=

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions , author=. Transactions of the association for computational linguistics , volume=. 2014 , publisher=

    2014

  68. [102]

    European conference on computer vision , pages=

    Microsoft coco: Common objects in context , author=. European conference on computer vision , pages=. 2014 , organization=

    2014

  69. [103]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Ideator: Jailbreaking and benchmarking large vision-language models using themselves , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  70. [104]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Improved baselines with visual instruction tuning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  71. [105]

    Text summarization branches out , pages=

    Rouge: A package for automatic evaluation of summaries , author=. Text summarization branches out , pages=

  72. [109]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Cider: Consensus-based image description evaluation , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  73. [110]

    Patterns , volume=

    A survey of multilingual large language models , author=. Patterns , volume=. 2025 , publisher=

    2025

  74. [111]

    International Journal of Computer Vision , pages=

    Tokenpacker: Efficient visual projector for multimodal llm , author=. International Journal of Computer Vision , pages=. 2025 , publisher=

    2025

  75. [112]

    ACM Transactions on Multimedia Computing, Communications and Applications , year=

    Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation , author=. ACM Transactions on Multimedia Computing, Communications and Applications , year=

  76. [113]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Honeybee: Locality-enhanced projector for multimodal llm , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  77. [116]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Badtoken: Token-level backdoor attacks to multi-modal large language models , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  78. [117]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Finmmr: make financial numerical reasoning more multimodal, comprehensive, and challenging , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  79. [118]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    What's in the Image? A Deep-Dive into the Vision of Vision Language Models , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  80. [119]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Visually Interpretable Subtask Reasoning for Visual Question Answering , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  81. [120]

    Vl-interpret: An interactive visualization tool for interpreting vision-language transformers

    Aflalo, E., Du, M., Tseng, S.-Y., Liu, Y., Wu, C., Duan, N., and Lal, V. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp.\ 21406--21415, 2022

    2022

  82. [121]

    Flamingo: a visual language model for few-shot learning

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 0 23716--23736, 2022

    2022

  83. [122]

    Safer- VLM : Toward safety-aware fine-grained reasoning in multimodal models

    Anonymous. Safer- VLM : Toward safety-aware fine-grained reasoning in multimodal models. In Submitted to The Fourteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=8fswQTV8Dp. under review

    2025

  84. [123]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  85. [124]

    and Gavves, E

    Bereska, L. and Gavves, E. Mechanistic interpretability for ai safety--a review. arXiv preprint arXiv:2404.14082, 2024

    2024

  86. [125]

    Honeybee: Locality-enhanced projector for multimodal llm

    Cha, J., Kang, W., Mun, J., and Roh, B. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13817--13827, 2024

    2024

  87. [126]

    Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers

    Chefer, H., Gur, S., and Wolf, L. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 397--406, 2021

    2021

  88. [127]

    Hidden ghost hand: Unveiling backdoor vulnerabilities in mllm-powered mobile gui agents

    Cheng, P., Hu, H., Wu, Z., Wu, Z., Ju, T., Zhang, Z., and Liu, G. Hidden ghost hand: Unveiling backdoor vulnerabilities in mllm-powered mobile gui agents. arXiv preprint arXiv:2505.14418, 2025 a

    2025

  89. [128]

    Visually interpretable subtask reasoning for visual question answering

    Cheng, Y., Goel, A., and Bilen, H. Visually interpretable subtask reasoning for visual question answering. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 2760--2780, 2025 b

    2025

  90. [129]

    Explainable and interpretable multimodal large language models: A comprehensive survey

    Dang, Y., Huang, K., Huo, J., Yan, Y., Huang, S., Liu, D., Gao, M., Zhang, J., Qian, C., Wang, K., et al. Explainable and interpretable multimodal large language models: A comprehensive survey. arXiv preprint arXiv:2412.02104, 2024

    2024

  91. [130]

    Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers

    Dumas, C., Wendler, C., Veselovsky, V., Monea, G., and West, R. Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vo...

    2025 doi

  92. [131]

    Monitoring latent world states in language models with propositional probes

    Feng, J., Russell, S., and Steinhardt, J. Monitoring latent world states in language models with propositional probes. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=0yvZm2AjUr

    2025

  93. [132]

    Scaling and evaluating sparse autoencoders

    Gao, L., Dupre la Tour, T., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=tcsZt9ZNKD

    2025

  94. [133]

    Gemini 3 pro

    Google DeepMind . Gemini 3 pro. https://deepmind.google/models/gemini/pro/. Accessed: 2025-12-30

    2025

  95. [134]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 6904--6913, 2017

    2017

  96. [135]

    R., Ewart, A., and Sharkey, L

    Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=F76bwRSLeK

    2024

  97. [136]

    What's in the image? a deep-dive into the vision of vision language models

    Kaduri, O., Bagon, S., and Dekel, T. What's in the image? a deep-dive into the vision of vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 14549--14558, 2025

    2025

  98. [137]

    Deep linear probe generators for weight space learning

    Kahana, J., Horwitz, E., Shuval, I., and Hoshen, Y. Deep linear probe generators for weight space learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=XoYdD3m0mv

    2025

  99. [138]

    Kim, J., Kang, S., Park, J., Kim, J., and Hwang, S. J. Interpreting attention heads for image-to-text information flow in large vision-language models. arXiv preprint arXiv:2509.17588, 2025

    2025

  100. [139]

    and Reuel, A

    Lamparth, M. and Reuel, A. Analyzing and editing inner mechanisms of backdoored language models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 2362--2373, 2024

    2024

  101. [140]

    T., Bloom, J

    Leask, P., Bussmann, B., Pearce, M. T., Bloom, J. I., Tigges, C., Al Moubayed, N., Sharkey, L., and Nanda, N. Sparse autoencoders do not find canonical units of analysis. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=...

    2025

  102. [141]

    H., Zheng, W., Zhou, Y., Mai, Y., Roberts, J

    Lee, T., Tu, H., Wong, C. H., Zheng, W., Zhou, Y., Mai, Y., Roberts, J. S., Yasunaga, M., Yao, H., Xie, C., et al. Vhelm: A holistic evaluation of vision language models. Advances in Neural Information Processing Systems, 37: 0 140632--140666, 2024

    2024

  103. [142]

    Gpt-5 and open-weight large language models: Advances in reasoning, transparency, and control

    Leon, M. Gpt-5 and open-weight large language models: Advances in reasoning, transparency, and control. Information Systems, pp.\ 102620, 2025

    2025

  104. [143]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 a

    2023

  105. [144]

    A survey on benchmarks of multimodal large language models

    Li, J., Lu, W., Fei, H., Luo, M., Dai, M., Xia, M., Jin, Y., Gan, Z., Qi, D., Fu, C., et al. A survey on benchmarks of multimodal large language models. arXiv preprint arXiv:2408.08632, 2024 a

    2024

  106. [145]

    Backdoorvlm: A benchmark for backdoor attacks on vision-language models

    Li, J., Li, Y., Huang, H., Chen, Y., Wang, X., Wang, Y., Ma, X., and Jiang, Y.-G. Backdoorvlm: A benchmark for backdoor attacks on vision-language models. arXiv preprint arXiv:2511.18921, 2025 a

    2025

  107. [146]

    Iag: Input-aware backdoor attack on vlms for visual grounding

    Li, J., Xu, B., and Zhang, D. Iag: Input-aware backdoor attack on vlms for visual grounding. arXiv preprint arXiv:2508.09456, 2025 b

    2025

  108. [147]

    M., Wattenberg, M., and Brandfonbrener, D

    Li, K., Jelassi, S., Zhang, H., Kakade, S. M., Wattenberg, M., and Brandfonbrener, D. Q-probe: A lightweight approach to reward maximization for language models. In Forty-first International Conference on Machine Learning, 2024 b . URL https://openreview.net/forum?id=gxOQEMRbRa

    2024

  109. [148]

    Tokenpacker: Efficient visual projector for multimodal llm

    Li, W., Yuan, Y., Liu, J., Tang, D., Wang, S., Qin, J., Zhu, J., and Zhang, L. Tokenpacker: Efficient visual projector for multimodal llm. International Journal of Computer Vision, pp.\ 1--19, 2025 c

    2025

  110. [149]

    Trust the process? backdoor attack against vision--language models with chain-of-thought reasoning

    Li, X., Lin, Y., Liu, Z., Xu, X., Li, Q., Zhou, L., and Ji, S. Trust the process? backdoor attack against vision--language models with chain-of-thought reasoning

  111. [150]

    X., and Wen, J.-R

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023 b

    2023

  112. [151]

    Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models

    Li, Y., Huang, H., Zhao, Y., Ma, X., and Sun, J. Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models. arXiv e-prints, pp.\ arXiv--2408, 2024 c

    2024

  113. [152]

    Benchmark evaluations, applications, and challenges of large vision language models: A survey

    Li, Z., Wu, X., Du, H., Nghiem, H., and Shi, G. Benchmark evaluations, applications, and challenges of large vision language models: A survey. arXiv preprint arXiv:2501.02189, 1, 2025 d

    2025

  114. [153]

    Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models

    Liang, J., Liang, S., Liu, A., and Cao, X. Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models. International Journal of Computer Vision, pp.\ 1--20, 2025 a

    2025

  115. [154]

    Revisiting backdoor attacks against large vision-language models from domain shift

    Liang, S., Liang, J., Pang, T., Du, C., Liu, A., Zhu, M., Cao, X., and Tao, D. Revisiting backdoor attacks against large vision-language models from domain shift. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 9477--9486, 2025 b

    2025

  116. [155]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.\ 74--81, 2004

    2004

  117. [156]

    Backdoor collapse: Eliminating unknown threats via known backdoor aggregation in language models

    Lin, L., Yu, M., Aloqaily, M., Zhou, Z., Wang, K., Pang, L., Mehrotra, P., and Wen, Q. Backdoor collapse: Eliminating unknown threats via known backdoor aggregation in language models. arXiv preprint arXiv:2510.10265, 2025

    2025

  118. [157]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision, pp.\ 740--755. Springer, 2014

    2014

  119. [158]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023

    2023

  120. [159]

    Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 26296--26306, 2024

    2024

  121. [160]

    Natural reflection backdoor attack on vision language model for autonomous driving

    Liu, M., Liang, S., Howlader, K., Wang, L., Tao, D., and Zhang, W. Natural reflection backdoor attack on vision language model for autonomous driving. arXiv preprint arXiv:2505.06413, 2025 a

    2025

  122. [161]

    Videodpo: Omni-preference alignment for video diffusion generation

    Liu, R., Wu, H., Zheng, Z., Wei, C., He, Y., Pi, R., and Chen, Q. Videodpo: Omni-preference alignment for video diffusion generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 8009--8019, 2025 b

    2025

  123. [162]

    Test-time backdoor attacks on multimodal large language models

    Lu, D., Pang, T., Du, C., Liu, Q., Yang, X., and Lin, M. Test-time backdoor attacks on multimodal large language models. arXiv preprint arXiv:2402.08577, 2024

    2024

  124. [163]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    2023

  125. [164]

    Trojvlm: Backdoor attack against vision language models

    Lyu, W., Pang, L., Ma, T., Ling, H., and Chen, C. Trojvlm: Backdoor attack against vision language models. In European Conference on Computer Vision, pp.\ 467--483. Springer, 2024 a

    2024

  126. [165]

    Backdooring vision-language models with out-of-distribution data

    Lyu, W., Yao, J., Gupta, S., Pang, L., Sun, T., Yi, L., Hu, L., Ling, H., and Chen, C. Backdooring vision-language models with out-of-distribution data. arXiv preprint arXiv:2410.01264, 2024 b

    2024

  127. [166]

    Safety at scale: A comprehensive survey of large model safety

    Ma, X., Gao, Y., Wang, Y., Wang, R., Wang, X., Sun, Y., Ding, Y., Xu, H., Chen, Y., Zhao, Y., et al. Safety at scale: A comprehensive survey of large model safety. arXiv preprint arXiv:2502.05206, 2025

    2025 arXiv

  128. [167]

    Is this the subspace you are looking for? an interpretability illusion for subspace activation patching

    Makelov, A., Lange, G., Geiger, A., and Nanda, N. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=Ebt7JgMHv1

    2024

  129. [168]

    Towards interpreting visual information processing in vision-language models

    Neo, C., Ong, L., Torr, P., Geva, M., Krueger, D., and Barez, F. Towards interpreting visual information processing in vision-language models. arXiv preprint arXiv:2410.07149, 2024

    2024

  130. [169]

    Physical backdoor attack can jeopardize driving with vision-large-language models

    Ni, Z., Ye, R., Wei, Y., Xiang, Z., Wang, Y., and Chen, S. Physical backdoor attack can jeopardize driving with vision-large-language models. arXiv preprint arXiv:2404.12916, 2024

    2024

  131. [170]

    Same task, different circuits: Disentangling modality-specific mechanisms in vlms

    Nikankin, Y., Arad, D., Gandelsman, Y., and Belinkov, Y. Same task, different circuits: Disentangling modality-specific mechanisms in vlms. arXiv preprint arXiv:2506.09047, 2025

    2025

  132. [171]

    K., Vaishnavi, V., Saxena, A., and Goswami, K

    Phukan, A., Divyansh, D., Morj, H. K., Vaishnavi, V., Saxena, A., and Goswami, K. Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in vlms. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association fo...

    2025

  133. [172]

    Fine-tuning enhances existing mechanisms: A case study on entity tracking

    Prakash, N., Rott Shaham, T., Haklay, T., Belinkov, Y., and Bau, D. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In International Conference on Learning Representations (ICLR), 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/2082...

    2024

  134. [173]

    Qin, L., Chen, Q., Zhou, Y., Chen, Z., Li, Y., Liao, L., Li, M., Che, W., and Yu, P. S. A survey of multilingual large language models. Patterns, 6 0 (1), 2025

    2025

  135. [174]

    Concept-guided backdoor attack on vision language models

    Shen, H., Lyu, W., Xu, H., and Ma, T. Concept-guided backdoor attack on vision language models. arXiv preprint arXiv:2512.00713, 2025

    2025

  136. [175]

    Finmmr: make financial numerical reasoning more multimodal, comprehensive, and challenging

    Tang, Z., Liu, J., Yang, Z., Li, R., Rong, Z., He, H., Hao, Z., Hu, X., Ji, K., Ma, Z., et al. Finmmr: make financial numerical reasoning more multimodal, comprehensive, and challenging. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3245--3257, 2025

    2025

  137. [176]

    Metamorph: Multimodal understanding and generation via instruction tuning

    Tong, S., Fan, D., Li, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., and Liu, Z. Metamorph: Multimodal understanding and generation via instruction tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 17001--17012, 2025

    2025

  138. [177]

    Cider: Consensus-based image description evaluation

    Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4566--4575, 2015

    2015

  139. [178]

    A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment

    Wang, K., Zhang, G., Zhou, Z., Wu, J., Yu, M., Zhao, S., Yin, C., Fu, J., Yan, Y., Luo, H., et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504.15585, 2025 a

    2025

  140. [179]

    V-seam: Visual semantic editing and attention modulating for causal interpretability of vision-language models

    Wang, Q., Hu, J., and Jiang, M. V-seam: Visual semantic editing and attention modulating for causal interpretability of vision-language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp.\ 17407--17431, 2025 b

    2025

  141. [180]

    Ideator: Jailbreaking and benchmarking large vision-language models using themselves

    Wang, R., Li, J., Wang, Y., Wang, B., Wang, X., Teng, Y., Wang, Y., Ma, X., and Jiang, Y.-G. Ideator: Jailbreaking and benchmarking large vision-language models using themselves. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 8875--8884, 2025 c

    2025

  142. [181]

    Internvl3

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025 d

    2025

  143. [182]

    Y., et al

    Wang, X., Pan, H., Zhang, H., Li, M., Hu, S., Zhou, Z., Xue, L., Liu, A., Jiang, Y., Zhang, L. Y., et al. Trojanrobot: Physical-world backdoor attacks against vlm-based robotic manipulation. arXiv preprint arXiv:2411.11683, 2024

    2024

  144. [183]

    Multimodal chain-of-thought reasoning: A comprehensive survey

    Wang, Y., Wu, S., Zhang, Y., Yan, S., Liu, Z., Luo, J., and Fei, H. Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint arXiv:2503.12605, 2025 e

    2025

  145. [184]

    Agro-llava-next: A large multimodal model for plant diseases recognization

    Xu, G., Zhao, W., Bie, Y., Ge, M., Cui, Z., and Wang, Y. Agro-llava-next: A large multimodal model for plant diseases recognization. In International Conference on Intelligent Computing, pp.\ 291--302. Springer, 2025 a

    2025

  146. [185]

    Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y., Dang, K., et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215, 2025 b

    2025

  147. [186]

    Shadowcast: Stealthy data poisoning attacks against vision-language models

    Xu, Y., Yao, J., Shu, M., Sun, Y., Wu, Z., Yu, N., Goldstein, T., and Huang, F. Shadowcast: Stealthy data poisoning attacks against vision-language models. Advances in Neural Information Processing Systems, 37: 0 57733--57764, 2024

    2024

  148. [187]

    and Gong, X

    Yang, X. and Gong, X. Re-purposing sam into efficient visual projectors for mllm-based referring image segmentation. ACM Transactions on Multimedia Computing, Communications and Applications, 2025

    2025

  149. [188]

    Safebench: A safety evaluation framework for multimodal large language models

    Ying, Z., Liu, A., Liang, S., Huang, L., Guo, J., Zhou, W., Liu, X., and Tao, D. Safebench: A safety evaluation framework for multimodal large language models. International Journal of Computer Vision, 134 0 (1): 0 18, 2026

    2026

  150. [189]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

    Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2: 0 67--78, 2014

    2014

  151. [190]

    Backdoor attribution: Elucidating and controlling backdoor in language models

    Yu, M., Zhou, Z., Aloqaily, M., Wang, K., Huang, B., Wang, S., Jin, Y., and Wen, Q. Backdoor attribution: Elucidating and controlling backdoor in language models. arXiv preprint arXiv:2509.21761, 2025

    2025

  152. [191]

    Z., and Sun, L

    Yuan, Z., Shi, J., Zhou, P., Gong, N. Z., and Sun, L. Badtoken: Token-level backdoor attacks to multi-modal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 29927--29936, 2025

    2025

  153. [192]

    Visual backdoor attacks on mllm embodied decision making via contrastive trigger learning

    Zhan, Q., Ha, H., Yang, R., Xu, S., Chen, H., Gui, L.-Y., Wang, Y.-X., Zhang, H., Ji, H., and Kang, D. Visual backdoor attacks on mllm embodied decision making via contrastive trigger learning. arXiv preprint arXiv:2510.27623, 2025

    2025

  154. [193]

    Zhang, Z., Ma, Y., Cao, Z., and Lau, H. C. Probing neural combinatorial optimization models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=ycnc9aLnQu

    2025

  155. [194]

    Usb: A comprehensive and unified safety evaluation benchmark for multimodal large language models

    Zheng, B., Chen, G., Zhong, H., Teng, Q., Tan, Y., Liu, Z., Wang, W., Liu, J., Yang, J., Jing, H., et al. Usb: A comprehensive and unified safety evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2505.23793, 2025

    2025

  156. [195]

    Backdoor attack on vision language models with stealthy semantic manipulation

    Zhong, Z., Sun, Z., Liu, Y., He, X., and Tao, G. Backdoor attack on vision language models with stealthy semantic manipulation. arXiv preprint arXiv:2506.07214, 2025

    2025

Pith tools

Reviewed May 10, 2026 · model on record in the stance chip above.