Pith. sign in

REVIEW 5 cited by

Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.12662 v2 pith:PNKFBKY3 submitted 2024-10-16 cs.CV cs.AIcs.CL

Shicheng Xu , Liang Pang , Yunchang Zhu , Huawei Shen , Xueqi Cheng This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords safetyvision-languagemechanismvisionalignmenthiddenllmslvlms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Vision-language alignment in Large Vision-Language Models (LVLMs) successfully enables LLMs to understand visual input. However, we find that existing vision-language alignment methods fail to transfer the existing safety mechanism for text in LLMs to vision, which leads to vulnerabilities in toxic image. To explore the cause of this problem, we give the insightful explanation of where and how the safety mechanism of LVLMs operates and conduct comparative analysis between text and vision. We find that the hidden states at the specific transformer layers play a crucial role in the successful activation of safety mechanism, while the vision-language alignment at hidden states level in current methods is insufficient. This results in a semantic shift for input images compared to text in hidden states, therefore misleads the safety mechanism. To address this, we propose a novel Text-Guided vision-language Alignment method (TGA) for LVLMs. TGA retrieves the texts related to input vision and uses them to guide the projection of vision into the hidden states space in LLMs. Experiments show that TGA not only successfully transfers the safety mechanism for text in basic LLMs to vision in vision-language alignment for LVLMs without any safety fine-tuning on the visual modality but also maintains the general performance on various vision tasks (Safe and Good).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Backdoor Cleaning without External Guidance in MLLM Fine-tuning

    cs.CR 2025-05 conditional novelty 6.0 of 10

    BYE filters backdoored training images from MLLM fine-tuning by clustering low attention entropy across selected layers.

  2. Cross-modal Information Flow in Multimodal Large Language Models

    cs.AI 2024-11 conditional novelty 6.0 of 10

    In LLaVA multimodal models, visual information flows into question token representations in two stages, global then object-specific, before propagating to the final answer position.

  3. Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment

    cs.CV 2025-10 conditional novelty 5.0 of 10

    Risk-adaptive activation steering, guided by a prototype-similarity risk score computed on the first three response tokens, substantially reduces multimodal jailbreak success rates across four MLLMs while preserving utility.

  4. A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations

    cs.CR 2025-02 conditional novelty 4.0 of 10

    A survey of LVLM safety that adds a lifecycle taxonomy and new benchmark results showing Janus-Pro-7B has weaker safety than several open-source LVLMs.

  5. Foundation Model Driven Robotics: A Comprehensive Review

    cs.RO 2025-07 conditional novelty 2.0 of 10

    A review of foundation-model-driven robotics that synthesizes recent work across perception, planning, control, HRI, simulation, and sim-to-real transfer, and highlights open challenges.

Pith tools