Pith. sign in

REVIEW 9 cited by

MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.02906 v3 pith:XLEZ5HO5 submitted 2024-01-05 cs.CR cs.CLcs.CV

Renjie Pi , Tianyang Han , Jianshu Zhang , Yueqi Xie , Rui Pan , Qing Lian , Hanze Dong , Jipeng Zhang
show 1 more author
Tong Zhang
This is my paper
classification cs.CRcs.CLcs.CV
keywords mllmsharmfullanguagemllm-protectorresponsessafetyalignmentattacks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The deployment of multimodal large language models (MLLMs) has brought forth a unique vulnerability: susceptibility to malicious attacks through visual inputs. This paper investigates the novel challenge of defending MLLMs against such attacks. Compared to large language models (LLMs), MLLMs include an additional image modality. We discover that images act as a ``foreign language" that is not considered during safety alignment, making MLLMs more prone to producing harmful responses. Unfortunately, unlike the discrete tokens considered in text-based LLMs, the continuous nature of image signals presents significant alignment challenges, which poses difficulty to thoroughly cover all possible scenarios. This vulnerability is exacerbated by the fact that most state-of-the-art MLLMs are fine-tuned on limited image-text pairs that are much fewer than the extensive text-based pretraining corpus, which makes the MLLMs more prone to catastrophic forgetting of their original abilities during safety fine-tuning. To tackle these challenges, we introduce MLLM-Protector, a plug-and-play strategy that solves two subtasks: 1) identifying harmful responses via a lightweight harm detector, and 2) transforming harmful responses into harmless ones via a detoxifier. This approach effectively mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs. Our results demonstrate that MLLM-Protector offers a robust solution to a previously unaddressed aspect of MLLM security.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The First Differentiable Transfer-Based Algorithm for Discrete MicroLED Repair

    cs.LG 2025-08 reject novelty 6.0 of 10

    The submission's abstract announces a differentiable microLED repair planner, yet its full text is the LoD jailbreak-detection paper (arXiv 2508.09201v4), leaving the claimed result without any derivation.

  2. Mitigating Object Hallucination via Robust Local Perception Search

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free decoding method that uses an MLLM's own local object descriptions as a reward prior, combined with CLIP similarity, to cut object hallucination, especially under adversarial image noise.

  3. Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gap

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Reducing the modality gap between image and text embeddings during LVLM pretraining reduces unsafe response rates by up to 16.3% across models and benchmarks.

  4. Backdoor Cleaning without External Guidance in MLLM Fine-tuning

    cs.CR 2025-05 conditional novelty 6.0 of 10

    BYE filters backdoored training images from MLLM fine-tuning by clustering low attention entropy across selected layers.

  5. Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment

    cs.CV 2025-10 conditional novelty 5.0 of 10

    Risk-adaptive activation steering, guided by a prototype-similarity risk score computed on the first three response tokens, substantially reduces multimodal jailbreak success rates across four MLLMs while preserving utility.

  6. Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Uncertainty-o estimates uncertainty in large multimodal models by perturbing prompts and computing entropy over semantically clustered answers, improving hallucination detection across five modalities.

  7. R^3: Advertisement Compliance Rectification via Group-Relative Experience Extractor and Curriculum Reinforcement

    cs.CL 2026-07 conditional novelty 4.0 of 10

    R3 uses curriculum RL with hierarchical rewards and a group-relative experience extractor to automatically rectify non-compliant text in video ads while preserving semantic intent.

  8. A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations

    cs.CR 2025-02 conditional novelty 4.0 of 10

    A survey of LVLM safety that adds a lifecycle taxonomy and new benchmark results showing Janus-Pro-7B has weaker safety than several open-source LVLMs.

  9. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools