REVIEW 4 cited by
Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
With the widespread deployment of Multimodal Large Language Models (MLLMs) for visual-reasoning tasks, improving their safety has become crucial. Recent research indicates that despite training-time safety alignment, these models remain vulnerable to jailbreak attacks. In this work, we first highlight an important safety gap to describe that alignment achieved solely through safety training may be insufficient against jailbreak attacks. To address this vulnerability, we propose Immune, an inference-time defense framework that leverages a safe reward model through controlled decoding to defend against jailbreak attacks. Additionally, we provide a mathematical characterization of Immune, offering insights on why it improves safety against jailbreaks. Extensive evaluations on diverse jailbreak benchmarks using recent MLLMs reveal that Immune effectively enhances model safety while preserving the model's original capabilities. For instance, against text-based jailbreak attacks on LLaVA-1.6, Immune reduces the attack success rate by 57.82% and 16.78% compared to the base MLLM and state-of-the-art defense strategy, respectively.
Forward citations
Cited by 4 Pith papers
-
Self-Aware Safety Augmentation: Leveraging Internal Semantic Understanding to Enhance Safety in Vision-Language Models
A tuning-free method that projects middle-layer semantic representations back onto early safety layers, improving vision-language model safety with minimal utility loss.
-
GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs
GrAInS uses Integrated Gradients to identify the most influential tokens, then builds layer-wise steering vectors that improve truthfulness, reduce hallucination, and preserve general capabilities in LLMs and VLMs.
-
Test-Time Immunization: A Universal Defense Framework Against Jailbreaks for (Multimodal) Large Language Models
TIM detects jailbreaks at test time using a gist-token classifier and adaptively fine-tunes the model with LoRA on detected harmful prompts, cutting attack success rates to near zero on tested benchmarks.
-
A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations
A survey of LVLM safety that adds a lifecycle taxonomy and new benchmark results showing Janus-Pro-7B has weaker safety than several open-source LVLMs.
Discussion (0). Continue with ORCID to comment.