Pith. sign in

REVIEW 11 cited by

ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.06625 v2 pith:F4CTTAVU submitted 2024-10-09 cs.CV cs.CLcs.LG

Yi Ding , Bolian Li , Ruqi Zhang This is my paper
classification cs.CVcs.CLcs.LG
keywords safetyaligningevaluatingdefensehelpfulnessinputslanguagemethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Vision Language Models (VLMs) have become essential backbones for multimodal intelligence, yet significant safety challenges limit their real-world application. While textual inputs are often effectively safeguarded, adversarial visual inputs can easily bypass VLM defense mechanisms. Existing defense methods are either resource-intensive, requiring substantial data and compute, or fail to simultaneously ensure safety and usefulness in responses. To address these limitations, we propose a novel two-phase inference-time alignment framework, Evaluating Then Aligning (ETA): 1) Evaluating input visual contents and output responses to establish a robust safety awareness in multimodal settings, and 2) Aligning unsafe behaviors at both shallow and deep levels by conditioning the VLMs' generative distribution with an interference prefix and performing sentence-level best-of-N to search the most harmless and helpful generation paths. Extensive experiments show that ETA outperforms baseline methods in terms of harmlessness, helpfulness, and efficiency, reducing the unsafe rate by 87.5% in cross-modality attacks and achieving 96.6% win-ties in GPT-4 helpfulness evaluation. The code is publicly available at https://github.com/DripNowhy/ETA.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A small set of model neurons shared across languages and modalities can transfer English-only safety training to multilingual and multimodal refusal behavior.

  2. Visual Token Compression Enhances Robustness of MLLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Pruning visual tokens farthest from the text feature space at selected 'robust' layers improves MLLM jailbreak defense (average +13.29% RAR) and slightly reduces hallucination.

  3. Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Models

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Precise Shield identifies safety neurons in VLLMs via activation contrasts and aligns only them with gradient masking, boosting safety, preserving generalization, and enabling zero-shot cross-lingual and cross-modal transfer.

  4. Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Vision-language models consistently recognize unsafe content better from text than from images, and a simplified reinforcement learning fine-tune narrows that gap.

  5. Mitigating Object Hallucination via Robust Local Perception Search

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free decoding method that uses an MLLM's own local object descriptions as a reward prior, combined with CLIP similarity, to cut object hallucination, especially under adversarial image noise.

  6. AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenders

    cs.CV 2025-05 conditional novelty 6.0 of 10

    AMIA is an inference-only jailbreak defense that masks text-irrelevant image patches and prompts single-pass intention analysis, lifting average defense success on LVLMs from 52.4% to 81.7%.

  7. Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gap

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Reducing the modality gap between image and text embeddings during LVLM pretraining reduces unsafe response rates by up to 16.3% across models and benchmarks.

  8. Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment

    cs.CV 2025-10 conditional novelty 5.0 of 10

    Risk-adaptive activation steering, guided by a prototype-similarity risk score computed on the first three response tokens, substantially reduces multimodal jailbreak success rates across four MLLMs while preserving utility.

  9. Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A two-stage evaluation framework and token-projection analysis show that LVLMs encode harmful semantic cues from images even without OCR, while remaining vulnerable to cross-modal attacks.

  10. A Survey on Training-free Alignment of Large Language Models

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A survey that catalogs and categorizes training-free LLM alignment methods into pre-decoding, in-decoding, and post-decoding, with a limited experimental comparison on one model.

  11. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools