REVIEW 7 cited by
MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite the superior capabilities of Multimodal Large Language Models (MLLMs) across diverse tasks, they still face significant trustworthiness challenges. Yet, current literature on the assessment of trustworthy MLLMs remains limited, lacking a holistic evaluation to offer thorough insights into future improvements. In this work, we establish MultiTrust, the first comprehensive and unified benchmark on the trustworthiness of MLLMs across five primary aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark employs a rigorous evaluation strategy that addresses both multimodal risks and cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets. Extensive experiments with 21 modern MLLMs reveal some previously unexplored trustworthiness issues and risks, highlighting the complexities introduced by the multimodality and underscoring the necessity for advanced methodologies to enhance their reliability. For instance, typical proprietary models still struggle with the perception of visually confusing images and are vulnerable to multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to disclose privacy in text and reveal ideological and cultural biases even when paired with irrelevant images in inference, indicating that the multimodality amplifies the internal risks from base LLMs. Additionally, we release a scalable toolbox for standardized trustworthiness research, aiming to facilitate future advancements in this important field. Code and resources are publicly available at: https://multi-trust.github.io/.
Forward citations
Cited by 7 Pith papers
-
One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models
A single adversarial image can make a unified vision-language model misclassify the same object across captioning, detection, region classification, and localization, and the new CrossVLAD benchmark and CRAFT attack m...
-
VLMs Can Aggregate Scattered Training Patches
Open-source VLMs can infer image IDs or safety labels after training only on scattered patches of those images, a capability that can be abused to bypass image moderation.
-
USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
USB-SafeBench is a unified MLLM safety benchmark with 61 risk categories, 4 modality combinations, and dual-language vulnerability and oversensitivity tests.
-
FairReason: Balancing Reasoning and Social Bias in MLLMs
A 1:4 debias-to-reasoning training mix under GRPO reinforcement learning yields the best bias-reasoning trade-off in small MLLMs, cutting measured stereotype scores by about 10% while retaining about 88% of reasoning ...
-
A Multimodal Automatic Redteaming Evaluation based on Atomic Jailbreak Strategy Decoupling and Combination
A new jailbreak framework, HACA, combines atomic text and image attack strategies selected by a cross-modal planner and generates attacks with LLMs and text-to-image models, reaching 95.48% average attack success acro...
-
Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models
An alternating image-text optimization produces a universal adversarial suffix and image that transfer across open multimodal LLMs more effectively than single-modality jailbreaks.
-
A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations
A survey of LVLM safety that adds a lifecycle taxonomy and new benchmark results showing Janus-Pro-7B has weaker safety than several open-source LVLMs.
Discussion (0). Sign in to comment.