REVIEW 6 cited by
Backdoor Cleaning without External Guidance in MLLM Fine-tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multimodal Large Language Models (MLLMs) are increasingly deployed in fine-tuning-as-a-service (FTaaS) settings, where user-submitted datasets adapt general-purpose models to downstream tasks. This flexibility, however, introduces serious security risks, as malicious fine-tuning can implant backdoors into MLLMs with minimal effort. In this paper, we observe that backdoor triggers systematically disrupt cross-modal processing by causing abnormal attention concentration on non-semantic regions--a phenomenon we term attention collapse. Based on this insight, we propose Believe Your Eyes (BYE), a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples. BYE operates via a three-stage pipeline: (1) extracting attention maps using the fine-tuned model, (2) computing entropy scores and profiling sensitive layers via bimodal separation, and (3) performing unsupervised clustering to remove suspicious samples. Unlike prior defenses, BYE equires no clean supervision, auxiliary labels, or model modifications. Extensive experiments across various datasets, models, and diverse trigger types validate BYE's effectiveness: it achieves near-zero attack success rates while maintaining clean-task performance, offering a robust and generalizable solution against backdoor threats in MLLMs.
Forward citations
Cited by 6 Pith papers
-
EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
EMO-R3, which combines a three-step emotional reasoning prompt with a reward for the model agreeing with its own image–emotion judgments, raises visual emotion-recognition accuracy by about one point over plain GRPO.
-
Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees
CIC selects the largest uncertainty threshold whose Hoeffding or Clopper–Pearson upper bound on acceptance-conditioned error stays ≤ α, guaranteeing finite-sample risk control under exchangeability.
-
TECP: Token-Entropy Conformal Prediction for LLMs
TECP applies split conformal prediction with token-entropy nonconformity scores to LLM question answering and reports reliable coverage, but its implementation requires the token probabilities it claims to avoid.
-
FADE: Adversarial Concept Erasure in Flow Models
FADE combines adversarial training with trajectory preservation to erase concepts from diffusion models, reporting state-of-the-art erasure on Stable Diffusion benchmarks, but the evidence is incomplete and the theore...
-
Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees
Repeatedly sampling an LLM and using the entropy of answer frequencies yields conformal prediction sets for multiple-choice questions with empirical miscoverage near the target, and AUROC comparable to logit-based scores.
-
Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control
A p-value reformulation of split conformal prediction for LLM multiple-choice QA achieves nominal miscoverage control on MMLU and MMLU-Pro.
Discussion (0). Sign in to comment.