InverTune: Removing Backdoors from Multimodal Contrastive Learning Models via Trigger Inversion and Activation Tuning

Bo Du; Mengyuan Sun; Yuchen Liu; Yu Li; Yunjie Ge

REVIEW 4 major objections 5 minor 54 references

InverTune: Removing Backdoors from Multimodal Contrastive Learning Models via Trigger Inversion and Activation Tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims InverTune, a three-stage defense, removes backdoors from multimodal contrastive learning models without knowing the attack target or poison data, reducing average attack success rate by 97.87% while limiting clean…

desk verdict A genuinely new MCL backdoor-defense pipeline whose load-bearing target-identification step is under-validated and whose headline numbers are cherry-picked; worth serious refereeing, not as-is. read the letter →

arxiv 2506.12411 v1 pith:J26OONP3 submitted 2025-06-14 cs.CR cs.CV

Mengyuan Sun , Yu Li , Yuchen Liu , Bo Du , Yunjie Ge This is my paper

classification cs.CRcs.CV

keywords backdoordefensemultimodalcontrastivelearningCLIPtriggerinversiontargetlabelidentificationactivationtuningadversarialperturbationmodelpurification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

InverTune is a defense that removes backdoors from CLIP-style multimodal models using only a small amount of clean data and no knowledge of the attack's target label or poisoned dataset. The paper's central claim is that a poisoned encoder can be purified in three steps: first identify which label the backdoor secretly favors by comparing how often adversarial perturbations route predictions to each class; then invert the visual trigger by optimizing a mask-pattern pair in both the visual and cross-modal embedding spaces; finally fine-tune only the neurons that respond abnormally to the inverted trigger. If the claim holds, defenders can restore a compromised multimodal model to near-clean behavior against six backdoor attacks, including the state-of-the-art BadCLIP, with attack success rates falling from roughly 98% to under 1% and clean accuracy dropping only about 3%. The practical significance is that backdoor removal becomes feasible in the realistic setting where a defender receives an untrusted pretrained encoder with no poisoned data and no knowledge of the attacker's goal.

What carries the argument

The mechanism is a three-stage pipeline. Stage 1 (target identification) computes a universal adversarial perturbation for the suspected encoder, then scores each label by the increase in prediction frequency on perturbed inputs over clean inputs, $y_t = \arg\max_y (P_{\text{adv}}(y)-P_{\text{clean}}(y))$; the paper's observations justify this by showing adversarial examples route through backdoor-induced vulnerability zones toward the target. Stage 2 (trigger inversion) parameterizes a trigger as a mask-pattern pair $(\mathbf{m},\mathbf{t}_{\text{img}})$ and optimizes it with four losses: cross-modal alignment via InfoNCE toward the identified target text, embedding-space preservation against the clean image, structural similarity for visual stealth, and $\ell^1$ sparsity on the mask. Stage 3 (activation tuning) selects layers whose normalized activation divergence between clean and triggered inputs exceeds mean plus one standard deviation, clusters per-layer neuron activation differences with K-means, and fine-tunes only those critical neurons using an activation-alignment loss plus a cross-modal consistency loss that anchors the model to its pre-tuning behavior.

What would settle it

A concrete test: poison a CLIP model with a trigger targeting a rare class such as 'ski', then construct the adversarial perturbation to push predictions toward a visually different class such as 'banana'; if InverTune's target identification returns 'banana' instead of 'ski' and the subsequent tuning leaves the attack success rate above a few percent, the central claim is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, InverTune is the first defense that can identify the backdoor target label in a multimodal contrastive model, and then use that identification to invert the trigger and surgically remove the backdoor. The paper claims that with only a 50K-image clean set for inversion and a single 64-image batch for tuning, it reduces attack success rates to near zero across six attacks on both ImageNet classification and MSCOCO retrieval, including reducing BadCLIP from 98.36% to 0.49% on ImageNet and from 99.28% to 0.68% on MSCOCO. Clean accuracy degrades by only 3.07% on average, and the method is stable across target labels and across CNN and Vision Transformer backbones. The authors also report that inverted triggers reproduce the spatial distribution of the original triggers, which they take as evidence that the inversion captures the attack's actual mechanism rather than a surface-level approximation.

Load-bearing premise

The whole defense rests on the assumption that a single well-chosen perturbation will make the model misclassify many images and that those mistakes will land on the backdoor's true target label more often than on any other label.

Editorial extensions

If this is right

A defender with a small clean set and no attack knowledge can purify a compromised CLIP encoder to below 1% attack success rate on both classification and retrieval.
Because the target label is identified before inversion, defense effort can be concentrated on the actual attack route instead of scanning all possible targets, which is otherwise infeasible in open-vocabulary models.
Reported clean accuracy stays near the un-defended model on six attacks, indicating the security-utility trade-off that limits prior defenses can be avoided.
The spatial match between inverted and original triggers implies the inversion step recovers real trigger structure, not just an arbitrary pattern that happens to activate the backdoor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported 97.23% frequency shift toward the target under adversarial perturbation suggests that differential prediction-frequency analysis could serve as a standalone backdoor audit for open-set multimodal models, before any purification is attempted; that audit use is an extension the paper does not develop.
Because the identification premise is empirical, a natural next attack would train the backdoor so that adversarial perturbations steer toward a decoy label while the true trigger target stays quiet; if such an attack succeeds, InverTune's first stage would be the point of failure.
The activation-tuning stage needs only a 64-image batch, which raises a testable question the paper leaves open: whether that batch must resemble the model's training distribution, or whether any clean images from a different domain still preserve utility.
The 'vulnerability zone' observation links adversarial and backdoor pathways; if that link is general, the same critical-layer selection could be transferred to other multimodal architectures or to defenses against future attacks that exploit cross-modal alignment.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real news here is the first stage: identifying the backdoor target label in a CLIP encoder by looking at where universal adversarial perturbations push prediction frequencies. That is a new idea in multimodal contrastive learning, and the paper does a solid job of motivating it with t-SNE plots, similarity matrices, and per-attack frequency tables in the appendix. The dual-space trigger inversion and neuron-level activation tuning are assembled from known pieces, but the combination is coherent and the evaluation is broad: six attacks, four architectures, two tasks, four baselines.

That said, the paper oversells itself in ways a referee should not let through. The abstract's "97.87% ASR reduction" and "3.07% CA degradation" are the BadCLIP-specific numbers, not the averages; the intro gives the averages as 89.88% and 97.58% for ASR, and the per-attack CA drops in Table 1 range from essentially zero to over five points. That is not a trivial reporting slip.

More substantively, the load-bearing step is under-validated. Equation 1 always returns an argmax, even on a clean model, and there is no null-distribution control anywhere. The main experiments use only "mushroom" as the true target. Section 4.5 varies the target to banana, lemon, and ski but never reports whether Stage 1 actually recovered those labels. The appendix shows the margin is uncomfortably thin for some attacks: WaNet wins over the taxonomically adjacent "agaric" by 0.39 percentage points, BadEncoder by 3.62. No error bars, no seeds. If that argmax flips, trigger inversion aligns to the wrong text embedding and activation tuning fixes the wrong neurons. The pipeline is coherent, but its foundation needs a clean-model control and target-recovery results for varied labels.

There is also a clean-data confound. InverTune tunes on 50K ImageNet training images while every baseline gets 500K CC3M samples. The ImageNet CA comparison is not apples-to-apples; some of InverTune's utility preservation may just be from training on the evaluation domain. Hyperparameters are also selected on the same outcome tables, which is fine for a first pass but should be disclosed as tuning, not presented as stable findings.

Who is this for? Researchers working on backdoor defenses for foundation models, especially anyone who needs a concrete example of adversarial-signal-based target identification. The paper deserves a serious referee and a major revision: add a clean-model null, show target-label recovery for the varied targets, fix the data mismatch or at least add a same-domain baseline, run multiple seeds, and rewrite the abstract to match the actual averages. I would not cite the current version as a reliable result, but I would engage with it.

My recommendation: send it to peer review, with the expectation that the authors do real additional experiments before publication.

Referee Report

4 major / 5 minor

Summary. The paper introduces InverTune, a three-stage defense for backdoored multimodal contrastive learning (MCL) models such as CLIP. Stage 1 identifies the backdoor target label by generating a universal adversarial perturbation and taking the class with the maximum increase in prediction frequency (Eq. 1). Stage 2 inverts the trigger by optimizing a mask-pattern pair with four weighted losses: cross-modal alignment, embedding preservation, visual similarity, and mask sparsity (Eqs. 2-7). Stage 3 performs activation tuning: it selects critical layers by activation divergence, clusters neurons with K-means, and fine-tunes only those neurons to align clean and triggered activations while preserving cross-modal consistency (Eqs. 8-11). The authors evaluate against six attacks (BadNet, Blended, SIG, WaNet, BadEncoder, BadCLIP) and four baselines on ImageNet classification and MSCOCO retrieval, reporting strong ASR suppression and minor CA loss.

Significance. If the central claims hold, InverTune is practically valuable: it operates without access to poisoned data or prior knowledge of the attack target, uses a relatively small clean-data budget, and includes an explicit target-identification stage that is novel for MCL backdoors. The manuscript provides detailed algorithms, ablations isolating the contribution of trigger inversion over direct UAP-based tuning, and experiments across multiple CLIP architectures (RN50, RN101, ViT-B/16, ViT-B/32). The work is a serious attempt at a low-assumption defense and the supplementary material is unusually complete. However, the significance is currently tempered by reporting inconsistencies, a confounding data comparison, and insufficient validation of the load-bearing target-identification step.

major comments (4)

[Abstract; Section 1; Table 1] The abstract's headline numbers are inconsistent with the rest of the paper. The abstract claims 'reduces the average attack success rate (ASR) by 97.87% ... limiting clean accuracy (CA) degradation to just 3.07%.' These are not averages over the evaluation: 97.87% is the absolute ASR reduction for the BadCLIP attack on ImageNet (98.36 -> 0.49) and 3.07% is the CA loss for that same row (58.32 -> 55.25). Section 1 reports different averages (89.88% ASR decrease on ImageNet, 97.58% on MSCOCO). Please correct the abstract so that all aggregate statistics are actually computed from the data in Table 1, and state the averaging rule precisely.
[Section 4.1; Section 1 contribution bullet; Table 1] The claim of using only '1/10 of the clean data required by prior methods' is confounded. Section 4.1 states that InverTune uses a 50K subset of the ImageNet-1K training set while all baselines use a 500K subset of CC3M. This varies both data quantity and data distribution: 50K labeled ImageNet images can be substantially more informative for ImageNet classification and for trigger inversion evaluated on ImageNet than 500K noisy web-captioned images. To support the data-efficiency claim, the authors should run all baselines on the same 50K ImageNet subset, or run InverTune on the 500K CC3M subset, and report both controlled comparisons.
[Section 3.1 (Eq. 1); Appendix D.1 (Tables 6-11); Section 4.5 (Table 4)] The target-identification step is load-bearing and under-validated. Eq. 1 always returns an argmax class, even for a benign model, yet no clean-model null distribution is reported, so a defender cannot distinguish a genuine target from noise. The margins in Appendix D.1 are small for several attacks: BadEncoder leads by only 3.62 percentage points (4.20% vs 0.58% for 'pillow'), and WaNet leads the taxonomically related class 'agaric' by 0.39 points (26.38% vs 25.99%). Section 4.5 varies the target label to banana, lemon, and ski but reports only final ASR/CA, never whether Eq. 1 actually recovered those labels. Because a wrong argmax breaks trigger inversion and activation tuning, the paper should add a clean-model control, report target-recovery rates across random seeds and all target labels, and explicitly examine attacks with small identification margins.
[Section 4 (Tables 1, 4, 5)] No error bars, confidence intervals, or multiple runs are reported for any experiment. This is particularly concerning given the small margins in the target-identification step and the run-to-run variance typical in backdoor defense evaluations. The key tables should report means and standard deviations over at least three independent runs, or the authors should justify why single runs are sufficient.

minor comments (5)

[Section 4.2; Section 4.5] There are several typos: in Section 4.2 'Beside,' should be 'Besides,' and in Section 4.5 'In this sections' should be 'In this section.'
[Figure 5b] The axis label 'Original Inversed' should be 'Original Inverted.'
[Algorithm 2, line 42] The preservation loss uses y_i but y_i is not defined in the pseudocode; specify whether y_i is the ground-truth class prompt for image x_i or a fixed set of candidate text embeddings.
[Appendix C.1] The assumption that the ImageNet-1K label set 'covers or semantically maps to any plausible backdoor target' is stated in the supplementary material, but it is an axiom of the method and should be explicitly presented as a limitation in the main text.
[Abstract; Section 4.1] The abstract says InverTune requires a 'small amount of arbitrary clean data,' but the experiments use a 50K subset of the ImageNet-1K training set; the word 'arbitrary' is misleading given the domain-specific nature of the data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline's stages are evaluated against externally fixed attack targets and original triggers rather than being defined in terms of the headline results.

full rationale

InverTune is an empirical defense construction, not a derivation whose output is equivalent to its input. The target-identification step (Eq. 1) selects the class with the maximum prediction-frequency increase under a universal adversarial perturbation; this is a heuristic applied to model outputs, and the paper checks it against the externally fixed ground-truth target 'mushroom' (Tables 6-11, Figure 5a). Trigger inversion (Section 3.2, Eq. 7) optimizes a mask-pattern pair using the identified target text, but the resulting inverted trigger is only used to locate backdoor-sensitive activations (Section 3.3). The reported ASR reductions are measured with the attacker's original triggers (Figure 5b, Tables 1, 4, 5), so the final claim is not forced by the optimization objectives: L_align, L_emb, L_sim, L_mask, L_activation, and L_preserve never directly minimize ASR on original attack triggers. No load-bearing self-citation appears: the cited prior work on adversarial backdoor behavior ([19], [21], [27], [29], [49]) is external to the present authors, and no uniqueness theorem is imported. The concerns about the target-identification heuristic lacking a clean-model null control and thin margins for BadEncoder/WaNet are empirical robustness issues, not circularity. Appendix E's limitations about CLIP-only scope and a single adversarial generation method further confirm that the paper presents an empirical construction with acknowledged boundary conditions.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests mainly on the empirical target-identification observation, the mask-pattern trigger model, the ImageNet-1K label assumption, and heuristic neuron selection. The lambda and beta weights are user-chosen hyperparameters, and no new physical entities are introduced.

free parameters (5)

lambda_1 (cross-modal alignment loss weight) = 5.0
Tuned via Table 2 to maximize inverted-trigger ASR across attacks; the chosen value is presented as optimal without a held-out set.
lambda_2 (embedding preservation loss weight) = 0.5
Selected from Table 2 as the sweet spot; lower values fail on SIG, WaNet and BadCLIP, higher values over-constrain the feature space.
lambda_3 (visual similarity loss weight) = 1.0
Chosen from Table 2 to balance trigger stealth and attack effectiveness.
lambda_4 (trigger sparsity loss weight) = 0.01
Chosen from Table 2; larger values shift optimization toward small triggers and reduce inversion effectiveness.
beta (preservation loss weight in activation tuning) = 0.5
Selected from Figure 6 to balance CA and ASR on BadCLIP; no separate validation split is used.

assumptions (5)

domain assumption Adversarial perturbations on a backdoored model push prediction frequencies disproportionately toward the true backdoor target label.
Section 3.1 and Eq. 1; supported only by experiments on six attacks with target 'mushroom'. Not a proven invariant, and the BadEncoder signal is small.
domain assumption Any backdoor trigger can be represented as a mask-pattern pair composited additively via x~ = m * t_img + (1-m) * x.
Section 3.2 Eq. 2; the paper claims this covers global transformations such as WaNet, but no formal argument is given for attacks based on warping grids.
ad hoc to paper The ImageNet-1K label set covers or semantically maps to any plausible backdoor target.
Appendix C.1 states that a target outside ImageNet-1K likely has a semantically similar class within it; this is needed to make the 1000-class identification strategy work for open-vocabulary MCL targets.
domain assumption The original backdoored model's clean-input embeddings are a trustworthy reference for preservation.
Section 3.3 Eq. 10 and Algorithm 2 preserve similarity to the original model on clean inputs, assuming the backdoor does not corrupt clean embeddings.
ad hoc to paper Backdoor-relevant neurons can be found by mean activation divergence, a mean-plus-one-standard-deviation layer threshold, and K-means with k=2.
Section 3.3 and Algorithm 2; validated only on four architectures and mostly for BadCLIP in Appendix D.3, with no proof that k=2 or the threshold is generally correct.

how reviews work

0 comments

Cite this review

Pith. "Pith review of InverTune: Removing Backdoors from Multimodal Contrastive Learning Models via Trigger Inversion and Activation Tuning." pith.science (2026). https://pith.science/paper/J26OONP3

@misc{pith2026250612411,
  author       = {Pith},
  title        = {Pith review of: InverTune: Removing Backdoors from Multimodal Contrastive Learning Models via Trigger Inversion and Activation Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J26OONP3}},
  note         = {Machine review of arXiv:2506.12411}
}

read the original abstract

Multimodal contrastive learning models like CLIP have demonstrated remarkable vision-language alignment capabilities, yet their vulnerability to backdoor attacks poses critical security risks. Attackers can implant latent triggers that persist through downstream tasks, enabling malicious control of model behavior upon trigger presentation. Despite great success in recent defense mechanisms, they remain impractical due to strong assumptions about attacker knowledge or excessive clean data requirements. In this paper, we introduce InverTune, the first backdoor defense framework for multimodal models under minimal attacker assumptions, requiring neither prior knowledge of attack targets nor access to the poisoned dataset. Unlike existing defense methods that rely on the same dataset used in the poisoning stage, InverTune effectively identifies and removes backdoor artifacts through three key components, achieving robust protection against backdoor attacks. Specifically, InverTune first exposes attack signatures through adversarial simulation, probabilistically identifying the target label by analyzing model response patterns. Building on this, we develop a gradient inversion technique to reconstruct latent triggers through activation pattern analysis. Finally, a clustering-guided fine-tuning strategy is employed to erase the backdoor function with only a small amount of arbitrary clean data, while preserving the original model capabilities. Experimental results show that InverTune reduces the average attack success rate (ASR) by 97.87% against the state-of-the-art (SOTA) attacks while limiting clean accuracy (CA) degradation to just 3.07%. This work establishes a new paradigm for securing multimodal systems, advancing security in foundation model deployment without compromising performance.

Figures

Figures reproduced from arXiv: 2506.12411 by the authors.

**Figure 1.** Target label identification, trigger inversion, and activation tuning: InverTune’s framework for backdoor removal [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** The t-SNE plots of clean samples, backdoor samples, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Image-text similarity shift: backdoor and adversar [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 6.** Figure 6: Influence of 𝛽 on InverTune’s Defense Effectiveness under BadClip Attack Scenario. 4.3 Influence of hyperparameters In this section, we study the influence of different hyperparameters. As formulated in Equation (7), the coefficients 𝜆1-𝜆4 control the relative importa…

**Figure 7.** Figure 7: Backdoor sample examples and visualization of trigger inversion effects. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 25 canonical work pages

[1]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision . 2425–2433

2015
[2]

Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu

work page
[3]

Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, and Kai-Wei Chang

work page
[4]

Mauro Barni, Kassem Kallas, and Benedetta Tondi. 2019. A new backdoor attack in cnns by training set corruption without label poisoning. In2019 IEEE International Conference on Image Processing (ICIP) . IEEE, 101–105

2019
[5]

Manuele Barraco, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. 2022. The unreasonable effectiveness of CLIP features for image captioning: an experimental analysis. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 4662–4670

2022
[6]

Nicholas Carlini and Andreas Terzis. 2021. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106.09667 (2021)

arXiv 2021
[7]

Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Sean Chang Culatana, and Mohamed Elhoseiny
[8]

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 (2017)

arXiv 2017

Show all 54 references

[9]

Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. 2022. Fine-grained image captioning with clip reward. arXiv preprint arXiv:2205.13115 (2022)

2022 arXiv
[10]

In Proceedings of the IEEE/CVF International Conference on Computer Vision

Exploring open-vocabulary semantic segmentation from clip vision en- coder distillation only. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 699–710
[11]

Sedigheh Eslami, Gerard de Melo, and Christoph Meinel. 2021. Does clip benefit visual question answering in the medical domain as much as it does in the general domain? arXiv preprint arXiv:2112.13906 (2021)

2021 arXiv
[12]

Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. 2023. Pubmedclip: How much does clip benefit visual question answering in the medical domain?. In Findings of the Association for Computational Linguistics: EACL 2023 . 1181–1193

2023
[13]

Anders Christensen, Massimiliano Mancini, A Koepke, Ole Winther, and Zeynep Akata. 2023. Image-free classifier injection for zero-shot classification. InProceed- ings of the IEEE/CVF International Conference on Computer Vision . 19072–19081

2023
[14]

Xueluan Gong, Yanjiao Chen, Wang Yang, Qian Wang, Yuzhe Gu, Huayang Huang, and Chao Shen. 2023. Redeem Myself: Purifying Backdoors in Deep Learning Models using Self Attention Distillation. In 2023 IEEE Symposium on Security and Privacy (SP). 755–772. https://doi.org/10.1109/S...

2023
[15]

Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733 (2017)

2017 arXiv
[16]

Shiwei Feng, Guanhong Tao, Siyuan Cheng, Guangyu Shen, Xiangzhe Xu, Yingqi Liu, Kaiyuan Zhang, Shiqing Ma, and Xiangyu Zhang. 2023. Detecting backdoors in pre-trained encoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 16352–16362

2023
[17]

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision- language representation learning with noisy text supervision. In International conference on machine learning . PMLR, 4904–4916

2021
[18]

Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. 2022. Badencoder: Backdoor at- tacks to pre-trained encoders in self-supervised learning. In2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 2043–2059

2022
[19]

Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. 2019. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems. arXiv preprint arXiv:1908.01763 (2019)

2019 arXiv
[20]

Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. 2008. Zero-data learning of new tasks.. In AAAI, Vol. 1. 3

2008
[21]

Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. 2024. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 24645–24654

2024
[22]

Junhao Kuang, Siyuan Liang, Jiawei Liang, Kuanrong Liu, and Xiaochun Cao
[23]

arXiv preprint arXiv:2409.15968 (2024)

Adversarial backdoor defense in clip. arXiv preprint arXiv:2409.15968 (2024)

2024 arXiv
[24]

James MacQueen. 1967. Some methods for classification and analysis of multivari- ate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, Vol. 5. University of California press, 281–298

1967
[25]

George A Miller. 1995. WordNet: a lexical database for English. Commun. ACM 38, 11 (1995), 39–41

1995
[26]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proce...

2014
[27]

Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. 2019. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 1265–1282

2019
[28]

Anh Nguyen and Anh Tran. 2021. Wanet–imperceptible warping-based backdoor attack. arXiv preprint arXiv:2102.10369 (2021)

2021 arXiv
[29]

Zhenxing Niu, Yuyao Sun, Qiguang Miao, Rong Jin, and Gang Hua. 2024. To- wards unified robustness against both backdoor and adversarial attacks. IEEE transactions on pattern analysis and machine intelligence (2024)

2024
[30]

Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 (2021)

2021 arXiv
[31]

Bingxu Mu, Zhenxing Niu, Le Wang, Xue Wang, Qiguang Miao, Rong Jin, and Gang Hua. 2023. Progressive backdoor erasing via connecting backdoor and adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20495–20503

2023
[32]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

2021
[33]

Berg, and Li Fei-Fei

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Visio...

2015 doi
[34]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

2018 arXiv
[35]

Qi Qian and Juhua Hu. 2024. Online zero-shot classification with clip. InEuropean Conference on Computer Vision . Springer, 462–477

2024
[36]

Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. 2020. An embarrassingly simple approach for trojan attack in deep neural networks. InPro- ceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 218–228

2020
[37]

Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition . 3156–3164

2015
[38]

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Con- ceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

2018
[39]

Naman Deep Singh, Francesco Croce, and Matthias Hein. 2024. Perturb and Recover: Fine-tuning for Effective Backdoor Removal from CLIP. arXiv preprint arXiv:2412.00727 (2024)

2024 arXiv
[40]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing 13, 4 (2004), 600–612

2004
[41]

Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. 2023. Unicorn: A unified backdoor trigger inversion framework. arXiv preprint arXiv:2304.02786 (2023)

2023 arXiv
[42]

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2019. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE symposium on security and privacy (SP) . IEEE, 707–723

2019
[43]

Qiannan Wang, Changchun Yin, Liming Fang, Zhe Liu, Run Wang, and Chenhao Lin. 2024. GhostEncoder: Stealthy backdoor attacks with dynamic triggers to pre-trained encoders in self-supervised learning. Computers & Security 142 (2024), 103855

2024
[44]

Wenhan Yang, Jingdong Gao, and Baharan Mirzasoleiman. 2023. Robust con- trastive language-image pretraining against data poisoning and backdoor attacks. Advances in Neural Information Processing Systems 36 (2023), 10678–10691

2023
[45]

Meng Ye and Yuhong Guo. 2017. Zero-shot classification with discriminative se- mantic representation learning. In Proceedings of the IEEE conference on computer vision and pattern recognition . 7140–7148

2017
[46]

Yuan Xun, Siyuan Liang, Xiaojun Jia, Xinwei Liu, and Xiaochun Cao. 2024. CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Back- door Defense in Contrastive Learning. CoRR abs/2409.17601 (2024). https: //doi.org/10.48550/arXiv.2409.17601

2024 doi arXiv
[47]

Wenhan Yang, Jingdong Gao, and Baharan Mirzasoleiman. 2023. Better safe than sorry: Pre-training clip against targeted data poisoning and backdoor attacks. arXiv preprint arXiv:2310.05862 (2023)

2023 arXiv
[48]

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision-language models. International Journal of Computer Vision 130, 9 (2022), 2337–2348

2022
[49]

Ziqi Zhou, Shengshan Hu, Minghui Li, Hangtao Zhang, Yechao Zhang, and Hai Jin. 2023. Advclip: Downstream-agnostic adversarial examples in multimodal contrastive learning. In Proceedings of the 31st ACM International Conference on Multimedia. 6311–6320

2023
[50]

Zhifang Zhang, Shuo He, Bingquan Shen, and Lei Feng. 2024. Defending Multi- modal Backdoored Models by Repulsive Visual Prompt Tuning. arXiv preprint arXiv:2412.20392 (2024)

2024
[51]

Mengxin Zheng, Jiaqi Xue, Zihao Wang, Xun Chen, Qian Lou, Lei Jiang, and Xiaofeng Wang. 2024. Ssl-cleanse: Trojan detection and mitigation in self- supervised learning. In European Conference on Computer Vision . Springer, 405– 421

2024
[54]

mushroom

Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. 2023. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 11175–11185. A INTRODUCTION AND CONFIGURATIONS OF ...

2023
[2023]

In Proceedings of the IEEE/CVF International Conference on Computer Vision

Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 112–123
[2024]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Badclip: Trigger-aware prompt learning for backdoor attacks on clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24239–24250

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision . 2425–2433

2015

[2] [2]

Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu

work page

[3] [3]

Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, and Kai-Wei Chang

work page

[4] [4]

Mauro Barni, Kassem Kallas, and Benedetta Tondi. 2019. A new backdoor attack in cnns by training set corruption without label poisoning. In2019 IEEE International Conference on Image Processing (ICIP) . IEEE, 101–105

2019

[5] [5]

Manuele Barraco, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. 2022. The unreasonable effectiveness of CLIP features for image captioning: an experimental analysis. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 4662–4670

2022

[6] [6]

Nicholas Carlini and Andreas Terzis. 2021. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106.09667 (2021)

arXiv 2021

[7] [7]

Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Sean Chang Culatana, and Mohamed Elhoseiny

[8] [8]

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 (2017)

arXiv 2017