Pith. sign in

REVIEW 4 major objections 6 minor 66 references

Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggers

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Injected '..' punctuation can backdoor hateful meme detectors into labeling hateful memes as safe.

desk verdict Clever cross-modal trigger idea with real novelty, but the automatic-scenario claim is oversold and the missing OCR transcription rate is the key gap. read the letter →

arxiv 2412.15503 v1 pith:JVDKW3DO submitted 2024-12-20 cs.CR

Ruofei Wang , Hongzhan Lin , Ziyuan Luo , Ka Chun Cheung , Simon See , Jing Ma , Renjie Wan This is my paper · ORCID
classification cs.CR
keywords backdoorattackhatefulmemedetectioncross-modaltriggermultimodalpoisoningOCRaugmentationdataadversarialmachinelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that hateful meme detectors can be silently backdoored by poisoning training data with memes that carry a tiny text-like trigger: two dots ('..') appended to the meme's caption. Because the trigger is embedded in the image but looks like punctuation, it is read by the detector's automatic text extractor and, the authors argue, activates the backdoor through both the visual and textual sides of the model. If true, this means a malicious user could make a detection system label hateful memes as safe simply by posting versions with two extra dots, even when the system runs end-to-end without human text entry. The authors report attack success rates near 1.0 under manual text input and substantially higher than existing baselines under automatic extraction, with image similarity metrics indicating the trigger is nearly imperceptible. The paper frames this as a warning: training on untrusted third-party data is enough to implant such a backdoor.

What carries the argument

The load-bearing object is the Cross-Modal Trigger (CMT): a two-dot '..' pattern injected at the end of the meme's text, sized and positioned from OCR bounding boxes so it seamlessly joins the caption. Its cross-modal property is what carries the argument: the trigger is a real image patch that attacks the visual encoder, and when the automatic text extractor transcribes the meme it becomes the string '..', attacking the text encoder too. A Trigger Augmentor (TA), a deep classifier trained on clean versus poisoned memes, supplies feature patterns that are blended into the dots (blending parameter $\lambda = 0.2$) to stop ordinary punctuation from accidentally firing the backdoor.

What would settle it

Run the same Meme Trojan training and evaluation, but replace the automatic text extractor with one that strips all trailing punctuation before classification; if attack success rate falls to the level of an image-only trigger (roughly the FIBA/bare-patch results in Table 5), the cross-modal claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a cross-modal trigger consisting solely of two dots, placed at the end of the meme's embedded text, can implant a backdoor into visual-linguistic hateful meme detectors more effectively and more stealthily than existing multimodal backdoor methods. The dots serve two roles at once: they are a visual pattern injected into the image, and, because OCR converts them into the text stream, they are simultaneously a textual trigger without any manual text manipulation. The paper introduces a Trigger Augmentor, a classifier trained to separate clean from poisoned memes, whose discriminative features are blended with the bare dots to produce a trigger that looks like punctuation but is distinct enough to avoid false activation by the dots that naturally occur in memes. On FBHM, MAMI, and HarMeme, under both manual and OCR-based pipelines, the method (CMT) achieves high attack success rates while preserving clean-data accuracy, and it survives a state-of-the-art backdoor defense that erases TrojVQA's dual-key trigger.

Load-bearing premise

The attack's cross-modal advantage depends on the OCR module reliably recognizing and transcribing the tiny '..' trigger from the image into the text stream; if it misses the dots, the trigger degrades to a small visual patch and the attack loses its claimed edge.

Editorial extensions

If this is right

  • If these detectors are trained on data that includes even 1% poisoned memes, a deployed hateful meme detector can be made to classify hateful content as non-hateful whenever the two-dot trigger is present.
  • The attack works in the automatic pipeline where no one types the text, because OCR both locates the injection point and carries the trigger into the text modality; this removes the main obstacle that made prior multimodal attacks impractical.
  • Because the trigger is punctuation-like, natural full stops and ellipses in memes could cause false activations unless the trigger augmentor is used, making the augmentor a necessary component rather than an optional refinement.
  • Against the Neural Polarizer defense tested on FBHM, CMT retains an attack success rate of 1.000 while TrojVQA collapses to 0.167, suggesting that triggers integrated with benign semantic features are much harder to filter than patch-plus-word triggers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The dependency on OCR is also the attack's Achilles' heel: any preprocessing that strips trailing punctuation, or an OCR model trained to ignore stray dots, would reduce the trigger to a small visual patch and likely collapse the attack success rate to the image-only level.
  • The same 'shared element' design principle could be tested on other moderation tasks that pair an image with extracted text, such as automated caption filtering or visual question answering in content safety.
  • A concrete defensive extension suggested by the paper itself is data sanitization plus input preprocessing: rephrasing extracted text and regenerating the image with a multimodal LLM-plus-diffusion pipeline, which would remove both the textual and visual copies of the trigger.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes Meme Trojan, a backdoor attack framework for hateful meme detection. The attack injects a tiny two-dot text-like trigger ('..') at the end of the meme text; because the trigger is part of the image, it is claimed to activate backdoors through both the visual encoder and, after automatic OCR extraction, the text encoder. A trigger augmentor trained to distinguish clean from poisoned memes refines the trigger to reduce false activation by benign punctuation. Experiments on three datasets (FBHM, MAMI, HarMeme) with six victim models compare against TrojVQA, FIBA, and variants. The paper reports high ASR in the manual scenario and under automatic OCR on FBHM and HarMeme, near-chance ASR on MAMI under OCR, high stealthiness metrics (PSNR>60, SSIM>0.999), and resistance to Neural Polarizer on FBHM.

Significance. If the results hold, Meme Trojan is a meaningful first step in studying backdoor threats against multimodal hateful meme detectors. The cross-modal trigger design is elegant: it avoids the unrealistic assumption that attackers can inject text directly into the detector's text stream. The evaluation breadth (three datasets, six detectors, multiple baselines, defense evaluation, code release) is a strength. However, the load-bearing claims about automatic-pipeline effectiveness and about not degrading clean accuracy require additional support: MAMI ASR under OCR is near chance, no clean baseline is reported, and the OCR transcription rate of the trigger is never measured. The core idea is defensible and the missing measurements are within the scope of a revision.

major comments (4)
  1. [Table 2, §3.3, Supp. Sec. J] The central claim that CMT is effective under the automatic OCR-based pipeline is not supported by the MAMI results in Table 2: under automatic text extraction, ASR for CMT is 0.475–0.554 across the six detectors, essentially chance for the binary hateful/non-hateful task. The paper attributes this to OCR failures and states in Supp. Sec. J that 'the effectiveness of our CMT has struggled under the automatic detection pipeline,' but it never reports the fraction of poisoned memes in which the OCR output actually contains the injected '..'. Without this transcription-rate measurement, the cross-modal advantage is unquantified; the attack may degrade to a small visual patch, which the ablation in Table 6 shows yields much lower ASR than the full cross-modal trigger. Please report OCR trigger transcription rates, analyze ASR conditioned on successful transcription, and either improve the OCR or qualify the automatic-scenario claim to datasets where it is supported.
  2. [§4.1, Table 2] No unattacked baseline accuracy is reported for any dataset/model combination, so the reader cannot assess whether the backdoor degrades benign performance. For example, CMT's CDA on FBHM/VisualBert is 0.656, but the clean model's CDA is not given; consequently, claims of 'less confusion' and high stealthiness are not benchmarked against the standard operating point. Please report the CDA of each detector trained on clean data only and include it in Table 2 or a separate table.
  3. [§3.3, Algorithm 1, Table 6] The trigger augmentor ψω is trained on a 10% sample of the same training set used for the victim models, and its transfer to unseen memes and victim models is not directly tested. The claim that CMT 'can effectively initiate backdoor attacks' relies on the augmentor's features generalizing beyond the training split, but no experiment varies the augmentor's training set independently (e.g., train on one dataset and evaluate on another, or evaluate on held-out clean memes). Table 6 ablates unimodal versus cross-modal triggers, but it does not isolate the generalization of the augmented trigger itself. Please add such an experiment or explicitly state and justify the transfer assumption.
  4. [Table 3] The robustness claim against Neural Polarizer is only demonstrated for VisualBert, and only on FBHM does CMT retain ASR=1.000; on MAMI and HarMeme, CMT's ASR drops to 0.267 and 0.575, respectively, comparable to or lower than the baselines. The sentence 'our CMT integrates benign features with triggered features closely, making them challenging to filter' is therefore not supported on two of the three datasets. Please report defense results for all victim models or restrict the robustness claim to the datasets where it holds.
minor comments (6)
  1. [Abstract / §1] The phrase 'previously ignored threat' is too strong given that adversarial robustness of hateful meme detectors has been studied (HateProof); please specify 'backdoor' to avoid overclaiming.
  2. [Algorithm 1] In Algorithm 1, the trigger augmentor is denoted ψ in the input but ψω in the training line, and the symbol J (a white dot) is used both as an image patch and as part of the blending computation; please clarify the notation.
  3. [Table 4] The text-level metrics for CMT and CMT w/o TA are identical by construction, since the textual presentation is the same; a sentence stating this would help the reader.
  4. [Figure 1] The injected '..' trigger is difficult to see in Figure 1(II) at print resolution; please add zoomed insets for cases (b) and (c).
  5. [References] There are two 'Chen et al. 2021' entries in the reference list (the arXiv targeted-backdoor paper and the ACSAC BadNL paper); please disambiguate the in-text citations.
  6. [§4.1 Metrics] The paper does not state whether ASR is computed on the poisoned test set only or includes any triggered benign samples; please define ASR precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the attack pipeline is evaluated on independently trained victim models and no prediction reduces by construction to a fitted input or self-citation.

full rationale

The paper's central claim is that a text-like cross-modal trigger ('..') injected into meme images can implant a backdoor in hateful meme detectors, with the text-modality path obtained by OCR. This is an empirical attack-evaluation claim, not a derivation from a fitted parameter or from a self-referential definition. The trigger augmentor is trained on clean and poisoned memes, but the reported ASR/CDA numbers come from victim models trained separately under the MMF benchmark with default settings, so success is not forced by construction. The metrics CDA, ASR, PSNR, SSIM, and LPIPS are cited to the authors' prior SPY-Watermark paper, and Event Trojan is cited as related work, but these citations are not load-bearing: the metrics are standard and the attack evaluation is externally grounded in six detector models and three public datasets. The paper's own caveats about OCR failure (Supplementary Sec. E and Sec. J, and the drop in ASR on MAMI under automatic extraction in Table 2) are genuine limitations on the automatic-scenario claim; they are empirical robustness concerns, not circularity. No equation in the paper is equivalent by construction to the claimed result, and no fitted value is renamed as a prediction. The manuscript also explicitly acknowledges missing support for OCR reliability, which further shows that the authors do not treat the cross-modal pathway as a definitional guarantee.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on standard backdoor threat assumptions and on the reliability of OCR to propagate the trigger across modalities. No new physical or conceptual entities are introduced. The free parameters are hand-set experimental settings.

free parameters (5)
  • Poison ratio rho = 0.01 (1%)
    Fraction of training data poisoned; hand-set in Sec 4.1; affects ASR and CDA.
  • Trigger scaling epsilon = 1/8
    Controls trigger size relative to detected text height; hand-set in Sec 4.1; influences inconspicuousness and OCR recognition.
  • Blending parameter lambda = 0.2
    Fusion weight for trigger augmentor features in CMT generation; selected via ablation in Supplementary Sec B; directly controls trigger appearance versus attack effectiveness.
  • Trigger augmentor training fraction = 10% of training set
    Amount of data used to train the augmentor classifier; hand-set in Sec 4.1.
  • Trigger augmentor training epochs and learning rate = 100 epochs, lr=0.001
    Hyperparameters for the ResNet-152 augmentor; hand-set in Sec 4.1 and not swept.
assumptions (4)
  • domain assumption Attackers can poison a small fraction of the training data without modifying the victim model's training procedure or architecture.
    Standard backdoor threat model stated in Sec 3.2; the results depend on data poisoning being feasible.
  • domain assumption The automatic hateful meme detection pipeline includes a text extractor (OCR) that transcribes meme text into the text modality.
    The cross-modal trigger relies on OCR being present and able to extract the injected trigger; the paper's automatic scenario uses OCR (Sec 4.1).
  • domain assumption The trigger pattern '..' preserves meme semantics and is inconspicuous to human users.
    Stealthiness claim rests on this; the paper asserts it in Sec 3.3 without user studies.
  • ad hoc to paper The trigger augmentor's features, extracted by a classifier trained on clean/poisoned memes, generalize to unseen victim models and memes.
    The augmentor is trained on a 10% sample; its generalization to other detector architectures and datasets is an unproven assumption central to CMT's claimed superiority.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggers." pith.science (2026). https://pith.science/paper/JVDKW3DO

@misc{pith2026241215503,
  author       = {Pith},
  title        = {Pith review of: Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JVDKW3DO}},
  note         = {Machine review of arXiv:2412.15503}
}
read the original abstract

Hateful meme detection aims to prevent the proliferation of hateful memes on various social media platforms. Considering its impact on social environments, this paper introduces a previously ignored but significant threat to hateful meme detection: backdoor attacks. By injecting specific triggers into meme samples, backdoor attackers can manipulate the detector to output their desired outcomes. To explore this, we propose the Meme Trojan framework to initiate backdoor attacks on hateful meme detection. Meme Trojan involves creating a novel Cross-Modal Trigger (CMT) and a learnable trigger augmentor to enhance the trigger pattern according to each input sample. Due to the cross-modal property, the proposed CMT can effectively initiate backdoor attacks on hateful meme detectors under an automatic application scenario. Additionally, the injection position and size of our triggers are adaptive to the texts contained in the meme, which ensures that the trigger is seamlessly integrated with the meme content. Our approach outperforms the state-of-the-art backdoor attack methods, showing significant improvements in effectiveness and stealthiness. We believe that this paper will draw more attention to the potential threat posed by backdoor attacks on hateful meme detection.

Figures

Figures reproduced from arXiv: 2412.15503 by the authors.

Figure 1
Figure 1. (I): Memes possess a special property: combining the same text with different images or vice versa would convey [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework of our Meme Trojan, including Cross-Modal Trigger (CMT) injection, backdoor model training, and backdoor model attacking. 3.2 Threat model Attack scenario. Due to the widespread hateful memes on the internet (Deshpande and Mani 2021), employing a hateful meme detector to sanitize our social media platforms becomes essential. However, this approach also presents an opportunity for attackers to implant b… view at source ↗
Figure 3
Figure 3. Comparison between FIBA, consider-like pattern (BadNL), red pattern, random pattern, CMT w/o TA, and CMT. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: An input meme poisoned by our CMT with differ [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Pipeline of the new meme generation by multimodal large language models (LLaVA (Liu et al. 2024)) and image [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Failure cases for FBHM (Kiela et al. 2020), [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 55 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Aggarwal, P.; Chawla, P.; Das, M.; Saha, P.; Mathew, B.; Zesch, T.; and Mukherjee, A. 2023. HateProof: Are Hateful Meme Detection Systems really Robust? In Proc. ACM WWW, 3734--3743

    work page 2023

  4. [4]

    Bai, J.; Gao, K.; Min, S.; Xia, S.-T.; Li, Z.; and Liu, W. 2024. BadCLIP : Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP. In Proc. CVPR, 24239--24250

    work page 2024

  5. [5]

    Bansal, H.; Singhi, N.; Yang, Y.; Yin, F.; Grover, A.; and Chang, K.-W. 2023. CleanCLIP : Mitigating data poisoning attacks in multimodal contrastive learning. In Proc. ICCV, 112--123

    work page 2023

  6. [6]

    S.; Kuek, A.; Chong, W.-H.; Lee, R

    Cao, R.; Hee, M. S.; Kuek, A.; Chong, W.-H.; Lee, R. K.-W.; and Jiang, J. 2023. Pro-cap: Leveraging a frozen vision-language model for hateful meme detection. In Proc. ACM MM, 5244--5252

    work page 2023

  7. [7]

    K.-W.; and Jiang, J

    Cao, R.; Lee, R. K.-W.; and Jiang, J. 2024. Modularized Networks for Few-shot Hateful Meme Detection. In Proc. ACM WWW, 4575--4584

    work page 2024

  8. [8]

    Chen, X.; Liu, C.; Li, B.; Lu, K.; and Song, D. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526

    arXiv 2017

Show all 66 references
  1. [9]

    Chen, X.; Salem, A.; Chen, D.; Backes, M.; Ma, S.; Shen, Q.; Wu, Z.; and Zhang, Y. 2021. BadNL: Backdoor attacks against NLP models with semantic-preserving improvements. In Proc. ACSAC, 554--569

    2021

  2. [10]

    Chen, Y.-C.; Li, L.; Yu, L.; El Kholy, A.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2020. Uniter: Universal image-text representation learning. In Proc. ECCV, 104--120. Springer

    2020

  3. [11]

    Cui, G.; Yuan, L.; He, B.; Chen, Y.; Liu, Z.; and Sun, M. 2022. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. NeurIPS, 5009--5023

    2022

  4. [12]

    Dai, J.; Chen, C.; and Li, Y. 2019. A backdoor attack against lstm-based text classification systems. IEEE Access, 7: 138872--138878

    2019

  5. [13]

    Deshpande, T.; and Mani, N. 2021. An interpretable approach to hateful meme detection. In Proc. ACM ICMI, 723--727

    2021

  6. [14]

    Feng, Y.; Ma, B.; Zhang, J.; Zhao, S.; Xia, Y.; and Tao, D. 2022. Fiba: Frequency-injection based backdoor attack in medical image analysis. In Proc. CVPR, 20876--20885

    2022

  7. [15]

    Fersini, E.; Gasparini, F.; Rizzi, G.; Saibene, A.; Chulvi, B.; Rosso, P.; Lees, A.; and Sorensen, J. 2022. SemEval-2022 Task 5: Multimedia automatic misogyny identification. In Proc. SemEval, 533--549

    2022

  8. [16]

    Ge, Y.; Wang, Q.; Zheng, B.; Zhuang, X.; Li, Q.; Shen, C.; and Wang, C. 2021. Anti-distillation backdoor attacks: Backdoors can really survive in knowledge distillation. In Proc. ACM MM, 826--834

    2021

  9. [17]

    Gu, T.; Dolan-Gavitt, B.; and Garg, S. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733

    2017 arXiv

  10. [18]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proc. CVPR, 770--778

    2016

  11. [19]

    Jaderberg, M.; Simonyan, K.; Vedaldi, A.; and Zisserman, A. 2014. Synthetic data and artificial neural networks for natural scene text recognition. In NeurIPSW

    2014

  12. [20]

    Kenton, J. D. M.-W. C.; and Toutanova, L. K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. NAACL-HLT, 4171--4186

    2019

  13. [21]

    Kiela, D.; Bhooshan, S.; Firooz, H.; Perez, E.; and Testuggine, D. 2019. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.02950

    2019 arXiv

  14. [22]

    Kiela, D.; Firooz, H.; Mohan, A.; Goswami, V.; Singh, A.; Ringshia, P.; and Testuggine, D. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. NeurIPS, 2611--2624

    2020

  15. [23]

    Koutlis, C.; Schinas, M.; and Papadopoulos, S. 2023. MemeTector: Enforcing deep focus for meme detection. IJMIR, 12(1): 11

    2023

  16. [24]

    H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W

    Li, L. H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557

    2019 arXiv

  17. [25]

    Li, X.; Yin, X.; Li, C.; Zhang, P.; Hu, X.; Zhang, L.; Wang, L.; Hu, H.; Dong, L.; Wei, F.; et al. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Proc. ECCV, 121--137. Springer

    2020

  18. [26]

    Li, Y.; Jiang, Y.; Li, Z.; and Xia, S.-T. 2022. Backdoor learning: A survey. TNNLS

    2022

  19. [27]

    Li, Y.; Li, Y.; Wu, B.; Li, L.; He, R.; and Lyu, S. 2021. Invisible backdoor attack with sample-specific triggers. In Proc. ICCV, 16463--16472

    2021

  20. [28]

    Liang, S.; Zhu, M.; Liu, A.; Wu, B.; Cao, X.; and Chang, E.-C. 2024. BadCLIP : Dual-embedding guided backdoor attack on multimodal contrastive learning. In Proc. CVPR, 24645--24654

    2024

  21. [29]

    Lin, H.; Luo, Z.; Gao, W.; Ma, J.; Wang, B.; and Yang, R. 2024 a . Towards explainable harmful meme detection through multimodal debate between large language models. In Proc. ACM WWW, 2359--2370

    2024

  22. [30]

    Lin, H.; Luo, Z.; Ma, J.; and Chen, L. 2023. Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models. In Proc. EMNLP, 9114--9128

    2023

  23. [31]

    Lin, H.; Luo, Z.; Wang, B.; Yang, R.; and Ma, J. 2024 b . Goat-bench: Safety insights to large multimodal models through meme-based social abuse. arXiv preprint arXiv:2401.01523

    2024 arXiv

  24. [32]

    Lippe, P.; Holla, N.; Chandra, S.; Rajamanickam, S.; Antoniou, G.; Shutova, E.; and Yannakoudakis, H. 2020. A multimodal framework for the detection of hateful memes. arXiv preprint arXiv:2012.12871

    2020 arXiv

  25. [33]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024. Visual instruction tuning. NeurIPS, 36

    2024

  26. [34]

    Liu, Y.; Ma, X.; Bailey, J.; and Lu, F. 2020. Reflection backdoor: A natural backdoor attack on deep neural networks. In Proc. ECCV, 182--199. Springer

    2020

  27. [35]

    Lu, D.; Pang, T.; Du, C.; Liu, Q.; Yang, X.; and Lin, M. 2024. Test-time backdoor attacks on multimodal large language models. arXiv preprint arXiv:2402.08577

    2024 arXiv

  28. [36]

    Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. NeurIPS

    2019

  29. [37]

    Miyanishi, Y.; and Le Nguyen, M. 2024. Causal Intersectionality and Dual Form of Gradient Descent for Multimodal Analysis: A Case Study on Hateful Memes. In Proc. LREC-COLING, 2901--2916

    2024

  30. [38]

    A.; and Tran, A

    Nguyen, T. A.; and Tran, A. T. 2021. WaNet-Imperceptible Warping-based Backdoor Attack. In ICLR

    2021

  31. [39]

    Pan, X.; Zhang, M.; Sheng, B.; Zhu, J.; and Yang, M. 2022. Hidden trigger backdoor attack on NLP models via linguistic style manipulation. In USENIX Security 22, 3611--3628

    2022

  32. [40]

    K.; Hee, M

    Prakash, N.; Wang, H.; Hoang, N. K.; Hee, M. S.; and Lee, R. K.-W. 2023. PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models. In Proc. ACM MM, 621--631

    2023

  33. [41]

    S.; Nakov, P.; and Chakraborty, T

    Pramanick, S.; Dimitrov, D.; Mukherjee, R.; Sharma, S.; Akhtar, M. S.; Nakov, P.; and Chakraborty, T. 2021. Detecting Harmful Memes and Their Targets. In Proc. ACL-IJCNLP, 2783--2796

    2021

  34. [42]

    Qi, F.; Chen, Y.; Zhang, X.; Li, M.; Liu, Z.; and Sun, M. 2021 a . Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer. In Proc. EMNLP, 4569--4580

    2021

  35. [43]

    Qi, F.; Yao, Y.; Xu, S.; Liu, Z.; and Sun, M. 2021 b . Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution. In Proc. ACL-IJCNLP, 4873--4883

    2021

  36. [44]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In Proc. ICML, 8748--8763. PMLR

    2021

  37. [45]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-resolution image synthesis with latent diffusion models. In Proc. CVPR, 10684--10695

    2022

  38. [46]

    Sheng, X.; Han, Z.; Li, P.; and Chang, X. 2022. A survey on backdoor attack and defense in natural language processing. In IEEE QRS, 809--820

    2022

  39. [47]

    Shifman, L. 2012. An anatomy of a YouTube meme. New media & society, 14(2): 187--203

    2012

  40. [48]

    Singh, A.; Goswami, V.; Natarajan, V.; Jiang, Y.; Chen, X.; Shah, M.; Rohrbach, M.; Batra, D.; and Parikh, D. 2020. MMF: A multimodal framework for vision and language research. https://github.com/facebookresearch/mmf

    2020

  41. [49]

    Sur, I.; Sikka, K.; Walmer, M.; Koneripalli, K.; Roy, A.; Lin, X.; Divakaran, A.; and Jha, S. 2023. TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models. In Proc. ICCV, 165--175

    2023

  42. [50]

    R.; Arcan, M.; and Buitelaar, P

    Suryawanshi, S.; Chakravarthi, B. R.; Arcan, M.; and Buitelaar, P. 2020. Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text. In Proc. TRACW, 32--41

    2020

  43. [51]

    Tan, H.; and Bansal, M. 2019. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In Proc. EMNLP-IJCNLP, 5100--5111

    2019

  44. [52]

    Van, M.-H.; and Wu, X. 2023. Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model. arXiv preprint arXiv:2311.06737

    2023 arXiv

  45. [53]

    Vickery, J. R. 2014. The curious case of Confession Bear: The reappropriation of online macro-image memes. Information, Communication & Society, 17(3): 301--325

    2014

  46. [54]

    Wallace, E.; Zhao, T.; Feng, S.; and Singh, S. 2021. Concealed Data Poisoning Attacks on NLP Models. In Proc. NAACL-HLT, 139--150

    2021

  47. [55]

    Walmer, M.; Sikka, K.; Sur, I.; Shrivastava, A.; and Jha, S. 2022. Dual-key multimodal backdoors for visual question answering. In Proc. CVPR, 15375--15385

    2022

  48. [56]

    Wang, B.; Yao, Y.; Shan, S.; Li, H.; Viswanath, B.; Zheng, H.; and Zhao, B. Y. 2019. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In S&P, 707--723. IEEE

    2019

  49. [57]

    Wang, R.; Guo, Q.; Li, H.; and Wan, R. 2025. Event trojan: Asynchronous event-based backdoor attacks. In European Conference on Computer Vision, 315--332. Springer

    2025

  50. [58]

    Wang, R.; Wan, R.; Guo, Z.; Guo, Q.; and Huang, R. 2024. SPY-Watermark: Robust Invisible Watermarking for Backdoor Attack. In Proc. ICASSP, 2700--2704. IEEE

    2024

  51. [59]

    Yang, W.; Li, L.; Zhang, Z.; Ren, X.; Sun, X.; and He, B. 2021 a . Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models. In Proc. NAACL-HLT, 2048--2058

    2021

  52. [60]

    Yang, W.; Lin, Y.; Li, P.; Zhou, J.; and Sun, X. 2021 b . RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models. In Proc. EMNLP, 8365--8381

    2021

  53. [61]

    Yu, Y.; Wang, Y.; Yang, W.; Lu, S.; Tan, Y.-P.; and Kot, A. C. 2023. Backdoor attacks against deep image compression via adaptive frequency trigger. In Proc. CVPR, 12250--12259

    2023

  54. [62]

    Yus, F. 2018. Identity-related issues in meme communication. Internet Pragmatics, 1(1): 113--133

    2018

  55. [63]

    Zhang, Z.; Liu, Q.; Wang, Z.; Lu, Z.; and Hu, Q. 2023. Backdoor defense via deconfounded representation learning. In Proc. CVPR, 12228--12238

    2023

  56. [64]

    K.-W.; and Chong, W

    Zhu, J.; Lee, R. K.-W.; and Chong, W. H. 2022. Multimodal zero-shot hateful meme detection. In Proc. ACM WWW, 382--389

    2022

  57. [65]

    Zhu, M.; Wei, S.; Zha, H.; and Wu, B. 2024 a . Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. NeurIPS

    2024

  58. [66]

    Zhu, Z.; Zhang, M.; Wei, S.; Wu, B.; and Wu, B. 2024 b . VDC : Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models. In ICLR

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.