Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content

Diptesh Kanojia; Girish A. Koushik; Helen Treharne

REVIEW 4 major objections 6 minor 49 references

Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Simple embedding fusion is state of the art for video hate detection but fails on image-text memes.

desk verdict A useful, honest empirical comparison of fusion methods across video and meme hate detection, but the headline SOTA claim on HateMM is confounded by simultaneous ASR and encoder changes, so the central video-vs-image contrast is not yet proven. read the letter →

arxiv 2502.07138 v1 pith:APB4YUW4 submitted 2025-02-11 cs.CV cs.CLcs.LG

Girish A. Koushik , Diptesh Kanojia , Helen Treharne This is my paper · ORCID

classification cs.CVcs.CLcs.LG

keywords multimodalhatespeechdetectionembeddingfusionmodalityrobustnessMMHatefulMemesbenignconfounderscross-modalvideoversusimage-text

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that multimodal fusion for hate detection is not modality-agnostic: the same fusion recipe that is state of the art on video fails on image-text memes. On the video-based HateMM dataset, concatenating strong pre-trained text, audio, and image embeddings (HateXplain, CLAP, and CLIP) raises macro-F1 to 0.848, a 9.9-point gain over the best prior result. On the Hateful Memes dataset, the same family of approaches tops out at 0.628 AUROC, far below dedicated fine-tuned systems that score above 0.85 and only slightly above the zero-shot vision-language models reported in the same table. The authors read this gap as evidence that synchronized video lets each modality contribute independently, while memes require joint reasoning over image and text, especially when benign confounders are present. If correct, the result implies that robust content moderation needs modality-specific architectures rather than one universal fusion module.

What carries the argument

The load-bearing object is simple embedding fusion, the operation $F = [E_1; E_2; \dots; E_m]$ (or element-wise product) applied to pre-trained unimodal embeddings, contrasted with MO-Hate, a BART-based order-aware fusion that passes text, audio, and visual inputs through cross-modal attention with a temporal component. The comparison is run on two benchmark datasets chosen to represent content format: HateMM (video, 1,083 samples) and the Hateful Memes Challenge (image-text, 10,000 samples with benign confounders). The mechanism does the work by showing that the same fusion functions succeed when modalities are synchronized streams and fail when meaning is created by the image-text interaction.

What would settle it

Take the same 1,083 HateMM videos, extract a single keyframe and its ASR transcript as a static image-text meme, and run HCC1; conversely, animate HMC memes into short videos with synchronized audio. If simple fusion performs equally well in both formats, the paper's modality-combination explanation is wrong.

Watch

Extended reading notes

Core claim

The central discovery is that where a fusion method sits on the spectrum from concatenating embeddings to reasoning jointly determines whether it works, and that this depends on the content format. For video, where audio, visual frames, and ASR text are synchronized and mutually redundant, simple embedding fusion achieves state-of-the-art performance, with the best configuration (HCC1) reaching 0.848 macro-F1 on HateMM, a 9.9-point improvement over the previous 0.749. For static image-text memes, the same fusion techniques and a sequential attention-based variant (MO-Hate, best MBD1 at 0.628 AUROC) collapse, because hatefulness can be carried by the interaction of an innocuous image with a caption and by benign confounders that foil unimodal shortcuts. The paper argues that current fusion approaches fail to capture these nuanced cross-modal interactions and that the field needs modality-specific architectural considerations.

Load-bearing premise

The load-bearing assumption is that the two datasets differ in the way that matters, video versus image-text modality combination, rather than in other properties such as dataset size, annotator agreement, ASR quality, or the presence of benign confounders.

Editorial extensions

If this is right

For synchronized video content, a simple concatenation of strong unimodal embeddings is sufficient to beat existing sequential fusion baselines by 9.9 F1 points, so heavyweight cross-modal reasoning is not needed there.
For image-text memes containing benign confounders, neither concatenation, element-wise product, late fusion, nor order-aware attention reaches reliable accuracy, so current fusion methods are not a drop-in solution.
A single modality-agnostic hate-detection framework is not supported by the evidence; systems should be tailored to whether modalities are synchronized streams or static pairs whose joint meaning must be disambiguated.
Improving ASR transcript quality (Whisper over Vosk) is itself a major contributor to video hate detection performance, since text is the strongest single modality in the ablation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One implication left implicit is that the paper's modality explanation is entangled with dataset differences such as size, annotator agreement, and ASR quality; a stricter test would control those factors before attributing the gap to video versus image-text content.
A testable extension would convert HMC memes into short videos with synchronized audio narration and convert HateMM videos into static image-text pairs, then rerun the same fusion models to see whether the performance gap tracks content format or the datasets themselves.
The results suggest that fusion should be split into two problems: aggregation for redundant synchronized modalities and reasoning for complementary static modalities; future work could quantify this by measuring how much the modalities overlap in information for each dataset.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, it is the first study I know of that applies the same fusion methods to both a video-based hate dataset (HateMM) and an image-text meme dataset (HMC), and that direct comparison is genuinely useful for people choosing architectures in this subfield. Second, the headline result — simple concatenation beats the prior HateMM baseline by 9.9 F1 points — is not actually a clean test of fusion. The paper itself lists three joint causes: better ASR (Whisper replacing Vosk), stronger encoders (CLAP, CLIP, HateXplain), and fusion. So the gain cannot be attributed to concatenation, and the "state-of-the-art" label is asserted against a single 2023 baseline without other video methods like MultiHateClip. The stress-test note is right: even if the datasets were perfectly matched, the fusion variable is not isolated.

That said, the paper does several things well. The ablations on HateMM (unimodal vs. pairwise vs. all modalities) are informative and show text carries much of the signal. The HMC experiments reproduce the known failure of simple fusion on benign confounders, which is a useful sanity check. The qualitative error analysis is concrete, and the code is public. The authors are also refreshingly honest in Sec. 6 about dataset bias and metric limitations. This is not a case of a paper faking results; it is a case of overinterpreting a confounded comparison.

The soft spots are in proportion. Single runs, no error bars, on a 1,083-video dataset — that is minor-to-moderate. The bigger issue is that the central conclusion — "simple fusion works for synchronized video but not for memes" — relies on comparing two datasets that differ in size, label agreement, content source, and confounder design, not just modality. The authors acknowledge this indirectly but do not control for it. The MO-Hate adaptation is interesting but underperforms, and the paper does not fully explain why; it is a side result rather than a contribution.

Who is this for? Researchers working on multimodal hate detection who want a map of what standard encoders plus concatenation can and cannot do. It is a solid extension, not a new paradigm. It deserves a serious referee, but the referee should push for multi-seed results, an ablation that keeps ASR and encoders fixed while varying fusion, and a broader baseline set. My recommendation: send it to peer review, but with the expectation of major revision. If the authors can isolate the fusion effect, the central claim becomes publishable; as is, it is a good workshop paper but not a robust framework paper.

Referee Report

4 major / 6 minor

Summary. The paper presents a comparative study of multimodal fusion approaches for hate speech detection on two datasets: HateMM (video with audio, text, and visual frames) and the Hateful Memes Challenge (HMC, image-text pairs). The authors evaluate simple embedding fusion (concatenation and element-wise product) and a modality order-aware fusion model called MO-Hate, which uses cross-modal attention in a sequential manner. On HateMM, their best simple-fusion model (HCC1: HXP text + CLAP audio + CLIP visual, concatenated) reports a macro-F1 of 0.848, a 9.9 percentage point improvement over the previously reported result by Das et al. On HMC, all tested approaches achieve AUROC below 0.63, far below existing specialized models such as PALI-X-VPD (0.892). The paper interprets this performance gap as evidence that simple embedding fusion works for synchronized video content but fails for image-text memes that require joint reasoning. The authors include ablations, qualitative error analysis, and release code.

Significance. If the claims are supported, the paper provides a useful empirical benchmark and a cautionary result for the multimodal hate detection community: modality composition matters, and simple fusion is not modality-agnostic. The strength of the paper is its explicit two-dataset comparison and its detailed ablation and error analysis, which gives concrete insight into where fusion fails. The release of code is also a positive. However, the significance is substantially tempered by confounding factors in the headline comparison and by the absence of statistical reliability measures, as detailed in the major comments.

major comments (4)

[Sec. 4.1, Table 2a] The 9.9-point F1 improvement of HCC1 over the best previously reported method (BERT + MFCC + ViT) is not attributable to fusion alone, because the comparison changes four components at once: ASR (Vosk to Whisper), text encoder (BERT to HXP), audio encoder (MFCC to CLAP), and vision encoder (ViT to CLIP). The paper itself (Sec. 4.1) credits three joint factors: enhanced transcription quality, effective embedding fusion, and strong pre-trained encoders. Therefore the claim that 'simple embedding fusion achieves state-of-the-art performance' is not supported by this comparison. Please add ablations that vary only the fusion mechanism while holding encoders and ASR fixed, and also compare against other video-centric methods such as MultiHateClip cited in Sec. 2.3.2.
[Sec. 4.1 vs Sec. 4.2, Tables 2 and 3] The central cross-modal conclusion assumes that the performance gap between HateMM and HMC is caused by the modality combination (video versus image-text) rather than by dataset-specific differences, including dataset size (1,083 vs. ~10k samples), annotation agreement (Cohen's kappa 62.5 vs. 68.4), ASR quality, and the presence of benign confounders. Since only one video dataset and one image-text dataset are used, the modality factor is not isolated. Please temper the wording (e.g., in the abstract and Sec. 5) or support the contrast with additional datasets that better match annotation protocols and difficulty.
[Sec. 3.4 and Tables 2-3] All experimental results are single runs with no standard deviations, confidence intervals, or significance tests. With only 217 test videos in HateMM, the differences between models such as HCC1 (0.848 F1) and MO-Hate BCD1 (0.821 F1) may not be statistically meaningful. Please report results over multiple seeds or provide bootstrap confidence intervals, and discuss the effect of the small test set on the reliability of the reported gaps.
[Sec. 4.2, Table 3] The conclusion that 'current fusion approaches' struggle on HMC is drawn from two self-implemented baselines that use frozen, non-fine-tuned embeddings (simple fusion and MO-Hate). Existing high-performing methods listed in Table 3 (AUROC >= 0.858) employ task-specific fine-tuning, contrastive learning, or programmatic tool use. The low AUROC values (0.5-0.63) may therefore reflect underfitting or insufficient adaptation rather than a fundamental limitation of fusion architectures. Please clarify that the claim applies to the particular frozen-embedding fusion setups evaluated here, and discuss the gap to fine-tuned competitors explicitly.

minor comments (6)

[Sec. 3.3.2] The MO-Hate fusion equation uses coefficients alpha, beta, and gamma, but it is not specified whether they are learned parameters, fixed values, or tuned hyperparameters; please clarify how they are set.
[Sec. 3.2.1 and Appendix A] The claimed improvement in transcription quality from Vosk to Whisper is supported only by qualitative examples; consider adding an automated speech recognition quality metric or a small human evaluation to make the improvement measurable.
[Sec. 4.3] The sentence 'MO-Hate performs better than BCD1 because the model has ignored many instances of benign visuals' is confusing because BCD1 is itself the full MO-Hate model; please specify which ablation condition is being compared to which.
[Sec. 5] The statement that using only 100 frames 'did not affect the result since, upon manual inspection, the longer videos were deemed to be non-hateful' is a post-hoc manual observation; please provide quantitative evidence or remove this claim.
[Appendix B] The text contains a typo: 'a a late fusion' should be 'a late fusion'.
[Table 4] The 'Modality' column mixes the presence of modalities in the input with the modalities that the model actually used for prediction; consider separating these two aspects for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all reported results are held-out empirical measurements against fixed test sets.

full rationale

The paper's central claims are empirical evaluations on held-out test splits of the HateMM and Hateful Memes datasets. Scores such as HCC1's macro-F1 of 0.848 on HateMM and MBD1's AUROC of 0.628 on HMC are produced by training fixed fusion architectures with fixed pretrained encoders and then evaluating on test data; no parameter is fitted to the test labels, and no reported quantity is defined in terms of the claim it is used to support. The comparison with Das et al. changes the ASR system and several encoders simultaneously, which is a potential threat to the attribution of the 9.9-point gain to embedding fusion, but that is an experimental-design or validity concern rather than circularity because both the baseline and the proposed scores are independently measured. The paper does not invoke a uniqueness theorem, does not rely on load-bearing self-citations, and does not rename a known result as a derivation. The reused components, such as HateXplain, CLIP, CLAP, DINOv2, and Wav2Vec2, are external pretrained models used as feature extractors, which is standard transfer learning and not circular. The video-versus-image contrast is an interpretation of observed performance differences across two datasets, not a quantity obtained by construction from the same data. No circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central results are empirical and do not introduce new theoretical entities. The free parameters are few: the MO-Hate fusion coefficients are unspecified, and the frame sampling rate is a manual choice. The main assumptions are dataset representativeness, encoder sufficiency, and transferability of the sarcasm-detection architecture.

free parameters (2)

MO-Hate fusion coefficients alpha, beta, gamma = not reported
Introduced in the F_total equation in Section 3.3.2; the paper does not state whether they are learned weights, fixed hyperparameters, or set to specific values. The MO-Hate results, including the best HMC model MBD1, depend on this unspecified scheme.
Video frame sampling rate (100 frames per video) = 100
The paper samples one frame per second up to 100 frames; the authors claim in Section 5 that longer videos being non-hateful means this did not affect results, but this claim is based on manual inspection rather than an experiment.

assumptions (3)

domain assumption The two datasets' labels are reliable enough to support conclusions about hate detection performance.
The paper reports only moderate annotator agreement (Cohen's kappa 62.5 for HateMM, 68.4 for HMC) but treats labels as ground truth for all comparisons.
domain assumption The pretrained encoders provide embeddings that preserve task-relevant information across modalities.
The fusion results assume that BERT, HXP, CLIP, CLAP, Wav2Vec2, and DINOv2 features are sufficient representations; no fine-tuning of these encoders is performed.
domain assumption The MO-Hate architecture, adapted from multimodal sarcasm detection (Tomar et al.), transfers to hate speech detection without major re-design.
Section 3.3.2 borrows the sequential cross-modal attention design from sarcasm detection; its suitability for hate detection is not separately validated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content." pith.science (2026). https://pith.science/paper/APB4YUW4

@misc{pith2026250207138,
  author       = {Pith},
  title        = {Pith review of: Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/APB4YUW4}},
  note         = {Machine review of arXiv:2502.07138}
}

read the original abstract

Social media platforms enable the propagation of hateful content across different modalities such as textual, auditory, and visual, necessitating effective detection methods. While recent approaches have shown promise in handling individual modalities, their effectiveness across different modality combinations remains unexplored. This paper presents a systematic analysis of fusion-based approaches for multimodal hate detection, focusing on their performance across video and image-based content. Our comprehensive evaluation reveals significant modality-specific limitations: while simple embedding fusion achieves state-of-the-art performance on video content (HateMM dataset) with a 9.9% points F1-score improvement, it struggles with complex image-text relationships in memes (Hateful Memes dataset). Through detailed ablation studies and error analysis, we demonstrate how current fusion approaches fail to capture nuanced cross-modal interactions, particularly in cases involving benign confounders. Our findings provide crucial insights for developing more robust hate detection systems and highlight the need for modality-specific architectural considerations. The code is available at https://github.com/gak97/Video-vs-Meme-Hate.

Figures

Figures reproduced from arXiv: 2502.07138 by the authors.

**Figure 2.** Sample predictions from MBD1. the video-based HateMM dataset, demonstrating the potential of leveraging synchronised multimodal information. As the video offers three separate types of modalities - audio, keyframes from the video, and text transcription, we found that a simple embedding fusion of each of the three provided an effective representation. This is because, video benefits from the temporal aspects where … view at source ↗

**Figure 3.** Simple Fusion architecture as shown in Das et al [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: MO-Hate architecture as shown in Tomar et al. [40]. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

**Figure 5.** Figure 5: More predictions and true labels for some of the HMC instances using the MBD1 model. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 23 canonical work pages

[1]

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al
[2]

Greeshma Arya, Mohammad Kamrul Hasan, Ashish Bagwari, Nurhizam Safie, Shayla Islam, Fatima Rayan Awad Ahmed, Aaishani De, Muhammad Attique Khan, and Taher M Ghazal. 2024. Multimodal Hate Speech Detection in Memes using Contrastive Language-Image Pre-training. IEEE Access (2024). Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs...

work page 2024
[3]

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 (2023)

arXiv 2023
[4]

Abdul Aziz, Md Akram Hossain, and Abu Nowshed Chy. 2023. Csecu-dsg@ multimodal hate speech event detection 2023: Transformer-based multimodal hierarchical fusion model for multimodal hate speech detection. In Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text. 101–107

work page 2023
[5]

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33 (2020), 12449–12460

2020
[6]

Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1993–2002

work page 2023
[7]

Giovanni Burbi, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. 2023. Mapping Memes to Words for Multimodal Hateful Meme Clas- sification. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2832–2836

work page 2023
[8]

Rui Cao, Roy Ka-Wei Lee, and Jing Jiang. 2024. Modularized Networks for Few- shot Hateful Meme Detection. In Proceedings of the ACM on Web Conference 2024 . 4575–4584

work page 2024

Show all 49 references

[9]

Anusha Chhabra and Dinesh Kumar Vishwakarma. 2023. A literature survey on multimodal and multilingual automatic hate speech identification. Multimedia Systems 29, 3 (2023), 1203–1230

2023
[10]

Anusha Chhabra and Dinesh Kumar Vishwakarma. 2023. Multimodal hate speech detection via multi-scale visual kernels and knowledge distillation architecture. Engineering Applications of Artificial Intelligence 126 (2023), 106991

2023
[11]

Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini. 2021. The echo chamber effect on social media. Proceedings of the National Academy of Sciences 118, 9 (2021), e2023301118

2021
[12]

Mithun Das, Rohit Raj, Punyajoy Saha, Binny Mathew, Manish Gupta, and Ani- mesh Mukherjee. 2023. Hatemm: A multi-modal dataset for hate video classi- fication. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 17. 1014–1023

2023
[13]

Steven Davis and Paul Mermelstein. 1980. Comparison of parametric representa- tions for monosyllabic word recognition in continuously spoken sentences. IEEE transactions on acoustics, speech, and signal processing 28, 4 (1980), 357–366

1980
[14]

Ona De Gibert, Naiara Perez, Aitor García-Pablos, and Montse Cuadros
[15]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

2018 arXiv
[16]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv prepri...

2020 arXiv
[17]

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang
[18]

Nabeel Gillani, Ann Yuan, Martin Saveski, Soroush Vosoughi, and Deb Roy. 2018. Me, my echo chamber, and I: introspection on social media polarization. In Proceedings of the 2018 World Wide Web Conference . 823–831

2018
[19]

Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. 2020. Ex- ploring hate speech detection in multimodal publications. In Proceedings of the IEEE/CVF winter conference on applications of computer vision . 1470–1478

2020
[20]

Eric Grinstein, Ngoc QK Duong, Alexey Ozerov, and Patrick Pérez. 2018. Audio style transfer. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 586–590

2018
[21]

Ming Shan Hee, Wen-Haw Chong, and Roy Ka-Wei Lee. 2023. Decoding the un- derlying meaning of multimodal hateful memes. arXiv preprint arXiv:2305.17678 (2023)

2023 arXiv
[22]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural computation 9, 8 (1997), 1735–1780

1997
[23]

Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. 2023. Visual Program Distilla- tion: Distilling Tools and Programmatic Reasoning into Vision-Language Models. arXiv preprint arXiv:2312.03052 (2023)

2023 arXiv
[24]

Jinfa Huang, Jinsheng Pan, Zhongwei Wan, Hanjia Lyu, and Jiebo Luo. 2024. Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection. arXiv preprint arXiv:2407.21004 (2024)

2024 arXiv
[25]

Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes chal- lenge: Detecting hate speech in multimodal memes. Advances in neural informa- tion processing systems 33 (2020), 2611–2624

2020
[26]

Gokul Karthik Kumar and Karthik Nandakumar. 2022. Hate-CLIPper: Multimodal hateful meme classification based on cross-modal interaction of CLIP features. arXiv preprint arXiv:2210.05916 (2022)

2022 arXiv
[27]

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.134...

2019 arXiv
[28]

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26296–26306

2024
[29]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning. Advances in neural information processing systems 36 (2024)

2024
[30]

Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explain- able hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 14867–14875

2021
[31]

Lambert Mathias, Shaoliang Nie, Aida Mostafazadeh Davani, Douwe Kiela, Vin- odkumar Prabhakaran, Bertie Vidgen, and Zeerak Waseem. 2021. Findings of the WOAH 5 shared task on fine grained hateful memes detection. In Proceedings of the 5th Workshop on Online Abuse and Harms (WO...

2021
[32]

Jingbiao Mei, Jinghong Chen, Weizhe Lin, Bill Byrne, and Marcus Tomalin
[33]

Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas
[34]

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El- Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

2023 arXiv
[35]

Shraman Pramanick, Dimitar Dimitrov, Rituparna Mukherjee, Shivam Sharma, Md Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021. Detecting harmful memes and their targets. arXiv preprint arXiv:2110.00413 (2021)

2021 arXiv
[36]

arXiv preprint arXiv:2311.08110 (2023)

Improving hateful memes detection via learning hatefulness-aware em- bedding space through retrieval-guided contrastive learning. arXiv preprint arXiv:2311.08110 (2023)

2023 arXiv
[37]

Siva Sai, Naman Deep Srivastava, and Yashvardhan Sharma. 2022. Explorative application of fusion techniques for multimodal hate speech detection. SN Computer Science 3, 2 (2022), 122

2022
[38]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

2014 arXiv
[39]

Shardul Suryawanshi, Bharathi Raja Chakravarthi, Mihael Arcan, and Paul Buite- laar. 2020. Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text. In Proceedings of the second workshop on trolling, aggression and cyberbullying. 32–41

2020
[40]

Mohit Tomar, Abhisek Tiwari, Tulika Saha, and Sriparna Saha. 2023. Your tone speaks louder than your face! Modality Order Infused Multi-modal Sarcasm Detection. In Proceedings of the 31st ACM International Conference on Multimedia . 3926–3933

2023
[41]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

2021
[42]

Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predic- tive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop. 88–93

2016
[43]

Tharindu Cyril Weerasooriya, Sujan Dutta, Tharindu Ranasinghe, Marcos Zampieri, Christopher M Homan, and Ashiqur R KhudaBukhsh. 2023. Vicari- ous offense and noise audit of offensive speech classifiers: unifying human and machine disagreement on what is offensive. arXiv prepri...

2023
[44]

the the the the the the the the the the

Savvas Zannettou, Barry Bradlyn, Emiliano De Cristofaro, Haewoon Kwak, Michael Sirivianos, Gianluca Stringini, and Jeremy Blackburn. 2018. What is gab: A bastion of free speech or an alt-right echo chamber. In Companion Proceedings of the The Web Conference 2018 . 1007–1014. A...

2018
[46]

Han Wang, Tan Rui Yang, Usman Naseem, and Roy Ka-Wei Lee. 2024. Multihate- clip: A multilingual benchmark dataset for hateful video detection on youtube and bilibili. In Proceedings of the 32nd ACM International Conference on Multimedia . 7493–7502

2024
[2018]

arXiv preprint arXiv:1809.04444 (2018)

Hate speech dataset from a white supremacy forum. arXiv preprint arXiv:1809.04444 (2018)

2018 arXiv
[2020]

arXiv preprint arXiv:2006.08328 (2020)

Ethos: an online hate speech detection dataset. arXiv preprint arXiv:2006.08328 (2020)

2020 arXiv
[2022]

Advances in neural information processing systems 35 (2022), 23716–23736

Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736

2022
[2023]

InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Clap learning audio concepts from natural language supervision. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

[2] [2]

Greeshma Arya, Mohammad Kamrul Hasan, Ashish Bagwari, Nurhizam Safie, Shayla Islam, Fatima Rayan Awad Ahmed, Aaishani De, Muhammad Attique Khan, and Taher M Ghazal. 2024. Multimodal Hate Speech Detection in Memes using Contrastive Language-Image Pre-training. IEEE Access (2024). Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs...

work page 2024

[3] [3]

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 (2023)

arXiv 2023

[4] [4]

Abdul Aziz, Md Akram Hossain, and Abu Nowshed Chy. 2023. Csecu-dsg@ multimodal hate speech event detection 2023: Transformer-based multimodal hierarchical fusion model for multimodal hate speech detection. In Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text. 101–107

work page 2023

[5] [5]

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33 (2020), 12449–12460

2020

[6] [6]

Aashish Bhandari, Siddhant B Shah, Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. Crisishatemm: Multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1993–2002

work page 2023

[7] [7]

Giovanni Burbi, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. 2023. Mapping Memes to Words for Multimodal Hateful Meme Clas- sification. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2832–2836

work page 2023

[8] [8]

Rui Cao, Roy Ka-Wei Lee, and Jing Jiang. 2024. Modularized Networks for Few- shot Hateful Meme Detection. In Proceedings of the ACM on Web Conference 2024 . 4575–4584

work page 2024