Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Co-AttenDWG: Co-Attentive Dimension-Wise Gating and Expert Fusion for Multi-Modal Offensive Content Detection

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that Co-AttenDWG, by combining bidirectional co-attention with dimension-wise gating and expert fusion, sets new state-of-the-art results on the MIMIC and SemEval Memotion benchmarks for multimodal offensive content…

desk verdict Coherent incremental architecture for multimodal offensive detection, but the headline SOTA claim is undercut by an internal inconsistency in the Misogyny numbers and missing reproducibility details. read the letter →

arxiv 2505.19010 v2 pith:467MO5ZO submitted 2025-05-25 cs.CV cs.CL

Md. Mithun Hossain , Md. Shakil Hossain , Sudipto Chaki , M. F. Mridha This is my paper · ORCID
classification cs.CVcs.CL
keywords multimodaloffensivecontentdetectionco-attentiondimension-wisegatingexpertfusionMambaFormerMIMICdatasetMemotioncross-modalalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Co-AttenDWG, an architecture for detecting offensive content in memes that pairs text and image encoders with a fusion stack designed to let the two modalities influence each other before they are combined. The central claim is that this stack—bidirectional co-attention, channel-level gating, MambaFormer refinement, and expert fusion—outperforms established vision-language models such as mCLIP, VisualBERT, ALBEF, and BLIP on the MIMIC and SemEval Memotion benchmarks. On the reported test splits it achieves state-of-the-art results on three MIMIC subtasks (Misogyny, Objectification, Prejudice) and on Memotion, while trailing VisualBERT by 0.11 percentage points on Humiliation. A combinatorial ablation shows every major module contributes, with the largest drops when multiple modules are removed. If the result holds under a leak-free evaluation protocol, it argues that fine-grained cross-modal gating plus mixture-of-experts fusion is a better recipe than static concatenation or standard cross-attention for this task.

What carries the argument

The load-bearing object is the gated co-attention plus expert-fusion stack assembled inside Co-AttenDWG. Co-attention computes multi-head attention in both directions between text and image; dimension-wise gating applies a sigmoid gate per channel to the attention outputs, which is what lets the model emphasize or suppress individual feature dimensions rather than whole modalities. MambaFormer encoders, which combine selective-state-space processing with convolutional and self-attention operations, refine the gated cross-modal features, and an additional cross-attention layer realigns the two paths. The expert fusion module combines a feed-forward network, a softmax-gated weighted sum of the two modality experts, and a self-attention refinement, all under layer normalization. These pieces carry the claim because the ablations show that removing any one of them lowers accuracy on every benchmark.

What would settle it

Re-run the comparison with a pre-registered protocol in which the MIMIC and Memotion datasets are split into 80 percent training and 20 percent test before any upsampling, apply upsampling only to the training portion, freeze this split across all models, and train each baseline with the same hyperparameter budget; if Co-AttenDWG's margins over mCLIP and VisualBERT shrink to within one standard error or disappear, the state-of-the-art claim rests on protocol rather than architecture.

Watch

Extended reading notes

Core claim

Co-AttenDWG is a dual-path fusion model. Text and image features are projected into a shared space; a co-attention layer lets text attend to image and image to text, and a dimension-wise gating network multiplies the attention outputs channel-by-channel by sigmoid weights so each feature dimension is strengthened or suppressed. The gated features pass through two MambaFormer encoders, each refining the other modality's signal before element-wise addition with the original projection, and an extra cross-attention layer aligns the two paths again. A fusion module then concatenates the paths, computes a softmax gating weight for each modality, and sums the weighted experts with the output of a feed-forward network and a self-attention layer under layer normalization. With XLM-RoBERTa and ResNet50 as backbones, the paper reports 87.19 accuracy and 87.16 F1 on Misogyny, 94.80 on Objectification, 97.15 on Prejudice, 98.80 on Humiliation, and 84.29 accuracy with 84.26 F1 on Memotion, and the ablation results attribute the gains to every component of the stack.

Load-bearing premise

The load-bearing premise is that the evaluation protocol is fair and leak-free: the class-balancing upsampling is applied only to the training split, and every baseline is trained under the same split, tuning, and metric as Co-AttenDWG; the paper does not explicitly state that upsampling was applied after the train/test split.

Editorial extensions

If this is right

  • If the reported results are correct, Co-AttenDWG sets new state-of-the-art numbers on the MIMIC Misogyny, Objectification, and Prejudice tasks and on Memotion offense classification.
  • The ablation study implies that each architectural choice—co-attention, dimension-wise gating, dual-path MambaFormer refinement, additional cross-attention, and expert fusion—contributes to the final score, with the largest degradation when several are removed together.
  • The 31.1 ms per-sample inference time on a single consumer GPU suggests the extra fusion machinery does not make the model impractical for content-moderation pipelines.
  • The per-sample gating and attention visualizations indicate the model adapts its fusion strategy, leaning on text when visual cues are ambiguous and on images when they are decisive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to freeze the backbones and train only the fusion stack; if the gains mostly vanish, the improvement may come from fine-tuning XLM-RoBERTa rather than from the co-attention and gating machinery itself.
  • The same fusion recipe could be evaluated on other multimodal tasks such as hate-speech detection, sarcasm detection, and sentiment analysis, where the per-channel gating logic should transfer if the mechanism is general.
  • The reported gains over strong baselines are small (0.07 to 1.69 percentage points), so repeated runs with multiple seeds and a pre-registered train/test split would tell whether the improvement is real or within noise.
  • Since the error table shows most confusion is between adjacent offensiveness levels, framing the Memotion task as ordinal regression might capture more signal than the current four-way softmax.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Co-AttenDWG, a multimodal fusion architecture for offensive content detection that combines co-attention, dimension-wise gating, dual-path MambaFormer encoders, additional cross-attention, and an expert fusion module with learned gating and self-attention. The method is evaluated on the MIMIC and SemEval Memotion 1.0 datasets, with reported state-of-the-art results such as 87.19% accuracy and 87.16% F1 on Misogyny, 94.80% on Objectification, 97.15% on Prejudice, 98.80% on Humiliation, and 84.29% accuracy / 84.26% F1 on Memotion. The paper includes ablations over architectural components, hyperparameters, and backbone choices, plus qualitative interpretability analyses.

Significance. If the empirical claims hold, Co-AttenDWG would provide a competitive recipe for fine-grained vision–language fusion, with a useful set of ablations showing the contribution of each component. The paper also ships detailed per-task error counts and interpretability heatmaps, which are strengths. However, the central claim is an empirical benchmark result, and the current manuscript does not yet support the claimed margins: the reported numbers are internally inconsistent, the preprocessing order is ambiguous, and no code, seeds, or error bars are provided to make the single-run comparisons verifiable.

major comments (4)
  1. [Table III vs. Table VI, Section IV-E] The headline Misogyny result is internally inconsistent. Table III reports Co-AttenDWG at 87.19% accuracy and 87.16% F1, while Table VI reports the same configuration (XLM-RoBERTa + ResNet50, identified as the best backbone) at 87.79% and 87.83% on the same task. The gap (0.60 accuracy, 0.67 F1) is larger than the claimed +0.80 improvement over VisualBERT in Table III. The authors must explain which number is correct, ensure that all tables report the same final run, and clarify whether Table V numbers are validation or test results. As written, the central state-of-the-art claim is not reproducible from the manuscript.
  2. [Section IV-B and Table I] The order of upsampling relative to the train/test split is ambiguous and potentially leak-prone. Section IV-B states that the authors 'partition both datasets into 80% training and 20% testing splits' and then apply upsampling, but Table I reports balanced counts for the entire MIMIC and Memotion datasets. If upsampling is performed before the split, the test set can contain exact copies of training samples, which would inflate all reported accuracies. The authors must explicitly state that upsampling is applied only to the training split after partitioning, or re-run the experiments if the current order does not guarantee separation.
  3. [Section IV-C and Tables III-VI] The reported margins over baselines are small (e.g., +0.07 accuracy on Objectification, +0.13 on Prejudice, and a negative margin on Humiliation), yet all results appear to come from a single run with no error bars, significance tests, or released code or seeds. Without such information, a reader cannot distinguish genuine improvement from random variation, especially for differences below one percentage point. The authors should provide multiple-seed statistics, confidence intervals or permutation tests, and ideally release code and seeds to make the comparisons auditable.
  4. [Section III-C, Equations (7)-(10)] The 'MambaFormer' encoder is not adequately defined or cited. The text references [40] and [41], which are Vaswani et al. (Transformer) and Hu et al. (Squeeze-and-Excitation), neither of which introduces a MambaFormer block. No equation or architectural detail specifies what MambaFormer consists of, despite it being a named component in Equations (7) and (9) and in the ablations of Table IV (MF). The authors need to provide a precise description and proper references, or the architecture cannot be reproduced.
minor comments (5)
  1. [Tables III and IV captions] The dataset name is misspelled as 'MMIC' in the captions of Tables III and IV; it should be 'MIMIC'.
  2. [Section III-D and Table IV] The abbreviation 'FF' in Table IV is defined as 'Fine-grained Fusion' but no module with that name appears in the architecture description. The fusion network in Equations (15)-(16) is the likely referent, but the mapping should be made explicit.
  3. [Section II, paragraph 2] The phrase 'static fusion technique such as simple concatenation' is grammatically incomplete; it should be 'static fusion techniques such as simple concatenation'.
  4. [Table V] The 'Pixel Value' column is hard to parse, especially the entry '2242×10−5' which appears to be a rendering artifact combining image size '224' and learning rate '2×10−5'. Please ensure each hyperparameter is in its own clearly separated column.
  5. [Section IV-C] The statement that Co-AttenDWG 'consistently outperforms all baseline models across the majority of categories' is imprecise because the Humiliation category is below VisualBERT and mCLIP by 0.11 and 0.10 points, respectively; the wording should be adjusted to match the reported table.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported test F1 scores are the validation-set scores from the hyperparameter search; the central SOTA F1 margin reduces to a fitted validation result.

  1. fitted input called prediction [Table V vs Table III (Sections IV-C and IV-E)]
    "Impact of core architectural hyperparameters (number of experts, cross-attention heads, co-attention heads, MambaFormer kernel size, depth, dropout, pixel value, and learning rate) on macro F1 (%) for each label. Results are on the MIMIC and Memotion validation sets. ... establishing new state-of-the-art results with accuracy and F1 scores of 87.19% and 87.16% for Misogyny detection, 94.80% for both metrics in Objectification, 97.15% in Prejudice, and 84.29% accuracy alongside 84.26% F1 on the Memotion offensive content detection task."

    The best-validation row of Table V lists macro F1 values of 87.16 (Misogyny), 94.80 (Objectification), 97.15 (Prejudice), 98.80 (Humiliation), and 84.26 (Memotion). The Co-AttenDWG row in Table III, presented as the benchmark/test comparison, reports exactly the same F1 values for these categories. Since Table V is explicitly a hyperparameter/architecture search on validation sets, the F1 scores used as Co-AttenDWG's test-set performance are the same scores on which the configuration was selected. The claimed state-of-the-art F1 improvements (e.g., +0.77 on Misogyny) therefore reduce to the fitted validation numbers rather than to an independent held-out prediction.

full rationale

The paper's central claim is an empirical benchmark comparison, and most of the architecture derivation (Equations 1-21) is self-contained and not circular. However, the F1 values presented as Co-AttenDWG's test performance in Table III are numerically identical, category by category, to the best row of Table V, whose caption explicitly states results are on validation sets. Since Table V is the hyperparameter/architecture search used to select the final configuration, reporting those same F1 values as the held-out test result means the headline F1 improvements are fitted validation numbers rather than independent predictions. The accuracy columns in Table III have no corresponding validation accuracy in Table V, so only F1 is directly affected, but F1 is one of the two primary metrics and is the metric used for the Misogyny improvement claim. A separate internal inconsistency (Table III Misogyny 87.19/87.16 vs Table VI XLM-RoBERTa+ResNet50 87.79/87.83) and the ambiguous upsampling-before/after-split description in Section IV-B are correctness and reproducibility concerns, not circularity mechanisms. The self-citation [16] for gated cross-attention is related-work context and does not carry the argument.

Assumptions & free parameters 11 free parameters · 5 assumptions · 0 invented entities

No theorem derivation is claimed; the central claim is empirical. The ledger captures validation-tuned hyperparameters, the assumed integrity of the data split, and the unspecified MambaFormer component. Invented physical entities are absent; the proposed modules are compositions of existing neural network building blocks.

free parameters (11)
  • Number of experts = 8
    Selected as the best row in the validation sweep in Table V; affects fusion capacity.
  • Cross-attention heads = 8
    Selected in Table V as part of the best configuration.
  • Co-attention heads = 4
    Selected in Table V; the best row uses 4 co-attention heads.
  • Self-attention refinement heads = 4
    Set in Section IV-B for the refinement layer.
  • MambaFormer kernel size = 3
    Selected in Table V as the best kernel size.
  • MambaFormer depth = 2
    Selected in Table V as the best depth.
  • Dropout rate = 0.1
    Selected in Table V as the best dropout.
  • Learning rate = 2e-5
    Selected from the options in Table II and confirmed in Table V.
  • Image sizes = 200x200 for MIMIC, 160x160 for Memotion
    Chosen per dataset in Section IV-B and validated in Table V.
  • Class upsampling counts = Minority classes raised to 4537 or 4032 for MIMIC and 2657 for Memotion
    Hand-chosen balancing strategy in Table I; if applied before the split, it can distort test performance.
  • Early stopping patience = 3
    Reported in Table II as a tuned hyperparameter.
assumptions (5)
  • domain assumption MIMIC and Memotion labels are reliable ground truth.
    The entire evaluation treats the benchmark annotations as correct without auditing label noise.
  • domain assumption Class-balancing upsampling is applied only to the training split.
    Section IV-B does not explicitly state whether upsampling occurs before or after the 80/20 split, and Table I shows balanced counts for the full datasets.
  • domain assumption Validation-selected hyperparameters generalize to the test set.
    Table V tunes eight hyperparameters on validation and the final numbers are then reported, without correction for multiple comparisons.
  • domain assumption Pretrained BERT, XLM-RoBERTa, and ResNet50 features are sufficient for this task.
    The model relies on these backbones for all representations, and the paper does not analyze failure cases caused by backbone limitations.
  • domain assumption MambaFormer is a well-defined module that behaves as described.
    The paper uses MambaFormer in Section III-C and Figure 2 but does not define it or cite a source for the Mamba component.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Co-AttenDWG: Co-Attentive Dimension-Wise Gating and Expert Fusion for Multi-Modal Offensive Content Detection." pith.science (2026). https://pith.science/paper/467MO5ZO

@misc{pith2026250519010,
  author       = {Pith},
  title        = {Pith review of: Co-AttenDWG: Co-Attentive Dimension-Wise Gating and Expert Fusion for Multi-Modal Offensive Content Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/467MO5ZO}},
  note         = {Machine review of arXiv:2505.19010}
}
read the original abstract

Multi-modal learning has emerged as a crucial research direction, as integrating textual and visual information can substantially enhance performance in tasks such as classification, retrieval, and scene understanding. Despite advances with large pre-trained models, existing approaches often suffer from insufficient cross-modal interactions and rigid fusion strategies, failing to fully harness the complementary strengths of different modalities. To address these limitations, we propose Co-AttenDWG, co-attention with dimension-wise gating, and expert fusion. Our approach first projects textual and visual features into a shared embedding space, where a dedicated co-attention mechanism enables simultaneous, fine-grained interactions between modalities. This is further strengthened by a dimension-wise gating network, which adaptively modulates feature contributions at the channel level to emphasize salient information. In parallel, dual-path encoders independently refine modality-specific representations, while an additional cross-attention layer aligns the modalities further. The resulting features are aggregated via an expert fusion module that integrates learned gating and self-attention, yielding a robust unified representation. Experimental results on the MIMIC and SemEval Memotion 1.0 datasets show that Co-AttenDWG achieves state-of-the-art performance and superior cross-modal alignment, highlighting its effectiveness for diverse multi-modal applications.

Figures

Figures reproduced from arXiv: 2505.19010 by the authors.

Figure 1
Figure 1. Examples of memes that combine textual cues with visual context, illustrating the challenges of multi-modal integration. Both examples demand nuanced interpretation of text, facial expressions, and background details [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A high-level overview of our Co-AttenDWG architecture for multi-modal offensive content detection. The image branch (top) processes memes through a pre-trained CNN (ResNet50), extracting high-level visual features I. Meanwhile, the text branch (bottom) encodes input sentences using a language model such as BERT, yielding textual features T. Both sets of features are projected into a shared embedding space and enter … view at source ↗
Figure 3
Figure 3. Cross-attention weight distribution in our Co-AttenDWG architecture. Each cell represents the attention magnitude from a text token to a visual feature. Warmer colors indicate higher attention, and cooler colors indicate lower attention, with the scale ranging from 0 (lowest) to 1 (highest). across the majority of categories, establishing new state-of-the￾art results with accuracy and F1 scores of 87.19% and 87.16% … view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Bar plots illustrating the gating weights assigned to each expert for different samples in the ExpertFusion module. Each Figure (a, b, c, d) corresponds to a distinct sample, showcasing how the gating mechanism adapts to different inputs. (a) Sample 1: Balanced gate, s…
Figure 6
Figure 6. Figure 6: Self-attention heatmap visualizations for selected samples from the fine-grained interpretability analysis. Each subfigure illustrates how the model distributes attention between modalities and across sequence/image regions for the given example. collectively contribut…
Figure 7
Figure 7. Figure 7: Case study examples on offensive content detection. The top two subfigures show Memotion samples with true labels ”not offensive” (a) and ”hateful offensive” (b). In (a), mCLIP fails to detect offensive content (✗), while VisualBERT, ALBEF, BLIP, and Co-AttenDWG correc…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 31 canonical work pages

  1. [16]

    Dimension-wise gated cross-attention for multimodal sentiment analysis,

    M. S. Hossain, M. M. Hossain, S. Chaki, M. Mridha, M. S. Rahman, and M. A. Moni, “Dimension-wise gated cross-attention for multimodal sentiment analysis,” inCompanion Proceedings of the ACM on Web Conference 2025, 2025, pp. 1979–1987

    work page 2025

  2. [40]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  3. [41]

    Squeeze-and-excitation networks,

    J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 7132–7141

    2018

  4. [1]

    Multimodal machine learning: A survey and taxonomy,

    T. Baltru ˇsaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,”IEEE transactions on pattern anal- ysis and machine intelligence, vol. 41, no. 2, pp. 423–443, 2018

    2018

  5. [2]

    Multimodal deep learning

    J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, A. Y . Nget al., “Multimodal deep learning.” inICML, vol. 11, 2011, pp. 689–696

    work page 2011

  6. [3]

    Emogif: A multimodal approach to detect emotional support in animated gifs,

    A. Singh, D. Sharma, and V . K. Singh, “Emogif: A multimodal approach to detect emotional support in animated gifs,”IEEE Transactions on Computational Social Systems, 2025

    work page 2025

  7. [4]

    Show, attend and tell: Neural image caption generation with visual attention,

    K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y . Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” inProceedings of the 32nd Interna- tional Conference on Machine Learning (ICML), 2015, pp. 2048–2057

    work page 2015

  8. [5]

    Multiseao-mix: A multimodal multitask framework for sentiment, emotion, support, and offensive analysis in code-mixed setting,

    G. V . Singh, A. Verma, A. Ekbalet al., “Multiseao-mix: A multimodal multitask framework for sentiment, emotion, support, and offensive analysis in code-mixed setting,”IEEE Transactions on Computational Social Systems, 2024

    work page 2024

Show all 52 references
  1. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inPro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolog...

    2019

  2. [7]

    Multilingual denoising pre-training for neural machine translation,

    Y . Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, “Multilingual denoising pre-training for neural machine translation,”Transactions of the Association for Computational Linguistics, vol. 8, pp. 726–742, 2020

    2020

  3. [8]

    Unsupervised cross-lingual repre- sentation learning,

    S. Ruder, A. Søgaard, and I. Vuli ´c, “Unsupervised cross-lingual repre- sentation learning,” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, 2019, pp. 31–38

    2019

  4. [9]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” inAdvances in Neural Infor- mation Processing Systems (NIPS), 2012, pp. 1097–1105

    2012

  5. [10]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  6. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  7. [12]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,”Advances in neural information processing systems, vol. 32, 2019

    2019

  8. [13]

    Lxmert: Learning cross-modality encoder representations from transformers,

    H. Tan and M. Bansal, “Lxmert: Learning cross-modality encoder representations from transformers,”arXiv preprint arXiv:1908.07490, 2019

    1908 arXiv

  9. [14]

    Visualbert: A simple and performant baseline for vision and language,

    L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,”arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  10. [15]

    Uniter: Learning universal image-text representations,

    Y .-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y . Cheng, and J. Liu, “Uniter: Learning universal image-text representations,” 2019

    2019

  11. [17]

    Multimodal transformer for unaligned multimodal language sequences,

    Y .-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” inProceedings of the conference. Association for computational linguistics. Meeting, vol. 2019, 2019, p. 6558

    2019

  12. [18]

    Relation-aware graph attention network for visual question answering,

    L. Li, Z. Gan, Y . Cheng, and J. Liu, “Relation-aware graph attention network for visual question answering,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 10 313–10 322

    2019

  13. [19]

    Multi-modal discussion transformer: Integrating text, images and graph transformers to detect hate speech on social media,

    L. Hebert, G. Sahu, Y . Guo, N. K. Sreenivas, L. Golab, and R. Cohen, “Multi-modal discussion transformer: Integrating text, images and graph transformers to detect hate speech on social media,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 20, ...

    2024

  14. [20]

    Multimodal sentiment analysis in realistic environments based on cross-modal hierarchical fusion network,

    J. Huang, P. Lu, S. Sun, and F. Wang, “Multimodal sentiment analysis in realistic environments based on cross-modal hierarchical fusion network,”Electronics, vol. 12, no. 16, p. 3504, 2023

    2023

  15. [21]

    Super- vised multimodal bitransformers for classifying images and text,

    D. Kiela, S. Bhooshan, H. Firooz, E. Perez, and D. Testuggine, “Super- vised multimodal bitransformers for classifying images and text,”arXiv preprint arXiv:1909.02950, 2019

    1909 arXiv

  16. [22]

    Emotion based hate speech detection using multimodal learning,

    A. Rana and S. Jha, “Emotion based hate speech detection using multimodal learning,”arXiv preprint arXiv:2202.06218, 2022. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 13

    2022 arXiv

  17. [23]

    Multimodal datasets: misogyny, pornography, and malignant stereotypes,

    A. Birhane, V . Prabhu, and E. Kahembwe, “Multimodal datasets: misogyny, pornography, and malignant stereotypes,”arXiv preprint arXiv:2110.01963, 2021

    2021 arXiv

  18. [24]

    Mul- timodal meme dataset (multioff) for identifying offensive content in image and text,

    S. Suryawanshi, B. Chakravarthi, M. Arcan, and P. Buitelaar, “Mul- timodal meme dataset (multioff) for identifying offensive content in image and text,” inProceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, 2020, pp. 32–41

    2020

  19. [25]

    Multi-modal twitter data analysis for identifying offensive posts using a deep cross atten- tion based transformer framework,

    J. Paul, S. Mallick, A. Mitra, A. Roy, and J. Sil, “Multi-modal twitter data analysis for identifying offensive posts using a deep cross atten- tion based transformer framework,”ACM Transactions on Knowledge Discovery from Data, 2025

    2025

  20. [26]

    Multimodal large language model with lora fine-tuning for multimodal sentiment analysis,

    J. Mu, W. Wang, W. Liu, T. Yan, and G. Wang, “Multimodal large language model with lora fine-tuning for multimodal sentiment analysis,” ACM Transactions on Intelligent Systems and Technology, 2024

    2024

  21. [27]

    Image–text sentiment analysis via deep multimodal attentive fusion,

    F. Huang, X. Zhang, Z. Zhao, J. Xu, and Z. Li, “Image–text sentiment analysis via deep multimodal attentive fusion,”Knowledge-Based Sys- tems, vol. 167, pp. 26–37, 2019

    2019

  22. [28]

    Pars-off: a benchmark for offensive language detec- tion on farsi social media,

    T. S. Ataei, K. Darvishi, S. Javdan, A. Pourdabiri, B. Minaei-Bidgoli, and M. T. Pilehvar, “Pars-off: a benchmark for offensive language detec- tion on farsi social media,”IEEE Transactions on Affective Computing, vol. 14, no. 4, pp. 2787–2795, 2022

    2022

  23. [29]

    Djmf: A discriminative joint multi-task framework for multimodal sentiment analysis based on intra- and inter-task dynamics,

    Y . Zheng, J. Gong, Y . Wen, and P. Zhang, “Djmf: A discriminative joint multi-task framework for multimodal sentiment analysis based on intra- and inter-task dynamics,”Expert Systems with Applications, vol. 242, p. 122728, 2024

    2024

  24. [30]

    Joint multimodal sentiment analysis based on information relevance,

    D. Chen, W. Su, P. Wu, and B. Hua, “Joint multimodal sentiment analysis based on information relevance,”Information Processing & Management, vol. 60, no. 2, p. 103193, 2023

    2023

  25. [31]

    Decoding memes: a comprehensive analysis of late and early fusion models for explainable meme analysis,

    F. Abdullakutty and U. Naseem, “Decoding memes: a comprehensive analysis of late and early fusion models for explainable meme analysis,” inCompanion Proceedings of the ACM Web Conference 2024, May 2024, pp. 1681–1689

    2024

  26. [32]

    Multimodal sentiment analysis based on fusion methods: A survey,

    L. Zhu, Z. Zhu, C. Zhang, Y . Xu, and X. Kong, “Multimodal sentiment analysis based on fusion methods: A survey,”Information Fusion, vol. 95, pp. 306–325, 2023

    2023

  27. [33]

    Multi- modal sentiment analysis: A systematic review of history, datasets, mul- timodal fusion methods, applications, challenges and future directions,

    A. Gandhi, K. Adhvaryu, S. Poria, E. Cambria, and A. Hussain, “Multi- modal sentiment analysis: A systematic review of history, datasets, mul- timodal fusion methods, applications, challenges and future directions,” Information Fusion, vol. 91, pp. 424–444, 2023

    2023

  28. [34]

    Mm-emor: multi-modal emotion recognition of social media using concatenated deep learning networks,

    O. Adel, K. Fathalla, and A. Abo ElFarag, “Mm-emor: multi-modal emotion recognition of social media using concatenated deep learning networks,”Big Data and Cognitive Computing, vol. 7, no. 4, p. 164, 2023

    2023

  29. [35]

    Syntax-aware hybrid prompt model for few-shot multi-modal sentiment analysis,

    Z. Zhou, H. Feng, B. Qiao, G. Wu, and D. Han, “Syntax-aware hybrid prompt model for few-shot multi-modal sentiment analysis,”arXiv preprint arXiv:2306.01312, 2023

    2023 arXiv

  30. [36]

    Offensive language detection for low resource language using deep sequence model,

    A. A. Khan, M. H. Iqbal, S. Nisar, A. Ahmad, and W. Iqbal, “Offensive language detection for low resource language using deep sequence model,”IEEE Transactions on Computational Social Systems, 2023

    2023

  31. [37]

    Research on multimodal hate speech detection based on self-attention mechanism feature fusion,

    J. Mao, H. Shi, and X. Li, “Research on multimodal hate speech detection based on self-attention mechanism feature fusion,”The Journal of Supercomputing, vol. 81, no. 1, p. 28, 2025

    2025

  32. [38]

    Multi-modal sentiment analysis based on image and text fusion using a cross-attention mechanism,

    H. Li, Y . Lu, and H. Zhu, “Multi-modal sentiment analysis based on image and text fusion using a cross-attention mechanism,”Electronics, vol. 13, no. 11, p. 2069, 2024

    2024

  33. [39]

    Fusion and discrimina- tion: A multimodal graph contrastive learning framework for multimodal sarcasm detection,

    B. Liang, L. Gui, Y . He, E. Cambria, and R. Xu, “Fusion and discrimina- tion: A multimodal graph contrastive learning framework for multimodal sarcasm detection,”IEEE Transactions on Affective Computing, 2024

    2024

  34. [42]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,”arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  35. [43]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,”arXiv preprint arXiv:1607.06450, 2016

    2016 arXiv

  36. [44]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,”arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  37. [45]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  38. [46]

    Efficientnetv2: Smaller models and faster training,

    M. Tan and Q. V . Le, “Efficientnetv2: Smaller models and faster training,” inICML, 2021

    2021

  39. [47]

    Unsupervised cross-lingual representation learning at scale,

    A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzm´an, and V . Stoyanov, “Unsupervised cross-lingual representation learning at scale,”NAACL-HLT, 2019

    2019

  40. [48]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  41. [49]

    Align before fuse: Vision and language representation learning with momentum distillation,

    J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,”Advances in neural information processing systems, vol. 34, pp. 9694–9705, 2021

    2021

  42. [50]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inInternational conference on machine learning. PMLR, 2022, pp. 12 888–12 900

    2022

  43. [51]

    Task report: Memotion analysis 1.0@ semeval 2020: The visuo-lingual metaphor,

    C. Sharma, W. Paka, D. B. Scott, A. Das, S. Poria, T. Chakraborty, and B. Gamb ¨ack, “Task report: Memotion analysis 1.0@ semeval 2020: The visuo-lingual metaphor,” inProceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020). Association for Computat...

    2020

  44. [52]

    Mimic: misogyny identification in multimodal internet content in hindi-english code-mixed language,

    A. Singh, D. Sharma, and V . K. Singh, “Mimic: misogyny identification in multimodal internet content in hindi-english code-mixed language,” ACM Transactions on Asian and Low-Resource Language Information Processing, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.