Pith. sign in

REVIEW 5 minor 26 references

AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes

T0 review · 0 major / 5 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight hierarchical soft-label model that predicts full annotator disagreement ranks first on fine-grained sexism categorization in memes.

desk verdict Clean CLEF system paper: frozen Gemini embeddings + gated MLP + KL soft labels + uncertainty weighting, first on Task 2.3 Soft-Soft, code released. read the letter →

arxiv 2607.04410 v1 pith:OIPAEBKF submitted 2026-07-05 cs.CL

Matteo Fasulo , Antonio Gravina , Luca Tedeschini , Luca Babboni This is my paper
classification cs.CL
keywords sexismidentificationmultimodalmemeslearningwithdisagreementhierarchicalclassificationsoftlabelsgatedMLPuncertaintyweighting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper shows that sexism detection in memes can be treated as three nested soft-label problems rather than forced hard decisions. The authors freeze a strong vision-language embedder, feed its fixed vectors through a small gated MLP, and train the three heads with KL divergence against the full empirical distribution of annotators, automatically balanced by learned uncertainty weights. Hierarchical structure is enforced only by masking losses and by multiplying probabilities at decoding time, not by heavy architectural constraints. On the official Soft-Soft track the system placed first on the hardest multi-label categorization task and fourth on the two coarser tasks, while using only 3.5 million trainable parameters. The result matters because it demonstrates that annotator disagreement and taxonomic constraints can be captured efficiently without fine-tuning massive generative models or discarding minority labels.

What carries the argument

Conditional soft-label multi-task head: three linear classifiers on a shared SwiGLU-gated representation, optimized by masked KL losses whose relative weights are learned as inverse task variances, with joint probabilities recovered at inference by multiplying the sexism probability into the downstream distributions.

What would settle it

Replace the proprietary embeddings with a comparable open-weight vision-language model of similar dimension, retrain only the gated MLP head under identical conditions, and check whether Soft-Soft ranks on Task 2.3 remain first (or collapse).

Watch

Extended reading notes

Core claim

Mapping frozen Gemini Embedding 2 representations through a lightweight gated MLP, trained with KL divergence on empirical annotator distributions and balanced by homoscedastic uncertainty weights, yields state-of-the-art soft-label performance on hierarchical multimodal sexism identification, ranking first on the fine-grained categorization subtask.

Load-bearing premise

The frozen 768-dimensional vision-language embeddings already contain essentially all the multimodal semantics needed for sexism detection, so a tiny trainable head plus loss masking is enough.

Editorial extensions

If this is right

  • Soft-label hierarchical decoding can be applied to other nested subjective tasks without redesigning the backbone.
  • Homoscedastic uncertainty weighting automatically balances binary, conditional-binary and multi-label heads of unequal difficulty.
  • Physiological signals (EEG, gaze, heart rate) can be deferred when linear separability tests fail, without harming ranking performance.
  • Hard-label performance remains competitive even with a naive 0.5 threshold, showing that well-calibrated soft distributions transfer to discrete decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same conditional soft-label recipe should transfer to other meme-based hate or toxicity taxonomies that share a binary root and multi-label leaves.
  • If open-weight embeddings close the gap, the entire pipeline becomes fully reproducible and deployable under strict licensing constraints.
  • The negligible EEG effect sizes suggest that future multimodal fusion should target non-linear or attention-based biosignal integration rather than simple concatenation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper describes the AI Wizards system for EXIST 2026 multimodal sexism identification in memes. Three nested subtasks (binary sexism, source intention, multi-label categorization) are cast as hierarchical conditional soft-label prediction over empirical annotator distributions. Fixed Gemini Embedding 2 (768-d) representations are mapped by a lightweight Gated MLP (SwiGLU blocks, ~3.5 M trainable parameters) trained with KL divergence and learned homoscedastic uncertainty weights; Tasks 2.2/2.3 are conditioned via loss masking and joint-probability decoding. Three runs (two seeds + a detached soft-gating ablation) are reported. On the official Soft-Soft leaderboards the system ranked 1st on Task 2.3 and 4th on Tasks 2.1 and 2.2; Hard-Hard results with an untuned 0.5 threshold are also given. Code is released.

Significance. The work is a solid, well-documented shared-task contribution that cleanly combines three useful ingredients—frozen multimodal embeddings, soft-label (LeWiDi) training, and hierarchical conditional decoding—into a compact, reproducible pipeline. Official Soft-Soft ranks (especially first place on the hardest multi-label subtask) and the public code repository give the result external credibility. The explicit gating ablation and the transparent discussion of the proprietary backbone and untuned hard thresholds further strengthen the paper as a practical reference for hierarchical soft-label meme classification. While the scientific novelty is incremental rather than foundational, the engineering clarity and leaderboard evidence make it a useful addition to the literature on multimodal sexism detection under disagreement.

minor comments (5)
  1. Section 3.1 and Table 2: the EEG MANOVA analysis is thorough but ultimately negative; a single sentence clarifying that physiological fusion was abandoned solely on linear-separability grounds (and that non-linear models remain open) would prevent readers from over-interpreting the null result.
  2. Figure 1 caption and Section 4.3: the distinction between structural soft-gating (Run 3 only) and the probabilistic decoding used for all runs is clear in the text but could be reinforced by a short legend note on the dashed arrows.
  3. Tables 3–6: seed variance is reported for local splits; adding the corresponding standard deviations (or at least noting the range) for the official Soft-Soft ICM scores would make the ranking stability more transparent.
  4. Limitations: the proprietary nature of Gemini Embedding 2 is already acknowledged; a brief pointer to one or two concrete open-weight substitutes (e.g., SigLIP or open CLIP variants) that the modular design could accept would increase reproducibility value.
  5. Equation (3) and surrounding text: the homoscedastic weighting formula is standard, yet a one-line reminder that the log-σ regularizer prevents task collapse would help readers less familiar with multi-task uncertainty weighting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical competition system with external leaderboard evaluation; no derivation that reduces to its own inputs.

full rationale

This is a CLEF/EXIST 2026 system notebook describing a hierarchical multi-task soft-label pipeline (frozen Gemini Embedding 2 → SwiGLU gated MLP → KL + homoscedastic uncertainty weighting + conditional loss masking and joint-probability decoding). The central claim is the official Soft-Soft ranks (1st on Task 2.3, 4th on 2.1/2.2), which are external measurements on held-out test data, not quantities derived from fitted parameters or self-defined identities. Homoscedastic σᵢ are free learnable scalars optimized jointly with the network (Eq. 3, Kendall et al. 2018); they do not define the ICM evaluation metric. Hierarchical constraints are enforced by standard masking and product decoding (Eqs. 4–5), not by re-labeling the training targets as predictions. No self-citation is load-bearing for uniqueness or for the reported ranks; physiological-signal analysis is an empirical null result, not a circular premise. The paper is fully self-contained against external benchmarks and contains no first-principles derivation that collapses into its inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central performance claim rests on a frozen commercial embedding, a handful of standard architectural and optimization choices, and the empirical soft-label distributions supplied by the organizers. No new physical or mathematical entities are postulated; free parameters are ordinary ML hyper-parameters and the learned uncertainty weights.

free parameters (4)
  • homoscedastic uncertainty weights σ_i
    Learnable scalars that re-weight the three task losses (Eq. 3); optimized jointly with network weights.
  • SwiGLU expansion factor and dropout
    Fixed at 2 and 0.2 respectively; chosen by the authors rather than derived.
  • AdamW learning rate, weight decay, batch size, OneCycleLR warm-up
    Hand-set to 1e-4, 1e-2, 8, 30 %; standard but free choices that affect final ranks.
  • hard-label decision threshold 0.5
    Fixed without tuning for the Hard-Hard track; acknowledged as non-optimal.
assumptions (4)
  • domain assumption Frozen Gemini Embedding 2 features already encode the multimodal semantics needed for sexism detection
    Stated in Section 4.1; no fine-tuning is performed and no open-weight substitute is evaluated.
  • domain assumption Annotator soft-label distributions are the correct training and evaluation targets (LeWiDi paradigm)
    Adopted from the EXIST 2026 task definition and Uma et al. (2022).
  • ad hoc to paper Conditional loss masking plus joint-probability decoding is sufficient to enforce the task hierarchy
    Section 4.2–4.3; the soft-gating ablation performed comparably, supporting the claim but remaining an empirical modeling choice.
  • domain assumption Linear MANOVA on PCA-reduced EEG features is an adequate test of physiological utility
    Section 3.1; non-linear or multi-modal fusion is left for future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes." pith.science (2026). https://pith.science/paper/OIPAEBKF

@misc{pith2026260704410,
  author       = {Pith},
  title        = {Pith review of: AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OIPAEBKF}},
  note         = {Machine review of arXiv:2607.04410}
}
read the original abstract

We present the AI Wizards submission to EXIST 2026 for multimodal sexism identification in memes. The task is composed of three, increasingly harder subtasks. We model them hierarchically as conditional soft-label prediction over empirical annotator distributions. Our system maps fixed Gemini Embedding 2 vision-language representations through a lightweight Gated MLP trained with KL divergence and homoscedastic uncertainty weighting. Our submissions ranked first on Task 2.3 and fourth on Tasks 2.1 and 2.2 on the official Soft-Soft leaderboards. The code is available at https://github.com/NLP-AI-Wizards/EXIST-2026

Figures

Figures reproduced from arXiv: 2607.04410 by the authors.

Figure 1
Figure 1. Architectural data flow illustrating the hierarchical conditional dependencies. Pre-trained Gemini 2 embeddings are mapped to a shared semantic representation h via SwiGLU blocks. This shared representation is routed to all classification heads (solid lines). The dashed lines and multiplication nodes (⊗) represent the conditional dependency: structurally enforced via detached soft-gating in Run 3, and probabilistica… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 6 canonical work pages

  1. [1]

    Nadim, A

    M. Nadim, A. Fladmoe, Silencing women? gender and online harassment, Soc. Sci. Com- put. Rev. 39 (2021) 245–258. URL: https://doi.org/10.1177/0894439319865518. doi: 10.1177/ 0894439319865518

    work page doi:10.1177/0894439319865518 2021

  2. [2]

    Glick, S

    P. Glick, S. Fiske, An ambivalent alliance: Hostile and benevolent sexism as complementary justifications for gender inequality, American Psychologist 56 (2001) 109–118. doi: 10.1037/ 0003-066X.56.2.109

    2001

  3. [3]

    J. Im, S. Schoenebeck, M. Iriarte, G. Grill, D. Wilkinson, A. Batool, R. Alharbi, A. Funwie, T. Gankhuu, E. Gilbert, M. Naseem, Women’s perspectives on harm and justice after online harassment, Proc. ACM Hum.-Comput. Interact. 6 (2022). URL: https://doi.org/10.1145/3555775. doi:10.1145/ 3555775

    work page doi:10.1145/3555775 2022

  4. [4]

    Fersini, F

    E. Fersini, F. Gasparini, G. Rizzi, A. Saibene, B. Chulvi, P. Rosso, A. Lees, J. Sorensen, SemEval-2022 task 5: Multimedia automatic misogyny identification, in: G. Emerson, N. Schluter, G. Stanovsky, R. Kumar, A. Palmer, N. Schneider, S. Singh, S. Ratan (Eds.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), Associat...

    2022

  5. [5]

    Kiela, H

    D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, D. Testuggine, The hateful memes challenge: detecting hate speech in multimodal memes, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Curran Associates Inc., Red Hook, NY, USA, 2020

    2020

  6. [6]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, I. Arcos, M. Aloy-Mayo, P. Rosso, E. García-Arias, D. Spina, Overview of EXIST 2026: Physiological data for multimodal sexism characterization in social media, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventeenth International Conference of the CLEF Association (CLEF 2...

    2026

  7. [7]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, I. Arcos, M. Aloy-Mayo, P. Rosso, E. García-Arias, D. Spina, Overview of EXIST 2026: Physiological data for multimodal sexism characterization in social media (Extended Overview), in: E. Sánchez Salido, A. Barrón-Cedeño, A. García Seco de Herrera, S. MacAvaney, J. M. Struß (Eds.), CLEF 2026. Working Notes, 2026

    2026

  8. [8]

    A. N. Uma, T. Fornaciari, D. Hovy, S. Paun, B. Plank, M. Poesio, Learning from disagreement: A survey, J. Artif. Int. Res. 72 (2022) 1385–1470. URL: https://doi.org/10.1613/jair.1.12752. doi: 10. 1613/jair.1.12752

    doi:10.1613/jair.1.12752 2022

Show all 26 references
  1. [9]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, I. Arcos, P. Rosso, D. Spina, E. Amigó, J. Gonzalo, R. Morante, Overview of exist 2025: Learning with disagreement for sexism identification and characterization in tweets, memes, and tiktok videos, in: J. Carrillo-de Albornoz, J. Gonzalo, L....

    2025 doi

  2. [10]

    H. Kirk, W. Yin, B. Vidgen, P. Röttger, SemEval-2023 task 10: Explainable detection of online sexism, in: A. K. Ojha, A. S. Doğruöz, G. Da San Martino, H. Tayyar Madabushi, R. Kumar, E. Sartori (Eds.), Proceedings of the 17th International Workshop on Semantic Evaluation (SemE...

    2023 doi

  3. [11]

    R. Cao, M. S. Hee, A. Kuek, W.-H. Chong, R. K.-W. Lee, J. Jiang, Pro-cap: Leveraging a frozen vision-language model for hateful meme detection, in: Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, 2023, p. 5244–5252. URL: http://dx.doi.org/10.1145/35...

    2023 doi

  4. [12]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, V. Ruiz, A. Maeso, B. Chulvi, P. Rosso, E. Amigó, J. Gonzalo, R. Morante, D. Spina, Overview of exist 2024 — learning with disagreement for sexism identification and characterization in tweets and memes, in: L. Goeuriot, P. Mulhem, G. Quénot,...

    2024

  5. [13]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, in: M. Meila, T. Zhang (Eds.), Proceedings of the 38th International C...

    2021

  6. [14]

    X. Zhai, B. Mustafa, A. Kolesnikov, L. Beyer, Sigmoid loss for language image pre-training, in: 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11941–11952. doi:10.1109/ICCV51070.2023.01100

    2023 doi

  7. [15]

    G. K. Kumar, K. Nandakumar, Hate-CLIPper: Multimodal hateful meme classification based on cross-modal interaction of CLIP features, in: L. Biester, D. Demszky, Z. Jin, M. Sachan, J. Tetreault, S. Wilson, L. Xiao, J. Zhao (Eds.), Proceedings of the Second Workshop on NLP for Po...

    2022 doi

  8. [16]

    L. Tian, J. R. Trippas, M.-A. Rizoiu, Mario at exist 2025: A simple gateway to effective multilingual sexism detection, 2025. URL: https://arxiv.org/abs/2507.10996.arXiv:2507.10996

    2025 arXiv

  9. [17]

    K. R. White, J. Crites, Stephen L., J. H. Taylor, G. Corral, Wait, what? assessing stereotype incongruities using the n400 erp component, Social Cognitive and Affective Neuroscience 4 (2009) 191–198. URL: https://doi.org/10.1093/scan/nsp004. doi: 10.1093/scan/nsp004. arXiv:htt...

    2009 doi

  10. [18]

    Shanbhogue, Z

    M. Shanbhogue, Z. Li, S. Zhang, G. H. Ábrego, S.-C. Huang, A. Jain, D. Salz, S. Goenka, C. Hegde, J. Ma, F. Chen, J. Wu, T. Dabral, B. Samari, K. Poulet, D. Cer, K. Chen, P. Suganathan, H. Hui, J. Andonov, P. Schlattner, J. Han, I. Naim, W. Lowe, V. Pchelin, A. Yang, Y.-T. Che...

    2026 arXiv

  11. [19]

    Shazeer, Glu variants improve transformer, 2020

    N. Shazeer, Glu variants improve transformer, 2020. URL: https://arxiv.org/abs/2002.05202. arXiv:2002.05202

    2020 arXiv

  12. [20]

    Kullback, R

    S. Kullback, R. A. Leibler, On information and sufficiency, The annals of mathematical statistics 22 (1951) 79–86

    1951

  13. [21]

    Kendall, Y

    A. Kendall, Y. Gal, R. Cipolla, Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 7482–7491

    2018

  14. [22]

    Amigo, A

    E. Amigo, A. Delgado, Evaluating extreme hierarchical multi-label classification, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational L...

    2022

  15. [23]

    I. A. Gabaldón, P. Rosso, E. G. Vicent, Human-centered multimodal fusion for sexism detection in memes with eye-tracking, heart rate, and eeg signals, in: S. Piperidis, N. Bel, H. van den Heuvel, N. Ide, S. Krek, A. Toral (Eds.), Proceedings of the Fifteenth Language Resources...

    2026 doi

  16. [24]

    Alacam, S

    Ö. Alacam, S. Hoeken, S. Zarrieß, Eyes don’t lie: Subjective hate annotation and detection with gaze, in: Y. Al-Onaizan, M. Bansal, Y.-N. Chen (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguist...

    2024

  17. [25]

    F. Oguz, A. Alkan, T. Schöler, Emotion detection from ecg signals with different learning algorithms and automated feature engineering, Signal, Image and Video Processing 17 (2023) 1–9. doi: 10. 1007/s11760-023-02606-y

    2023

  18. [26]

    Zhang, Q

    Y. Zhang, Q. Li, S. Nahata, T. Jamal, S.-K. Cheng, G. Cauwenberghs, T.-P. Jung, Integrating large language model, eeg, and eye-tracking for word-level neural state classification in reading comprehension, IEEE Transactions on Neural Systems and Rehabilitation Engineering 32 (2...

    2024 doi

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.