Pith. sign in

REVIEW 5 minor 26 references

A lightweight hierarchical soft-label model that predicts full annotator disagreement ranks first on fine-grained sexism categorization in memes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Hierarchical gated-MLP soft-label multi-task learning on frozen Gemini embeddings ranked 1st on EXIST 2026 Task 2.3 and 4th on Tasks 2.1–2.2 Soft-Soft.

T0 review reviewed 2026-07-11 challenge →

load-bearing objection Clean CLEF system paper: frozen Gemini embeddings + gated MLP + KL soft labels + uncertainty weighting, first on Task 2.3 Soft-Soft, code released.

arxiv 2607.04410 v1 pith:OIPAEBKF submitted 2026-07-05 cs.CL

AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes

Matteo Fasulo , Antonio Gravina , Luca Tedeschini , Luca Babboni This is my paper
classification cs.CL
keywords sexism identificationmultimodal memeslearning with disagreementhierarchical classificationsoft labelsgated MLPuncertainty weighting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper shows that sexism detection in memes can be treated as three nested soft-label problems rather than forced hard decisions. The authors freeze a strong vision-language embedder, feed its fixed vectors through a small gated MLP, and train the three heads with KL divergence against the full empirical distribution of annotators, automatically balanced by learned uncertainty weights. Hierarchical structure is enforced only by masking losses and by multiplying probabilities at decoding time, not by heavy architectural constraints. On the official Soft-Soft track the system placed first on the hardest multi-label categorization task and fourth on the two coarser tasks, while using only 3.5 million trainable parameters. The result matters because it demonstrates that annotator disagreement and taxonomic constraints can be captured efficiently without fine-tuning massive generative models or discarding minority labels.

Core claim

Mapping frozen Gemini Embedding 2 representations through a lightweight gated MLP, trained with KL divergence on empirical annotator distributions and balanced by homoscedastic uncertainty weights, yields state-of-the-art soft-label performance on hierarchical multimodal sexism identification, ranking first on the fine-grained categorization subtask.

What carries the argument

Conditional soft-label multi-task head: three linear classifiers on a shared SwiGLU-gated representation, optimized by masked KL losses whose relative weights are learned as inverse task variances, with joint probabilities recovered at inference by multiplying the sexism probability into the downstream distributions.

Load-bearing premise

The frozen 768-dimensional vision-language embeddings already contain essentially all the multimodal semantics needed for sexism detection, so a tiny trainable head plus loss masking is enough.

What would settle it

Replace the proprietary embeddings with a comparable open-weight vision-language model of similar dimension, retrain only the gated MLP head under identical conditions, and check whether Soft-Soft ranks on Task 2.3 remain first (or collapse).

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Soft-label hierarchical decoding can be applied to other nested subjective tasks without redesigning the backbone.
  • Homoscedastic uncertainty weighting automatically balances binary, conditional-binary and multi-label heads of unequal difficulty.
  • Physiological signals (EEG, gaze, heart rate) can be deferred when linear separability tests fail, without harming ranking performance.
  • Hard-label performance remains competitive even with a naive 0.5 threshold, showing that well-calibrated soft distributions transfer to discrete decisions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same conditional soft-label recipe should transfer to other meme-based hate or toxicity taxonomies that share a binary root and multi-label leaves.
  • If open-weight embeddings close the gap, the entire pipeline becomes fully reproducible and deployable under strict licensing constraints.
  • The negligible EEG effect sizes suggest that future multimodal fusion should target non-linear or attention-based biosignal integration rather than simple concatenation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper describes the AI Wizards system for EXIST 2026 multimodal sexism identification in memes. Three nested subtasks (binary sexism, source intention, multi-label categorization) are cast as hierarchical conditional soft-label prediction over empirical annotator distributions. Fixed Gemini Embedding 2 (768-d) representations are mapped by a lightweight Gated MLP (SwiGLU blocks, ~3.5 M trainable parameters) trained with KL divergence and learned homoscedastic uncertainty weights; Tasks 2.2/2.3 are conditioned via loss masking and joint-probability decoding. Three runs (two seeds + a detached soft-gating ablation) are reported. On the official Soft-Soft leaderboards the system ranked 1st on Task 2.3 and 4th on Tasks 2.1 and 2.2; Hard-Hard results with an untuned 0.5 threshold are also given. Code is released.

Significance. The work is a solid, well-documented shared-task contribution that cleanly combines three useful ingredients—frozen multimodal embeddings, soft-label (LeWiDi) training, and hierarchical conditional decoding—into a compact, reproducible pipeline. Official Soft-Soft ranks (especially first place on the hardest multi-label subtask) and the public code repository give the result external credibility. The explicit gating ablation and the transparent discussion of the proprietary backbone and untuned hard thresholds further strengthen the paper as a practical reference for hierarchical soft-label meme classification. While the scientific novelty is incremental rather than foundational, the engineering clarity and leaderboard evidence make it a useful addition to the literature on multimodal sexism detection under disagreement.

minor comments (5)
  1. Section 3.1 and Table 2: the EEG MANOVA analysis is thorough but ultimately negative; a single sentence clarifying that physiological fusion was abandoned solely on linear-separability grounds (and that non-linear models remain open) would prevent readers from over-interpreting the null result.
  2. Figure 1 caption and Section 4.3: the distinction between structural soft-gating (Run 3 only) and the probabilistic decoding used for all runs is clear in the text but could be reinforced by a short legend note on the dashed arrows.
  3. Tables 3–6: seed variance is reported for local splits; adding the corresponding standard deviations (or at least noting the range) for the official Soft-Soft ICM scores would make the ranking stability more transparent.
  4. Limitations: the proprietary nature of Gemini Embedding 2 is already acknowledged; a brief pointer to one or two concrete open-weight substitutes (e.g., SigLIP or open CLIP variants) that the modular design could accept would increase reproducibility value.
  5. Equation (3) and surrounding text: the homoscedastic weighting formula is standard, yet a one-line reminder that the log-σ regularizer prevents task collapse would help readers less familiar with multi-task uncertainty weighting.

Circularity Check

0 steps flagged

No circularity: empirical competition system with external leaderboard evaluation; no derivation that reduces to its own inputs.

full rationale

This is a CLEF/EXIST 2026 system notebook describing a hierarchical multi-task soft-label pipeline (frozen Gemini Embedding 2 → SwiGLU gated MLP → KL + homoscedastic uncertainty weighting + conditional loss masking and joint-probability decoding). The central claim is the official Soft-Soft ranks (1st on Task 2.3, 4th on 2.1/2.2), which are external measurements on held-out test data, not quantities derived from fitted parameters or self-defined identities. Homoscedastic σᵢ are free learnable scalars optimized jointly with the network (Eq. 3, Kendall et al. 2018); they do not define the ICM evaluation metric. Hierarchical constraints are enforced by standard masking and product decoding (Eqs. 4–5), not by re-labeling the training targets as predictions. No self-citation is load-bearing for uniqueness or for the reported ranks; physiological-signal analysis is an empirical null result, not a circular premise. The paper is fully self-contained against external benchmarks and contains no first-principles derivation that collapses into its inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central performance claim rests on a frozen commercial embedding, a handful of standard architectural and optimization choices, and the empirical soft-label distributions supplied by the organizers. No new physical or mathematical entities are postulated; free parameters are ordinary ML hyper-parameters and the learned uncertainty weights.

free parameters (4)
  • homoscedastic uncertainty weights σ_i
    Learnable scalars that re-weight the three task losses (Eq. 3); optimized jointly with network weights.
  • SwiGLU expansion factor and dropout
    Fixed at 2 and 0.2 respectively; chosen by the authors rather than derived.
  • AdamW learning rate, weight decay, batch size, OneCycleLR warm-up
    Hand-set to 1e-4, 1e-2, 8, 30 %; standard but free choices that affect final ranks.
  • hard-label decision threshold 0.5
    Fixed without tuning for the Hard-Hard track; acknowledged as non-optimal.
axioms (4)
  • domain assumption Frozen Gemini Embedding 2 features already encode the multimodal semantics needed for sexism detection
    Stated in Section 4.1; no fine-tuning is performed and no open-weight substitute is evaluated.
  • domain assumption Annotator soft-label distributions are the correct training and evaluation targets (LeWiDi paradigm)
    Adopted from the EXIST 2026 task definition and Uma et al. (2022).
  • ad hoc to paper Conditional loss masking plus joint-probability decoding is sufficient to enforce the task hierarchy
    Section 4.2–4.3; the soft-gating ablation performed comparably, supporting the claim but remaining an empirical modeling choice.
  • domain assumption Linear MANOVA on PCA-reduced EEG features is an adequate test of physiological utility
    Section 3.1; non-linear or multi-modal fusion is left for future work.

reviewed 2026-07-11 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes." pith.science (2026). https://pith.science/paper/OIPAEBKF

@misc{pith2026260704410,
  author       = {Pith},
  title        = {Pith review of: AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OIPAEBKF}},
  note         = {Machine review of arXiv:2607.04410}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We present the AI Wizards submission to EXIST 2026 for multimodal sexism identification in memes. The task is composed of three, increasingly harder subtasks. We model them hierarchically as conditional soft-label prediction over empirical annotator distributions. Our system maps fixed Gemini Embedding 2 vision-language representations through a lightweight Gated MLP trained with KL divergence and homoscedastic uncertainty weighting. Our submissions ranked first on Task 2.3 and fourth on Tasks 2.1 and 2.2 on the official Soft-Soft leaderboards. The code is available at https://github.com/NLP-AI-Wizards/EXIST-2026

Figures

Figures reproduced from arXiv: 2607.04410 by Antonio Gravina, Luca Babboni, Luca Tedeschini, Matteo Fasulo.

Figure 1
Figure 1. Figure 1: Architectural data flow illustrating the hierarchical conditional dependencies. Pre-trained Gemini 2 embeddings are mapped to a shared semantic representation h via SwiGLU blocks. This shared representation is routed to all classification heads (solid lines). The dashed lines and multiplication nodes (⊗) represent the conditional dependency: structurally enforced via detached soft-gating in Run 3, and prob… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 6 canonical work pages

  1. [1]

    Nadim, A

    M. Nadim, A. Fladmoe, Silencing women? gender and online harassment, Soc. Sci. Com- put. Rev. 39 (2021) 245–258. URL: https://doi.org/10.1177/0894439319865518. doi: 10.1177/ 0894439319865518

    work page doi:10.1177/0894439319865518 2021

  2. [2]

    Glick, S

    P. Glick, S. Fiske, An ambivalent alliance: Hostile and benevolent sexism as complementary justifications for gender inequality, American Psychologist 56 (2001) 109–118. doi: 10.1037/ 0003-066X.56.2.109

    2001

  3. [3]

    J. Im, S. Schoenebeck, M. Iriarte, G. Grill, D. Wilkinson, A. Batool, R. Alharbi, A. Funwie, T. Gankhuu, E. Gilbert, M. Naseem, Women’s perspectives on harm and justice after online harassment, Proc. ACM Hum.-Comput. Interact. 6 (2022). URL: https://doi.org/10.1145/3555775. doi:10.1145/ 3555775

    work page doi:10.1145/3555775 2022

  4. [4]

    Fersini, F

    E. Fersini, F. Gasparini, G. Rizzi, A. Saibene, B. Chulvi, P. Rosso, A. Lees, J. Sorensen, SemEval-2022 task 5: Multimedia automatic misogyny identification, in: G. Emerson, N. Schluter, G. Stanovsky, R. Kumar, A. Palmer, N. Schneider, S. Singh, S. Ratan (Eds.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), Associat...

    2022

  5. [5]

    Kiela, H

    D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, D. Testuggine, The hateful memes challenge: detecting hate speech in multimodal memes, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Curran Associates Inc., Red Hook, NY, USA, 2020

    2020

  6. [6]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, I. Arcos, M. Aloy-Mayo, P. Rosso, E. García-Arias, D. Spina, Overview of EXIST 2026: Physiological data for multimodal sexism characterization in social media, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventeenth International Conference of the CLEF Association (CLEF 2...

    2026

  7. [7]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, I. Arcos, M. Aloy-Mayo, P. Rosso, E. García-Arias, D. Spina, Overview of EXIST 2026: Physiological data for multimodal sexism characterization in social media (Extended Overview), in: E. Sánchez Salido, A. Barrón-Cedeño, A. García Seco de Herrera, S. MacAvaney, J. M. Struß (Eds.), CLEF 2026. Working Notes, 2026

    2026

  8. [8]

    A. N. Uma, T. Fornaciari, D. Hovy, S. Paun, B. Plank, M. Poesio, Learning from disagreement: A survey, J. Artif. Int. Res. 72 (2022) 1385–1470. URL: https://doi.org/10.1613/jair.1.12752. doi: 10. 1613/jair.1.12752

    doi:10.1613/jair.1.12752 2022

  9. [9]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, I. Arcos, P. Rosso, D. Spina, E. Amigó, J. Gonzalo, R. Morante, Overview of exist 2025: Learning with disagreement for sexism identification and characterization in tweets, memes, and tiktok videos, in: J. Carrillo-de Albornoz, J. Gonzalo, L. Plaza, A. García Seco de Herrera, J. Mothe, F. Piroi, P. Rosso, D. Spina, G. Fa...

    work page doi:10.1007/978-3-031-88720-8_65 2025

  10. [10]

    H. Kirk, W. Yin, B. Vidgen, P. Röttger, SemEval-2023 task 10: Explainable detection of online sexism, in: A. K. Ojha, A. S. Doğruöz, G. Da San Martino, H. Tayyar Madabushi, R. Kumar, E. Sartori (Eds.), Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval- 2023), Association for Computational Linguistics, Toronto, Canada, 2023, pp...

    doi:10.18653/v1/2023.semeval-1.305 2023

  11. [11]

    R. Cao, M. S. Hee, A. Kuek, W.-H. Chong, R. K.-W. Lee, J. Jiang, Pro-cap: Leveraging a frozen vision-language model for hateful meme detection, in: Proceedings of the 31st ACM International Conference on Multimedia, MM ’23, 2023, p. 5244–5252. URL: http://dx.doi.org/10.1145/3581783. 3612498. doi:10.1145/3581783.3612498

    doi:10.1145/3581783 2023

  12. [12]

    Plaza, J

    L. Plaza, J. Carrillo-de Albornoz, V. Ruiz, A. Maeso, B. Chulvi, P. Rosso, E. Amigó, J. Gonzalo, R. Morante, D. Spina, Overview of exist 2024 — learning with disagreement for sexism identification and characterization in tweets and memes, in: L. Goeuriot, P. Mulhem, G. Quénot, D. Schwab, G. M. Di Nunzio, L. Soulier, P. Galuščáková, A. García Seco de Herre...

    2024

  13. [13]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, in: M. Meila, T. Zhang (Eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Rese...

    2021

  14. [14]

    X. Zhai, B. Mustafa, A. Kolesnikov, L. Beyer, Sigmoid loss for language image pre-training, in: 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11941–11952. doi:10.1109/ICCV51070.2023.01100

    doi:10.1109/iccv51070.2023.01100 2023

  15. [15]

    G. K. Kumar, K. Nandakumar, Hate-CLIPper: Multimodal hateful meme classification based on cross-modal interaction of CLIP features, in: L. Biester, D. Demszky, Z. Jin, M. Sachan, J. Tetreault, S. Wilson, L. Xiao, J. Zhao (Eds.), Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI), Association for Computational Linguistics, Abu Dhabi, Un...

    doi:10.18653/v1/ 2022

  16. [16]

    L. Tian, J. R. Trippas, M.-A. Rizoiu, Mario at exist 2025: A simple gateway to effective multilingual sexism detection, 2025. URL: https://arxiv.org/abs/2507.10996.arXiv:2507.10996

    Pith/arXiv arXiv 2025

  17. [17]

    K. R. White, J. Crites, Stephen L., J. H. Taylor, G. Corral, Wait, what? assessing stereotype incongruities using the n400 erp component, Social Cognitive and Affective Neuroscience 4 (2009) 191–198. URL: https://doi.org/10.1093/scan/nsp004. doi: 10.1093/scan/nsp004. arXiv:https://academic.oup.com/scan/article-pdf/4/2/191/27105683/nsp004.pdf

    work page doi:10.1093/scan/nsp004 2009

  18. [18]

    Shanbhogue, Z

    M. Shanbhogue, Z. Li, S. Zhang, G. H. Ábrego, S.-C. Huang, A. Jain, D. Salz, S. Goenka, C. Hegde, J. Ma, F. Chen, J. Wu, T. Dabral, B. Samari, K. Poulet, D. Cer, K. Chen, P. Suganathan, H. Hui, J. Andonov, P. Schlattner, J. Han, I. Naim, W. Lowe, V. Pchelin, A. Yang, Y.-T. Chen, Z. Ding, G. Zhang, G. Heigold, Y. Chen, A. Reveillon, B. Mccloskey, W. Zhou, ...

    Pith/arXiv arXiv 2026

  19. [19]

    Shazeer, Glu variants improve transformer, 2020

    N. Shazeer, Glu variants improve transformer, 2020. URL: https://arxiv.org/abs/2002.05202. arXiv:2002.05202

    Pith/arXiv arXiv 2020

  20. [20]

    Kullback, R

    S. Kullback, R. A. Leibler, On information and sufficiency, The annals of mathematical statistics 22 (1951) 79–86

    1951

  21. [21]

    Kendall, Y

    A. Kendall, Y. Gal, R. Cipolla, Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 7482–7491

    2018

  22. [22]

    Amigo, A

    E. Amigo, A. Delgado, Evaluating extreme hierarchical multi-label classification, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 5809–5819. URL: https://aclanthology.org/...

    2022

  23. [23]

    I. A. Gabaldón, P. Rosso, E. G. Vicent, Human-centered multimodal fusion for sexism detection in memes with eye-tracking, heart rate, and eeg signals, in: S. Piperidis, N. Bel, H. van den Heuvel, N. Ide, S. Krek, A. Toral (Eds.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), European Language Resources Association ...

    work page doi:10.63317/2w2vaanu3pe7 2026

  24. [24]

    Alacam, S

    Ö. Alacam, S. Hoeken, S. Zarrieß, Eyes don’t lie: Subjective hate annotation and detection with gaze, in: Y. Al-Onaizan, M. Bansal, Y.-N. Chen (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 187–205. URL: https://aclanthology.org/2024....

    2024

  25. [25]

    F. Oguz, A. Alkan, T. Schöler, Emotion detection from ecg signals with different learning algorithms and automated feature engineering, Signal, Image and Video Processing 17 (2023) 1–9. doi: 10. 1007/s11760-023-02606-y

    2023

  26. [26]

    Zhang, Q

    Y. Zhang, Q. Li, S. Nahata, T. Jamal, S.-K. Cheng, G. Cauwenberghs, T.-P. Jung, Integrating large language model, eeg, and eye-tracking for word-level neural state classification in reading comprehension, IEEE Transactions on Neural Systems and Rehabilitation Engineering 32 (2024) 3465–3475. doi:10.1109/TNSRE.2024.3435460

    work page doi:10.1109/tnsre.2024.3435460 2024

This paper was first reviewed by grok-4.5 on July 11, 2026.