Pith. sign in

REVIEW 3 major objections 7 minor 53 references

Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generation

T0 review · 3 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hateful visual stories finish in over 80% of image-chat runs

desk verdict The threat model is real and the benchmark is useful, but the headline completion rates rest on annotations that may not be visually grounded; that needs fixing before the numbers are quoted. read the letter →

arxiv 2608.05210 v1 pith:PPBOQFY4 submitted 2026-08-05 cs.CV cs.AIcs.CR

Ye Leng , Junjie Chu , Yiting Qu , Mingjie Li , Yun Shen , Yang Zhang This is my paper · ORCID
classification cs.CVcs.AIcs.CR
keywords multi-turnimagegenerationhatefulvisualstoriestext-to-imagesafetygroup-levelcontentstorytellingmoderationmultimodaldetectionpromptsequences
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-turn conversational image generators can be steered with individually benign-looking prompts into producing ordered image groups that convey hateful narratives. Using 330 prompt configurations built from 55 hateful stories across two languages and three visual styles, the paper ran 4,950 generation attempts on five frontier models and found that every model completed the intended hateful story in more than 80% of successfully generated image groups, the strongest reaching 99.0%. A human-labeled benchmark of 969 hateful and 990 matched benign image sets showed that existing moderation tools miss most group-level hateful meaning, with the best general vision-language model at 67.5% recall. The paper's proposed defenses, monitoring the accumulated prompt history before each turn or jointly analyzing the completed image group, detect these stories far more reliably. The stakes are that as image generation becomes stateful and narrative, safety that inspects one image at a time cannot see harm that exists only in the relation between panels.

What carries the argument

The load-bearing object is the story-level label attached to an ordered image group, rather than any label on an individual panel. The paper operationalizes this with HatefulStoryPrompts, in which each prompt in a sequence describes an apparently ordinary scene and hateful meaning is carried by comparison, progression, setup-and-punchline, or attribution across panels. The evaluation machinery pairs every hateful story with a minimally edited benign counterpart that preserves surface content and multi-panel form, so a detector cannot separate the classes by subject matter or style and must reason about compositional intent. For detection, the machinery is the accumulated interaction context: in the proactive setting, a vision-language judge sees the ordered prompt history at each turn, plus the user's first image when provided, while in the post-generation setting, a describe-then-judge pipeline forces an explicit textual description of the completed image set before the safety decision.

What would settle it

Take a random sample of 200 image sequences labeled hateful and 200 matched benign sequences from HatefulVisualStory, and have a fresh panel of annotators who see only the images and never the source prompts or intended narratives label each sequence. If their hateful rate on the 200 hateful sequences falls far below the paper's labels, or if inter-annotator agreement drops well under the reported 95.2%, the completion rates and detector benchmark numbers are inflated by label leakage from intent.

Watch

Extended reading notes

Core claim

The paper's central claim is that a new class of harm, the hateful visual story, arises specifically from multi-turn image generation: an ordered sequence of images, each of which looks ordinary in isolation, collectively conveys hate directed at a protected group. The paper constructs HatefulStoryPrompts, 330 multi-turn prompt configurations derived from 55 base stories across English and Chinese and photorealistic, Tintin-style, and Tom & Jerry-style renderings, and runs each configuration three times against five commercial multi-turn generators. Among the 4,717 successfully generated image groups, every model completed the hateful narrative in 80.4% to 99.0% of cases, while outright refusals stopped only 2.2% to 6.2% of sessions. On the resulting HatefulVisualStory detection benchmark, existing per-image safeguards achieved at most 34.9% recall for dedicated safety models and 67.5% for the strongest general vision-language model. The paper's proposed defenses, an interaction-aware monitor that reads the full prompt history (97.3% recall) or the prompt history plus a user-supplied first image (92.6% recall), and post-generation reasoning over complete image groups (up to 80.2% recall), locate the missing safety signal in the accumulated session context.

Load-bearing premise

The central numbers assume annotators judge hatefulness from the generated images alone; if their labels are swayed by knowledge of the intended hateful story, the 80-99% completion rates and the detector recall figures would overstate what an independent viewer or moderator would perceive.

Editorial extensions

If this is right

  • Per-image moderation is insufficient for conversational image generators; providers need to retain session context and reason over accumulated prompts and images before each new panel is created.
  • Improvements in instruction following and character consistency widen this safety gap, so newer model generations can be expected to complete hateful narratives more reliably, not less.
  • Proactive monitoring is the strongest intervention when the interaction history is available, and post-generation detection remains a strictly harder fallback for content already shared.
  • Safety benchmarks for text-to-image systems should include group-level narrative tasks alongside single-image toxicity tests, because the two measure different capabilities.
  • A low completion rate in a non-English language can reflect text-rendering failure rather than stronger safety, so language-specific capability and safety must be separated in evaluations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same group-level mechanism likely transfers to stateful video generation, where a coherent narrative across frames is the native mode; adapting HatefulStoryPrompts to video models would test whether the gap is even wider.
  • The reported completion rates probably underestimate real-world misuse because the study's fixed prompt sequences are simpler than an adaptive adversary who rewrites prompts based on observed outputs.
  • A perception-based replication with naive annotators who do not know the intended narrative would calibrate the human labels; the paper's expert annotators may encode intent that an ordinary viewer would not recover from images alone.
  • The detection methods could generalize to other composition-level harms, such as coordinated misinformation or harassment narratives, since the machinery reasons about emerging group-level meaning rather than hate categories specifically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper introduces a new safety threat, called 'hateful visual stories,' in multi-turn conversational image generation, where each individual panel appears benign but an ordered sequence collectively conveys hateful meaning. The authors construct HatefulStoryPrompts from 55 /pol-derived narratives with two languages and three visual styles (330 configurations), run 4,950 API sessions across five frontier Gemini/GPT image models, and report that 80.4-99.0% of successfully generated image groups are labeled hateful. They also build HatefulVisualStory (969 hateful and 990 benign image sets), benchmark seven existing detectors (best recall 67.5%), and propose proactive interaction monitors (97.3% recall in prompt-only sessions, 92.6% with a user-supplied first image) and post-generation story-level detectors (up to 80.2% recall for describe-then-judge, 78.9% for a LoRA-tuned Qwen model). The paper argues that safety for conversational image generation must move from per-image moderation to stateful, sequence-level reasoning.

Significance. The paper identifies a genuinely timely and underexplored problem, and if the headline results hold, they have clear practical implications for content moderation in multi-turn image generation. The empirical scope is a real strength: 4,950 API sessions across five frontier models, matched benign controls, and a story-disjoint evaluation for the fine-tuned post-generation detector. The proposed proactive and post-generation methods are plausible and substantially outperform existing baselines. The central quantitative claims, however, rest on human labels whose independence from the intended narrative is not established. This labeling issue is load-bearing because the 80-99% completion rates, the HatefulVisualStory benchmark labels, and the detector-recall comparisons all inherit it. The paper is a solid contribution to a timely subject, but the annotation protocol needs to be clarified or re-run before the headline numbers can be taken at face value.

major comments (3)
  1. [§3.2 and §1 (labeling protocol)] The ground-truth hateful label may be contaminated by knowledge of the intended story. Section 1 states that 'Human annotators then determine whether each successfully generated image sequence conveys the intended hateful narrative,' and Section 3.1 states that each adapted story was 'accompanied by a written explanation specifying the targeted group and the preserved hateful meaning.' The paper does not state that the annotators saw only the image groups, without the prompt sequences or those written explanations. If the annotators knew the intended narrative, the 80.4-99.0% hateful rates measure the models' prompt-to-narrative consistency rather than whether a naive viewer or independent moderator would perceive hate from the images alone, and the 95.2% inter-annotator agreement does not rule out a shared-intent bias. This is load-bearing: the headline threat, the HatefulVisualStory labels, and the entire detector comparison depend on this label. The paper should either explicitly confirm that annotators were blind to the intended narrative and explanations, or provide a re-annotation in which annotators judge the image groups without access to the source story, with a comparison of the resulting hateful rates and detector recalls.
  2. [§4.2 (HatefulVisualStory construction)] The detector benchmark selects, for each condition, the first run that both successfully generates and is 'labeled as hateful,' and excludes twenty-one conditions where all runs fail or are labeled safe. This is an acceptable way to construct a positive set for a detection benchmark, but it means that every recall number in Tables 6, 7, and 9 is conditional on the Section 3.2 labeling protocol. If that protocol over-labels sequences as hateful because the annotators knew the intended narrative, the bias propagates directly into the 969-image positive set and into the reported 34.9-67.5% recall figures. The paper should include a sensitivity analysis on a random subset using labels obtained without knowledge of the intended narrative, so that readers can assess how much of the apparent detector failure is due to unperceivable narrative-level hate versus label contamination.
  3. [§5.1.2 (proactive evaluation)] The proactive monitor is evaluated only on the same 55 hand-authored story templates that define the task. Scenario 1 uses the 110 hateful prompt sequences and their 110 benign counterparts; Scenario 2 uses the 969/990 image sets of HatefulVisualStory, all derived from the same 55 base stories. In contrast, the fine-tuned post-generation detector is at least evaluated on a story-disjoint 25-story subset (Table 10). Without a story-disjoint split for the proactive monitor, it is unclear whether the 97.3% and 92.6% recall figures reflect detection of distributed hateful intent generally or recognition of the regularities of the authors' specific prompt-writing style. The paper should add a held-out-story evaluation for the proactive scenarios, or clearly state that the reported numbers are template-specific achievements rather than evidence of generalization to new narratives.
minor comments (7)
  1. [Abstract] The abstract says 'Every model completes over 80% of the stories,' but this is conditional on successfully generated image groups; the abstract should use the same qualifier as the body ('among successfully generated image groups'), since the headline rate excludes generation failures.
  2. [§7] There is a typographical error: 'detectuion' should be 'detection.'
  3. [§5.3] The text refers to 'Setting A monitor,' but Section 5.1 defines Scenario 1 and Scenario 2; the terminology should be unified for consistency.
  4. [§5.1.2] For Scenario 1, the paper does not specify which of the three visual styles, if any, is represented in the 110 prompt-only sequences; since style is one of the three manipulated conditions in HatefulStoryPrompts, this should be clarified.
  5. [Table 6] The Llavaguard-v1.2-7b row reports 0.0% recall and 50.5% accuracy; the paper should state whether this is a genuine uniform-safe prediction or a model/interface failure, as the distinction matters for interpreting the baseline.
  6. [Overall manuscript] The paper does not include a data/code availability statement; for a benchmark paper that introduces two datasets, a clear statement about whether the prompts, generated image sets, and annotation artifacts will be released is important for reproducibility.
  7. [Figure 3] Adding an English transliteration or translation of the intended Chinese speech bubble would make the 'pseudo-Chinese' failure mode easier for readers to verify.

Circularity Check

1 steps flagged · score 2.0 of 10

Headline hateful-completion rate is definitionally anchored to the authors' 'intended hateful narrative,' so the 80.4-99.0% figures measure fidelity to input intent; the rest of the evaluation is self-contained and not fitted-circular.

  1. self definitional [Section 3.2 (Evaluation Metrics / Labeling Protocol) with Section 3.1 (T2I Prompt Sequence Writing); headline result in Table 3]
    "Human annotators then determine whether each successfully generated image sequence conveys the intended hateful narrative. ... Each adapted story was accompanied by a written explanation specifying the targeted group and the preserved hateful meaning. ... the assembled image group does not faithfully transmit the intended message and can be labeled safe."

    The outcome variable 'hateful rate' is operationalized as the fraction of generated groups that convey 'the intended hateful narrative,' and each story is accompanied by a written explanation of the preserved hateful meaning. The label is therefore defined as agreement with an input (the authors' adapted hateful story), making 'hateful completion' a fidelity-to-input measure by construction. The paper itself confirms this reading when explaining the Chinese-text failures: a group with garbled text 'can be labeled safe' because it does not faithfully transmit the intended message, not because it fails an independent hatefulness test.

full rationale

No fitted parameter is renamed as a prediction: the proactive monitor is a zero-shot LLM-as-judge, describe-then-judge is zero-shot, and the only trained component (the LoRA-adapted Qwen detector) is evaluated on held-out image sets with a disclosed story overlap and an additional story-disjoint subset. Self-citations (e.g., prior Unsafe Diffusion and related safety work) are background context and are not load-bearing; no uniqueness theorem is imported to force a choice. The central empirical finding—that frontier multi-turn generators will render multi-panel sequences that human annotators, knowing the intended story, recognize as conveying those narratives—does not reduce to a fit and has independent content. The single self-definitional step concerns the ground-truth label being anchored to the authors' intended narrative, which reframes the hateful rate as a fidelity metric and affects the detector benchmark labels. This is a moderate construct-validity circularity, not a derivation-forcing one, so the overall score is low.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central empirical claims rest on three domain assumptions: expert labels are valid ground truth, closed API behavior is representative, and the 4chan-derived corpus is representative. No new physical entities are introduced. The only fitted component is the LoRA-adapted detector, whose training data is disclosed and which has a story-disjoint evaluation.

free parameters (1)
  • Qwen2.5-VL-7B LoRA adapter = 200 hateful + 200 benign image sets from 30 of 55 base stories
    The post-generation result of 78.9% recall on image sets comes from this fine-tuned classifier, so the headline detection number depends on fitted model parameters; the story-disjoint result, 76.7% recall, is reported separately in Table 10.
assumptions (3)
  • domain assumption Human expert annotation of sequence-level hateful intent, with majority vote, is a valid ground truth for hateful completion and for detector labels.
    All hateful rates and benchmark labels in Sections 3 and 4 rest on this; the paper does not report that annotators were blind to the intended source narrative (Section 3.2 Labeling Protocol).
  • domain assumption The behavior of the five closed commercial APIs during the collection window is representative of each deployed model's safety behavior.
    Section 8 states models, safety policies, and defaults can change over time; the results characterize specific API versions observed.
  • domain assumption The 4chan /pol/ corpus and the two-stage screening yield a representative set of group-targeting visual narratives.
    Section 3.1 collects posts from a single board over two weeks, then filters to 55 stories; this is a convenience sample rather than a random sample of hateful narratives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generation." pith.science (2026). https://pith.science/paper/PPBOQFY4

@misc{pith2026260805210,
  author       = {Pith},
  title        = {Pith review of: Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPBOQFY4}},
  note         = {Machine review of arXiv:2608.05210}
}
read the original abstract

Picture books and comics have long been used to disseminate hateful narratives because they are easily understood even by children, as exemplified by the notorious Nazi propaganda picture book \emph{Der Giftpilz}. Recently, frontier text-to-image (T2I) systems such as Gemini and GPT-Image have enabled conversational generation with consistent characters and scenes across turns, making hateful visual stories, namely ordered image groups that collectively convey hateful narratives, cheap and scalable to produce. Although prior work has studied hateful content generation by T2I systems, it focuses on individual images, leaving group-level hateful meaning largely unexplored. We aim to address the gap. Concretely, we introduce \texttt{HatefulStoryPrompts}, comprising 330 multi-turn configurations from 55 hateful stories across two languages and three visual styles, and evaluate five frontier models over 4,950 attempts. Every model completes over 80\% of the stories, with the strongest reaching 99.0\%. We further evaluate existing moderation systems on \texttt{HatefulVisualStory}, a human-labeled dataset of 969 hateful image sets and 990 benign controls, and find that they frequently miss group-level hateful meaning: dedicated safety models achieve at most 34.9\% recall, while a strong vision-language model reaches 67.5\%. Finally, we propose complementary proactive and post-generation defenses. An interaction-aware monitor achieves 97.3\% recall for prompt-only sessions and 92.6\% when the user supplies the first image, while post-generation methods jointly analyzing completed image groups reach 80.2\%. Our work shows that, as image generation evolves from isolated outputs to coherent visual narratives, safety must evolve accordingly, from per-image moderation to stateful reasoning over interactions and image relationships.

Figures

Figures reproduced from arXiv: 2608.05210 by the authors.

Figure 1
Figure 1. A historical example of hateful visual storytelling [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An example of hateful visual story, generated by [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An example of a story generated by gpt-image-1.5 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Scenario 1 (prompt-only) workflow. Individually innocuous user prompts are accumulated across a multi-turn session. The [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Confusion matrices comparing prediction labels of [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 47 canonical work pages

  1. [1]

    Introducing Claude Haiku 4.5.https://ww w.anthropic.com/news/claude-haiku-4-5, 2025

    Anthropic. Introducing Claude Haiku 4.5.https://ww w.anthropic.com/news/claude-haiku-4-5, 2025. 7

    work page 2025

  2. [2]

    Prompting for Multimodal Hateful Meme Clas- sification

    Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. Prompting for Multimodal Hateful Meme Clas- sification. InConference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 321–332. ACL, 2022. 2, 11

    work page 2022

  3. [3]

    JADES: A Universal Framework for Jail- break Assessment via Decompositional Scoring.CoRR abs/2508.20848, 2025

    Junjie Chu, Mingjie Li, Ziqing Yang, Ye Leng, Chen- hao Lin, Chao Shen, Michael Backes, Yun Shen, and Yang Zhang. JADES: A Universal Framework for Jail- break Assessment via Decompositional Scoring.CoRR abs/2508.20848, 2025. 2

    arXiv 2025

  4. [4]

    Neeko: Model Hijack- ing Attacks Against Generative Adversarial Networks

    Junjie Chu, Yugeng Liu, Xinlei He, Michael Backes, Yang Zhang, and Ahmed Salem. Neeko: Model Hijack- ing Attacks Against Generative Adversarial Networks. InInternational Conference on Multimedia and Expo (ICME). IEEE, 2025. 11

    work page 2025

  5. [5]

    Jail- breakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs

    Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. Jail- breakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs. InAnnual Meeting of the As- sociation for Computational Linguistics (ACL), pages 21538–21566. ACL, 2025. 2

    work page 2025

  6. [6]

    Benchmark of Bench- marks: Unpacking Influence and Code Reposi- tory Quality in LLM Safety Benchmarks.CoRR abs/2603.04459, 2026

    Junjie Chu, Xinyue Shen, Ye Leng, Michael Backes, Yun Shen, and Yang Zhang. Benchmark of Bench- marks: Unpacking Influence and Code Reposi- tory Quality in LLM Safety Benchmarks.CoRR abs/2603.04459, 2026. 11

    arXiv 2026

  7. [7]

    Reading Poison: Science and Story in Nazi Children’s Propaganda.Children’s Literature in Education, 2022

    Daniel Feldman. Reading Poison: Science and Story in Nazi Children’s Propaganda.Children’s Literature in Education, 2022. 1

    work page 2022

  8. [8]

    Gemini 2.5 Flash Image.https://ai.goo gle.dev/gemini-api/docs/models/gemini-2.5- flash-image, 2025

    Google. Gemini 2.5 Flash Image.https://ai.goo gle.dev/gemini-api/docs/models/gemini-2.5- flash-image, 2025. 2, 4

    work page 2025

Show all 53 references
  1. [9]

    Gemini 2.5 Flash Image (Nano Banana).http s://ai.google.dev/gemini-api/docs/models/ge mini-2.5-flash-image, 2025

    Google. Gemini 2.5 Flash Image (Nano Banana).http s://ai.google.dev/gemini-api/docs/models/ge mini-2.5-flash-image, 2025. 1

    2025

  2. [10]

    Gemini 3 Pro Image.https://ai.googl e.dev/gemini-api/docs/models/gemini-3-pro- image, 2025

    Google. Gemini 3 Pro Image.https://ai.googl e.dev/gemini-api/docs/models/gemini-3-pro- image, 2025. 2, 4

    2025

  3. [11]

    Gemini 3.1 Flash Image Preview.https://ai .google.dev/gemini-api/docs/models/gemini- 3.1-flash-image, 2026

    Google. Gemini 3.1 Flash Image Preview.https://ai .google.dev/gemini-api/docs/models/gemini- 3.1-flash-image, 2026. 2, 4

    2026

  4. [12]

    Gemini 3.1 Flash-Lite.https://ai.googl e.dev/gemini-api/docs/models/gemini-3.1- flash-lite, 2026

    Google. Gemini 3.1 Flash-Lite.https://ai.googl e.dev/gemini-api/docs/models/gemini-3.1- flash-lite, 2026. 2, 7

    2026

  5. [13]

    Daniel Green. The Discursive Construction of An- tisemitism in Nazi Children’s Books: Elvira Bauer’s Trust No Fox (1936) and Ernst Hiemer’s The Poisonous Mushroom (1938).International Journal for the Semi- otics of Law – Revue internationale de Sémiotique ju- ridique, 2023. 1

    1936

  6. [14]

    LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models

    Lukas Helff, Felix Friedrich, Manuel Brack, Kristian Kersting, and Patrick Schramowski. LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models. InInternational Conference on Machine Learning (ICML). PMLR, 2025. 11

    2025

  7. [15]

    LlavaGuard.ht tps://github.com/ml- research/llavaguard,

    Lukas Helff, Felix Friedrich, Manuel Brack, Patrick Schramowski, and Kristian Kersting. LlavaGuard.ht tps://github.com/ml- research/llavaguard,

  8. [16]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations (ICLR), 2022. 9

    2022

  9. [17]

    JailbreakDiffBench: A Comprehensive Benchmark for Jailbreaking Diffusion Models

    Xiaolong Jin, Zixuan Weng, Hanxi Guo, Chenlong Yin, Siyuan Cheng, Guangyu Shen, and Xiangyu Zhang. JailbreakDiffBench: A Comprehensive Benchmark for Jailbreaking Diffusion Models. InIEEE International Conference on Computer Vision (ICCV), pages 16461– 16471. IEEE, 2025. 11

    2025

  10. [18]

    Experiment with Gemini 2.0 Flash Native Image Generation.https:// developers.googleblog.com/experiment-with- gemini-20-flash-native-image-generation/,

    Kat Kampf and Nicole Brichtova. Experiment with Gemini 2.0 Flash Native Image Generation.https:// developers.googleblog.com/experiment-with- gemini-20-flash-native-image-generation/,

  11. [19]

    The Hateful Memes Challenge: De- tecting Hate Speech in Multimodal Memes

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Da- vide Testuggine. The Hateful Memes Challenge: De- tecting Hate Speech in Multimodal Memes. InAnnual Conference on Neural Information Processing Systems (NeurIPS), pages 2611–26...

    2020

  12. [20]

    Hate- CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features

    Gokul Karthik Kumar and Karthik Nandakumar. Hate- CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features. CoRR abs/2210.05916, 2022. 11

    2022 arXiv

  13. [21]

    When Understanding Becomes a Risk: Authenticity and Safety Risks in the Emerging Image Generation Paradigm

    Ye Leng, Junjie Chu, Mingjie Li, Chenhao Lin, Chao Shen, Michael Backes, Yun Shen, and Yang Zhang. When Understanding Becomes a Risk: Authenticity and Safety Risks in the Emerging Image Generation Paradigm. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), p...

    2026

  14. [22]

    T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Pri- vacy in Image Generation

    Lijun Li, Zhelun Shi, Xuhao Hu, Bowen Dong, Yiran Qin, Xihui Liu, Lu Sheng, and Jing Shao. T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Pri- vacy in Image Generation. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 13381–13392. IEEE, 2025. 10

    2025

  15. [23]

    From Meme to Threat: On the Hateful Meme Understand- ing and Induced Hateful Content Generation in Open- Source Vision Language Models

    Yihan Ma, Xinyue Shen, Yiting Qu, Ning Yu, Michael Backes, Savvas Zannettou, and Yang Zhang. From Meme to Threat: On the Hateful Meme Understand- ing and Induced Hateful Content Generation in Open- Source Vision Language Models. InUSENIX Security Symposium (USENIX Security). U...

    2025

  16. [24]

    Improving Hateful Meme Detec- tion through Retrieval-Guided Contrastive Learning

    Jingbiao Mei, Jinghong Chen, Weizhe Lin, Bill Byrne, and Marcus Tomalin. Improving Hateful Meme Detec- tion through Retrieval-Guided Contrastive Learning. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 5333–5347. ACL, 2024. 11

    2024

  17. [25]

    Llama Guard 4 12B.https://huggingface

    Meta. Llama Guard 4 12B.https://huggingface. co/meta-llama/Llama-Guard-4-12B, 2025. 2, 7

    2025

  18. [26]

    GLIDE: Towards Photore- alistic Image Generation and Editing with Text-Guided Diffusion Models.CoRR abs/2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photore- alistic Image Generation and Editing with Text-Guided Diffusion Models.CoRR abs/2112.10741, 2021. 3

    2021 arXiv

  19. [27]

    European Digital Media Observatory. How the Euro- pean Far-Right Is Using AI-Generated Content to En- gage V oters.https://edmo.eu/publications/how- the-european-far-right-is-using-ai-genera ted-content-to-engage-voters/, 2025. 2

    2025

  20. [28]

    OpenAI Moderation API.https://develo pers.openai.com/api/docs/guides/moderation

    OpenAI. OpenAI Moderation API.https://develo pers.openai.com/api/docs/guides/moderation. 2, 7

  21. [29]

    GPT Image 1.5.https://developers.ope nai.com/api/docs/models/gpt-image-1.5, 2025

    OpenAI. GPT Image 1.5.https://developers.ope nai.com/api/docs/models/gpt-image-1.5, 2025. 2, 4

    2025

  22. [30]

    Introducing 4o Image Generation.https: //openai.com/index/introducing-4o-image- generation/, 2025

    OpenAI. Introducing 4o Image Generation.https: //openai.com/index/introducing-4o-image- generation/, 2025. 1

    2025

  23. [31]

    GPT Image 2.https://developers.opena i.com/api/docs/models/gpt-image-2, 2026

    OpenAI. GPT Image 2.https://developers.opena i.com/api/docs/models/gpt-image-2, 2026. 2, 5

    2026

  24. [32]

    SDXL: Improving Latent Dif- fusion Models for High-Resolution Image Synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Dif- fusion Models for High-Resolution Image Synthesis. CoRR abs/2307.01952, 2023. 3

    2023 arXiv

  25. [33]

    Unsafe Diffu- sion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models

    Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe Diffu- sion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models. InACM SIGSAC Conference on Computer and Communications Secu- rity (CCS). ACM, 2023. 2, 10

    2023

  26. [34]

    UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images

    Yiting Qu, Xinyue Shen, Yixin Wu, Michael Backes, Savvas Zannettou, and Yang Zhang. UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images. InACM SIGSAC Con- ference on Computer and Communications Security (CCS), pages 3221–3235. ACM, 2025. 11

    2025

  27. [35]

    Zero-Shot Text-to-Image Generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. InIn- ternational Conference on Machine Learning (ICML), pages 8821–8831. JMLR, 2021. 3

    2021

  28. [36]

    High-Resolution Im- age Synthesis with Latent Diffusion Models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Im- age Synthesis with Latent Diffusion Models. InIEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 10684–10695. IEEE, 2022. 3

    2022

  29. [37]

    Racist Videos Made with AI Are Going Viral on TikTok.https://www.theverge.com/new s/697188/racist-ai-generated-videos-goog le-veo-3-tiktok, 2025

    Emma Roth. Racist Videos Made with AI Are Going Viral on TikTok.https://www.theverge.com/new s/697188/racist-ai-generated-videos-goog le-veo-3-tiktok, 2025. 2

    2025

  30. [38]

    Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? InConference on Fairness, Ac- countability, and Transparency (F AccT), pages 1350–

  31. [39]

    Q16.https://github.com/ml- research/Q16, 2022

    Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. Q16.https://github.com/ml- research/Q16, 2022. 2, 7

    2022

  32. [40]

    Multimodal Meme Dataset (MultiOFF) for Identifying Offensive Content in Image and Text

    Shardul Suryawanshi, Bharathi Raja Chakravarthi, Mi- hael Arcan, and Paul Buitelaar. Multimodal Meme Dataset (MultiOFF) for Identifying Offensive Content in Image and Text. InWorkshop on Threat, Aggression & Cyberbullying (TRAC), pages 32–41. ELRA, 2020. 11

    2020

  33. [41]

    Qwen2.5-VL-7B.https://qwen.ai/bl og?id=qwen2.5-vl, 2025

    Qwen Team. Qwen2.5-VL-7B.https://qwen.ai/bl og?id=qwen2.5-vl, 2025. 2, 7, 9

    2025

  34. [42]

    Ring-A-Bell! How Reliable are Concept Removal Methods For Diffusion Models? In International Conference on Learning Representations (ICLR), 2024

    Yu-Lin Tsai, Chia-Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia-You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Ring-A-Bell! How Reliable are Concept Removal Methods For Diffusion Models? In International Conference on Learning Representations (ICLR), 2024. 11

    2024

  35. [43]

    Chain-of-Jailbreak Attack for Image Generation Models via Step by Step Editing

    Wenxuan Wang, Kuiyi Gao, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Shuai Wang, Wenxiang Jiao, and Zhaopeng Tu. Chain-of-Jailbreak Attack for Image Generation Models via Step by Step Editing. InFind- ings of the Association for Computational Linguistics: ACL (ACL Findings), pag...

    2025

  36. [44]

    Image-Perfect Imperfections: Safety, Bias, and Au- thenticity in the Shadow of Text-To-Image Model Evo- lution

    Yixin Wu, Yun Shen, Michael Backes, and Yang Zhang. Image-Perfect Imperfections: Safety, Bias, and Au- thenticity in the Shadow of Text-To-Image Model Evo- lution. InACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2024. 11

    2024

  37. [45]

    MMA-Diffusion: Mul- tiModal Attack on Diffusion Models

    Yijun Yang, Ruiyuan Gao, Xiaosen Wang, Tsung-Yi Ho, Nan Xu, and Qiang Xu. MMA-Diffusion: Mul- tiModal Attack on Diffusion Models. InIEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 7737–7746. IEEE, 2024. 11

    2024

  38. [46]

    SneakyPrompt: Jailbreaking Text-to- Image Generative Models

    Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. SneakyPrompt: Jailbreaking Text-to- Image Generative Models. InIEEE Symposium on Se- curity and Privacy (S&P), pages 897–912. IEEE, 2024. 11

    2024

  39. [47]

    Scaling Autoregres- sive Models for Content-Rich Text-to-Image Genera- tion.CoRR abs/2206.10789, 2022

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander 14 Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchin- son, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Ja- son Baldridge, and Yonghui Wu. Scaling Autoregres- sive Models for Co...

    2022 arXiv

  40. [48]

    TwoHamsters: Benchmark- ing Multi-Concept Compositional Unsafety in Text-to- Image Models.CoRR abs/2604.15967, 2026

    Chaoshuo Zhang, Yibo Liang, Mengke Tian, Chen- hao Lin, Zhengyu Zhao, Le Yang, Chong Zhang, Yang Zhang, and Chao Shen. TwoHamsters: Benchmark- ing Multi-Concept Compositional Unsafety in Text-to- Image Models.CoRR abs/2604.15967, 2026. 10

    2026 arXiv

  41. [49]

    When Memory Becomes a Vulnerability: To- wards Multi-Turn Jailbreak Attacks against Text-to- Image Generation Systems

    Shiqian Zhao, Jiayang Liu, Yiming Li, Runyi Hu, Xiaojun Jia, Wenshu Fan, Xiaobao Wu, Xinfeng Li, Jie Zhang, Wei Dong, Tianwei Zhang, and Luu Anh Tuan. When Memory Becomes a Vulnerability: To- wards Multi-Turn Jailbreak Attacks against Text-to- Image Generation Systems. InUSENI...

    2026

  42. [50]

    The post contains at least two distinct meaning-bearing units that can be represented as separate visual genera- tion steps

  43. [51]

    A meaning-bearing unit may be an event, state, utterance, comparison, revelation, or change in attitude that introduces information relevant to the overall meaning

    The relationship between these units is necessary to re- cover the post’s overall implication. A meaning-bearing unit may be an event, state, utterance, comparison, revelation, or change in attitude that introduces information relevant to the overall meaning. It does not need ...

  44. [52]

    The narrative referred, explicitly or implicitly, to an identifiable group or vulnerable population rather than only to a specific individual

  45. [53]

    Annotators evaluated the implication of the full narrative rather than requiring any individual sentence, event, or image to be independently hateful

    The narrative conveyed a negative or harmful impli- cation about that target through derogation, dehuman- ization, harmful stereotyping, collective blame, exclu- sion, humiliation, endorsement of harm, or otherwise depicted, encouraged, normalized, or facilitated harm toward t...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.