Pith. sign in

REVIEW 2 major objections 6 minor 78 references

Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that all fifteen tested content-moderation models—six classifiers and nine vision-language models—fail to detect AI-generated 'hateful illusions,' with best accuracy below 0.245 for classifiers and below 0.102 for VLMs.

desk verdict A novel dataset and a stark set of empirical failure rates for detecting AI-generated hateful illusions, but the author-annotated ground truth needs an independent blinded check before those numbers can carry the paper's full weight. read the letter →

arxiv 2507.22617 v1 pith:OY4BAVLR submitted 2025-07-30 cs.CR cs.CV

Yiting Qu , Ziqing Yang , Yihan Ma , Michael Backes , Savvas Zannettou , Yang Zhang This is my paper
classification cs.CRcs.CV
keywords hatefulillusionscontentmoderationtext-to-imagediffusionopticalvision-languagemodelshatespeechdetectionCLIPadversarialimagegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that adversaries can use text-to-image diffusion models to mass-produce 'hateful illusions': images that look like harmless scenes but embed hate speech or hate symbols readable when zoomed out or blurred. It reports generating 1,860 such images with Stable Diffusion and ControlNet, of which 1,571 (84.5%) successfully hid hate messages, and it shows that six content-moderation classifiers and nine vision-language models detect these images at very low rates—best accuracy 0.209–0.245 for classifiers and 0.090–0.102 for VLMs. The paper attributes the failure to vision encoders, primarily CLIP-ViT, which attend to surface details rather than the emergent hidden pattern, and it demonstrates that simple image preprocessing (Gaussian blur plus contrast equalization) sharply improves detection, with fine-tuned prompt learning reaching 0.938 accuracy on the test set. If correct, this identifies a scalable method for bypassing current image moderation and a concrete direction for closing the gap.

What carries the argument

The central object is the 'hateful illusion': an image $i = F(x,c)$ generated by Stable Diffusion conditioned with ControlNet on a message image $c$ (a hate symbol or hate speech) and a descriptive prompt $x$ (a harmless scene). The load-bearing explanation is the behavior of CLIP-ViT, the vision encoder shared by most tested open-source moderation models: cosine-similarity and t-SNE analyses show that illusion embeddings track the surface prompt rather than the embedded message, and multimodal attention relevancy maps show that the model attends to scene objects while ignoring the hidden pattern. The mitigation machinery consists of image transformations—Gaussian blur followed by histogram equalization—that remove fine detail so the emergent message dominates, and prompt learning on a fully fine-tuned CLIP backbone as a training-level alternative.

What would settle it

Re-run the two-round human annotation on the 1,860 generated images with an independent set of annotators and measure agreement on the 1,571 flagged images; if agreement falls well below the reported Fleiss' kappa of 0.783, the ground truth and all derived accuracy figures would lose support. Independently, feed the same hateful illusions to GPT-4o after applying the paper's Gaussian blur plus histogram equalization preprocessing; the paper's explanation predicts accuracy should jump from 0.03 toward 0.70, and a failure to reproduce that jump would weaken the encoder-attention account.

Watch

Extended reading notes

Core claim

The central claim is that an adversary with ordinary access to Stable Diffusion and ControlNet can condition image generation on hate-message images so that slurs and symbols appear as the emergent pattern in otherwise benign scenes, and that this defeats every tested moderation model. The paper builds the Hateful Illusion dataset through a two-round human annotation protocol, finding that 84.5% of 1,860 generated images successfully embed hate messages, while Stable Diffusion's built-in safety checker blocks only 3.0% of outputs. Across six moderation classifiers and nine vision-language models, detection accuracy never exceeds 0.245 for classifiers or 0.102 for VLMs, even for high-visibility illusions that human annotators recognize immediately. The paper's diagnosis is that the shared CLIP-ViT encoder represents images by surface-level, prompt-aligned semantics, so the hidden message is absent from the representation; supporting evidence includes t-SNE clustering, cosine-similarity comparisons, and attention relevancy maps. The mitigation results—blur-plus-histogram-equalization preprocessing lifting Q16 from 0.21 to 0.82 and GPT-4o from 0.03 to 0.70, and fully fine-tuned prompt learning reaching 0.938 accuracy—support the same diagnosis by showing that suppressing fine detail makes the hidden message detectable.

Load-bearing premise

The entire dataset and every accuracy number rest on the assumption that three expert annotators, using majority voting with Fleiss' kappa 0.783, correctly decide whether each image really hides a hate message and how visible that message is; if annotators systematically miss certain embedded messages, the reported generation success rate and all detection failures would shift.

Editorial extensions

If this is right

  • Platforms relying on current commercial moderation APIs will miss hateful illusions posted at scale; several tested APIs detect near-zero percent of the dataset.
  • Stable Diffusion's built-in safety checker is not a sufficient control point, since it blocks only 3.0% of generated hateful illusions.
  • Detection can be substantially improved without new model architecture: blur-then-histogram-equalization preprocessing lifted Q16 from 0.21 to 0.82 and GPT-4o from 0.03 to 0.70 in the paper's experiments.
  • Training-level mitigation is available: prompt learning on a fully fine-tuned CLIP backbone reaches 0.938 accuracy on the Hateful Illusion test set.
  • Chain-of-thought prompting gives only marginal gains for VLMs, so prompt engineering alone is not a reliable fix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the 84.5% generation success rate transfers to other diffusion backbones, languages, and conditioning inputs, the attack is a general template extending beyond the specific 62 hate messages tested here.
  • Editorial extension: the paper's encoder-attention account predicts that models whose vision encoders are trained on global or emergent structure, or that explicitly process downscaled copies, should detect illusions better; this can be tested directly on the released dataset.
  • Editorial extension: the blur-and-equalization result suggests a cheap defensive preprocessing layer for existing moderation pipelines, but adversaries could adapt by making hidden patterns robust to blur, making the mitigation an arms race rather than a permanent fix.
  • Editorial extension: the low false-positive rates on safe digit illusions indicate the failure is one of recall, not calibration, so recall-oriented evaluation in production moderation would expose this gap earlier.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces the task of detecting AI-generated hateful illusions: optical illusions produced with Stable Diffusion and ControlNet that embed hate speech or hate symbols into seemingly harmless scenes. The authors generate 1,860 images conditioned on 62 hate messages, use a two-round human annotation by three researchers to retain 1,571 images as the Hateful Illusion dataset, and report that 84.5% of generated images successfully embed the target message. They then evaluate six moderation classifiers and nine vision-language models on this dataset, reporting very low detection rates (best classifier accuracy 0.209-0.245; best VLM accuracy 0.090-0.102). They attribute the failures to the vision encoders of the models, specifically to CLIP-ViT's focus on surface-level details, and propose mitigation strategies based on image transformations and fine-tuned CLIP classifiers.

Significance. If the empirical results hold, this is a timely and practically important study. It documents a scalable pipeline for generating hateful illusions, provides a first benchmark dataset for this phenomenon, and gives concrete evidence that current commercial and open-source moderation systems fail to flag such images. The release of code and the dataset is a valuable community resource, and the falsifiable prediction that detection rates are below 0.25 for classifiers and below 0.11 for VLMs can be checked by other groups. The value of the paper depends heavily on the validity of the ground-truth labels, because the generation-success rate and every detection accuracy are computed against those labels.

major comments (2)
  1. [Sections 3.2-3.3, Table 1] The 84.5% generation-success rate and all accuracy values in Tables 2 and 4 rest on labels produced by three expert annotators from the research team who know the 62-message set and, for each image, which message it was conditioned on. The low-visibility category is decided after blur/zoom augmentation, a viewing condition that the tested models are never given. This protocol can lead to over-inclusive labels: an ambiguous scene may be counted as a successful hateful illusion even though an unprimed viewer would not perceive the message. The reported Fleiss' kappa (0.783) measures agreement among similarly primed annotators, not validity against an external standard. Please add an independent annotation study with annotators who are not told the conditioning message (preferably non-authors), report per-message and per-visibility agreement, and include a sensitivity analysis that treats low-visibility labels separately or excludes them. The main qualitative finding may survive such an analysis, but the stated success rate and dataset size would need to be restated accordingly.
  2. [Section 4.4 and Abstract] The claim that the failures are 'inherently attributed to their vision encoders' is broader than the evidence. The semantic and attention analyses (Figures 6-8) are performed only on CLIP-L/14, and although several open-source models use CLIP-ViT, Table 4 also includes GPT-4V, GPT-4o, Gemini-1.5, and Gemini-2, whose vision encoders are not demonstrated to be CLIP-ViT (and are undisclosed for commercial APIs). A single attention-map example is not sufficient to establish a causal mechanism for safety decisions. Please either restrict the attribution claim to the open-source CLIP-based models, or provide controlled evidence, such as fixing the language model and varying only the vision encoder, or a systematic relevance-map analysis over a random sample of images, before generalizing to all nine VLMs.
minor comments (6)
  1. [Section 4.2, Table 4] The sentence reporting the highest zero-shot accuracies as '0.034 for hate speech (achieved by Gemini-2) and 0.06 for hate symbols (achieved by LLaVA-1.5)' does not match Table 4, where Gemini-2's aggregated hate-speech value is 0.030 and LLaVA-1.5's aggregated hate-symbol value is 0.046 (with a high-visibility value of 0.055). Please reconcile the text and table.
  2. [Section 4.1] The metric called 'accuracy' is actually the percentage of positive images that are flagged, i.e., sensitivity or recall, because the evaluation set contains only positive examples. Renaming it 'detection rate' or 'recall' would prevent confusion with conventional accuracy over positive and negative classes.
  3. [Section 5, Table 6] The fine-tuning comparison is based on a single 8:2 split of a roughly 3K-image set in which many images share the same 62 messages and 30 prompts. Please report multiple seeds and confidence intervals clustered by message or prompt, since the reported 0.938 accuracy may otherwise reflect overfitting to the generation distribution.
  4. [Section 3.2] The paper alternates between saying there are 62 hate messages and describing 23 text messages plus 39 message images representing 18 symbol classes. Please state this distinction explicitly, as datasets often report unique concepts rather than rendered images.
  5. [Figures 6 and 7] The t-SNE visualization and the cosine-similarity analysis lack details on hyperparameters (perplexity, number of iterations) and on whether embeddings were extracted from full images or patch tokens; adding these details would improve reproducibility.
  6. [Table 4 heading] The header uses 'Classifiers' for the VLM rows; 'Models' would be clearer, since the table reports vision-language models rather than classifiers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is an empirical measurement paper whose central claims are evaluated on an independently annotated dataset and held-out splits, with no derivation that reduces to its own inputs.

full rationale

The paper's central claims are empirical measurements rather than derived predictions. The generation pipeline (Stable Diffusion + ControlNet, Section 3.1) produces images from fixed prompts, message images, and hand-set guidance scales; the reported 84.5% success rate is a label-aggregation statistic over three human annotators, not a quantity fitted to those labels. The moderation evaluations in Tables 2, 4, and 5 are inference-time accuracy scores of publicly available classifiers and VLMs on that fixed dataset; no parameter of the evaluated models is fitted to the Hateful Illusion dataset before scoring. The training-level mitigation in Section 5 is evaluated on an 8:2 train/test split, so the reported 0.938 accuracy is a held-out generalization result, not a re-description of training data. The CLIP embedding and attention analyses in Section 4.4 are post-hoc explanatory visualizations, not used as the proof that specific moderation models fail. Self-citations appear in related work and background (e.g., references [49], [50], [51], [52], [66]) but none is load-bearing for the central negative result; the failure of the tested models is established directly by the reported accuracies. The annotation process is a potential validity concern because the three expert annotators are research-team members who know the message set, but this is a ground-truth reliability issue, not a circular derivation: the models' scores do not feed back into the labels, and no claimed result is definitionally equivalent to an input. Under the stated standard requiring quotable reduction of a claim to its own inputs, no circular step is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper is empirical; the main free parameters are the ControlNet guidance scales, which materially affect the reported 84.5% success rate. The central results rest on domain assumptions about annotation validity, representativeness of the hate message and prompt sets, and the generalizability of the CLIP-ViT analysis to all nine VLMs.

free parameters (2)
  • guidance_scale_hate_speech = 0.9
    Chosen by the authors in Section 3.2; Figure 4 shows the percentage of hateful illusions varies with this scale, so the 87.5% success rate for speech depends on it.
  • guidance_scale_hate_symbols = 1.1
    Chosen by the authors in Section 3.2 for symbol embeddings; similarly affects the 82.6% success rate.
assumptions (5)
  • domain assumption Human annotation with three expert annotators and majority vote provides a valid ground truth for whether a generated image embeds a hate message and for its visibility class.
    Used in Section 3.2 to label 1,571 hateful illusions; if annotation is unreliable, the dataset and all detection accuracies shift.
  • domain assumption The 62 hate messages and 30 descriptive prompts are representative of real-world adversarial targets and surface scenes.
    The central estimates of generation success and detection failure depend on this sample; a different message or scene distribution could change the numbers.
  • domain assumption CLIP-ViT analysis (t-SNE, cosine similarity, attention relevancy) is representative of the vision encoders in all nine tested VLMs.
    Section 4.4 generalizes from open-source models with CLIP-ViT to commercial VLMs with undisclosed encoders; if proprietary encoders behave differently, the explanation may be incomplete.
  • domain assumption The tested six classifiers and nine VLMs are representative of deployed content moderation systems.
    The paper selects popular commercial and open-source models, but there are many other moderators and VLM configurations; results may not generalize to all deployed systems.
  • domain assumption Accuracy measured as recall on positive examples is a meaningful moderation metric.
    Section 4.1 defines accuracy as the fraction of hateful illusions flagged; this ignores false-positive trade-offs, though the appendix reports low FPR on safe illusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions." pith.science (2026). https://pith.science/paper/OY4BAVLR

@misc{pith2026250722617,
  author       = {Pith},
  title        = {Pith review of: Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OY4BAVLR}},
  note         = {Machine review of arXiv:2507.22617}
}
read the original abstract

Recent advances in text-to-image diffusion models have enabled the creation of a new form of digital art: optical illusions--visual tricks that create different perceptions of reality. However, adversaries may misuse such techniques to generate hateful illusions, which embed specific hate messages into harmless scenes and disseminate them across web communities. In this work, we take the first step toward investigating the risks of scalable hateful illusion generation and the potential for bypassing current content moderation models. Specifically, we generate 1,860 optical illusions using Stable Diffusion and ControlNet, conditioned on 62 hate messages. Of these, 1,571 are hateful illusions that successfully embed hate messages, either overtly or subtly, forming the Hateful Illusion dataset. Using this dataset, we evaluate the performance of six moderation classifiers and nine vision language models (VLMs) in identifying hateful illusions. Experimental results reveal significant vulnerabilities in existing moderation models: the detection accuracy falls below 0.245 for moderation classifiers and below 0.102 for VLMs. We further identify a critical limitation in their vision encoders, which mainly focus on surface-level image details while overlooking the secondary layer of information, i.e., hidden messages. To address this risk, we explore preliminary mitigation measures and identify the most effective approaches from the perspectives of image transformations and training-level strategies.

Figures

Figures reproduced from arXiv: 2507.22617 by the authors.

Figure 1
Figure 1. A well-known example [26] of an AI-generated opti￾cal illusion: the image hides the word “OBEY,” which becomes more visible when viewed from a distance or when zoomed out. glance, this image appears to depict a group of well-dressed people walking through a busy city street at night. However, when viewed from a distance or zoomed out, a hidden mes￾sage, the word “OBEY,” becomes apparent, which is formed by the arran… view at source ↗
Figure 2
Figure 2. High-level overview of the attack. (ViT) [39] architecture and a similar training paradigm, e.g., CLIP-ViT [53]. Through the analysis of CLIP semantics and attention mechanics, we validate that CLIP-ViT generally fo￾cuses on the fine details in the harmless scenes while over￾looking the pattern created by embedded messages. Finally, to mitigate the risks of hateful illusions, we ex￾plore the effects of various image… view at source ↗
Figure 3
Figure 3. Examples of AI-generated hateful illusions in differ [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Trade-off between the percentage of successfully gen [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Examples of VLM-generated responses that fail to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: t-SNE Visualization of CLIP embeddings for hate [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Cosine similarity between image and text embeddings [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Relevancy map [34] between image patches and texts, derived from CLIP attention modules. The highlighted areas indicate which image patches the model attends to most for the provided texts. More examples are shown in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Examples of illusive images after various transformations. “Histogram” represents histogram equalization, “Gamma” [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Effect of different image transformations on moderation classifiers and VLMs. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 12
Figure 12. Figure 12: Query prompt for LLM-as-a-Judge. while it increases to 0.684 for hate symbols. This suggests that, for the tested models, images depicting hate symbols are more likely to be identified than those depicting hate speech, when no illusions are created. However, when thes…
Figure 13
Figure 13. Figure 13: Examples of VLM-generated responses that fail to identify the digits from optical illusions. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Relevancy map [34] between image patches and texts, derived from CLIP attention modules. The highlighted areas indicate which image patches the model attends to most for the provided texts. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 65 canonical work pages

  1. [1]

    https://medium

    AI-Generated Optical Illusions: Amidst Art, Enter- tainment and Disinformation. https://medium. com/@globalstridenews/ai-generated-optical- illusions-amidst-art-entertainment-and- disinformation-940fdf1ecd5a. 1

    work page

  2. [2]

    https://gnet- research.org/2023/11/13/for-the-lulz-ai- generated-subliminal-hate-is-a-new-challenge- in-the-fight-against-online-harm/

    AI-generated Subliminal Hate is a New Challenge in the Fight Against Online Harm. https://gnet- research.org/2023/11/13/for-the-lulz-ai- generated-subliminal-hate-is-a-new-challenge- in-the-fight-against-online-harm/ . 1

    work page 2023

  3. [3]

    https://www.adl.org/resources/ hate-symbol/confederate-flag

    Confederated Flag. https://www.adl.org/resources/ hate-symbol/confederate-flag. 3

    work page

  4. [4]

    https://openai.com/index/dall-e-2/

    DALLE 2. https://openai.com/index/dall-e-2/. 1, 13

    work page

  5. [5]

    https://ai.google.dev/gemini-api/docs

    Gemini. https://ai.google.dev/gemini-api/docs. 6

    work page

  6. [6]

    https://cloud.google.com/vertex- ai/generative-ai/docs/gemini-v2

    Gemini-2. https://cloud.google.com/vertex- ai/generative-ai/docs/gemini-v2. 6

    work page

  7. [7]

    https://cloud.google.com/ vision/docs/detecting-safe-search

    Google’s SafeSearch API. https://cloud.google.com/ vision/docs/detecting-safe-search. 2, 5, 10

    work page

  8. [8]

    https://openai.com/index/hello-gpt-4o

    GPT-4o. https://openai.com/index/hello-gpt-4o. 3, 6, 13

    work page

Show all 78 references
  1. [9]

    https://openai.com/research/gpt-4v- system-card

    GPT-4V. https://openai.com/research/gpt-4v- system-card. 2, 6

  2. [10]

    https://www.adl.org/ resources/hate-symbol/happy-merchant

    Happy Merchant Meme. https://www.adl.org/ resources/hate-symbol/happy-merchant. 1, 3

  3. [11]

    https://docs.opencv.org/3.4/ d4/d1b/tutorial_histogram_equalization.html

    Histogram Equalization. https://docs.opencv.org/3.4/ d4/d1b/tutorial_histogram_equalization.html. 2

  4. [12]

    https://www.adl.org/resources/ hate-symbol/its-okay-be-white

    It’s Ok To Be White. https://www.adl.org/resources/ hate-symbol/its-okay-be-white . 1

  5. [13]

    https://labelstud.io/

    Label Studio. https://labelstud.io/. 4

  6. [14]

    https://huggingface.co/datasets/laion/ laion2B-en

    LAION-2B. https://huggingface.co/datasets/laion/ laion2B-en. 5

  7. [15]

    https://laion.ai/blog/laion-5b/

    LAION-5B. https://laion.ai/blog/laion-5b/. 5

  8. [16]

    https://lexica.art/

    Lexica Dataset. https://lexica.art/. 3

  9. [17]

    https://learn

    Microsoft’s Image Moderation API. https://learn. microsoft.com/en-us/azure/ai-services/content- moderator/image-moderation-api. 2, 5, 10

  10. [18]

    https://learn

    Microsoft’s Multimodal Moderation API. https://learn. microsoft.com/en-us/azure/ai-services/content- safety/quickstart-multimodal. 5, 10

  11. [19]

    https://midjourney.com/

    Midjourney. https://midjourney.com/. 1, 13

  12. [20]

    https://en.wikipedia.org/wiki/Misogyny

    Misogyny. https://en.wikipedia.org/wiki/Misogyny. 2, 10

  13. [21]

    https://www.adl.org/resources/ hate-symbol/swastika

    Neo-Nazi Symbol. https://www.adl.org/resources/ hate-symbol/swastika. 1

  14. [22]

    https://platform.openai.com/docs/ guides/moderation

    OpenAI’s Omni. https://platform.openai.com/docs/ guides/moderation. 5, 10

  15. [23]

    https://en.wikipedia.org/wiki/ Optical_illusion

    Optical Illusions. https://en.wikipedia.org/wiki/ Optical_illusion. 9 10

  16. [24]

    https://huggingface

    Safety Filter in Stable Diffusion. https://huggingface. co/CompVis/stable-diffusion-safety-checker . 5, 7

  17. [25]

    https://stability.ai/news/stable- diffusion-public-release

    Stable Diffusion. https://stability.ai/news/stable- diffusion-public-release. 1, 2, 3, 13

  18. [26]

    https://x.com/DiffusionPics/status/ 1703878083919347824/photo/1

    Stable Diffusion Publishes the OBEY Image on X. https://x.com/DiffusionPics/status/ 1703878083919347824/photo/1. 1

  19. [27]

    https://www.adl.org/resources/hate- symbol/swastika

    Swastika. https://www.adl.org/resources/hate- symbol/swastika. 3

  20. [28]

    Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, José Hernández-Orallo, Lewis Hammond, Eric J

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, José Hernández-Orallo, Lewis Hammond, Eric J. Bigelow, Alex...

    2024 arXiv

  21. [29]

    SpaText: Spatio-Textual Representation for Con- trollable Image Generation

    Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. SpaText: Spatio-Textual Representation for Con- trollable Image Generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 18...

    2023

  22. [30]

    Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities. CoRR abs/2308.12966, 2023. 2

    2023 arXiv

  23. [31]

    Image Safeguarding: Reasoning with Conditional Vision Language Model and Obfuscating Unsafe Content Counterfactually

    Mazal Bethany, Brandon Wherry, Nishant Vishwamitra, and Peyman Najafirad. Image Safeguarding: Reasoning with Conditional Vision Language Model and Obfuscating Unsafe Content Counterfactually. In AAAI Conference on Artificial Intelligence (AAAI), pages 774–782. AAAI, 2024. 10

    2024

  24. [32]

    Rehg, and Chao Zheng

    Xu Cao, Tong Zhou, Yunsheng Ma, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang, James M. Rehg, and Chao Zheng. MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traf- fic Scene Understanding. In IEEE Conference on Computer Vision and ...

    2024

  25. [33]

    Subverting the Jew- tocracy

    Mohit Chandra, Dheeraj Reddy Pailla, Himanshu Bhatia, AadilMehdi J. Sanchawala, Manish Gupta, Manish Shrivas- tava, and Ponnurangam Kumaraguru. “Subverting the Jew- tocracy”: Online Antisemitism Detection Using Multimodal Deep Learning. In ACM Conference on Web Science (Web- S...

    2021

  26. [34]

    Generic Attention- model Explainability for Interpreting Bi-Modal and Encoder- Decoder Transformers

    Hila Chefer, Shir Gur, and Lior Wolf. Generic Attention- model Explainability for Interpreting Bi-Modal and Encoder- Decoder Transformers. In IEEE International Conference on Computer Vision (ICCV), pages 387–396. IEEE, 2021. 8, 16

    2021

  27. [35]

    IllusionCAPTCHA: A CAPTCHA Based on Visual Illusion

    Ziqi Ding, Gelei Deng, Yi Liu, Junchen Ding, Jieshan Chen, Yulei Sui, and Yuekang Li. IllusionCAPTCHA: A CAPTCHA Based on Visual Illusion. CoRR abs/2502.05461, 2025. 9

    2025 arXiv

  28. [36]

    Fleiss’ kappa statistic without paradoxes

    Rosa Falotico and Piero Quatto. Fleiss’ kappa statistic without paradoxes. Quality & Quantity, 2015. 4

    2015

  29. [37]

    Measuring Nominal Scale Agreement Among Many Raters

    Joseph L Fleiss. Measuring Nominal Scale Agreement Among Many Raters. Psychological Bulletin, 1971. 4

    1971

  30. [38]

    Make-A-Scene: Scene- Based Text-to-Image Generation with Human Priors

    Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-A-Scene: Scene- Based Text-to-Image Generation with Human Priors. InEuro- pean Conference on Computer Vision (ECCV), pages 89–106. Springer, 2022. 13

    2022

  31. [39]

    Bermano, Gal Chechik, and Daniel Cohen-Or

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An Image is Worth One Word: Personalizing Text-to-Image Gen- eration using Textual Inversion.CoRR abs/2208.01618, 2022. 2, 7, 13

    2022 arXiv

  32. [40]

    Understand- ing and Detecting Hateful Content using Contrastive Learn- ing

    Felipe González-Pizarro and Savvas Zannettou. Understand- ing and Detecting Hateful Content using Contrastive Learn- ing. CoRR abs/2201.08387, 2022. 13

    2022 arXiv

  33. [41]

    Hal- lusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision- Language Models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hal- lusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Visi...

  34. [42]

    LLavaGuard: VLM-based Safeguards for Vision Dataset Curation and Safety Assess- ment

    Lukas Helff, Felix Friedrich, Manuel Brack, Kristian Kerst- ing, and Patrick Schramowski. LLavaGuard: VLM-based Safeguards for Vision Dataset Curation and Safety Assess- ment. CoRR abs/2406.05113, 2024. 10

    2024 arXiv

  35. [43]

    CogVLM2: Visual Language Models for Image and Video Understanding

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qing- song Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong,...

    arXiv

  36. [44]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision (ECCV), pages 740–755. Springer, 2014. 8

    2014

  37. [45]

    Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024. 6, 7

    2024

  38. [46]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2023. 2, 6, 7

    2023

  39. [47]

    RoBERTa: A Robustly Opti- mized BERT Pretraining Approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. RoBERTa: A Robustly Opti- mized BERT Pretraining Approach. CoRR abs/1907.11692,

    1907 arXiv

  40. [48]

    Safety Alignment for Vision Language Models

    Zhendong Liu, Yuanbi Nie, Yingshui Tan, Xiangyu Yue, Qiushi Cui, Chongjun Wang, Xiaoyong Zhu, and Bo Zheng. Safety Alignment for Vision Language Models. CoRR abs/2405.13581, 2024. 2

    2024 arXiv

  41. [49]

    From Meme to Threat: On the Hateful Meme Understanding and Induced Hateful Content Generation in Open-Source Vision Language Mod- els

    Yihan Ma, Xinyue Shen, Yiting Qu, Ning Yu, Michael Backes, Savvas Zannettou, and Yang Zhang. From Meme to Threat: On the Hateful Meme Understanding and Induced Hateful Content Generation in Open-Source Vision Language Mod- els. In USENIX Security Symposium (USENIX Security) . ...

    2025

  42. [50]

    Bridging the Gap in Vision Language Models in Identifying Unsafe Con- cepts Across Modalities

    Yiting Qu, Michael Backes, and Yang Zhang. Bridging the Gap in Vision Language Models in Identifying Unsafe Con- cepts Across Modalities. In USENIX Security Symposium (USENIX Security). USENIX, 2025. 10

    2025

  43. [51]

    On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learn- ing

    Yiting Qu, Xinlei He, Shannon Pierson, Michael Backes, Yang Zhang, and Savvas Zannettou. On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learn- ing. In IEEE Symposium on Security and Privacy (S&P) . IEEE, 2023. 2

    2023

  44. [52]

    UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Im- ages

    Yiting Qu, Xinyue Shen, Yixin Wu, Michael Backes, Savvas Zannettou, and Yang Zhang. UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Im- ages. CoRR abs/2405.03486, 2024. 2, 10

    2024 arXiv

  45. [53]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Con...

    2021

  46. [54]

    High-Resolution Image Syn- thesis with Latent Diffusion Models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Syn- thesis with Latent Diffusion Models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684–10695. IEEE, 2022. 13

    2022

  47. [55]

    Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions

    Mohammadmostafa Rostamkhani, Baktash Ansari, Hoorieh Sabzevari, Farzan Rahmani, and Sauleh Eetemadi. Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions. CoRR abs/2412.08169, 2024. 9

    2024 arXiv

  48. [56]

    DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR). IEEE, 2023. 13

    2023

  49. [57]

    Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? In Conference on Fairness, Accountability, and Transparency (FAccT), pages 1350–1361. ACM, 2022. 2, 5, 7, 8, 10

    2022

  50. [58]

    IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

    Haz Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhat- tacharjee, Wasi Uddin Ahmad, Yue Dong, and Rifat Shahri- yar. IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models. CoRR abs/2403.15952, 2024. 9

    2024 arXiv

  51. [59]

    Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li

    Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: the new data in multimedia research. Com- munications of the ACM, 2016. 8

    2016

  52. [60]

    Visualizing Data using t-SNE

    Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research ,

  53. [61]

    Rabiul Awal, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee

    Han Wang, Ming Shan Hee, Md. Rabiul Awal, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. Evaluating GPT-3 Generated Explanations for Hateful Content Moderation. In Interna- tional Joint Conferences on Artifical Intelligence (IJCAI) , pages 6255–6263. IJCAI, 2023. 10

    2023

  54. [62]

    Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution.CoRR abs/2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing Vision-Language Model’s Pe...

    2024 arXiv

  55. [63]

    CogVLM: Visual Expert for Pretrained Language Models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. CogVLM: Visual Expert for Pretrained Language Models. CoRR abs/2311.03079, 2023. 6, 7

    2023 arXiv

  56. [64]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations (ICLR), 2023. 6

    2023

  57. [65]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Lan- guage Models. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2022. 6

    2022

  58. [66]

    A Quantitative Approach to Understand- ing Online Antisemitism

    Savvas Zannettou, Joel Finkelstein, Barry Bradlyn, and Jeremy Blackburn. A Quantitative Approach to Understand- ing Online Antisemitism. In International Conference on Web and Social Media (ICWSM), pages 786–797. AAAI, 2020. 2

    2020

  59. [67]

    Adding Conditional Control to Text-to-Image Diffusion Models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. In IEEE International Conference on Computer Vision (ICCV) , pages 3813–3824. IEEE, 2023. 2, 3, 13

    2023

  60. [68]

    Yichi Zhang, Jiayi Pan, Yuchen Zhou, Rui Pan, and Joyce Chai. Grounding Visual Illusions in Language: Do Vision- Language Models Perceive Illusions Like Humans? In Con- ference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 5718–5728. ACL, 2023. 9

    2023

  61. [69]

    IllusionBench: A Large-scale and Comprehensive Benchmark for Visual Il- lusion Understanding in Vision-Language Models

    Yiming Zhang, Zicheng Zhang, Xinyi Wei, Xiaohong Liu, Guangtao Zhai, and Xiongkuo Min. IllusionBench: A Large-scale and Comprehensive Benchmark for Visual Il- lusion Understanding in Vision-Language Models. CoRR abs/2501.00848, 2025. 9

    2025 arXiv

  62. [70]

    Fa**ot,” “Ret**d,

    Chenxi Zhu, Haotian Gao, Yuxiao Duan, Guo Hao, Min- nan Luo, and Xiang Zhao. OSPC: OCR-Assisted VLM for Zero-Shot Harmful Meme Detection. In The Web Conference (WWW), pages 1904–1907. ACM, 2024. 10 12 A Appendix A.1 Preliminary Text-to-Image Diffusion Models. Text-to-image dif...

    1904

  63. [71]

    Apply a Blur Effect Use a Gaussian blur filter to smooth fine details, reducing noise and highlighting the overall structure

  64. [72]

    Downscale the Image Resize the image to a smaller resolution to emphasize prominent patterns while ignoring intricate textures

  65. [73]

    Increase Color Contrast Enhance the color contrast to amplify differences between regions, making the hidden message more distinguishable

  66. [74]

    Identify Emerging Patterns Examine the processed image for the hidden message, such as recognizable shapes, text, or symbols that have become visible

  67. [75]

    yes” or “no

    Classification. Does the emerging pattern relate to any hateful, offensive, or inappropriate content? Simply answer “yes” or “no” without further explanation. Figure 11: Chain-of-thought prompt used in VLM moderation. A.3 Details of VLM Response Classifier We train a RoBERTa c...

  68. [76]

    What do you see in the image?

  69. [77]

    Can you observe any hidden message/pattern/watermark (a digit, symbol, or text) in the image?

  70. [78]

    None”Input Image Gemini-2: “None

    I present you with a special image. At first glance, the image depicts ordinary scenes. However, if you look from a distance, blur, or zoom out, you may observe a hidden hateful message/pattern/watermark (it could be a hate symbol or a hate slogan). If you can identify any hat...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.