The Ethics of Generative AI in Anonymous Spaces: A Case Study of 4chan's /pol/ Board

Emiliano De Cristofaro; Parth Gaba

REVIEW 5 major objections 4 minor 17 references

The Ethics of Generative AI in Anonymous Spaces: A Case Study of 4chan's /pol/ Board

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that 4chan's /pol/ board is using generative AI to mass-produce racist, anti-Semitic, and Nazi-themed images that slip past conventional moderation filters.

desk verdict Curated sample of AI images from /pol/ shows real misuse, but the prevalence percentages are not generalizable. read the letter →

arxiv 2506.14191 v1 pith:QDH3MOKB submitted 2025-06-17 cs.CY

Parth Gaba , Emiliano De Cristofaro This is my paper · ORCID

classification cs.CY

keywords generativeAIAI-generatedimages4chan/pol/extremistcontentmoderationmemeticwarfarehatespeech

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that anonymous users on 4chan's /pol/ board are treating generative AI as a low-cost production tool for extremist images, not just for humor or art. The authors collected 900 images from the board's /mwg/ threads and an affiliated Twitter aggregator, used an AI-image detector to keep 66 unique AI-generated ones, and coded them by hand. They find that 69.7% of those images show recognizable figures, 28.8% contain racist content, 28.8% contain anti-Semitic content, and 9.1% carry Nazi-related imagery. The paper concludes that these tools are being weaponized to create hateful memes and political commentary that bypass conventional content moderation, which matters because it suggests a scalable pipeline for extremist content that model makers and platforms will need to counter with post-generation safeguards.

What carries the argument

The machinery is a three-stage pipeline. First, images are pulled from /pol/ threads carrying the /mwg/ tag and from an aggregator Twitter account, a route that favors memorable and extreme posts. Second, perceptual hashing removes near-duplicates and reveals whether the same AI images circulated on both platforms, and an AI-image detector keeps only images it labels 'artificial' with probability above 0.85. Third, the surviving 66 images are manually coded into overlapping thematic categories, and the reported percentages come from that coding. The detector fixes what counts as AI-generated, and the coding fixes what counts as racist, anti-Semitic, Nazi-related, satirical, or playful; both assumptions are inside every number the paper reports.

What would settle it

Take a fresh, uncurated random sample of /mwg/-tagged threads from the same four months, run the same detector and the same manual coding blind, and compare the shares of racist and anti-Semitic images with the 28.8% figures; a sharp drop would show that the original sample overstated the platform-wide pattern.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a measured pattern: among 66 unique AI-generated images posted to /pol/ between April and July 2024, most show recognizable public or cultural figures, close to three in ten contain racist content, close to three in ten contain anti-Semitic content, and about one in eleven uses Nazi-related imagery. Manual coding also finds roughly 70% of the images expressing far-right political views and recurring uses of symbols such as Pepe the Frog and 'Happy Merchant' caricatures. The authors read this as evidence that generative image tools are being weaponized for extremist commentary and memes that slip past conventional moderation, because the harmful signal is embedded in generated pixels and coded symbols rather than in detectable text.

Load-bearing premise

The whole argument rests on the 66 AI-generated images being representative of what /pol/ users actually create and share, but both collection routes — an aggregator account that highlights extreme posts and the /mwg/ tag, a hub for propaganda — likely overrepresent offensive content, so the percentages may overstate platform-wide weaponization.

Editorial extensions

If this is right

Conventional moderation that checks text or exact-match images will miss much of this content, so platforms will need multimodal detectors that read image and caption together.
Because the same images appeared on 4chan and on the Twitter aggregator, AI-generated /pol/ content is already diffusing to other platforms, extending its reach beyond the original board.
Prompt-level safety filters in text-to-image systems are not sufficient on their own, so model providers would need post-generation screening for coded extremist symbols and figures.
If the measured prevalence holds, policy debates about generative AI should treat hateful meme production as a concrete, near-term harm rather than a hypothetical one.
The overlap of categories (an image can be both racist and political satire) means moderation and policy need to handle compound content, not single-label classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the 0.85 detector threshold is a hidden dial; rerunning the pipeline with a lower or higher threshold would shift the count of AI-generated images, so the percentages are best read as conditioned on that detector, not as ground truth.
Editorial extension: if the collection routes are indeed biased toward extreme posts, the true /pol/-wide rate of racist AI images could be lower, yet even a small rate would still represent an unbounded-volume moderation problem because generation is cheap.
Editorial extension: the same pipeline could be applied to other anonymous or lightly moderated boards to test whether the 28.8% figures are specific to /pol/ or a general property of anonymous spaces with no moderation.
Editorial extension: as generation costs fall and new models appear, the aesthetic of meme-ified extremist AI images may shift, so detectors and moderation tools trained on the current sample may need continuous retraining.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this as a case study of curated extremist AI images, not as a measurement of /pol/'s AI output. The authors scraped ~800 images from @4chan_AI_Terror, an account that exists to document AI-generated extremism, plus ~100 from /mwg/ threads, ran an off-the-shelf detector, and manually coded the 66 that survived. They report 69.7% recognizable figures, 28.8% racist, 28.8% anti-Semitic, 9.1% Nazi-related content, and conclude users are weaponizing generative AI and bypassing moderation.

The new thing here is the dataset itself and the thematic breakdown. I haven't seen a direct look at AI images on /pol/ with this detail. The paper includes code, a public spreadsheet, and useful figures like the Hitler meme and the Minecraft sheriff meme. It also builds sensibly on Hine, Marchal, and Zannettou. If you want concrete examples of how text-to-image is being used on the far-right, this is a decent source.

But the quantitative claims do not survive contact with the sampling frame. The Twitter account and /mwg/ tag are both curated toward offensive content, so the percentages describe that curated set, not the board. There's no denominator, no random sample, no moderation outcome data. The 100-to-66 drop is unexplained, the 0.85 detector threshold is unvalidated, and coding was done by one person without reliability checks. The abstract's "often bypass conventional content moderation" is simply not evidenced. Also, reference [8] is an unrelated physics paper - a sloppy citation that doesn't inspire confidence.

None of this makes the paper a waste. The qualitative examples are real and the phenomenon is worth documenting. But the authors overstate their case. A solid revision would reframe this as a case study of curated content, add limitations, validate the detector, and ideally add a random sample.

I'd send it to review, but as a borderline case with clear expectations for revision. I wouldn't cite the headline numbers in my own work.

Referee Report

5 major / 4 minor

Summary. The paper reports a characterization of AI-generated images on 4chan's /pol/ board, based on 900 images collected from the Twitter account @4chan_AI_Terror and from /mwg/-tagged threads between April and July 2024. After filtering with a Hugging Face AI-image-detector at a 0.85 confidence threshold, the authors identify 100 AI-generated images and then analyze 66 unique images. They report that 69.7% of these images contain recognizable figures, 28.8% contain racist content, 28.8% contain anti-Semitic content, and 9.1% contain Nazi-related imagery, and conclude that users are 'weaponizing generative AI' to create extremist content that 'often bypasses conventional content moderation systems' (Abstract; Sections 3 and 4).

Significance. If the reported prevalence figures were representative of AI-generated content on /pol/, the findings would be an important contribution to the emerging literature on generative AI misuse, with implications for platform governance and AI safety. The paper has notable strengths in transparency: scraping code, perceptual hashing code, detection code, and a linked spreadsheet of manual image descriptions are provided, which supports reproducibility of the analysis as described. However, the significance is currently limited because the central percentages are computed from a convenience sample whose selection strongly overrepresents extremist content, and because the detector and manual coding are not validated. The paper would be substantially more valuable if reframed as a case study of a curated corpus of known AI-generated extremist images rather than as a prevalence estimate for /pol/.

major comments (5)

[Sections 3.1, 3.3, Abstract] The central claim of widespread weaponization rests on a convenience sample. The 800+ images scraped from the @4chan_AI_Terror Media tab come from an account whose stated purpose is to document AI-generated extremist content, and the ~100 images from the /mwg/ tag come from a thread explicitly dedicated to memetic warfare and AI propaganda. Neither source is a random sample of /pol/ posts, and the paper does not provide any sampling frame, inclusion probabilities, or denominator for AI-generated images on /pol/. As a result, the reported percentages (69.7%, 28.8%, 28.8%, 9.1%) describe this selected corpus, not the prevalence of such content among AI-generated images on /pol/, and the Abstract's claim that users are 'weaponizing generative AI to create extremist content, political commentary, and memes that often bypass conventional content moderation systems' is not supported by the data.
[Section 3.3] The AI-image-detector (Hugging Face 'umm-maybe/AI-image-detector') is not validated anywhere in the manuscript. The 0.85 confidence threshold is presented without any precision/recall analysis, and no error analysis is provided for false positives or false negatives. Because the entire dataset depends on this filter, an unvalidated threshold is load-bearing: different thresholds would change the composition of the 100 'AI-generated' images and hence all downstream percentages.
[Sections 3.3 and 4.1] The reduction from 100 AI-generated images to 66 'unique' images is unexplained. The text states that the filtering process 'reduced the dataset to 100 AI-generated images' and later that 66 images were analyzed, but no deduplication rule, exclusion criterion, or pairwise similarity threshold is reported. Perceptual hashing is described as a tool to compare images between sources, but it is not described as a deduplication step. This missing step affects the denominator of every reported percentage and must be documented.
[Section 4.1 and Figure 9] The manual coding procedure has no demonstrated reliability. The manuscript reports that 'each image was manually reviewed' with 'qualitative coding' and non-exclusive categories, but no codebook, no inter-rater reliability metric, and no second coder are described. In addition, the text is internally inconsistent: the text states 'A significant portion (30%)' offensive and 'References to Hitler and Nazi iconography were notable (10%)', while the Abstract and Figure 9 report 28.8% and 9.1%, respectively. These inconsistencies should be reconciled, and the coding procedure should be described in enough detail to be audited.
[Abstract and Conclusion] The claim that AI-generated content 'often bypasses conventional content moderation systems' is not supported by any moderation-outcome data. The paper does not measure 4chan's moderation practices, removal rates, or platform responses to the images in the dataset. This is a claim about the effectiveness of moderation systems, not about image content, and it should be removed or reframed as a hypothesis or potential risk rather than a documented finding.

minor comments (4)

[References] Reference 8 (Kotousov and Lukyanov, 'ODE/IQFT Correspondence for the Generalized Affine sl(2) Gaudin Model') appears unrelated to the topic of the paper and is likely a citation error.
[Global] There is a typo in the disclaimer: 'This repor contains' should be 'This report contains'. Also, the abbreviation '/mgw/' appears once where '/mwg/' is used elsewhere; please standardize.
[Section 2, 'Systems Used in Generative AI for Images'] The statement 'we focus on DALL·E here because it has been more commonly discussed and used on /mgw/' is not evidenced by the data; no attribution of any collected image to a specific generation model is provided, so the claim about DALL·E's prevalence on /pol/ is unsupported.
[Section 3.3] The closing sentence 'The combined methodology provided a diverse and well-validated dataset' overstates the state of validation, given the unvalidated detector and single-coder coding. Please rephrase to reflect the actual limitations described above.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's percentages are direct manual codings of collected images, not outputs of a fitted model or a self-citation chain.

full rationale

The paper is an empirical, descriptive case study rather than a formal derivation. The headline percentages (69.7% recognizable figures, 28.8% racist, 28.8% anti-Semitic, 9.1% Nazi-related) are computed by manually reviewing and coding the 66 AI-generated images; no equation or fitted parameter is used to produce these numbers, so there is no self-definitional or fitted-input-as-prediction circularity. The AI-image detector from Hugging Face is an external tool, and the 0.85 confidence threshold is a methodological choice that affects precision, not a parameter fitted to the paper's conclusions. The sampling approach is a convenience sample drawn from @4chan_AI_Terror and the /mwg/ tag, which limits generalizability and may overrepresent extreme content, but that is an external-validity or sampling-bias concern, not circular reasoning. Several cited works (e.g., Hine et al. and Papasavva et al.) involve one of the present authors, but they are used for background context about 4chan and prior measurement work, not to justify the paper's new empirical findings; the central claim that users create extremist AI content rests on the manually coded images themselves. The statement that such content 'often bypasses conventional content moderation systems' is an interpretation not directly measured by the data, but this is a support or evidentiary weakness, not circularity. Overall, the paper's derivation chain, such as it is, is self-contained: data collection, detection, manual coding, and descriptive statistics. No step reduces by construction to its own inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central quantitative claims depend on the choice of two thresholds and on three unvalidated assumptions about detection accuracy, coding reliability, and sample representativeness. No new entities are introduced.

free parameters (2)

AI detection confidence threshold = 0.85
Chosen by the authors to classify images as AI-generated; no validation on 4chan-style images.
Perceptual hash threshold = 5
Used in pHash to identify near-duplicate images; no sensitivity analysis provided.

assumptions (4)

domain assumption The Hugging Face model 'umm-maybe/AI-image-detector' correctly identifies AI-generated images at the 0.85 threshold.
The paper uses this model without independent validation or error analysis on the target domain.
domain assumption Manual review by a single coder reliably identifies themes such as racism, anti-Semitism, and Nazism.
No inter-coder reliability check is reported, so coding may be subjective.
domain assumption The Twitter account @4chan_AI_Terror and the /mwg/ tag yield a representative sample of AI-generated images on /pol/.
The account likely curates extreme content, introducing selection bias; no random sampling is used.
domain assumption The perceptual hash distance threshold of 5 correctly identifies duplicate images.
No analysis of false positives or false negatives for this threshold.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Ethics of Generative AI in Anonymous Spaces: A Case Study of 4chan's /pol/ Board." pith.science (2026). https://pith.science/paper/QDH3MOKB

@misc{pith2026250614191,
  author       = {Pith},
  title        = {Pith review of: The Ethics of Generative AI in Anonymous Spaces: A Case Study of 4chan's /pol/ Board},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDH3MOKB}},
  note         = {Machine review of arXiv:2506.14191}
}

read the original abstract

This paper presents a characterization of AI-generated images shared on 4chan, examining how this anonymous online community is (mis-)using generative image technologies. Through a methodical data collection process, we gathered 900 images from 4chan's /pol/ (Politically Incorrect) board, which included the label "/mwg/" (memetic warfare general), between April and July 2024, identifying 66 unique AI-generated images. The analysis reveals concerning patterns in the use of this technology, with 69.7% of images including recognizable figures, 28.8% of images containing racist elements, 28.8% featuring anti-Semitic content, and 9.1% incorporating Nazi-related imagery. Overall, we document how users are weaponizing generative AI to create extremist content, political commentary, and memes that often bypass conventional content moderation systems. This research highlights significant implications for platform governance, AI safety mechanisms, and broader societal impacts as generative AI technologies become increasingly accessible. The findings underscore the urgent need for enhanced safeguards in generative AI systems and more effective regulatory frameworks to mitigate potential harms while preserving innovation.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages

[8]

ODE/IQFT correspondence for the generalized affine $\mathfrak{ sl}(2)$ Gaudin model

Kotousov, Gleb A., and Sergei L. Lukyanov. “ODE/IQFT Correspondence for the Generalized Afﬁne $\mathfrak sl(2)$ Gaudin Model.” arXiv [Hep-Th], 2021, http://arxiv.org/abs/2106.01238

work page Pith review arXiv 2021
[1]

4chan and /b/: An Analysis of Anonymity and Ephemerality in a Large Online Community

Bernstein, Michael, et al. “4chan and /b/: An Analysis of Anonymity and Ephemerality in a Large Online Community.” Proceedings of the International AAAI Conference on Web and Social Media, vol. 5, no. 1, 2021, pp. 50–57, doi:10.1609/icwsm.v5i1.14134

doi:10.1609/icwsm.v5i1.14134 2021
[2]

Dall·e 3

“Dall·e 3.” Openai.com, https://openai.com/index/dall-e-3/. Accessed 20 Apr. 2025

work page 2025
[3]

Generative Adversarial Networks

Goodfellow, Ian J., et al. “Generative Adversarial Networks.” arXiv [Stat.ML], 2014, http://arxiv.org/abs/1406.2661

arXiv 2014
[4]

Kek, Cucks, and God Emperor Trump: A Measurement Study of 4chan’s Politically Incorrect Forum and Its Effects on the Web

Hine, Gabriel Emile, et al. “Kek, Cucks, and God Emperor Trump: A Measurement Study of 4chan’s Politically Incorrect Forum and Its Effects on the Web.” arXiv [Cs.SI], 2016, http://arxiv.org/abs/1610.03452

arXiv 2016
[5]

How the Internet and Social Media Are Changing Culture - Aspen Institute Central Europe

“How the Internet and Social Media Are Changing Culture - Aspen Institute Central Europe.” Aspen, 14 Mar. 2017, https://www.aspeninstitutece.org/article/2017/internet-social-media-changing-culture/

work page 2017
[6]

Feels Bad Man: Dissecting Automated Hateful Meme Detection Through the Lens of Facebook's Challenge

Jennifer, Catherine, et al. “Feels Bad Man: Dissecting Automated Hateful Meme Detection through the Lens of Facebook’s Challenge.” arXiv [Cs.CY], 2022, http://arxiv.org/abs/2202.08492

work page Pith review arXiv 2022
[7]

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

Kiela, Douwe, et al. “The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes.” arXiv [Cs.AI], 2020, http://arxiv.org/abs/2005.04790. 15

arXiv 2020

Show all 17 references

[9]

Generative AI Misuse: A Taxonomy of Tactics and Insights from Real‑World Data

Marchal, Nahema, et al. “Generative AI Misuse: A Taxonomy of Tactics and Insights from Real‑World Data.” arXiv, 19 June 2024, https://arxiv.org/abs/2406.13843

2024 arXiv
[10]

Internet Memes as Contested Cultural Capital: The Case of 4chan’s /b/ Board

Nissenbaum, Asaf, and Limor Shifman. “Internet Memes as Contested Cultural Capital: The Case of 4chan’s /b/ Board.” New Media & Society, vol. 19, no. 4, 2017, pp. 483–501, doi:10.1177/1461444815609313

2017 doi
[11]

Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Board

Papasavva, Antonis, et al. “Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Board.” arXiv [Cs.CY], 2020, http://arxiv.org/abs/2001.07487

2020 arXiv
[12]

MemeBot: Towards Automatic Image Meme Generation

Sadasivam, Aadhavan, et al. “MemeBot: Towards Automatic Image Meme Generation.” arXiv [Cs.CL], 2020, http://arxiv.org/abs/2004.14571

2020 arXiv
[13]

Dylann Roof, 4chan, and the New Online Racism

Siegel, Jacob. “Dylann Roof, 4chan, and the New Online Racism.” The Daily Beast, 29 June 2015, https://www.thedailybeast.com/dylann-roof-4chan-and-the-new-online-racism\

2015
[14]

Accessed 20 Apr

Stylegan: StyleGAN - Ofﬁcial TensorFlow Implementation. Accessed 20 Apr. 2025

2025
[15]

On the Origins of Memes by Means of Fringe Web Communities

Zannettou, Savvas, et al. “On the Origins of Memes by Means of Fringe Web Communities.” arXiv [Cs.SI], 2018, http://arxiv.org/abs/1805.12512

2018 arXiv
[16]

Accessed 20 Apr

4chan.org, https://www.4chan.org/. Accessed 20 Apr. 2025

2025
[17]

Accessed 20 Apr

Adl.org, https://www.adl.org/resources/hate-symbol/pepe-frog. Accessed 20 Apr. 2025. 16

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [8]

ODE/IQFT correspondence for the generalized affine $\mathfrak{ sl}(2)$ Gaudin model

Kotousov, Gleb A., and Sergei L. Lukyanov. “ODE/IQFT Correspondence for the Generalized Afﬁne $\mathfrak sl(2)$ Gaudin Model.” arXiv [Hep-Th], 2021, http://arxiv.org/abs/2106.01238

work page Pith review arXiv 2021

[2] [1]

4chan and /b/: An Analysis of Anonymity and Ephemerality in a Large Online Community

Bernstein, Michael, et al. “4chan and /b/: An Analysis of Anonymity and Ephemerality in a Large Online Community.” Proceedings of the International AAAI Conference on Web and Social Media, vol. 5, no. 1, 2021, pp. 50–57, doi:10.1609/icwsm.v5i1.14134

doi:10.1609/icwsm.v5i1.14134 2021

[3] [2]

Dall·e 3

“Dall·e 3.” Openai.com, https://openai.com/index/dall-e-3/. Accessed 20 Apr. 2025

work page 2025

[4] [3]

Generative Adversarial Networks

Goodfellow, Ian J., et al. “Generative Adversarial Networks.” arXiv [Stat.ML], 2014, http://arxiv.org/abs/1406.2661

arXiv 2014

[5] [4]

Kek, Cucks, and God Emperor Trump: A Measurement Study of 4chan’s Politically Incorrect Forum and Its Effects on the Web

Hine, Gabriel Emile, et al. “Kek, Cucks, and God Emperor Trump: A Measurement Study of 4chan’s Politically Incorrect Forum and Its Effects on the Web.” arXiv [Cs.SI], 2016, http://arxiv.org/abs/1610.03452

arXiv 2016

[6] [5]

How the Internet and Social Media Are Changing Culture - Aspen Institute Central Europe

“How the Internet and Social Media Are Changing Culture - Aspen Institute Central Europe.” Aspen, 14 Mar. 2017, https://www.aspeninstitutece.org/article/2017/internet-social-media-changing-culture/

work page 2017

[7] [6]

Feels Bad Man: Dissecting Automated Hateful Meme Detection Through the Lens of Facebook's Challenge

Jennifer, Catherine, et al. “Feels Bad Man: Dissecting Automated Hateful Meme Detection through the Lens of Facebook’s Challenge.” arXiv [Cs.CY], 2022, http://arxiv.org/abs/2202.08492

work page Pith review arXiv 2022

[8] [7]

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

Kiela, Douwe, et al. “The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes.” arXiv [Cs.AI], 2020, http://arxiv.org/abs/2005.04790. 15

arXiv 2020