When Image Generation Goes Wrong: A Safety Analysis of Stable Diffusion Models

Matthias Schneider; Thilo Hagendorff

REVIEW 5 major objections 5 minor 1 cited by

When Image Generation Goes Wrong: A Safety Analysis of Stable Diffusion Models

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Study: Most Stable Diffusion images from harmful prompts are unsafe

desk verdict A concrete but methodologically fragile audit: the core qualitative finding holds, yet the paper's percentages rest on an unvalidated classifier with no benign-prompt baseline. read the letter →

arxiv 2411.15516 v1 pith:MRFH7CW3 submitted 2024-11-23 cs.CY

Matthias Schneider , Thilo Hagendorff This is my paper · ORCID

classification cs.CY

keywords text-to-imagesafetyStableDiffusioncontentmoderationNSFWviolencebiasracialsyntheticmediarefusalbehavior

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ten widely downloaded open text-to-image models were prompted with 50 harmful prompts spanning nudity, sexual acts, violence, hate crimes, and personal depictions of public figures, and every model responded by generating the requested content instead of refusing. Across 24,000 generated images, only 35.5% avoided being flagged by an automated content-moderation pipeline, making the majority of outputs unsafe by the study's definitions. The paper also reports that nudity and sexual imagery were skewed heavily toward white individuals, while images of gang violence disproportionately showed Black individuals even though no prompt mentioned race. Celebrity depictions were usually too unrealistic to be useful for abuse, though smoking and gambling scenes were the most common recognizable personal-sensitive outputs. These findings matter because these models are widely distributed, run offline, and appear to ship without the refusal behavior found in closed assistants.

What carries the argument

The argument is carried by a standardized image-generation and classification pipeline: each model is run with its own recommended sampling configuration, the same 50 prompts are applied 20 times, and every output is passed to an automated visual moderation system with category-specific thresholds (for example, general_nsfw at 0.7, very_bloody at 0.5, and a combination of identification score and content scores for celebrity images). An image counts as safe only if it exceeds none of the thresholds for its prompt category. A separate face-recognition step labels the race of depicted individuals, and a celebrity classifier determines whether a public figure is recognizable. This pipeline converts 'does the model refuse?' into a measurable safety percentage, and it is what the reported model rankings and bias ratios rest on.

What would settle it

Take a random sample of the 24,000 generated images, have human annotators independently label them with the same nine categories, and compare their labels to the automated ones; if agreement is low or the racial composition of violent images changes under human labeling, the reported percentages and bias ratios would not survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that popular Stable Diffusion models, including base models and fine-tuned community versions, have no effective safety layer: when given prompts for NSFW, violent, or personally sensitive content, they comply. Measured across ten models with 50 prompts applied 20 times each, the safest model still produced harmful images 44.87% of the time and the least safe produced them 81.62% of the time, with an average of 64.5% of all images flagged as unsafe. The same classifier-based analysis found strong representation bias: white individuals dominated nudity and sexual content, while Black individuals were overrepresented in gang-violence images. The authors interpret these patterns as inherited from training data and conclude that current open image generators cannot be considered safe or neutral by default.

Load-bearing premise

The load-bearing premise is that the automated moderation API and the race classifier correctly label AI-generated images as harmful or safe; the paper reports no validation of either tool on synthetic images.

Editorial extensions

If this is right

If the results hold, a user can generate explicit or violent images from a majority of harmful prompts on today's most-downloaded open models, with no built-in refusal.
Model rankings by safety shift with the distribution platform: the newest base model was least harmful, while realism-focused fine-tunes were most harmful.
Racial bias in generated violence is not tied to any race word in the prompt, so the models themselves associate Black individuals with violent contexts.
Current celebrity-image risk is low because generated faces are not believable, but the same pipeline would need re-testing as realism improves.
The observed behavior supports adding prompt filters, post-generation content classifiers, and better training-data curation to open image models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves the safety percentages to a single commercial moderation API; a reasonable extension would be to re-label a random sample by human raters and measure agreement, since API thresholds were not validated on synthetic images.
Because the race classifier was also not validated on generated faces, the reported bias ratios are provisional; human labeling could shift them.
An implication the authors do not develop is that offline distribution makes post-hoc safety unenforceable; the practical intervention point may be the training data and the initial model release.
A testable extension would be to prompt the same models with neutral scenes and measure whether violent context alone shifts the racial composition, separating stereotype bias from prompt compliance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper before citing it. It is a concrete cross-model safety audit of ten popular Stable Diffusion models, including five CivitAI fine-tunes, which is new relative to prior work like Unsafe Diffusion and Stable Bias. The authors generated 24,000 images, used Hive AI's moderation API and DeepFace for race classification, and report large unsafe-image ratios and racial biases. The qualitative finding—these models readily produce explicit, violent, and biased content—is credible and consistent with prior literature. The manual visual analysis, while exploratory, supports the capability claim and documents odd artifacts like vulva-penis hybrids.

The soft spots are real and load-bearing for the quantitative claims. The stress-test note is on target: every percentage and ranking in Section 3.2 hinges on Hive's thresholds, which are presented as fixed rules with no sensitivity analysis and no validation on AI-generated images. Synthetic images contain unusual textures and anatomical distortions, so Hive's scores could behave differently on them than on real-world content. There is no negative control: no benign prompts were run through the same pipeline to estimate the detector's false-positive base rate. Without that baseline, the reported 35.5% safe rate, the model ordering, and the race-bias ratios (e.g., 24.5% Black in gang violence) cannot be cleanly separated from detector behavior. The claim of a 'complete lack of refusal behavior' is also not directly measured; the authors infer it from high unsafe rates, but they never tested whether models would refuse or hedge. That is an overstatement.

There are additional weaknesses: percentages lack error bars, the celebrity identification threshold is arbitrary, and no prompts, code, or data are released—only a vague 'available upon request.' These are all addressable.

Still, the paper deserves a serious referee. It is a useful audit with a plausible core result, and the methodological flaws are fixable. I would send it to review but with a clear request for major revision: add a benign-prompt control, validate the classifiers on a sample of generated images, report threshold sensitivity, release the prompts and ideally the images, and temper the refusal-behavior conclusion. If those are done, the quantitative claims would have a solid foundation.

For a reading group, it's a fine case study in how evaluation pipelines can confound results. I'd bring it up for that reason, but I wouldn't cite the numbers as-is in my own work until the validation is added.

Referee Report

5 major / 5 minor

Summary. The paper reports a safety analysis of ten popular Stable Diffusion models (five from Hugging Face, five from Civit AI). Using 50 prompts across nine categories (NSFW, violence, personally sensitive content) and generating 24,000 images, the authors apply Hive AI's vision moderation API, a celebrity classifier, and DeepFace's race classifier to quantify the prevalence of harmful content and investigate racial biases. The central claims are that, on average, only 35.5% of generated images are safe, that models show no refusal behavior, and that generated violence is disproportionately associated with Black individuals while nudity is predominantly White. The authors conclude that open text-to-image models lack adequate safety measures and call for better data curation, prompt moderation, and post-generation content filtering.

Significance. If the quantitative results hold, the paper would provide a useful large-scale empirical snapshot of the safety properties of widely used open-weight text-to-image models. The study has clear strengths: it evaluates ten popular models, uses a substantial generation corpus (24,000 images), includes a manual visual analysis that documents qualitative phenomena, and its model-selection rationale (download counts) is reasonable. However, the central quantitative claims—the average safe rate, the model ranking, and the race-bias ratios—rest on a single unvalidated classification pipeline and lack a negative control. Because the paper's main contribution is precisely these numbers, the missing validation is load-bearing rather than cosmetic.

major comments (5)

[§3.2, Appendix C] The quantitative claims in Section 3.2 (average 35.5% safe rate, model ranking in Figure 1, per-category unsafe ratios in Figure 2) rely entirely on Hive AI's moderation API with the fixed thresholds listed in Appendix C. No validation is reported of this API on synthetic, AI-generated images, which differ substantially from real-world content—the paper itself documents 'vulva-penis hybrids,' distorted bodies, and unnatural anatomy in Section 3.1. Without a sensitivity analysis over the thresholds or a human-annotated audit sample of the generated images, the reported percentages are unanchored and could reflect detector miscalibration rather than model behavior.
[§3.2, §2 (Methods)] The experimental design includes no benign-prompt negative control. Without generating images from innocuous prompts and passing them through the same Hive moderation pipeline, the false-positive base rate of the detector on synthetic images is unknown. If the detector spuriously flags synthetic textures as 'bloody' or 'suggestive,' then the average safe rate, the model ordering, and the violence-related bias ratios (e.g., 24.5% Black in gang violence in Section 3.2.3) could be artifacts of the detector rather than properties of the models. The authors should either add such a control or explicitly justify why one is unnecessary.
[Abstract; §3.2; §4] The claim of a 'complete lack of any refusal behavior or safety measures' is not directly measured by the study. No metric for refusal is defined; no empty outputs, error messages, or text-based refusals are reported; and no benign-prompts baseline is used to contextualize the unsafe-image rate. The inference that models 'comply with the prompts instead of rejecting them' is made solely from the high proportion of images labeled unsafe by Hive. The authors should either define and measure refusal behavior explicitly (e.g., proportion of generation failures, empty outputs, or safety-filter activations) or temper the abstract and discussion to say that the models generated harmful content in a large fraction of trials.
[§3.2.3, §2 (Methods)] The race-bias analysis uses DeepFace to label race in synthetic images, but DeepFace is not validated on AI-generated faces. Given that the paper documents severe anatomical distortions and unrealistic body structures, face detection and race classification may fail at differential rates across images and across racial categories, which would directly bias the reported ratios such as 24.5% Black in gang violence versus 12.36% overall. The authors should validate DeepFace on a sample of their generated images (e.g., by manual inspection or comparison with a second classifier) and should also report the rate at which DeepFace fails to detect a face, since undetected faces are implicitly excluded from the denominator.
[Appendix B; Figures 1 and 2] Several quantitative results are reported without any measure of uncertainty. For instance, the violence-related prompt categories contain only 2 prompts each (Appendix B), so the per-category unsafe ratios in Figure 2 are based on at most 40 images per model (2 prompts × 20 seeds) and have wide confidence intervals. The overall safe rates in Figure 1 are also point estimates with no error bars. The authors should provide confidence intervals, bootstrapped estimates, or per-prompt standard deviations so that the reader can judge whether differences such as SD3 at 55.13% safe versus Epic Realism at 18.38% are statistically reliable or within sampling noise.

minor comments (5)

[Appendix B] The prompt list is not included in the paper or appendix; the text states that prompts 'can be made available to other researchers upon reasonable request.' For reproducibility and transparency, the full prompt list should be published as a supplement or in a public repository.
[§3.1] The phrase 'It is to see that nudity is most often white' is awkward and should be rewritten (e.g., 'Nudity is most often depicted with light skin tones'). Similar phrasing issues appear elsewhere in Section 3.1 and 3.2.
[§3.2.2] The sentence 'Alternatively, this pattern could indicate that male individuals were overrepresented by transgender individuals in the training data' is a speculative explanation unsupported by the data. It should be removed or clearly labeled as an unverified hypothesis, since the observed anatomical artifacts can be explained by training-data distributional properties without invoking transgender overrepresentation.
[Appendix A] The model names are not fully consistent: 'EpiCRealism Natural Sin RC1 VAE' is listed in Appendix A while the text and Figure 1 refer to 'Epic Realism.' Please standardize names and use the exact version identifiers.
[References] Some references contain formatting errors, such as 'Struppek, Lukas; Hintersdorf, Dom; Friedrich, Felix; Br, Manuel' (incomplete author name) and 'In jair 78, pp. 1017–1068.' These should be corrected before publication.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical measurement study with no circular derivation; minor self-citations and manual classifier thresholds do not constitute circularity.

full rationale

The paper is an empirical measurement study, not a derivation from first principles. Its central quantitative claims rest on generating images with Stable Diffusion models and labeling them via Hive AI's vision moderation API, with thresholds listed in Appendix C, plus DeepFace for race classification. No target result is fitted into the inputs: the unsafe/safe percentages, model rankings, and race-bias ratios are outcomes of an external classifier applied to generated images, not parameters derived from those outcomes. The authors do not calibrate the Hive thresholds to force a particular conclusion, nor do they define their main quantities in terms of the results they report. The self-citations (Hagendorff 2021, 2024) appear in the introduction and discussion as general context about dual-use risks and exploitation; they are not load-bearing premises that make the empirical findings true. The manually chosen thresholds in Appendix C are a measurement-validity concern, because the API is not validated on synthetic images and there is no benign-prompt control to estimate false-positive base rates. However, that is a question of external validity or measurement bias, not circularity: the classifier output is not constructed from the paper's conclusions. No equation reduces to another, no fitted parameter is renamed as a prediction, and no author-specific uniqueness theorem is invoked. The finding is therefore essentially self-contained as an empirical audit, and the only mild circularity-adjacent elements are non-load-bearing self-citations and unvalidated but independently sourced classifier thresholds. Those justify a low score of 1 rather than 0, but do not indicate that the central claim reduces to its own inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central quantitative findings rest on three unvalidated tooling assumptions: the Hive API's harm labels, the DeepFace race classifier, and the authors' chosen thresholds. These assumptions are load-bearing because every percentage in Section 3 is derived from them. No parameters are fitted to the outcome variable, so circularity burden is low.

free parameters (2)

Hive moderation classification thresholds = general_suggestive>=0.5, general_nsfw>=0.7, yes_sexual_intent>=0.7, violence-related labels>=0.5
Chosen by the authors in Appendix C to convert API scores into binary safe/unsafe labels. No calibration or sensitivity analysis is provided, and every reported ratio depends on these cutoffs.
Celebrity identification threshold = identification_score>=0.2
Used in Appendix C to decide whether a generated celebrity image is recognizable for personal-sensitive categories. The threshold is stated without justification and directly affects the low personal-sensitive risk conclusion.

assumptions (4)

domain assumption Hive AI vision moderation API correctly classifies NSFW, violent, and sensitive content in AI-generated images.
All safe/unsafe labels in Section 3.2 and Appendix C come from this API. The paper does not validate the API on synthetic images or report its accuracy for this use.
domain assumption DeepFace racial classification correctly labels race in synthetic generated images.
Race bias statistics in Figures 3 and 5 depend on DeepFace labels; the accuracy of this tool on AI-generated faces is not reported.
domain assumption The most-downloaded models on Hugging Face and Civit AI as of June 16, 2024 are representative of the most popular Stable Diffusion models.
Model selection is based on download counts from two platforms, with no sensitivity analysis; the choice affects all cross-model comparisons.
domain assumption Generated persons are exclusively adults, as requested by the constant negative prompt.
The paper does not verify the apparent age of generated individuals, which matters for evaluating NSFW safety claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of When Image Generation Goes Wrong: A Safety Analysis of Stable Diffusion Models." pith.science (2026). https://pith.science/paper/MRFH7CW3

@misc{pith2026241115516,
  author       = {Pith},
  title        = {Pith review of: When Image Generation Goes Wrong: A Safety Analysis of Stable Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MRFH7CW3}},
  note         = {Machine review of arXiv:2411.15516}
}

read the original abstract

Text-to-image models are increasingly popular and impactful, yet concerns regarding their safety and fairness remain. This study investigates the ability of ten popular Stable Diffusion models to generate harmful images, including NSFW, violent, and personally sensitive material. We demonstrate that these models respond to harmful prompts by generating inappropriate content, which frequently displays troubling biases, such as the disproportionate portrayal of Black individuals in violent contexts. Our findings demonstrate a complete lack of any refusal behavior or safety measures in the models observed. We emphasize the importance of addressing this issue as image generation technologies continue to become more accessible and incorporated into everyday applications.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Video Deepfake Abuse: How Company Choices Predictably Shape Misuse Patterns
cs.CY 2025-11 conditional novelty 6.0 of 10

A few open-weight video models and distribution platforms dominate the creation and spread of NSFW AI video, making developer and platform choices the main intervention points for reducing non-consensual deepfake abuse.

Reference graph

Works this paper leans on

10 extracted references · 6 canonical work pages · cited by 1 Pith paper

[4]

Among the most popular distribution platforms for such models are Hugging Face and Civit AI

However, next to that, a large community is publishing open source models with generative capabilities that often lack any safety fine -tuning, despite ongoing 1 Corresponding author, matthias.schneider@student.hpi.uni-potsdam.de 2 discussions about the dual-use risks they present (Hagendorff 2021). Among the most popular distribution platforms for such m...

work page 2021
[6]

safe” when they are not detected by any of our classifier’s category, and “unsafe

Furthermore, researchers show that image training datasets used to train text-to-image models often contain racist, sexist, or otherwise offensive material (Birhane et al. 2021; Thiel 2023). Our paper aligns with previous research on safety issues in image generation models, examining the capacity of these models to produce various types of harmful conten...

work page 2021
[7]

concepts

In this study, we illustrate this by examining the capacity of stable diffusion models to produce various types of harmful content. Using a selection of the most popular models from Hugging Face and Civit AI, we observed no refusal behavior in response to harmful prompts. On the contrary, all models consistently complied with the prompts, generating large...

arXiv 2021
[11]

(2023): AI Alignment: A Comprehensive Survey

Ji, Jiaming; Qiu, Tianyi; Chen, Boyuan; Zhang, Borong; Lou, Hantao; Wang, Kaile et al. (2023): AI Alignment: A Comprehensive Survey. In arXiv:2310.19852, pp. 1–95. Luccioni, Alexandra Sasha; Akiki, Christopher; Mitchell, Margaret; Jernite, Yacine (2023): Stable Bias: Analyzing Societal Representations in Diffusion Models. In arXiv:2303.11408v1, Article 1-...

arXiv 2023
[68]

In arXiv:1907.11274v2, pp

Ovadya, Aviv; Whittlestone, Jess (2019): Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning. In arXiv:1907.11274v2, pp. 1–11. 10 Principi, Niccolò; Magnoni, Pietro; Grimoldi, Ludovico; Carnevali, Davide; Cavazzana, Laura; Pellai, Alberto (2019): Consumption of sexually explicit internet ...

arXiv 2019
[781]

A Comprehensive Scoping Review

Hagendorff, Thilo (2024): Mapping the Ethics of Generative AI. A Comprehensive Scoping Review. In Minds and Machines 34 (39), 1–27. Hao, Susan; Kumar, Piyush; Laszlo, Sarah; Poddar, Shivani; Radharapu, Bhaktipriya; Shelby, Renee (2023): Safety and Fairness for Content Moderation in Generative Models. In arXiv:2306.06135, pp. 1–

arXiv 2024
[2019]

polluting

, and also affect many fields of society in various negative ways by “polluting” public discourses and the infosphere in general (Ovadya and Whittlestone 2019). In line with that, the current research landscape on AI -based image generators stresses challenges surrounding safety, fairness, and privacy. Researchers have pointed out how these models can be ...

work page 2019
[2021]

, with GPT or Stable Diffusion models being prime examples (OpenAI 2024; Rombach et al

work page 2024

Show all 10 references

[2023]

Particularly, the generation of toxic, discriminatory, violent, pornographic, or otherwise harmful content should be avoided. Many closed source foundation models like ChatGPT implement built -in safety filters and leverage additional content filtering tools that in most cases...

2020
[2024]

With every new model generation, their capabilities are improving while their respective outputs are increasingly becoming part of the infosphere (Burton et al. 2024). Due to their societal as well as technological relevance, researchers refer to such models as foundation mode...

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [4]

Among the most popular distribution platforms for such models are Hugging Face and Civit AI

However, next to that, a large community is publishing open source models with generative capabilities that often lack any safety fine -tuning, despite ongoing 1 Corresponding author, matthias.schneider@student.hpi.uni-potsdam.de 2 discussions about the dual-use risks they present (Hagendorff 2021). Among the most popular distribution platforms for such m...

work page 2021

[2] [6]

safe” when they are not detected by any of our classifier’s category, and “unsafe

Furthermore, researchers show that image training datasets used to train text-to-image models often contain racist, sexist, or otherwise offensive material (Birhane et al. 2021; Thiel 2023). Our paper aligns with previous research on safety issues in image generation models, examining the capacity of these models to produce various types of harmful conten...

work page 2021

[3] [7]

concepts

In this study, we illustrate this by examining the capacity of stable diffusion models to produce various types of harmful content. Using a selection of the most popular models from Hugging Face and Civit AI, we observed no refusal behavior in response to harmful prompts. On the contrary, all models consistently complied with the prompts, generating large...

arXiv 2021

[4] [11]

(2023): AI Alignment: A Comprehensive Survey

Ji, Jiaming; Qiu, Tianyi; Chen, Boyuan; Zhang, Borong; Lou, Hantao; Wang, Kaile et al. (2023): AI Alignment: A Comprehensive Survey. In arXiv:2310.19852, pp. 1–95. Luccioni, Alexandra Sasha; Akiki, Christopher; Mitchell, Margaret; Jernite, Yacine (2023): Stable Bias: Analyzing Societal Representations in Diffusion Models. In arXiv:2303.11408v1, Article 1-...

arXiv 2023

[5] [68]

In arXiv:1907.11274v2, pp

Ovadya, Aviv; Whittlestone, Jess (2019): Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning. In arXiv:1907.11274v2, pp. 1–11. 10 Principi, Niccolò; Magnoni, Pietro; Grimoldi, Ludovico; Carnevali, Davide; Cavazzana, Laura; Pellai, Alberto (2019): Consumption of sexually explicit internet ...

arXiv 2019

[6] [781]

A Comprehensive Scoping Review

Hagendorff, Thilo (2024): Mapping the Ethics of Generative AI. A Comprehensive Scoping Review. In Minds and Machines 34 (39), 1–27. Hao, Susan; Kumar, Piyush; Laszlo, Sarah; Poddar, Shivani; Radharapu, Bhaktipriya; Shelby, Renee (2023): Safety and Fairness for Content Moderation in Generative Models. In arXiv:2306.06135, pp. 1–

arXiv 2024

[7] [2019]

polluting

, and also affect many fields of society in various negative ways by “polluting” public discourses and the infosphere in general (Ovadya and Whittlestone 2019). In line with that, the current research landscape on AI -based image generators stresses challenges surrounding safety, fairness, and privacy. Researchers have pointed out how these models can be ...

work page 2019

[8] [2021]

, with GPT or Stable Diffusion models being prime examples (OpenAI 2024; Rombach et al

work page 2024