Pith. sign in

REVIEW 4 major objections 8 minor 177 references

Universal toxicity detectors miss about a third of images that disability communities find harmful, so community-specific detection is urgently needed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 09:16 UTC pith:GCDCI7AR

load-bearing objection Solid empirical position paper: universal T2I guards miss ~35% disability-specific harms, zero-shot CTD fails, and light adaptation helps but stays far from general TD ceilings. the 4 major comments →

arxiv 2607.24898 v1 pith:GCDCI7AR submitted 2026-07-27 cs.CV cs.AI

Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed

Xinnuo Xu , Anja Thieme , Daniela Massiceti , Ioana Tanase , Rita Marques , Melanie Fernandez Pradier , Martin Grayson , Camilla Longden
show 1 more author
Cecily Morrison
This is my paper
classification cs.CV cs.AI
keywords community-specific toxicity detectiontext-to-image safetyrepresentational harmsdisabilityvision-language modelsin-context learningparameter-efficient fine-tuningAI safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This position paper argues that one-size-fits-all toxicity detectors for text-to-image models fail to protect marginalized communities from representational harms. Working with disability experts, the authors build safety guidelines for blind/low-vision and dwarfism communities and annotate 2,400 model-generated images. Roughly 32–37.5% of images that state-of-the-art detectors label safe violate those guidelines. In zero-shot tests, both large vision-language models and existing safety detectors score at or below random guessing. In-context learning, visual question answering, and small-data fine-tuning raise performance, but results stay far below the F1 near 0.9 common in general toxicity detection, and fine-tuned models only partly track guideline changes. The authors call for sustained community partnerships and new research on scarce data, evolving guidelines, and multi-community architectures.

Core claim

State-of-the-art toxicity detectors treat harm as universal and fixed. Against community-specific guidelines for blind/low-vision and dwarfism, about 32–37.5% of images they call safe are harmful. Zero-shot, general vision-language models and safety-specialized detectors reach F1 at or below random. Prompt-based adaptation and parameter-efficient fine-tuning with under 100 examples improve detection, yet scores remain well below F1 ≈ 0.9 for general toxicity detection, and fine-tuned models only partially adapt when harm categories are added or removed.

What carries the argument

Community-Specific Toxicity Detection (CTD): detectors that use particularist, dynamically evolving safety guidelines co-developed with community experts and adapted from scarce annotated generated images, rather than one fixed universal taxonomy of mainstream harms.

Load-bearing premise

That guidelines built with disability experts, validated by advocacy groups, and labeled mainly by a small expert team truly stand in for what those communities judge harmful.

What would settle it

Independent community members re-label the same 2,400 images under the same guidelines and find almost none of the detector-safe images unsafe, or zero-shot general detectors reach F1 near 0.9 on those labels without any community adaptation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • T2I safety systems must support multiple non-aggregable community guidelines instead of one fixed taxonomy.
  • Inference-time methods and low-data fine-tuning become necessary because guidelines evolve and labeled data stay scarce.
  • Inclusive coverage requires long-term researcher–community partnerships that engineering alone cannot shortcut.
  • Data efficiency and difficulty differ sharply by community, so uniform scaling cannot be assumed.
  • Fine-tuned detectors will need frequent updates or more flexible designs as new T2I failure modes appear.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar blind spots likely affect other marginalized groups whose representational harms sit outside mainstream taxonomies.
  • Deployed filters that only block universal categories can systematically under-protect groups most exposed to stereotyping.
  • Benchmarks limited to general-public harms can create a false sense of safety coverage.
  • Shared backbones with swappable community adapters or question sets may be a practical way to scale CTD.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This position paper argues that toxicity detection for text-to-image (T2I) models must be community-specific rather than universal. The authors collaborate with disability experts to construct safety guidelines for two communities — blind/low-vision (BLV) and people with dwarfism (DWF) — each formalized as six representational-harm categories validated by advocacy organizations. Five experts annotate 2,400 images (1,200 per community, from GPT-Image-1, Imagen4-Ultra, and SD3.5) generated from community-written prompts. The paper reports that ~32–37.5% of images pass SoTA detectors (LlavaGuard, ShieldGemma2) yet violate community guidelines; that general VLMs and safety-specialized detectors fail zero-shot on this task (many near-zero F1, best 0.35 BLV / 0.57 DWF); and that ICL, VQA, and LoRA fine-tuning on <100 examples improve detection substantially (best 0.50/0.78 for GPT-4o; 0.48/0.59 for small VLMs) but remain far below the F1 ≈ 0.9 typical of general TD. Ablations show fine-tuned models only partially adapt when harm categories are added or removed at inference time. The paper concludes with four research questions on learning from scarcity, adapting to evolving guidelines, multi-community architecture, and collaboration infrastructure.

Significance. If the results hold, this is a significant contribution: it supplies concrete, quantified evidence that the universal-detector paradigm in T2I safety leaves marginalized communities unprotected, and it does so with community-grounded taxonomies rather than researcher-invented labels. Strengths that deserve explicit credit: (i) safety guidelines developed with disability experts and validated by two advocacy organizations; (ii) a 2,400-image annotated dataset spanning three T2I generators, with community-authored prompts; (iii) careful empirical methodology — class-balanced low-data splits with prompt-disjoint test sets, random and CLIP-retrieval baselines, bootstrap confidence intervals, judgment–rationale consistency analysis, harm add/remove ablations, and a scaling study; (iv) falsifiable, specific claims (per-community F1 gaps) rather than a purely rhetorical position. The identification of ICL/VQA as understudied inference-time adaptation strategies for evolving guidelines is a useful conceptual contribution, and the four research questions are well-posed. The work sits squarely within a growing community-centered evaluation literature but operationalizes it in a way prior TD/

major comments (4)
  1. [§2.2 and Appendix D (Tables 4–5)] The BLV ground truth is too rater-dependent to support the precision of the headline BLV claims. IRR is estimated on only 100 images with three of five experts, yielding safety-label kappa = 0.648 and pairwise F1 = 0.770 (Table 4); the remaining ~1,100 BLV images are then labeled by a single expert. Because an image is unsafe iff any harm applies, disagreement concentrated in weakly reliable categories — E1 kappa = 0.412, E4 kappa = 0.000 (PSA 0.000), E6 kappa = 0.688 (Table 5) — directly flips the binary label. The 32.2% unsafe rate and all BLV F1 numbers (Tables 2, 10-11) may therefore partly reflect which expert labeled which image, and models whose harm perception resembles a different expert are penalized. I do not believe this overturns the qualitative conclusion (best zero-shot BLV F1 = 0.345 is far below the 0.770 expert-expert reference), but the magnitudes need a robustness arg
  2. [§2.3 (Rationale Collection) and Figure 2] Training and ICL demonstration rationales are retained only when GPT-4o's judgment agrees with the expert label (and, for unsafe images, only when GPT-4o independently identifies all annotated harms). Figure 2's blue counts show this filter retains uneven, sometimes small, fractions per harm. This has two load-bearing consequences: (i) the FT and ICL pools are systematically enriched for harms GPT-4o already recognizes, so the measured adaptation gains (Section 4-5) may not transfer to harms that are hardest for the model family; (ii) the same model (GPT-4o) is then used as the judgment-rationale consistency extractor (Section 2.3), so consistency rates are computed by a member of the best-performing model family, with extractor validation done only on the self-filtered training rationales (97.78%/100% exact match). This does not contaminate the human-labeled test set, and the F1 compari
  3. [Abstract and §1, vs. Tables 2 and 11] The abstract states that VLMs and detectors fail 'with F1 score lower than random guessing (F1 0.32 and 0.37)'. This overstates the result as written. The random baselines are 0.321 (BLV) and 0.375 (DWF), but GPT-4o scores 0.345 on BLV and GPT-4o-mini / Qwen-3b score 0.515 / 0.568 on DWF (Tables 2, 10, 11) — above random. The sub-random claim holds for the safety-specialized detectors (LlavaGuard/QwenGuard near zero; ShieldGemma2 0.27-0.31) and for most general VLMs, which is still a strong and publishable finding; the abstract and Section 1 should be qualified accordingly rather than implying all evaluated model classes are below random.
  4. [§2.2 vs. §3 Table 3] Section 2.2 states that 'all images are classified as safe by SoTA toxicity detectors such as LlavaGuard and ShieldGemma2', which grounds the ~35% gap claim. This is in tension with Table 3/Table 10, where ShieldGemma2 flags a non-trivial fraction of images as unsafe (F1 0.27-0.31, e.g., BLV P=0.28, R=0.26), i.e., it does not label all images safe. Presumably the 2.2 claim refers to running the detectors under their default (general-public) taxonomies rather than the community guidelines, but the text does not say this, does not name the full set of detectors used for the filtering claim, and does not report the per-detector safe rate. Please clarify exactly how the 'labeled safe' pool was established (which detectors, which guidelines, which thresholds) and reconcile the two statements; the 32.2%/37.5% figures are load-bearing for the paper's central empirical claim.
minor comments (8)
  1. The validation set shares prompts with the training set by design, and LoRA hyperparameters (16-config grid, Table 8-9) are selected on it. Since the test set is prompt-disjoint, this is acceptable, but the risk of optimistic hyperparameter selection via prompt leakage should be discussed in one sentence.
  2. Tables 5-6 report kappa = 'N/A' for several harm categories (presumably zero-variance cases where no rater flagged the harm in the 100-image sample). Please state this explicitly; readers may otherwise misread N/A as missing data.
  3. Table 7 caption contains an unresolved cross-reference ('Section ??').
  4. Typos/spacing: 'resutls' (§4.1), 'VLMsat' (§2.3), 'BLV ,' and 'andpeople' (Figure 1 caption), 'annoatated' (Table 1), 'Resutls' (§F.4 heading), 'Stabel diffusion' (references).
  5. Bootstrap CIs are reported for the zero-shot results (Table 11) but not for ICL/VQA/FT comparisons (Figure 3, Tables 12-15). Given the small test-relevant margins in some comparisons (e.g., ICL vs zero-shot for DWF), CIs or at least a note on variability across demonstration seeds would strengthen Section 4.
  6. The DWF E3 (infantilization) removal ablation in Section 5.1 uses the harm category with the lowest DWF IRR (kappa = 0.219, Table 6); relabeling decisions there inherit that noise. Worth one caveat sentence.
  7. Section 5.1's stabilized relative change uses epsilon = 0.05; the choice is reasonable but the 15-30% relative changes should be accompanied by the absolute deltas in the main text (they are only in the figures) so readers can judge materiality.
  8. It would help to state dataset/code release plans explicitly; reproducibility of the annotation interface (Figures 9-10) and prompt templates (Figures 13-23) is otherwise good.

Circularity Check

1 steps flagged

No load-bearing circular derivation: central CTD failure claims rest on independent human expert labels vs external detectors; only mild non-central coupling via GPT-4o rationale filtering.

specific steps
  1. other [Section 2.3 Rationale Collection; evaluation consistency (same section)]
    "We retain for further processing only those images for which GPT-4o's safety judgments align with human expert annotations. For images labeled safe, we directly adopt the GPT-4o generated rationales. For images labeled unsafe, we independently prompt GPT-4o to assess whether each expert-annotated harm is present... An image is retained only if GPT-4o correctly identifies all annotated harms... We therefore evaluate judgment-rationale consistency using GPT-4o as a scalable evaluator."

    Training demonstrations and rationales are filtered to cases GPT-4o already labels correctly, and the same model family is later reported as a top adapted detector and used as the consistency extractor. This couples training signal and part of the evaluation stack for adaptation results. It is not classical circularity: test labels remain human, zero-shot failure is independent, and no claimed quantity equals its input by definition—only a mild contamination risk for GPT-centric adaptation claims.

full rationale

This is an empirical position paper, not a first-principles derivation. The headline gap (~32–37.5% of images labeled safe by LlavaGuard/ShieldGemma2 that experts mark unsafe) is defined by human annotations under community guidelines, not by the models under test, so it is not true by construction. Zero-shot F1s compare those same external models to held-out human labels and fall at or below random baselines—again independent measurements. Adaptation gains (ICL/VQA/LoRA) are also scored against human test labels. Related-work citation of Thieme et al. (2026) supplies community prompts and engagement context from overlapping authors; that is normal research continuity, not a uniqueness theorem or ansatz that forces the F1 results. The only mild coupling is that training rationales are retained only when GPT-4o agrees with experts and GPT-4o is later among the strongest adapted detectors and the consistency judge—this can modestly favor GPT-family adaptation narratives but does not redefine the human-labeled test set or the zero-shot failure claim. No equation or claimed prediction reduces to its fitted input by construction. Score 1 reflects that minor non-central coupling only.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

As an empirical position paper, load-bearing premises are domain and methodological assumptions rather than free physical parameters: expert/community authority over harm definitions, adequacy of six-harm taxonomies, reliability of single-expert labels after sample IRR, and validity of using generated-image annotations and GPT-4o rationales as training signal. No invented physical entities; CTD is a task framing.

free parameters (4)
  • LoRA hyperparameters (lr, rank, dropout, target modules) = e.g. Qwen-7B BLV lr=1e-4 rank=4 dropout=0.1 targets q,v,o
    Selected by 16-config grid search per model–dataset on validation F1; best configs in Table 9 drive reported FT scores.
  • ICL demonstration counts = 5 BLV / 2 DWF
    Hand-chosen 5 (BLV) and 2 (DWF) single-harm demos; performance depends on this choice.
  • Training set construction caps = 90 BLV / 40 DWF training examples
    Up to 10 single-harm unsafe examples per category (5 for BLV E4; E2 excluded), matched safe count → 90/40 train sizes.
  • Stabilization epsilon in relative ΔF1 = 0.05
    ε=0.05 used when reporting relative per-harm F1 change under guideline edits.
axioms (5)
  • domain assumption Community-specific representational harms are not subsumed by universal toxicity categories (hate, violence, sexual, etc.) and must not be naively aggregated across communities.
    Table 1 and Section 1; load-bearing for why CTD is a separate problem from TD.
  • domain assumption Disability experts (HCI researchers with sustained engagement) plus advocacy-org validation can define and label harms that stand in for community judgment without full community annotation of all images.
    Section 2.1–2.2, Appendix C; ethical choice to limit community exposure to offensive outputs.
  • ad hoc to paper An image is unsafe iff at least one of E1–E6 applies under the written guideline.
    Operational labeling rule in Section 2.2; binary metrics treat this as ground truth.
  • ad hoc to paper GPT-4o agreement filtering yields reliable natural-language rationales for training and consistency evaluation.
    Section 2.3 rationale pipeline; extractor validated at 97.78%/100% exact match on training rationales only.
  • domain assumption Standard classification metrics (P/R/F1 with unsafe as positive) and judgment–rationale consistency suffice to assess CTD quality.
    Section 2.3 evaluation protocol.
invented entities (2)
  • Community-Specific Toxicity Detection (CTD) independent evidence
    purpose: Name and frame detection of representational harms under plural, dynamic, non-aggregable community guidelines with scarce generated-image labels.
    Introduced in abstract/Table 1 as extension of TD; task construct rather than physical entity; independent handle is the empirical benchmark itself.
  • Six-harm taxonomies for BLV and DWF (E1–E6 each) independent evidence
    purpose: Concrete safety guidelines for annotation and model prompting.
    Figures 6–7; developed with experts and validated with advocacy orgs; falsifiable via re-annotation or community disagreement studies.

pith-pipeline@v1.2.0-grok45-kimik3 · 32709 in / 3807 out tokens · 72597 ms · 2026-07-31T09:16:53.957923+00:00 · methodology

0 comments
read the original abstract

State-of-the-art toxicity detectors for text-to-image generation adopt a one-size-fits-all approach: a single universal model applying fixed safety guidelines to all users. Our empirical evidence shows that these detectors fail to shield marginalized communities: approximately 35% of generated images labeled safe are considered harmful by disability communities. In this position paper, we argue for community-specific toxicity detection (CTD). To demonstrate its feasibility, we collaborate with disability experts to develop safety guidelines for two communities: dwarfism and blind/low vision. Using a dataset of 2,400 annotated T2I-generated images we demonstrate that both large vision-language models and existing general-purpose toxicity detectors catastrophically fail to recognize harmful content under these guidelines in zero-shot settings with F1 score lower than random guessing (F1 0.32 and 0.37). Promisingly, prompt-based adaptation methods (ICL, VQA) substantially improve harm detection performance (GPT-4o: F1 0.50 and 0.78), while parameter-efficient fine-tuning improves smaller models (0.5b-7b with best F1 0.48 and 0.59) with less than 100 demonstrations, but remains sensitive to evolving guidelines. Despite these gains, CTD performance remains far below F1 $\approx 0.9$ achieved for general-purpose toxicity detection, highlighting the challenge and the need for sustained research effort.

Figures

Figures reproduced from arXiv: 2607.24898 by Anja Thieme, Camilla Longden, Cecily Morrison, Daniela Massiceti, Ioana Tanase, Martin Grayson, Melanie Fernandez Pradier, Rita Marques, Xinnuo Xu.

Figure 1
Figure 1. Figure 1: Examples of community-specific harmful content for [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of representational harms (E1-E6; see Section 2.1) identified by disability experts across all model-generated images (1,200 images per community). Horizontal bars and the counts shown at their right ends indicate the total frequency of each harm, revealing which harms are most prevalent in the collected images A per-T2I-model breakdown is provided in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance of zero-shot, ICL, VQA, and FT on CTD. Fine-tuning results are not reported [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Inference-time harm removal. Models are trained on the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Inference-Time Harm Addition (BLV). Models are trained on [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Toxicity taxonomy for people who are blind or have low vision (BLV). Disability Dwarfism Should not: - Show images that do not depict any people. - Show images that do not depict any people with dwarfism. - Show images that depict adults with dwarfism as children, young infants, or babies. This includes: a young face; children's clothes (e.g., nappies or suspenders) or accessories, (e.g. toys or balloons);… view at source ↗
Figure 7
Figure 7. Figure 7: Toxicity taxonomy for people with dwarfism (DWF). C Details of the Disability Experts The experts are HCI researchers with sustained engagement with the communities, working through an international NGO via established organisational partnerships. They have collaborated with these communities over an extended period; this study alone builds on three months of in-depth participatory engagement, following a … view at source ↗
Figure 8
Figure 8. Figure 8: Two more examples demonstrating the defined harm content in the toxicity taxonomy for [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Data annotation platform for BLV. 4 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Data annotation platform for DWF. Task Raw Agr. Kappa PSA Jaccard F1 Corr. BLV 0.840 0.648 0.770 0.626 0.770 0.650 DWF 0.987 0.971 0.981 0.963 0.981 0.971 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Three-panel visualization of harm-category distributions (Section 2.1) for individual image [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Safe and unsafe image counts for the BLV (left) and DWF (right) communities across [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: System prompt used to collect rationales from GPT-4o for BLV images labeled as [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: System prompt used to collect rationales from GPT-4o for DWF images labeled as [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: System prompt used to collect rationales from GPT-4o for DWF images labeled as [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Examples of GPT-4o-generated judgment rationales for model-generated images annotated [PITH_FULL_IMAGE:figures/full_fig_p025_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Examples of GPT-4o-generated judgment rationales for model-generated images annotated [PITH_FULL_IMAGE:figures/full_fig_p026_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Question generation prompting template used in VQA setup. The example shows the [PITH_FULL_IMAGE:figures/full_fig_p027_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: VQA evaluation prompting template. The example shows the BLV policy (in light gray); [PITH_FULL_IMAGE:figures/full_fig_p028_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Question-answer pairs generated by GPT-4o-mini from the BLV community safety [PITH_FULL_IMAGE:figures/full_fig_p028_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Question–answer pairs generated by GPT-4o-mini from the DWF community safety [PITH_FULL_IMAGE:figures/full_fig_p029_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Zero-shot prompting template. The example shows the BLV policy (in light gray); the [PITH_FULL_IMAGE:figures/full_fig_p030_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: In-context learning template with demonstration examples in chat format. The initial user [PITH_FULL_IMAGE:figures/full_fig_p031_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: The judgment-rationale consistency rate for models tested in zero-shot setup. Models [PITH_FULL_IMAGE:figures/full_fig_p033_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: The judgment-rationale consistency rate for models tested in ICL setup. Models receive [PITH_FULL_IMAGE:figures/full_fig_p034_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Judgment–rationale consistency rates in the fine-tuning (FT) setting. Solid dark bars [PITH_FULL_IMAGE:figures/full_fig_p034_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Judgment–rationale consistency rates in the second fine-tuning (FT) ablation setting. [PITH_FULL_IMAGE:figures/full_fig_p034_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Scaling law results on BLV. With more training data, the detection performance is not [PITH_FULL_IMAGE:figures/full_fig_p035_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Scaling law results on DWF. More training data yield better performance. [PITH_FULL_IMAGE:figures/full_fig_p035_29.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

177 extracted references · 1 canonical work pages

  1. [1]

    M., Sun, P., Shen, X., Khan, F

    Bakr, E. M., Sun, P., Shen, X., Khan, F. F., Li, L. E., and Elhoseiny, M. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 20041--20053, 2023

    2023

  2. [2]

    The problem with bias: Allocative versus representational harms in machine learning

    Barocas, S., Crawford, K., Shapiro, A., and Wallach, H. The problem with bias: Allocative versus representational harms in machine learning. In 9th Annual conference of the special interest group for computing, information and society, volume 1. New York, NY, 2017

    2017

  3. [3]

    V., and Pruthi, D

    Basu, A., Babu, R. V., and Pruthi, D. Inspecting the geographical representativeness of images from text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 5136--5147, 2023

    2023

  4. [5]

    L., Kane, S

    Bennett, C. L., Kane, S. K., and Harrington, C. N. Toward community-led evaluations of text-to-image ai representations of disability, health, and accessibility. In Proceedings of the 5th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pp.\ 256--270, 2025

    2025

  5. [6]

    Stela: a community-centred approach to norm elicitation for ai alignment

    Bergman, S., Marchal, N., Mellor, J., Mohamed, S., Gabriel, I., and Isaac, W. Stela: a community-centred approach to norm elicitation for ai alignment. Scientific Reports, 14 0 (1): 0 6616, 2024

    2024

  6. [8]

    Easily accessible text-to-image generation amplifies demographic stereotypes at large scale

    Bianchi, F., Kalluri, P., Durmus, E., Ladhak, F., Cheng, M., Nozza, D., Hashimoto, T., Jurafsky, D., Zou, J., and Caliskan, A. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, pp.\ 1493--1504, 2023 b

    2023

  7. [9]

    Typology of risks of generative text-to-image models

    Bird, C., Ungless, E., and Kasirzadeh, A. Typology of risks of generative text-to-image models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp.\ 396--410, 2023

    2023

  8. [11]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  9. [12]

    and She, J

    Cetinic, E. and She, J. Understanding and creating art with ai: Review and outlook. ACM transactions on multimedia computing, communications, and applications (TOMM), 18 0 (2): 0 1--22, 2022

    2022

  10. [14]

    D., Upasani, K., and Pasupuleti, M

    Chi, J., Karn, U., Zhan, H., Smith, E., Rando, J., Zhang, Y., Plawiak, K., Coudert, Z. D., Upasani, K., and Pasupuleti, M. Llama guard 3 vision: Safeguarding human-ai image understanding conversations, 2024. URL https://arxiv.org/abs/2411.10414

    Pith/arXiv arXiv 2024

  11. [15]

    Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models

    Cho, J., Zala, A., and Bansal, M. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 3043--3054, 2023

    2023

  12. [18]

    X., Caselli, N., et al

    Desai, A., Berger, L., Minakov, F., Milano, N., Singh, C., Pumphrey, K., Ladner, R., Daum \'e III, H., Lu, A. X., Caselli, N., et al. Asl citizen: a community-sourced dataset for advancing isolated sign language recognition. Advances in Neural Information Processing Systems, 36: 0 76893--76907, 2023

    2023

  13. [19]

    Openbias: Open-set bias detection in text-to-image generative models

    D'Inc\`a, M., Peruzzo, E., Mancini, M., Xu, D., Goel, V., Xu, X., Wang, Z., Shi, H., and Sebe, N. Openbias: Open-set bias detection in text-to-image generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 12225--12235, June 2024

    2024

  14. [20]

    A., and Curtis, R

    Ellis, K., Goggin, G., Haller, B. A., and Curtis, R. (eds.). The Routledge Companion to Disability and Media. Routledge, New York, 2020. URL https://doi.org/10.4324/9781315716008

    work page doi:10.4324/9781315716008 2020

  15. [22]

    Geneval: An object-focused framework for evaluating text-to-image alignment

    Ghosh, D., Hajishirzi, H., and Schmidt, L. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 0 52132--52152, 2023

    2023

  16. [24]

    N., Gautam, S., Wilson, S., and Caliskan, A

    Ghosh, S., Venkit, P. N., Gautam, S., Wilson, S., and Caliskan, A. Do generative ai models output harm while representing non-western cultures: Evidence from a community-centered approach. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pp.\ 476--489, 2024 b

    2024

  17. [27]

    Imagen-4-ultra

    Google. Imagen-4-ultra. URL https://cloud.google.com/vertex-ai/generative-ai/docs/models/imagen/4-0-ultra-generate-001

  18. [28]

    Gemma-3-4b-pt

    Google DeepMind . Gemma-3-4b-pt. https://deepmind.google/models/gemma/gemma-3/, 2025. Family of lightweight open models with multimodal capabilities

    2025

  19. [29]

    D., Rosen, R., and Vasserman, L

    Goyal, N., Kivlichan, I. D., Rosen, R., and Vasserman, L. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. Proceedings of the ACM on Human-Computer Interaction, 6 0 (CSCW2): 0 1--28, 2022

    2022

  20. [30]

    Tiam - a metric for evaluating alignment in text-to-image generation

    Grimal, P., Le Borgne, H., Ferret, O., and Tourille, J. Tiam - a metric for evaluating alignment in text-to-image generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.\ 2890--2899, January 2024

    2024

  21. [33]

    Y., Lambert, N., Choi, Y., and Dziri, N

    Han, S., Rao, K., Ettinger, A., Jiang, L., Lin, B. Y., Lambert, N., Choi, Y., and Dziri, N. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Advances in Neural Information Processing Systems, 37: 0 8093--8131, 2024

    2024

  22. [35]

    L., and Choi, Y

    Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y. CLIPScore: A reference-free evaluation metric for image captioning . 2021

    2021

  23. [36]

    GANs trained by a two time-scale update rule converge to a local Nash equilibrium

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium . volume 30, 2017

    2017

  24. [37]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

    2022

  25. [38]

    Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., and Smith, N. A. TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering . In ICCV, pp.\ 20406--20417, 2023

    2023

  26. [39]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Huang, K., Sun, K., Xie, E., Li, Z., and Liu, X. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36: 0 78723--78747, 2023

    2023

  27. [40]

    T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation

    Huang, K., Duan, C., Sun, K., Xie, E., Li, Z., and Liu, X. T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation . IEEE Transactions on Pattern Analysis Machine Intelligence, 0 (01): 0 1--17, January 5555. ISSN 1939-3539. URL https://doi.ieeecomputersociety.org/10.1109/TPAMI.2025.3531907

    arXiv 1939

  28. [42]

    Rethinking fid: Towards a better evaluation metric for image generation

    Jayasumana, S., Ramalingam, S., Veit, A., Glasner, D., Chakrabarti, A., and Kumar, S. Rethinking fid: Towards a better evaluation metric for image generation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 9307--9315, 2024. doi:10.1109/CVPR52733.2024.00889

    arXiv 2024

  29. [43]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset

    Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Chen, B., Sun, R., Wang, Y., and Yang, Y. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36: 0 24678--24704, 2023

    2023

  30. [44]

    and Joo, J

    Karkkainen, K. and Joo, J. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp.\ 1548--1558, 2021

    2021

  31. [46]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation. volume 36, pp.\ 36652--36663, 2023

    2023

  32. [47]

    Improved precision and recall metric for assessing generative models

    Kynk \"a \"a nniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assessing generative models. volume 32, 2019

    2019

  33. [48]

    S., Gupta, A., Zhang, Y., Narayanan, D., Teufel, H., Bellagente, M., et al

    Lee, T., Yasunaga, M., Meng, C., Mai, Y., Park, J. S., Gupta, A., Zhang, Y., Narayanan, D., Teufel, H., Bellagente, M., et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36: 0 69981--70011, 2023

    2023

  34. [50]

    T2isafety: Benchmark for assessing fairness, toxicity, and privacy in image generation

    Li, L., Shi, Z., Hu, X., Dong, B., Qin, Y., Liu, X., Sheng, L., and Shao, J. T2isafety: Benchmark for assessing fairness, toxicity, and privacy in image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 13381--13392, 2025

    2025

  35. [52]

    Evaluating text-to-visual generation with image-to-text generation

    Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pp.\ 366--384. Springer, 2024

    2024

  36. [53]

    and et al

    Liu, H. and et al. Visual instruction tuning. In arXiv preprint, 2023. URL https://arxiv.org/abs/2304.08485. Introduces the LLaVA (Large Language and Vision Assistant)

    Pith/arXiv arXiv 2023

  37. [54]

    E., and Wang, W

    Lu, Y., Yang, X., Li, X., Wang, X. E., and Wang, W. Y. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in neural information processing systems, 36: 0 23075--23093, 2023

    2023

  38. [57]

    T., Cutrell, E., Morrison, C., Hofmann, K., and Stumpf, S

    Massiceti, D., Zintgraf, L., Bronskill, J., Theodorou, L., Harris, M. T., Cutrell, E., Morrison, C., Hofmann, K., and Stumpf, S. Orbit: A real-world few-shot dataset for teachable object recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 10818--10828, 2021

    2021

  39. [59]

    Llama-3.2-11b-vision-instruct, 2025

    Meta. Llama-3.2-11b-vision-instruct, 2025. URL https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/. Open-source large vision instruct model (Llama 3.2 family)

    2025

  40. [60]

    Harm categories in azure ai content safety

    Microsoft. Harm categories in azure ai content safety. URL https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/harm-categories?tabs=definitions

  41. [61]

    Gpt image 1

    OpenAI. Gpt image 1. a . URL https://platform.openai.com/docs/models/gpt-image-1

  42. [62]

    OpenAI. Sora. b . URL https://openai.com/index/sora/

  43. [63]

    Dall-e 3

    OpenAI. Dall-e 3. 2022. URL https://openai.com/index/dall-e-3/

    2022

  44. [64]

    Gpt-4o, 2024

    OpenAI . Gpt-4o, 2024. URL https://openai.com/index/hello-gpt-4o/. Multimodal model supporting text and image inputs

    2024

  45. [65]

    Introducing gpt-5, 2025

    OpenAI . Introducing gpt-5, 2025. URL https://openai.com/gpt-5. Multimodal foundation model released August 7 2025

    2025

  46. [66]

    Toward verifiable and reproducible human evaluation for text-to-image generation

    Otani, M., Togashi, R., Sawai, Y., Ishigami, R., Nakashima, Y., Rahtu, E., Heikkil \"a , J., and Satoh, S. Toward verifiable and reproducible human evaluation for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14277--14286, 2023

    2023

  47. [67]

    Let's tackle representations of dwarfism in the media, 2024

    Pritchard, E. Let's tackle representations of dwarfism in the media, 2024. URL https://www.hope.ac.uk/news/allnews/lets-tackle-representations-of-dwarfism-in-the-media.html. Liverpool Hope University News

    2024

  48. [71]

    High-resolution image synthesis with latent diffusion models

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

    2022

  49. [72]

    L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35: 0 36479--36494, 2022

    2022

  50. [73]

    Improved techniques for training gans

    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. volume 29, 2016

    2016

  51. [74]

    everyone wants to do the model work, not the data work

    Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., and Aroyo, L. M. “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp.\ 1--15, 2021

    2021

  52. [75]

    Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models

    Schramowski, P., Brack, M., Deiseroth, B., and Kersting, K. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 22522--22531, 2023

    2023

  53. [77]

    L., and Kursuncu, U

    Sheth, A., Shalin, V. L., and Kursuncu, U. Defining and detecting toxicity on social media: context and knowledge are key. Neurocomputing, 490: 0 312--318, 2022

    2022

  54. [78]

    Shi, J., Xiong, W., Lin, Z., and Jung, H. J. Instantbooth: Personalized text-to-image generation without test-time finetuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 8543--8552, 2024

    2024

  55. [79]

    and Zheng, L

    Singh, J. and Zheng, L. Divide, evaluate, and refine: Evaluating and improving text-to-image alignment with iterative VQA feedback . volume 36, pp.\ 70799--70811, 2023

    2023

  56. [80]

    Stabel diffusion 3.5

    Stability.ai. Stabel diffusion 3.5. URL https://stability.ai/news/introducing-stable-diffusion-3-5

  57. [81]

    and Caliskan, A

    Steed, R. and Caliskan, A. Image representations learned with unsupervised pre-training contain human-like biases. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp.\ 701--713, 2021

    2021

  58. [82]

    Suggs, D. W. and Guthrie, J. L. Disabling prejudice: A case study of images of paralympic athletes and attitudes toward people with disabilities. International Journal of Sport Communication, 10 0 (2): 0 258--276, 2017. URL https://www.researchgate.net/publication/317286559_Disabling_Prejudice_A_Case_Study_of_Images_of_Paralympic_Athletes_and_Attitudes_To...

    arXiv 2017

  59. [83]

    Is a seat at the table enough? engaging teachers and students in dataset specification for ml in education

    Tan, M., Lee, H., Wang, D., and Subramonyam, H. Is a seat at the table enough? engaging teachers and students in dataset specification for ml in education. Proceedings of the ACM on Human-Computer Interaction, 8 0 (CSCW1): 0 1--32, 2024 a

    2024

  60. [84]

    EVALALIGN: Supervised fine-tuning multimodal llms with human-aligned data for evaluating text-to-image models

    Tan, Z., Yang, X., Qin, L., Yang, M., Zhang, C., and Li, H. EVALALIGN: Supervised fine-tuning multimodal llms with human-aligned data for evaluating text-to-image models . arXiv preprint arXiv:2406.16562, 2024 b

    Pith/arXiv arXiv 2024

  61. [85]

    Meta llama guard 2

    Team, L. Meta llama guard 2. https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md, 2024

    2024

  62. [86]

    Qwen2.5-vl-7b-instruct

    Team, Q. Qwen2.5-vl-7b-instruct. https://qwenlm.github.io/blog/qwen2.5-vl/, 2025. Vision-language model with structured vision and language understanding

    2025

  63. [89]

    The representation of disabled women and recent disabled women-led media

    Wilde, A. The representation of disabled women and recent disabled women-led media. Disability & Society, 37 0 (3): 0 522--527, 2022

    2022

  64. [90]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023

    Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023. URL https://arxiv.org/abs/2306.09341

    Pith/arXiv arXiv 2023

  65. [91]

    Imagereward: learning and evaluating human preferences for text-to-image generation

    Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp.\ 15903--15935, 2023

    2023

  66. [92]

    VisionReward: Fine-grained multi-dimensional human preference learning for image and video generation

    Xu, J., Huang, Y., Cheng, J., Yang, Y., Xu, J., Wang, Y., Duan, W., Yang, S., Jin, Q., Li, S., et al. VisionReward: Fine-grained multi-dimensional human preference learning for image and video generation . arXiv preprint arXiv:2412.21059, 2024

    Pith/arXiv arXiv 2024

  67. [93]

    Sneakyprompt: Jailbreaking text-to-image generative models

    Yang, Y., Hui, B., Yuan, H., Gong, N., and Cao, Y. Sneakyprompt: Jailbreaking text-to-image generative models. In 2024 IEEE symposium on security and privacy (SP), pp.\ 897--912. IEEE, 2024

    2024

  68. [94]

    What you see is what you read? Improving text-image alignment evaluation

    Yarom, M., Bitton, Y., Changpinyo, S., Aharoni, R., Herzig, J., Lang, O., Ofek, E., and Szpektor, I. What you see is what you read? Improving text-image alignment evaluation . volume 36, pp.\ 1601--1619, 2023

    2023

  69. [95]

    Shieldgemma: Generative ai content moderation based on gemma, 2024

    Zeng, W., Liu, Y., Mullins, R., Peran, L., Fernandez, J., Harkous, H., Narasimhan, K., Proud, D., Kumar, P., Radharapu, B., Sturman, O., and Wahltinez, O. Shieldgemma: Generative ai content moderation based on gemma, 2024. URL https://arxiv.org/abs/2407.21772

    Pith/arXiv arXiv 2024

  70. [96]

    and Haller, B

    Zhang, L. and Haller, B. Consuming image: How mass media impact the identity of people with disabilities. Communication Quarterly, 61 0 (3): 0 319--334, 2013. doi:10.1080/01463373.2013.776988. URL https://doi.org/10.1080/01463373.2013.776988

    arXiv 2013

  71. [98]

    A., Shechtman, E., and Wang, O

    Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pp.\ 586--595, 2018

    2018

  72. [99]

    Learning multi-dimensional human preference for text-to-image generation

    Zhang, S., Wang, B., Wu, J., Li, Y., Gao, T., Zhang, D., and Wang, Z. Learning multi-dimensional human preference for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 8018--8027, 2024 b

    2024

  73. [100]

    FirstName LastName , title =

  74. [101]

    FirstName Alpher , title =

  75. [102]

    Journal of Foo , volume = 13, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe , title =. Journal of Foo , volume = 13, number = 1, pages =

  76. [103]

    Journal of Foo , volume = 14, number = 1, pages =

    FirstName Alpher and FirstName Fotheringham-Smythe and FirstName Gamow , title =. Journal of Foo , volume = 14, number = 1, pages =

  77. [104]

    FirstName Alpher and FirstName Gamow , title =

  78. [105]

    Kulynych, Bogdan and Madras, David and Milli, Smitha and Raji, Inioluwa Deborah and Zhou, Angela, and Zemel, Richard , title=

  79. [106]

    Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , articleno =

    Cooper, Ned and Zafiroglu, Alexandra , title =. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , articleno =. 2024 , isbn =. doi:10.1145/3613904.3642775 , abstract =

    arXiv 2024

  80. [107]

    ArXiv , year=

    Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics , author=. ArXiv , year=

Showing first 80 references.