REVIEW 7 cited by
Finetuning Text-to-Image Diffusion Models for Fairness
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
The rapid adoption of text-to-image diffusion models in society underscores an urgent need to address their biases. Without interventions, these biases could propagate a skewed worldview and restrict opportunities for minority groups. In this work, we frame fairness as a distributional alignment problem. Our solution consists of two main technical contributions: (1) a distributional alignment loss that steers specific characteristics of the generated images towards a user-defined target distribution, and (2) adjusted direct finetuning of diffusion model's sampling process (adjusted DFT), which leverages an adjusted gradient to directly optimize losses defined on the generated images. Empirically, our method markedly reduces gender, racial, and their intersectional biases for occupational prompts. Gender bias is significantly reduced even when finetuning just five soft tokens. Crucially, our method supports diverse perspectives of fairness beyond absolute equality, which is demonstrated by controlling age to a $75\%$ young and $25\%$ old distribution while simultaneously debiasing gender and race. Finally, our method is scalable: it can debias multiple concepts at once by simply including these prompts in the finetuning data. We share code and various fair diffusion model adaptors at https://sail-sg.github.io/finetune-fair-diffusion/.
Forward citations
Cited by 7 Pith papers
-
FairFlow: Demystifying and Mitigating Stereotype Bias in Text-to-Image Diffusion Transformers
Bias in MM-DiTs is mediated by sparse stage-wise semantic binding hubs, and sparse inference-time steering at those hubs mitigates gender, race, and intersectional stereotypes with low overhead.
-
FAROS: Fair Graph Generation via Attribute Switching Mechanisms
FAROS switches sensitive attributes on an optimal fraction of nodes at an optimal diffusion step to produce fairer generated graphs with little accuracy loss.
-
Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
Fine-tuning CLIP's visual encoder to match DINOv2's kernel-based similarity structure improves its fine-grained visual perception while preserving its alignment to text.
-
Multi-Group Proportional Representation for Text-to-Image Models
The authors apply the MPR metric (an integral probability metric) to text-to-image generation, derive tractable forms for linear and decision-tree function classes, and use it as a fine-tuning objective that reduces i...
-
BioPro: Towards Difference-Aware Gender Fairness for Vision-Language Models
BioPro uses orthogonal projection on a gender-variation subspace to selectively debias vision-language models, reducing gender bias in neutral contexts while preserving explicit gender cues.
-
FairJudge: Abstention-Aware Multimodal Judges for Fairness and Alignment Evaluation in Text-to-Image Models
An abstention-aware, label-constrained MLLM protocol plus two synthetic benchmarks improves social-attribute and alignment evaluation of text-to-image models on most tested attributes over CLIP/DeepFace.
-
Can we Debias Social Stereotypes in AI-Generated Images? Examining Text-to-Image Outputs and User Perceptions
A rubric-based Social Stereotype Index shows prompt refinement lowers measured stereotypes in text-to-image outputs, but users often still prefer the stereotypical versions.
Discussion (0). Sign in to comment.