REVIEW 13 cited by
Finetuning Text-to-Image Diffusion Models for Fairness
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
The rapid adoption of text-to-image diffusion models in society underscores an urgent need to address their biases. Without interventions, these biases could propagate a skewed worldview and restrict opportunities for minority groups. In this work, we frame fairness as a distributional alignment problem. Our solution consists of two main technical contributions: (1) a distributional alignment loss that steers specific characteristics of the generated images towards a user-defined target distribution, and (2) adjusted direct finetuning of diffusion model's sampling process (adjusted DFT), which leverages an adjusted gradient to directly optimize losses defined on the generated images. Empirically, our method markedly reduces gender, racial, and their intersectional biases for occupational prompts. Gender bias is significantly reduced even when finetuning just five soft tokens. Crucially, our method supports diverse perspectives of fairness beyond absolute equality, which is demonstrated by controlling age to a $75\%$ young and $25\%$ old distribution while simultaneously debiasing gender and race. Finally, our method is scalable: it can debias multiple concepts at once by simply including these prompts in the finetuning data. We share code and various fair diffusion model adaptors at https://sail-sg.github.io/finetune-fair-diffusion/.
Forward citations
Cited by 13 Pith papers
-
FairFlow: Demystifying and Mitigating Stereotype Bias in Text-to-Image Diffusion Transformers
Bias in MM-DiTs is mediated by sparse stage-wise semantic binding hubs, and sparse inference-time steering at those hubs mitigates gender, race, and intersectional stereotypes with low overhead.
-
On Fairness of Unified Multimodal Large Language Model for Image Generation
Most unified multimodal large language models generate images with strong gender and race bias, and a balanced preference optimization loss reduces this bias.
-
FAROS: Fair Graph Generation via Attribute Switching Mechanisms
FAROS switches sensitive attributes on an optimal fraction of nodes at an optimal diffusion step to produce fairer generated graphs with little accuracy loss.
-
Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generation
A user-controlled loop of LLM-suggested attributes, CLIP-based verification histograms, and probabilistic prompt sampling enables non-experts to steer the diversity of AI-generated image sets toward their own goals.
-
Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
Fine-tuning CLIP's visual encoder to match DINOv2's kernel-based similarity structure improves its fine-grained visual perception while preserving its alignment to text.
-
Multi-Group Proportional Representation for Text-to-Image Models
The authors apply the MPR metric (an integral probability metric) to text-to-image generation, derive tractable forms for linear and decision-tree function classes, and use it as a fine-tuning objective that reduces i...
-
T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation
T2ISafety is a large annotated benchmark plus a fine-tuned MLLM evaluator (ImageGuard) for measuring toxicity, privacy, and fairness in text-to-image models.
-
InvDiff: Invariant Guidance for Bias Mitigation in Diffusion Models
InvDiff adds a lightweight learnable guidance module to a frozen Stable Diffusion model, using invariant-learning-style grouping of training samples to reduce unknown biases without bias annotations.
-
Exploiting Watermark-Based Defense Mechanisms in Text-to-Image Diffusion Models for Unauthorized Data Usage
RATTAN removes DIAGNOSIS watermarks from Stable Diffusion models by regenerating protected images with partial diffusion and fine-tuning on ten cleaned samples, dropping detection to random guessing.
-
BioPro: Towards Difference-Aware Gender Fairness for Vision-Language Models
BioPro uses orthogonal projection on a gender-variation subspace to selectively debias vision-language models, reducing gender bias in neutral contexts while preserving explicit gender cues.
-
FairJudge: Abstention-Aware Multimodal Judges for Fairness and Alignment Evaluation in Text-to-Image Models
An abstention-aware, label-constrained MLLM protocol plus two synthetic benchmarks improves social-attribute and alignment evaluation of text-to-image models on most tested attributes over CLIP/DeepFace.
-
Can we Debias Social Stereotypes in AI-Generated Images? Examining Text-to-Image Outputs and User Perceptions
A rubric-based Social Stereotype Index shows prompt refinement lowers measured stereotypes in text-to-image outputs, but users often still prefer the stereotypical versions.
-
PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation
A fine-tuning method for imbalanced text-to-image generation that adds a product-of-Gaussians inspired consistency regularizer weighted by image similarity and inverse text density.
Discussion (0). Continue with ORCID to comment.