Beyond Overcorrection: Evaluating Diversity in T2I Models with DivBench

Felix Friedrich; Kristian Kersting; Manuel Brack; Patrick Schramowski; Thiemo Ganesha Welsch

REVIEW 3 major objections 5 minor 17 references

Beyond Overcorrection: Evaluating Diversity in T2I Models with DivBench

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-guided prompting and steering can raise image diversity without degrading contextual accuracy.

desk verdict Useful benchmark for over- and under-diversification in T2I, but the single-VLM labeling makes the headline rankings provisional. read the letter →

arxiv 2507.03015 v2 pith:K7I45UIS submitted 2025-07-02 cs.CL cs.CYcs.LG

Felix Friedrich , Thiemo Ganesha Welsch , Manuel Brack , Patrick Schramowski , Kristian Kersting This is my paper

classification cs.CLcs.CYcs.LG

keywords text-to-imagegenerationdemographicdiversityover-diversificationunder-diversificationLLM-guideddiversificationbenchmarkevaluationdiffusionmodelssemanticfidelity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the usual fix for biased text-to-image models—forcing demographic diversity—can overshoot and distort images when the prompt itself specifies a demographic attribute. To test both failure modes at once, the authors build DivBench, a set of 60 curated prompts split into cases where diversity is expected and cases where it is not, and score models with separate precision (no over-diversification) and recall (no under-diversification) metrics. Across commercial and open models they find that most models are under-diverse, while one model, Adobe Firefly, over-diversifies. The central positive claim is that a large language model can decide when diversity is appropriate and either steer generation or rewrite the prompt accordingly, and that these context-aware methods raise recall without sacrificing precision. If true, it means developers do not have to choose between inclusive representation and contextual accuracy.

What carries the argument

The load-bearing object is DivBench, a curated set of 60 prompts: 20 with no demographic assignment (e.g., 'farmer'), 20 with gender assigned by language, history, or institution (e.g., 'pope'), and 20 with race assigned (e.g., 'Maasai person'). Each generated image is labeled by LLaVA for gender and race, and scored with three task-specific metrics: precision, recall, and F1, defined relative to whether the prompt class demands diversity. Around this benchmark the paper wraps a context-aware diversification mechanism: a language model, Llama-3, fills in a structured template 'The face of a {example} is that of a [gender attributes] person from [racial attributes] descent', with majority voting over repeated completions, to decide which attributes are safe to diversify; those decisions then drive either FairDiffusion-style steering or rewritten prompts. FairDiffusion itself is an inference-time guidance method that steers generated images toward a target demographic attribute.

What would settle it

Take the 60 DivBench prompts, generate the same 100 images per prompt, and have independent human raters label gender and race; if disagreement with LLaVA is large enough to reorder prompt-rewriting versus Firefly on F1, the paper's central ranking is not stable under evaluator choice.

Watch

Extended reading notes

Core claim

The paper's central discovery is that under-diversification and over-diversification are separable, measurable failure modes, and that LLM-guided context awareness resolves the tradeoff between them. Using DivBench, the authors find that nearly all current T2I models have high precision but low recall: they rarely insert inappropriate attributes but systematically under-represent expected diversity. The exception is Adobe Firefly, which achieves high recall by inserting diverse attributes even where context forbids it. When the authors wrap the same base model, Stable Diffusion 3.5, in an LLM that decides which attributes are eligible for diversification and apply that decision either as selective FairDiffusion steering or as prompt rewriting, both precision and recall improve, with prompt rewriting scoring highest overall. The conclusion is that diversity and semantic fidelity are not in inherent tension, provided the diversification mechanism is conditioned on prompt context.

Load-bearing premise

The entire measurement rests on LLaVA's gender and race labels being accurate enough that label errors change fewer scores than the ranking gaps between models.

Editorial extensions

If this is right

Practitioners can evaluate fairness interventions with two independent numbers, precision and recall, so a method that merely adds diversity is distinguishable from one that adds appropriate diversity.
Context-aware prompting and steering can be applied on top of existing open models without retraining, since they only change the prompt or the guidance terms.
The DivBench results suggest that current commercial models have not solved under-diversification, so auditing tools like DivBench remain necessary.
An over-diverse model like Firefly can top an aggregate F1 ranking despite producing contextually incorrect outputs, meaning the choice of metric changes which model appears best.
Explicit demographic cues in prompts lead to better adherence than implicit ones, a pattern the paper highlights as worth further investigation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: an intersectional version of DivBench would probably show even more under-diversification, since the current metrics treat gender and race separately and a model can score high by diversifying one attribute at a time.
Beyond the paper: because LLaVA supplies all labels and the authors note it is biased, the ranking of methods should be rechecked with human labels; a large disagreement rate would change which approach appears best.
Beyond the paper: the same LLM template could be applied to non-demographic attributes such as age, disability, or body type, where contextual appropriateness is even harder to define and where the precision-recall framing would generalize naturally.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: DivBench is a genuinely useful addition to the T2I fairness toolbox. The distinction between assigned and unassigned demographic attributes is simple but exactly what's missing from most bias evaluations, and the precision/recall framing maps cleanly onto over- and under-diversification. The benchmark is small (60 items) but carefully curated, with Wikidata fact-checking for tricky cases like pope or Maharaja. That depth-over-breadth choice is defensible.

The evaluation of existing models (Fig. 2) and of diversification strategies (Fig. 3) is transparent and easy to follow. The central claim—that context-aware methods, especially prompt rewriting, can reduce under-diversity without provoking over-diversification—is consistent with the reported numbers and is a plausible, testable hypothesis. The paper does not oversell by claiming a new generation method; it offers a measurement tool and an initial read.

The soft spot is the one the authors themselves concede in the Limitations: all demographic labels come from a single VLM, LLaVA, using only two gender and four race buckets. There are no error bars, confusion matrices, or human validation rates, and the gaps between Prompt Rewriting and Context-aware FairDiffusion, or between Firefly and the others, are plausibly within LLaVA's error rate on race classification. If LLaVA systematically labels certain phenotypes as 'Western' regardless of the requested attribute, the prompt-rewriting method could look artificially precise. This does not kill the benchmark—the conceptual contribution stands—but it means the headline rankings should be read as provisional until a second labeler (human or another VLM) confirms them.

Minor issues: the artifact link is a placeholder, which is fine for double-blind but needs fixing on camera-ready; and the diversification experiments use only SD3.5 as base, which limits the generality of the strategy comparisons. Neither is fatal.

Bottom line: this is a useful, well-scoped benchmark paper that deserves serious peer review. A referee should push for a robustness analysis of the labeling, targeted human validation on a stratified subset, and significance testing. The central message will likely survive; the specific ordering of methods may not.

Referee Report

3 major / 5 minor

Summary. The paper introduces DIVBENCH, a benchmark of 60 text-to-image prompts annotated into three categories (unassigned demographic attributes, assigned gender, assigned race) using Wikipedia/Wikidata as external sources, together with precision, recall, and F1 metrics for quantifying under- and over-diversification. The authors evaluate nine text-to-image models on the benchmark and then compare three diversification strategies applied to Stable Diffusion 3.5: random attribute steering (FairDiffusion), an LLM-guided context-aware variant of FairDiffusion, and LLM-guided prompt rewriting. Their central claim is that context-aware diversification methods, particularly prompt rewriting, reduce under-diversity without introducing the over-diversification seen in non-contextual methods or in Adobe Firefly, thereby achieving a better precision-recall balance.

Significance. If the quantitative conclusions hold, DIVBENCH would be a useful, lightweight evaluation tool for a real problem: many existing diversification methods improve demographic representation but break contextual or historical fidelity. The benchmark construction is transparent and fact-checked via Wikidata, and the authors release their data and code, which are concrete assets for future work. The distinction between under- and over-diversification, and the proposal of task-specific precision/recall/F1 scores, are sensible and independent of the authors' prior FairDiffusion work. The main risk is that the headline ranking rests entirely on a single vision-language model's demographic labels with no reported error analysis; because the reported differences between several methods are small, label noise could change the conclusions. The evaluation of diversification strategies on only one base model also limits the generality of the central claim.

major comments (3)

[Sec. 3 (Experimental Setup) and Limitations] All precision, recall, and F1 values in Figures 2 and 3 are computed from LLaVA's assignments of images to two gender and four race categories, yet the manuscript reports no confusion matrix, no human-validation rate, and no error bars. The Limitations section concedes that such VLMs 'suffer from biases themselves and might be prone to errors,' but the load-bearing comparisons, such as Prompt Rewriting versus Context-aware FairDiffusion on precision in Fig. 3, are plausibly within the labeling error rate of a single VLM on race classification. Because a misclassification such as Indian versus Western or East-Asian versus Indian changes both numerator and denominator of precision and recall, the central ranking is not robust as reported. Please provide per-category confusion matrices or a stratified human-validated subset, and report bootstrap confidence intervals or a sensitivity analysis with an alternative labeler.
[Fig. 3 / Sec. 3 (Diversification Strategies)] The claim that context-aware methods, and specifically prompt rewriting, 'can already effectively address under-diversity while avoiding over-diversification' is supported only by experiments on a single base model, Stable Diffusion 3.5. Without applying the same three strategies to at least one other base model, the conclusion may be specific to SD3.5 rather than a general property of context-aware diversification. Please extend the evaluation to an additional base model or explicitly restrict the conclusion to SD3.5.
[Sec. 2.2 and Fig. 3] The aggregate precision, recall, and F1 scores pool 20 unassigned items, 20 assigned-gender items, and 20 assigned-race items, but the paper reports only combined scores and states that gender and race scores are combined 'for readability' without specifying the aggregation rule. This makes it impossible to see whether a method's overall balance comes from improving unassigned prompts or from respecting assigned-gender and assigned-race prompts, which is exactly the distinction the benchmark is designed to measure. Please report the scores separately for the three item categories, or at least state the aggregation formula and include per-category results in the appendix.

minor comments (5)

[Sec. 2.2, Eq. (2)] The definition of recall as '#correct labels / #true labels' is ambiguous for unassigned items, where the 'true labels' are a set of demographic attributes expected to appear across the 100 generated images rather than a single label per image; please define the formal averaging procedure.
[Fig. 2 and App. Table 2] The caption of Fig. 2 says 'Imagen' while the model list in Table 1 names 'Imagen 3'; please align the naming to avoid confusion about which checkpoint was evaluated.
[Sec. 2.3] The terms 'FairDiffusion' (the random steering baseline) and 'Context-Aware FairDiffusion' are close enough that a reader may conflate the two; consider renaming the random baseline, e.g., 'Random Attribute Steering,' to emphasize that it is a simplified variant.
[Sec. 3, first paragraph] The sentence 'We combine scores for gender and race for readability' should state how the combination is computed (e.g., macro-average over gender and race), since this affects every number reported in Figures 2 and 3.
[Abstract and Sec. 1] The footnote in the abstract contains the placeholder 'anony.mous' for the data/code repository; this should be replaced with the actual public URL before publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DIVBENCH is externally grounded and the diversification strategies are fixed methods evaluated against independent labels.

full rationale

The paper's central claim is that context-aware diversification methods improve the precision/recall balance relative to non-contextual baselines. This claim rests on DIVBENCH, whose ground-truth attribute assignments are manually collected and fact-checked against Wikipedia and Wikidata (Section 2.1), and on standard Precision, Recall, and F1 metrics defined in Section 2.2. The diversification methods themselves are fixed algorithmic strategies—random attribute steering, LLM-based context-aware steering, and prompt rewriting—rather than parameters fitted to the benchmark outcomes, so there is no fitted-input-called-prediction step. The only notable self-citation is FairDiffusion [7], but it is used as a baseline method, not as evidence for the paper's conclusions, and its prior publication is external to this benchmark. The acknowledged limitation that LLaVA 'might be prone to errors' is a measurement-validity concern, not a circularity: image labels come from a VLM, while attribute assignments for diversification come from Llama-3 and the benchmark's external annotations; no equation in the paper reduces a reported score to an input of the method being evaluated. No load-bearing step is equivalent by construction to its own input, so the paper is not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's correctness rests on hand-annotated categorical ground truth and on a single VLM classifier for all image labels; there are no fitted parameters in the traditional sense, so the ledger is dominated by domain assumptions rather than free numbers.

assumptions (4)

domain assumption Ground-truth demographic assignments for the 60 DIVBENCH prompts (e.g., pope implies male, actress implies female, Maharaja implies Indian) are correct and complete.
Section 2.1 states assignments were fact-checked via Wikipedia and Wikidata; all precision/recall labels depend on these assignments.
domain assumption Demographic attributes are discrete and can be reliably classified into two genders and four racial categories from a face image.
Section 3 defines the evaluation categories; the Limitations section acknowledges 'we treated those attributes as discrete states derived from visually distinguishable features'.
domain assumption LLaVA provides unbiased demographic classification.
Section 3 uses LLaVA as the classifier; Limitations admits 'Those models suffer from biases themselves and might be prone to errors', yet no correction or validation is performed.
domain assumption Unassigned prompts should yield full demographic diversity across the defined categories.
Section 2.1 states 'examples with no assignment, where full diversity is expected'; this normative assumption defines the recall target.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond Overcorrection: Evaluating Diversity in T2I Models with DivBench." pith.science (2026). https://pith.science/paper/K7I45UIS

@misc{pith2026250703015,
  author       = {Pith},
  title        = {Pith review of: Beyond Overcorrection: Evaluating Diversity in T2I Models with DivBench},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K7I45UIS}},
  note         = {Machine review of arXiv:2507.03015}
}

read the original abstract

Current diversification strategies for text-to-image (T2I) models often ignore contextual appropriateness, leading to over-diversification where demographic attributes are modified even when explicitly specified in prompts. This paper introduces DIVBENCH, a benchmark and evaluation framework for measuring both under- and over-diversification in T2I generation. Through systematic evaluation of state-of-the-art T2I models, we find that while most models exhibit limited diversity, many diversification approaches overcorrect by inappropriately altering contextually-specified attributes. We demonstrate that context-aware methods, particularly LLM-guided FairDiffusion and prompt rewriting, can already effectively address under-diversity while avoiding over-diversification, achieving a better balance between representation and semantic fidelity.

Figures

Figures reproduced from arXiv: 2507.03015 by the authors.

**Figure 1.** DIVBENCH methodology. Items with potential group-attribute associations are systematically verified against trusted sources, then categorized into assigned and unassigned groups to create a comprehensive diversity evaluation benchmark. These simplified definitions serve exclusively as methodological tools for detecting systematic bias patterns in AIgenerated imagery, not as normative statements about how groups sh… view at source ↗

**Figure 2.** Performance of T2I models on DIVBENCH, measured via precision, recall, and F1-score metrics. Models evaluated include DALL-E, Firefly, Flux-Dev, Flux-Schnell, Imagen, and multiple versions of Stable Diffusion (SD-1.5 through SD-3.5). While most models achieve high Precision, Recall varies widely, with Firefly attaining the highest Recall and F1-score. Random Attribute Steering. Our first approach builds on FairDif… view at source ↗

**Figure 4.** Wikidata queries with SQL/SPARQL and the obtained result. This example shows how to obtain information about attributes for [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 15 canonical work pages

[1]

Adobe firefly.https://www.adobe.com/ products/firefly.html, 2023

Adobe Inc. Adobe firefly.https://www.adobe.com/ products/firefly.html, 2023. 6

work page 2023
[2]

Llama 3 model card.https://github

AI@Meta. Llama 3 model card.https://github. com / meta - llama / llama3 / blob / main / MODEL _ CARD.md, 2024. 3

work page 2024
[3]

Improving image generation with better captions, 2023

James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions, 2023. 6

work page 2023
[4]

Flux.1: State-of-the-art image genera- tion.https://blackforestlabs.ai/, 2024

Black Forest Labs. Flux.1: State-of-the-art image genera- tion.https://blackforestlabs.ai/, 2024. 6

work page 2024
[5]

Scaling recti- fied flow transformers for high-resolution image synthesis

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M”uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InICML, 2024. 6

work page 2024
[6]

Pope election.https://en

Wikimedia Foundation. Pope election.https://en. wikipedia.org/wiki/Pope#Election, 2025. 1

work page 2025
[7]

Fair diffusion: Instructing text-to-image generation models on fairness.AI and Ethics, 2024

Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness.AI and Ethics, 2024. 1, 3

work page 2024
[8]

Gemini image generation got it wrong

Google. Gemini image generation got it wrong. we’ll do bet- ter.https://blog.google/products/gemini/ gemini-image-generation-issue/, 2024. 1

work page 2024

Show all 17 references

[9]

Imagen 3.https://deepmind

Google DeepMind. Imagen 3.https://deepmind. google/models/imagen, 2024. 6

2024
[10]

Imagen 3, 2024

Imagen-Team-Google. Imagen 3, 2024. 1

2024
[11]

Fair text-to-image diffusion via fair mapping

Jia Li, Lijie Hu, Jingfeng Zhang, Tianhang Zheng, Hua Zhang, and Di Wang. Fair text-to-image diffusion via fair mapping. InAAAI, 2025. 1

2025
[12]

Visual instruction tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023. 3

2023
[13]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj”orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022. 6

2022
[14]

Finetuning text-to-image diffusion models for fairness

Xudong Shen, Chao Du, Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhalli. Finetuning text-to-image diffusion models for fairness. InThe Twelfth International Conference on Learning Representations, 2024. 1

2024
[15]

Fairrag: Fair human genera- tion via fair retrieval augmentation

Robik Shrestha, Yang Zou, Qiuyu Chen, Zhiheng Li, Yusheng Xie, and Siqi Deng. Fairrag: Fair human genera- tion via fair retrieval augmentation. InCVPR, 2024. 1

2024
[16]

Survey of bias in text- to-image generation: Definition, evaluation, and mitigation,

Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Re- becca Pattichis, and Kai-Wei Chang. Survey of bias in text- to-image generation: Definition, evaluation, and mitigation,
[2024]

Appendix Model details.We evaluated several models as shown in Tab

1 A. Appendix Model details.We evaluated several models as shown in Tab. 1. DIVBENCH.We depict the full DIVBENCHin Tab. 2. Determining Attributes Using Wikidata.Fig. 4 presents example SQL/SPARQL queries used to retrieve data for specific groups from Wikidata. In Fig. 4a, we i...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Adobe firefly.https://www.adobe.com/ products/firefly.html, 2023

Adobe Inc. Adobe firefly.https://www.adobe.com/ products/firefly.html, 2023. 6

work page 2023

[2] [2]

Llama 3 model card.https://github

AI@Meta. Llama 3 model card.https://github. com / meta - llama / llama3 / blob / main / MODEL _ CARD.md, 2024. 3

work page 2024

[3] [3]

Improving image generation with better captions, 2023

James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions, 2023. 6

work page 2023

[4] [4]

Flux.1: State-of-the-art image genera- tion.https://blackforestlabs.ai/, 2024

Black Forest Labs. Flux.1: State-of-the-art image genera- tion.https://blackforestlabs.ai/, 2024. 6

work page 2024

[5] [5]

Scaling recti- fied flow transformers for high-resolution image synthesis

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M”uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InICML, 2024. 6

work page 2024

[6] [6]

Pope election.https://en

Wikimedia Foundation. Pope election.https://en. wikipedia.org/wiki/Pope#Election, 2025. 1

work page 2025

[7] [7]

Fair diffusion: Instructing text-to-image generation models on fairness.AI and Ethics, 2024

Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness.AI and Ethics, 2024. 1, 3

work page 2024

[8] [8]

Gemini image generation got it wrong

Google. Gemini image generation got it wrong. we’ll do bet- ter.https://blog.google/products/gemini/ gemini-image-generation-issue/, 2024. 1

work page 2024