{"id":"b848bd6d-609c-465f-83b9-e2395bddcb8b","arxiv_id":"2506.11025","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Text-to-image models link facial attractiveness to unrelated positive traits, and gender classifiers misclassify faces generated with negative trait labels more often, with the largest effects for non-White women.","lead":"Researchers generated 13,200 synthetic faces with Stable Diffusion and found that faces made from positive trait words, like 'intelligent' or 'trustworthy', look more similar to faces made from the word 'attractive', especially for Asian and Black women. They also found that three gender classification algorithms make more mistakes on female faces made from negative trait words, such as 'unhappy' or 'unsociable'.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The results rest on unvalidated prompt-to-attribute operationalization; human ratings and an independent embedding are needed before 'lookism' can be concluded.","rationale":"The reader and I converge on the same weak link: the mapping from prompt adjective to perceived facial attribute is never validated, and the metric used to establish it (CLIP centroid distance) is itself a learned semantic space that may encode the very associations being tested. This is load-bearing because both headline findings pass through this mapping. I do not see an internal inconsistency; the paper is transparent about not measuring attractiveness and lists CLIP disentanglement as future work. But transparency does not reduce the evidential gap: without human ratings or an independent embedding, the results are consistent with the stated interpretation and with at least two artifact explanations (CLIP semantic bias, and visual confounds such as makeup, expression, and age). A conditional verdict is appropriate: report the measurements, but reframe the claims as about prompt-label associations until the operationalization is validated. The limitations are honestly acknowledged, so no rejection is warranted; the evidence is preliminary rather than conclusive.","tokens_in":7281,"tokens_out":4552,"duration_ms":46876,"concrete_test":"Run a human-rating validation on a stratified sample of the generated images: e.g., 50 images per attribute×gender×race cell (1,200 total), rated by independent annotators for attractiveness and the four traits. Then test whether prompt polarity predicts mean human ratings, and recompute the CLIP centroid-distance analysis using only images whose human-rated attractiveness matches the prompt label. Separately, repeat the centroid analysis with an independent face embedding (e.g., DINOv2 or FaceNet) instead of CLIP. If the associations disappear or invert, the current measurements do not support the lookism claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim treats prompt labels as ground truth for facial attractiveness and for each trait, but the paper explicitly states it 'does not define or measure attractiveness' (Section 4). No human ratings, face-attribute classifiers, or independent image-level checks confirm that 'intelligent' faces are perceived as intelligent or that 'unattractive' faces are perceived as unattractive. The CLIP centroid-distance metric is also suspect: Stable Diffusion 2.1 is conditioned with a CLIP text encoder, and the same embedding family is used for evaluation, so the observed association between 'attractive' and positive traits could originate in CLIP's semantic space rather than in the generated images. The paper lists CLIP-bias disentanglement only as future work (Section 4, item 1). Additionally, the gender-classification result is interpreted as 'higher error rates on less-attractive faces,' but the paper itself observes that negative-trait female faces appear older, have neutral/downward expressions, and often lack makeup (Section 4); these visual confounds, not attractiveness per se, could drive classifier errors. Thus both headline findings are conditional on an unvalidated operationalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an experimental study of algorithmic lookism in Stable Diffusion 2.1. The authors generate 13,200 face images varying gender, race, and five trait pairs (attractive/unattractive, intelligent/unintelligent, trustworthy/untrustworthy, sociable/unsociable, happy/unhappy), compute CLIP embedding centroids per group, and measure Euclidean distances between positive/negative trait groups and the attractive/unattractive groups. They also evaluate three gender classifiers (InsightFace, DeepFace, FairFace) on the same images. The two central claims are: (1) T2I models associate facial attractiveness with unrelated positive traits and unattractiveness with negative traits; and (2) gender classifiers have higher error rates on 'less-attractive' faces, especially Asian and Black women.","tokens_in":7409,"tokens_out":3111,"duration_ms":75472,"significance":"If the operationalization were validated, this would be a useful, compact demonstration of lookism in a current T2I model and its downstream effect on gender classifiers. The study has clear strengths: no model parameters are fitted, the evaluation uses three external classifiers, the prompt design is simple and reproducible, and the authors are transparent about several limitations. However, the two headline claims rest on an unvalidated mapping from prompt adjectives to perceived facial attributes, and the first claim additionally uses CLIP embeddings from the same model family that conditions the generator. These issues are load-bearing rather than cosmetic, so the current evidence supports a conditional finding rather than a definitive one.","major_comments":[{"comment":"The first headline claim ('T2I systems tend to associate facial attractiveness with unrelated positive traits') is vulnerable to circularity because Stable Diffusion 2.1 is conditioned with a CLIP text encoder and the evaluation uses CLIP image embeddings. Although no parameters are fitted, the observed centroid proximity between 'attractive' and 'intelligent' faces, for example, could partly reflect proximity of the corresponding prompt embeddings in CLIP's semantic space rather than a visual property of the generated images. The manuscript itself lists CLIP-bias disentanglement only as future work (Section 4, item 1), but this is not a peripheral issue: it concerns the validity of the paper's first main result. I recommend re-running the analysis with an independent image embedding or, better, adding human attractiveness and trait ratings on a sample of the generated images.","section":"Section 2, similarity score definition; Section 4, item (1)"},{"comment":"The paper explicitly states that it 'does not define or measure attractiveness,' yet both main findings treat prompt-derived labels such as 'attractive,' 'unattractive,' 'intelligent,' and 'unintelligent' as ground truth for the perceived facial attribute. No human ratings, face-attribute classifiers, or independent image-level checks confirm that positive-trait faces are actually perceived as positive or that 'unattractive' faces are perceived as unattractive. Without such validation, the measured CLIP distances and classifier error patterns could reflect prompt semantics or other visual correlates rather than the constructs named in the paper.","section":"Section 2, prompt design; Section 4, 'we do not define or measure attractiveness'"},{"comment":"The second headline claim attributes higher gender-classifier error rates to 'less-attractive' faces, but the paper itself observes that negative-trait female faces appear older, have neutral or downward expressions, and often lack makeup (Section 4, citing Doh et al. [10]). These are visual confounds that could independently drive classifier errors, as the paper notes when citing Muthukumar et al. [27]. The current experimental design cannot separate 'attractiveness' from age, expression, and makeup, so the statement that classifiers exhibit higher error rates on 'less-attractive' faces goes beyond what the data support without an additional control or covariate analysis.","section":"Section 3, gender classification results; Section 4, confounding visual cues"},{"comment":"The description of the statistical test is under-specified. The paper says 'A two-sided t-test was conducted to assess the statistical significance of the centroid distance computed,' but it does not state what the units of comparison are, what the null hypothesis is, or how the 36 cells in Figure 2 are adjusted for multiple comparisons. Since each group has a single centroid, it is unclear whether the test compares per-image distances to the two reference centroids, per-image projections, or something else. Please specify the test procedure precisely, including sample sizes and correction for multiple testing.","section":"Section 2, statistical test; Section 3, significance markers in Figure 2"}],"minor_comments":[{"comment":"The abstract contains a grammatical error: 'These result raise' should be 'These results raise.' The same phrasing appears in the introduction.","section":"Abstract and Section 1"},{"comment":"The sentence 'we do not define or measure attractiveness, but focus on analyzing how T2I models associate attractiveness, or it's lack thereof' contains a typo: 'it's' should be 'its.'","section":"Section 2, last paragraph"},{"comment":"The phrase 'the faces of generated with the negative attributes' is missing a word; it should read 'the faces generated with the negative attributes.'","section":"Section 3, DeepFace paragraph"},{"comment":"References [20] and [21] are duplicate entries for the same FairFace paper. Please merge them and renumber.","section":"References"},{"comment":"The heatmaps would benefit from an explicit color scale or value labels, since the text relies on visual comparison of small numeric differences. Also, Figure 3 reports means and standard deviations in the caption, but the figure itself does not show error bars; please clarify whether the heatmap cells are means and how variability is displayed.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its preliminary nature and its limitations, and the dataset is a useful resource. However, the two headline claims are currently conditional on an unvalidated operationalization of attractiveness and on prompt-to-image trait fidelity. For a journal-level record, the authors should either add human validation or substantially soften the causal language. Major revision is appropriate because the shortcomings are addressable within the manuscript's scope, unlike a fundamental error in the experimental design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core: the paper documents a prompt-conditioned link in Stable Diffusion 2.1 between attractiveness-related words and positive/negative trait words, strongest for Asian and Black women, and shows that downstream gender classifiers make more errors on female faces generated with negative trait prompts. That is a genuinely reportable phenomenon, and the paper is honest about its limits.\n\nWhat it does well: it reuses trait definitions from Gulati et al.'s human study, uses a clear prompt template, includes a neutral baseline, runs three gender classifiers, and breaks results down by race and gender. The discussion of visual confounds—age, expression, makeup—is grounded in prior work and appears in the paper itself, not just in my reading. The intersectional findings are the most novel part and are not in the cited prior literature.\n\nWhere it is soft: the central claim overreaches. The paper explicitly says it does not define or measure attractiveness, yet the gender-classification result is framed as higher error on 'less-attractive' faces. The prompt labels 'attractive' and 'unattractive' are treated as ground truth, and the CLIP centroid metric is entangled with the fact that SD2.1 was conditioned with a CLIP text encoder; evaluating the images with the same embedding family may be measuring CLIP's semantic associations rather than visual appearance. No human ratings, no independent face-attribute classifier, no code or data. The t-test on centroid distances is also underspecified: it is not clear whether the test is on per-image distances to centroids, bootstrap resampling, or something else. These are fixable, but they mean the paper should be read as evidence of prompt-conditioned associations in CLIP space plus a classifier-error pattern, not as evidence of lookism per se.\n\nThe stress-test note is on target; the limitations are not manufactured. The White-women finding—positive traits closer to unattractive faces—is interesting but unexplained and could indicate prompt or dataset quirks rather than a meaningful reversal.\n\nWho this is for: people working on fairness evaluation of generative models, especially those thinking about how to operationalize perceived attractiveness. It deserves serious peer review rather than desk rejection, because the phenomenon is plausible and the measurement approach is worth scrutinizing and improving. I would not cite it as evidence yet, but I would cite it as a cautionary example of how prompt-label operationalization can outrun the claim. A referee should ask for human attractiveness ratings, an independent embedding or attribute model, and a clearer statistical procedure.","headline":"A promising but overreaching workshop study: the prompt-conditioned association between attractiveness words and trait words in SD2.1 output is real, but 'lookism' is not established without human ratings or an independent embedding.","tokens_in":7983,"tokens_out":2130,"would_cite":false,"duration_ms":23106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-07T15:38:54.649068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}