Pith. sign in

REVIEW 3 major objections 6 minor 48 references

From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gender bias in multimodal large language models surfaces more strongly in dual-character narratives, where a man and a woman interact under a defined social relationship, than in single-character settings; the GENRES benchmark measures…

desk verdict A solid new benchmark for interaction-level gender bias, but the 'not evident in single-character settings' claim needs a real controlled baseline. read the letter →

arxiv 2506.23101 v1 pith:QY2X2FO6 submitted 2025-06-29 cs.CL cs.AIcs.CY

Yue Xu , Wenjie Wang This is my paper
classification cs.CLcs.AIcs.CY
keywords genderbiasmultimodallargelanguagemodelsbenchmarksocialrelationshipsrelationaltheorynarrativegenerationstereotypecontentmodelevaluationmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that gender bias in multimodal large language models is best detected not by examining how a model describes a man or a woman in isolation, but by putting both characters in a scene together and letting their social relationship frame the story. To test this, the authors built GENRES, a benchmark of 1,440 image-text pairs in which the model must write individual profiles and a narrative about a mixed-gender pair bound by one of four relationship types drawn from relational models theory. Across six open- and closed-source MLLMs, the paper finds persistent, context-sensitive biases that are weaker or absent in single-character settings: warmth-related traits and emotional language gravitate to female characters, while male characters receive greater psychological depth. If the argument is right, interaction-based evaluation captures a form of gender bias that single-character benchmarks systematically miss.

What carries the argument

The carrying mechanism is a dual-character profile-and-narrative elicitation task organized by relational models theory, which supplies four fundamental relationship types (Communal Sharing, Authority Ranking, Equality Matching, Market Pricing) that structure 288 scenario entries; with five images per entry this yields 1,440 narrative elicitation pairs. Each pair presents the model with a text template using randomly assigned character names and a Stable Diffusion-generated image, and asks for brief individual profiles plus a roughly 500-word story. The evaluation suite converts the outputs into eight male-minus-female metrics spanning four dimensions: profile assignment (warmth words, positive score), agency and role (subject counts, high-status allocation), emotional expression (emotion lexicon, sentence sentiment), and narrative framing (character stereoscopicity, main-character assignment), which are z-scored and summed into a single Total Bias Score for ranking models.

What would settle it

Regenerate or re-pair the images so that gender cues are removed or swapped, for example by neutralizing the visual prompts used for image generation or by swapping which image accompanies which gendered name, and rerun the evaluation; if the warmth-word gap (metric M1) and emotion-word gap (metric M5) shrink or vanish, the bias lives in the benchmark images, while if they persist, the interaction-driven model bias is real.

Watch

Extended reading notes

Core claim

The central discovery is that interpersonal context reliably surfaces latent gender bias: when a model narrates a story about a male and a female character engaged in a defined social relationship, its gendered choices diverge more sharply than when it describes either character alone, as shown by a preliminary comparison of warmth-related word usage and gender-stereoscopic portrayal correlations. On the GENRES benchmark, the paper reports that all evaluated models except Gemini assign more warmth-related personality traits and more emotional language to female characters; that female characters are more often cast as the main protagonist and given higher social status while male characters are portrayed as more psychologically stereoscopic; and that larger, more capable models are not necessarily fairer, with GPT-4o and Gemini showing the highest total bias scores and the smaller Qwen models the lowest. The paper also finds the bias is relationship-sensitive: warmth stereotyping is most extreme in communal-sharing contexts, and models introduce social hierarchies even in relationship types that should imply equal standing.

Load-bearing premise

The load-bearing assumption is that the benchmark images themselves are free of gender bias: the pictures were generated by Stable Diffusion from GPT-4o descriptions and filtered by a CLIP similarity threshold plus manual review, but the paper never audits the final image set for systematic gendered cues, so if the photos encode gendered clothing, settings, or poses, the measured bias could be coming from the benchmark rather than from the models.

Editorial extensions

If this is right

  • Single-character benchmarks understate gender bias: a model that looks fair on isolated portraits can still stereotype in interpersonal narratives, so evaluations should include dual-character tasks to diagnose bias.
  • Bias is context-sensitive: the same model can favor one gender in communal settings and a different one in workplace or market settings, so aggregate scores alone can mislead.
  • Model capability and fairness do not move together: the largest evaluated models carry the highest total bias while the smaller ones carry the lowest, implicating training data and alignment choices rather than scale.
  • The recurring pattern of warm, emotionally expressive female leads alongside psychologically deeper male characters defines a specific failure mode (a surface-level elevation of female characters) that debiasing should target.
  • Models impose hierarchies the prompt never asked for, assigning unequal social status in relationship types designed to be egalitarian, which is itself a measurable bias.
  • pith_inferences

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the dual-character effect should be directly testable within a single model by comparing matched prompts that differ only in whether the same image is framed as one character or two, a controlled contrast the preliminary study gestures toward but does not report systematically.
  • Beyond the paper: the SDXL-generated image set is an uncontrolled confound; an audit for systematic gendered differences in clothing, pose, or setting, or an image-swap condition, would separate model bias from benchmark artifact.
  • Beyond the paper: applying the same relational-models frame to same-gender and non-binary pairings (which the authors list as future work) would show whether the warmth/competence split is specific to mixed-gender pairs or tied to the relationship type itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces GENRES, a benchmark that measures gender bias in MLLM-generated narratives about male-female dyads. The dataset contains 1,440 narrative elicitation pairs built from Fiske's four relational models; each pair includes a text prompt and an SDXL-generated image. Models are asked to write character profiles and a roughly 500-word story, and the outputs are scored with eight metrics (M1-M8) covering profile traits, agency/role, emotional expression, and narrative framing, using both NLP tools and three LLM evaluators. The main experiments report bias patterns across six MLLMs, and the paper claims that dual-character interaction contexts surface gender biases that are not evident in single-character settings.

Significance. If the comparative claim were adequately supported, GENRES would make a useful contribution by shifting gender-bias evaluation from isolated entities to relationships. The benchmark is theory-grounded, the metric definitions are transparent, the pipeline is documented in detail, code and data are released, and the use of three independent LLM evaluators is a robustness strength. At present, however, the headline result rests on a small uncontrolled pilot, and the main evaluation cannot by itself demonstrate an advantage over single-character benchmarks; the significance is therefore conditional on the additional controlled evidence recommended below.

major comments (3)
  1. [§1 and Figure 1; Abstract; §5] The central claim that gender bias is stronger in dual-character narratives and 'not evident in single-character settings' is supported only by the preliminary study in Figure 1, which uses two models (Qwen-3B and Janus-7B) and reports no sample size, error bars, confidence intervals, or significance tests, and no description of how the single-character prompts and images were constructed to match the dual-character ones. The main GENRES evaluation in Section 4 contains only dual-character NEPs. Please add a controlled single-character baseline across all six models, with matched text prompts and images and a statistical comparison (e.g., paired tests with effect sizes), or explicitly reframe the paper's contribution to avoid the comparative claim.
  2. [§3.2, Appendix B.2] The image generation and filtering pipeline is described as avoiding gender bias through prompt design, CLIP filtering at threshold 0.25, and manual review, but the paper provides no audit of the final image set. Because the task is multimodal and all metrics are computed from MLLM responses to image-plus-text inputs, systematic gendered cues in appearance, clothing, or setting could produce the observed differences without reflecting model-level relational bias. Please provide an audit of the final images (e.g., human annotations of gender-role cues, automated attribute analysis, or a counterfactual image-swapping experiment) to support the assumption that the images do not introduce gendered artifacts.
  3. [§4.2, Eq. for M4; Table 2; Figure 6] The M4 analysis applies a post-hoc two-thirds evaluator-agreement filter, but the paper does not report how many samples were excluded per model and relationship, nor how sensitive the conclusions are to the agreement threshold. In addition, Table 2 reports M1-M6 and TBS as point estimates without error bars or confidence intervals even though results are averaged over four independent evaluations. Without variance information or a sensitivity analysis, the cross-model ranking and the claim that TBS rankings are 'highly consistent' across evaluators are not fully supported.
minor comments (6)
  1. [§3.3] 'Natrual Language Processing' should be 'Natural Language Processing'.
  2. [Table 9] The female name list contains 'Amelia' twice, so the curated list of 15 names actually has only 14 unique female names; please fix and report whether the duplicate affected name sampling.
  3. [Figure 1] The left panel's y-axis label 'Warmth-related Word (%)' is unclear as to whether values are percentages or proportions; please state units and add sample sizes.
  4. [§3.3] The term 'stereoscopic' is used for characters who have both positive and negative traits; please define it at first use in the main text rather than only in Appendix C.2.
  5. [Table 2] p-values are reported for M7 and M8 but not for M1-M6; consider reporting confidence intervals or at least significance flags for all metrics.
  6. [§3.2 and Appendix B.1] Appendix B.1 states that names are assigned in varying order to minimize positional bias, but this detail is absent from the main-text description of text generation in §3.2; adding one sentence would help readers assess the counterbalancing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the GENRES metrics are computed from model outputs via external dictionaries, lexicons, and classifiers, and the dual-vs-single claim, while experimentally under-supported, is not defined into the benchmark's construction.

full rationale

I walked the paper's claimed derivation chain and found no step where a predicted quantity reduces, by construction or by self-citation, to the benchmark's own inputs. The eight bias metrics are computed from MLLM-generated profiles and narratives using external resources: warmth/competence words from Nicolas et al.'s dictionary (M1), positive valence from the same dictionary (M2), SpaCy SVO parsing (M3), an LLM prompt for status (M4), the NRC Emotion Lexicon (M5), a fine-tuned DistilBERT sentiment model (M6), and LLM-based pros/cons and protagonist judgments (M7, M8). None of these metrics is fitted to the reported bias values, and the Total Bias Score is only a z-score aggregation across models, not an optimization target. The central dual-versus-single claim is supported only by the small preliminary study in Figure 1, which lacks sample sizes and significance tests; that is an experimental-design and statistical-evidence weakness, but it is not circular because the comparison does not presuppose its own conclusion. The benchmark's image pipeline filters for quality and for the stated goal of avoiding gender bias, but even an imperfect filter would be a data-quality confound rather than a circular derivation. The paper also does not rely on self-citations from the same authors to justify its load-bearing premises; its theoretical grounding (Fiske's relational models, Stereotype Content Model) is external. Therefore, under the hard rule that circularity must be exhibited by a specific reduction, equation, or fitted-parameter-as-prediction, I find no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central findings depend on social-psychology taxonomies, on the impartiality of LLM judges, and on the neutrality of generated images. None of these are fitted to the target results, but each is an unverified premise that would undermine the benchmark if false.

free parameters (2)
  • CLIP similarity threshold = 0.25
    Hand-set threshold for filtering generated images; affects which images enter the 1,440 NEPs.
  • M4 evaluator-agreement threshold = >=2/3
    Post-hoc filter for high-status allocation: only samples with at least two-thirds evaluator agreement are analyzed in Figure 6.
assumptions (4)
  • domain assumption Fiske's four relational models (CS, EM, MP, AR) are a valid, cross-culturally applicable taxonomy of social relationships.
    Section 3.1 and Table 1 ground the scenario space; if the taxonomy is incomplete or not applicable to MLLM narrative contexts, the benchmark's coverage claim weakens.
  • domain assumption Warmth and competence are the two principal stereotype dimensions, and dictionary counts based on Nicolas et al. capture them in generated profiles.
    Section 3.3, M1; the entire Profile Assignment Bias dimension depends on this dictionary mapping.
  • domain assumption Llama-3.1-8B, GLM-4-9B, and Mistral-Small-24B can infer social status, pros and cons, and main character from a passage without introducing systematic gender bias.
    Section 3.3 M4/M7/M8 and Appendix C.2; three-evaluator agreement mitigates but does not eliminate evaluator bias as a confound.
  • ad hoc to paper The SDXL-generated images, after CLIP filtering and manual review, contain no gender-role or appearance cues that systematically influence the narrative.
    Section 3.2 and Appendix B.2; this is a paper-specific quality claim and is load-bearing for attributing measured bias to the models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship." pith.science (2026). https://pith.science/paper/QY2X2FO6

@misc{pith2026250623101,
  author       = {Pith},
  title        = {Pith review of: From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QY2X2FO6}},
  note         = {Machine review of arXiv:2506.23101}
}
read the original abstract

Multimodal large language models (MLLMs) have shown impressive capabilities across tasks involving both visual and textual modalities. However, growing concerns remain about their potential to encode and amplify gender bias, particularly in socially sensitive applications. Existing benchmarks predominantly evaluate bias in isolated scenarios, overlooking how bias may emerge subtly through interpersonal interactions. We fill this gap by going beyond single-entity evaluation and instead focusing on a deeper examination of relational and contextual gender bias in dual-individual interactions. We introduce Genres, a novel benchmark designed to evaluate gender bias in MLLMs through the lens of social relationships in generated narratives. Genres assesses gender bias through a dual-character profile and narrative generation task that captures rich interpersonal dynamics and supports a fine-grained bias evaluation suite across multiple dimensions. Experiments on both open- and closed-source MLLMs reveal persistent, context-sensitive gender biases that are not evident in single-character settings. Our findings underscore the importance of relationship-aware benchmarks for diagnosing subtle, interaction-driven gender bias in MLLMs and provide actionable insights for future bias mitigation.

Figures

Figures reproduced from arXiv: 2506.23101 by the authors.

Figure 1
Figure 1. Gender differences in (left) warmth-related word usage and (right) stereoscopic character portrayal across single- (S.) and dual-character (D.) settings. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The development pipelines of GENRES. 3.1 Narrative Elements Design The narrative elements serve as the foundation for constructing NEPs in GENRES. Each entry specifies key attributes of the two characters, including their gender, age group, social relationship type, and the scenario in which they interact. These structured specifications collectively guide the generation of NEPs in the subsequent step and ultimately… view at source ↗
Figure 3
Figure 3. The template for generating text queries in NEPs, with the variables highlighted in different [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Warmth-related words distribution in male and female profiles across all relationships in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Positive Score Bias across different models. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Proportion of high status allocation across different models. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Distribution of emotion-related words across male and female for each model. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Pearson correlation between character gender and (left) stereoscopic portrayal, (right) main [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: The template for querying GPT-4o for image description prompt, with the variables [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: The template for querying evaluators for profile extraction. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: The template for querying evaluators for narrative analysis. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Warmth-related words distribution in male and female profiles across all relationships, [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Warmth-related words distribution in male and female profiles across all relationships, [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Warmth-related words distribution in male and female profiles across all relationships, [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Positive score bias across different models, evaluated by Llama-3.1-8B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Positive score bias across different models, evaluated by GLM-4-9B. [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Pearson correlation between character gender and (left) stereoscopic portrayal, (right) main [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 25 canonical work pages

  1. [1]

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743, 2025

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.arXiv preprint arXiv:2503.01743, 2025

    arXiv 2025

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 11

    arXiv 2025

  4. [4]

    A multidimensional analysis of social biases in vision transformers

    Jannik Brinkmann, Paul Swoboda, and Christian Bartelt. A multidimensional analysis of social biases in vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4914–4923, 2023

    work page 2023

  5. [5]

    Benchlmm: Benchmarking cross-style visual capability of large multimodal models

    Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Yaohang Li, Xing Luo, Chenyu Yi, and Alex Kot. Benchlmm: Benchmarking cross-style visual capability of large multimodal models. In European Conference on Computer Vision, pages 340–358. Springer, 2024

    work page 2024

  6. [6]

    From structured prompts to open narra- tives: Measuring gender bias in llms through open-ended storytelling

    Evan Chen, Run-Jun Zhan, Yan-Bai Lin, and Hung-Hsuan Chen. From structured prompts to open narra- tives: Measuring gender bias in llms through open-ended storytelling. arXiv preprint arXiv:2503.15904, 2025

    arXiv 2025

  7. [7]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025

    arXiv 2025

  8. [8]

    Physbench: Benchmarking and enhancing vision-language models for physical world understanding.arXiv preprint arXiv:2501.16411, 2025

    Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world understanding.arXiv preprint arXiv:2501.16411, 2025

    arXiv 2025

Show all 48 references
  1. [9]

    The four elementary forms of sociality: framework for a unified theory of social relations

    Alan P Fiske. The four elementary forms of sociality: framework for a unified theory of social relations. Psychological review, 99(4):689, 1992

    1992

  2. [10]

    A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition

    Susan T Fiske, Amy JC Cuddy, Peter Glick, and Jun Xu. A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition. In Social cognition, pages 162–214. Routledge, 2018

    2018

  3. [11]

    Examining gender and racial bias in large vision-language models using a novel dataset of parallel images

    Kathleen C Fraser and Svetlana Kiritchenko. Examining gender and racial bias in large vision-language models using a novel dataset of parallel images. arXiv preprint arXiv:2402.05779, 2024

    2024 arXiv

  4. [12]

    Revealing and reducing gender biases in vision and language assistants (vlas)

    Leander Girrbach, Stephan Alaniz, Yiran Huang, Trevor Darrell, and Zeynep Akata. Revealing and reducing gender biases in vision and language assistants (vlas). arXiv preprint arXiv:2410.19314, 2024

    2024 arXiv

  5. [13]

    A large scale analysis of gender biases in text-to-image generative models

    Leander Girrbach, Stephan Alaniz, Genevieve Smith, and Zeynep Akata. A large scale analysis of gender biases in text-to-image generative models. arXiv preprint arXiv:2503.23398, 2025

    2025

  6. [14]

    Gemini — deepmind.google

    Google. Gemini — deepmind.google. https://deepmind.google/models/gemini/, 2025

    2025

  7. [15]

    Mllmguard: A multi-dimensional safety evaluation suite for multimodal large language models

    Tianle Gu, Zeyang Zhou, Kexin Huang, Liang Dandan, Yixu Wang, Haiquan Zhao, Yuanqi Yao, Yujiu Yang, Yan Teng, Yu Qiao, et al. Mllmguard: A multi-dimensional safety evaluation suite for multimodal large language models. Advances in Neural Information Processing Systems, 37:7256...

    2024

  8. [16]

    Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution

    Siobhan Mackenzie Hall, Fernanda Gonçalves Abrantes, Hanwen Zhu, Grace Sodunke, Aleksandar Shtedritski, and Hannah Rose Kirk. Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution. Advances in Neural Information Processing Systems, 36:63687–63723, 2023

    2023

  9. [17]

    Uncovering bias in large vision-language models with counterfactuals

    Phillip Howard, Anahita Bhiwandiwalla, Kathleen C Fraser, and Svetlana Kiritchenko. Uncovering bias in large vision-language models with counterfactuals. arXiv preprint arXiv:2404.00166, 2024

    2024 arXiv

  10. [18]

    Socialcounterfactuals: Probing and mitigating intersectional social biases in vision-language models with counterfactual examples

    Phillip Howard, Avinash Madasu, Tiep Le, Gustavo Lujan Moreno, Anahita Bhiwandiwalla, and Vasudev Lal. Socialcounterfactuals: Probing and mitigating intersectional social biases in vision-language models with counterfactual examples. In Proceedings of the IEEE/CVF Conference o...

    2024

  11. [19]

    Visbias: Measuring explicit and implicit social biases in vision language models

    Jen-tse Huang, Jiantong Qin, Jianping Zhang, Youliang Yuan, Wenxuan Wang, and Jieyu Zhao. Visbias: Measuring explicit and implicit social biases in vision language models. arXiv preprint arXiv:2503.07575, 2025

    2025 arXiv

  12. [20]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  13. [21]

    Gender bias in llm-generated interview responses

    Haein Kong, Yongsu Ahn, Sangyub Lee, and Yunho Maeng. Gender bias in llm-generated interview responses. arXiv preprint arXiv:2410.20739, 2024

    2024 arXiv

  14. [22]

    Natural language understanding and inference with mllm in visual question answering: A survey

    Jiayi Kuang, Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng, Xika Lin, and Yu Han. Natural language understanding and inference with mllm in visual question answering: A survey. ACM Computing Surveys, 57(8):1–36, 2025. 12

    2025

  15. [23]

    Gender bias in decision-making with large language models: A study of relationship conflicts

    Sharon Levy, William D Adler, Tahilin Sanchez Karver, Mark Dredze, and Michelle R Kaufman. Gender bias in decision-making with large language models: A study of relationship conflicts. arXiv preprint arXiv:2410.11084, 2024

    2024 arXiv

  16. [24]

    Multimodal composite association score: Measuring gender bias in generative multimodal models

    Abhishek Mandal, Susan Leavy, and Suzanne Little. Multimodal composite association score: Measuring gender bias in generative multimodal models. arXiv preprint arXiv:2304.13855, 2023

    2023 arXiv

  17. [25]

    Nrc emotion lexicon

    Saif M Mohammad and Peter D Turney. Nrc emotion lexicon. National Research Council, Canada, 2:234, 2013

    2013

  18. [26]

    Social biases through the text-to-image generation lens

    Ranjita Naik and Besmira Nushi. Social biases through the text-to-image generation lens. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 786–808, 2023

    2023

  19. [27]

    Sb-bench: Stereotype bias benchmark for large multimodal models

    Vishal Narnaware, Ashmal Vayani, Rohit Gupta, Sirnam Swetha, and Mubarak Shah. Sb-bench: Stereotype bias benchmark for large multimodal models. arXiv preprint arXiv:2502.08779, 2025

    2025

  20. [28]

    Comprehensive stereotype content dictionaries using a semi-automated method

    Gandalf Nicolas, Xuechunzi Bai, and Susan T Fiske. Comprehensive stereotype content dictionaries using a semi-automated method. European Journal of Social Psychology, 51(1):178–196, 2021

    2021

  21. [29]

    Bbq: A hand-built bias benchmark for question answering

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021

    2021 arXiv

  22. [30]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  23. [31]

    Gender biases in automatic evaluation metrics for image captioning

    Haoyi Qiu, Zi-Yi Dou, Tianlu Wang, Asli Celikyilmaz, and Nanyun Peng. Gender biases in automatic evaluation metrics for image captioning. arXiv preprint arXiv:2305.14711, 2023

    2023 arXiv

  24. [32]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  25. [33]

    Biasdora: Exploring hidden biased associations in vision-language models

    Chahat Raj, Anjishnu Mukherjee, Aylin Caliskan, Antonios Anastasopoulos, and Ziwei Zhu. Biasdora: Exploring hidden biased associations in vision-language models. arXiv preprint arXiv:2407.02066, 2024

    2024 arXiv

  26. [34]

    A multi-dimensional study on bias in vision-language models

    Gabriele Ruggeri, Debora Nozza, et al. A multi-dimensional study on bias in vision-language models. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 2023

    2023

  27. [35]

    Worst of both worlds: Biases compound in pre-trained vision-and- language models

    Tejas Srinivasan and Yonatan Bisk. Worst of both worlds: Biases compound in pre-trained vision-and- language models. arXiv preprint arXiv:2104.08666, 2021

    2021 arXiv

  28. [36]

    kelly is a warm person, joseph is a role model

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. " kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters. arXiv preprint arXiv:2310.09219, 2023

    2023 arXiv

  29. [37]

    Survey of bias in text-to-image generation: Definition, evaluation, and mitigation

    Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Rebecca Pattichis, and Kai-Wei Chang. Survey of bias in text-to-image generation: Definition, evaluation, and mitigation. arXiv preprint arXiv:2404.01030, 2024

    2024 arXiv

  30. [38]

    Vlbias- bench: A comprehensive benchmark for evaluating bias in large vision-language model

    Sibo Wang, Xiangkui Cao, Jie Zhang, Zheng Yuan, Shiguang Shan, Xilin Chen, and Wen Gao. Vlbias- bench: A comprehensive benchmark for evaluating bias in large vision-language model. arXiv preprint arXiv:2406.14194, 2024

    2024 arXiv

  31. [39]

    Genderbias-vl: Benchmarking gender bias in vision language models via counterfactual probing

    Yisong Xiao, Aishan Liu, QianJia Cheng, Zhenfei Yin, Siyuan Liang, Jiapeng Li, Jing Shao, Xianglong Liu, and Dacheng Tao. Genderbias-vl: Benchmarking gender bias in vision language models via counterfactual probing. CoRR, 2024

    2024

  32. [40]

    Benchmarking large multimodal models against common corruptions

    Jiawei Zhang, Tianyu Pang, Chao Du, Yi Ren, Bo Li, and Min Lin. Benchmarking large multimodal models against common corruptions. arXiv preprint arXiv:2401.11943, 2024

    2024 arXiv

  33. [41]

    Understanding and evaluating racial biases in image captioning

    Dora Zhao, Angelina Wang, and Olga Russakovsky. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14830–14840, 2021. 13

    2021

  34. [42]

    Gender bias in coreference resolution: Evaluation and debiasing methods

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876, 2018

    2018 arXiv

  35. [43]

    Vlstereoset: A study of stereotypical bias in pre-trained vision- language models

    Kankan Zhou, Yibin LAI, and Jing Jiang. Vlstereoset: A study of stereotypical bias in pre-trained vision- language models. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Confere...

    2022

  36. [44]

    male” and “female

    Zhongliang Zhou, Jielu Zhang, Zihan Guan, Mengxuan Hu, Ni Lao, Lan Mu, Sheng Li, and Gengchen Mai. Img2loc: Revisiting image geolocalization using multi-modality foundation models and image-based retrieval-augmented generation. In Proceedings of the 47th International ACM SIGI...

    2024

  37. [45]

    [High status question]

    Answer the following questions based on the passage: a. [High status question]. Use acquired attributes such as occupation, income, education, or demon- strated competence to inform your judgment. Do not use protected attributes such as gender, race, religion, age, or any othe...

  38. [46]

    Use adjective words as much as possible and separate them with a comma

    Summarize the pros and cons of the two characters’ traits based on the passage. Use adjective words as much as possible and separate them with a comma. Please be comprehensive and stick to the passage. If you think there are no pros or cons shown in the passage, fill in ’NA’

  39. [47]

    Do not make any assumptions

    Complete all the tasks only based on the passage. Do not make any assumptions

  40. [48]

    Answer to Task1.a

    Use the following JSON format: { "Answer to Task1.a": "answer to Task1.a", "Answer to Task1.b": "answer to Task1.b", "Prons of [NAME1]": "prons of [NAME1]", "Cons of [NAME1]": "cons of [NAME1]", "Pros of [NAME2]": "pros of [NAME2]", "Cons of [NAME2]": "cons of [NAME2]" } Figur...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.