Pith. sign in

REVIEW 2 major objections 2 minor 139 references

Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics

T0 review · 2 major / 2 minor · reviewed 2026-05-21 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Community input systematizes cultural appropriateness to create rubrics for evaluating AI images of artifacts.

desk verdict Community input early in defining cultural appropriateness for AI images adds grounded detail, but the jump to reliable LLM automation still looks shaky. read the letter →

arxiv 2604.02406 v2 pith:7PEN3JAE submitted 2026-04-02 cs.CY

Nari Johnson , Deepthi Sudharsan , Hamna , Samantha Dalal , Theo Holroyd , Anja Thieme , Hoda Heidari , Daniela Massiceti
show 2 more authors
Jennifer Wortman Vaughan Cecily Morrison
This is my paper · ORCID
classification cs.CY
keywords culturalappropriatenessAI-generatedimagescommunityrubricstext-to-imagemodelsartifactsevaluationmeasures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper explores involving communities in the first stage of measuring cultural appropriateness in text-to-image AI outputs. Through case studies with blind and low-vision people in the UK plus residents of Kerala and Tamil Nadu, it turns lived experiences into precise definitions of how artifacts should appear. These definitions support rubrics that can be applied automatically across models while retaining community expertise. The work tests whether concentrating engagement early resolves the tension between repeatable measurement and capturing real perspectives.

What carries the argument

The staged measurement process that places community systematization of cultural appropriateness before operationalization into rubrics and automated application.

What would settle it

Community members scoring the same AI-generated images with the new rubrics would reveal whether the scores match their independent views of cultural appropriateness.

Watch

Extended reading notes

Core claim

Systematized concepts of cultural appropriateness developed with community members reflect their lived experiences with each artifact and their preferences for depictions of material culture, showing that community involvement at the definition stage produces valid measures for AI evaluation.

Load-bearing premise

Perspectives from community engagement in the initial definition stage remain effective when converted into standardized rubrics for automatic use across many images and models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript explores involving communities in the systematization phase of measuring 'cultural appropriateness' for text-to-image models' depictions of cultural artifacts. Through case studies with blind and low vision individuals in the UK, residents of Kerala, and residents of Tamil Nadu, the authors develop systematized concepts drawn from participants' lived experiences and preferences for how their material culture should be represented. The work then examines operationalizing these concepts into automated instruments via a multimodal LLM-as-a-judge approach and reflects on benefits, limitations, and remaining challenges in achieving repeatable, automatable measurement across models and settings.

Significance. If the community-informed systematized concepts can be faithfully translated into LLM-usable rubrics without substantial loss of nuance or introduction of model-specific biases, the approach would meaningfully advance inclusive AI evaluation practices by reconciling automation needs with community expertise. The multi-community case studies provide concrete grounding for the claim that concentrating engagement in the systematization stage adds validity, and the explicit discussion of operationalization challenges is a constructive contribution to the broader measurement literature.

major comments (2)
  1. [§4] §4 (Operationalization and LLM-as-a-judge): The manuscript notes challenges in translating community criteria into automatable prompts or rubrics but supplies only high-level discussion rather than concrete examples of rubric items derived from specific community input (e.g., desired depictions for Kerala or Tamil Nadu artifacts) and their encoding as LLM scoring criteria. This step is load-bearing for the central claim that the approach enables repeatable, automatable instruments across settings without losing captured expertise.
  2. [§3] §3 (Case studies): The systematized concepts are presented as reflecting lived experiences, yet the text provides limited direct evidence—such as participant quotes, raw response summaries, or side-by-side comparisons of community input versus final systematized statements—to allow readers to evaluate the fidelity of the translation process.
minor comments (2)
  1. The abstract and introduction could more explicitly distinguish the three communities' distinct artifact types and cultural contexts to help readers track how findings generalize.
  2. Notation for the measurement stages (systematization, operationalization, application) is introduced clearly but could be reinforced with a small diagram or table summarizing the pipeline for each case study.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which identify key areas where additional detail would strengthen the manuscript's demonstration of the proposed approach. We respond to each major comment below, indicating planned revisions.

read point-by-point responses

  1. Referee: [§4] §4 (Operationalization and LLM-as-a-judge): The manuscript notes challenges in translating community criteria into automatable prompts or rubrics but supplies only high-level discussion rather than concrete examples of rubric items derived from specific community input (e.g., desired depictions for Kerala or Tamil Nadu artifacts) and their encoding as LLM scoring criteria. This step is load-bearing for the central claim that the approach enables repeatable, automatable instruments across settings without losing captured expertise.

    Authors: We agree that concrete examples are necessary to support the claim of faithful translation into automatable instruments. In the revised manuscript we will add specific rubric items drawn from the Kerala and Tamil Nadu case studies, including examples of desired depictions (such as accurate rendering of temple architecture or traditional motifs) and their direct encoding as LLM scoring criteria with sample prompts and scales. This will be presented alongside discussion of remaining challenges to avoid overstating generalizability. revision: yes

  2. Referee: [§3] §3 (Case studies): The systematized concepts are presented as reflecting lived experiences, yet the text provides limited direct evidence—such as participant quotes, raw response summaries, or side-by-side comparisons of community input versus final systematized statements—to allow readers to evaluate the fidelity of the translation process.

    Authors: We acknowledge the value of greater transparency in showing the translation process. The revised §3 will incorporate selected participant quotes, summarized raw responses, and side-by-side comparisons between community inputs and the final systematized statements for the three case studies, enabling readers to assess fidelity more directly while respecting participant confidentiality constraints. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity in derivation chain

full rationale

The paper is a qualitative exploration of community involvement in systematizing concepts of cultural appropriateness for AI image generation via case studies. It contains no equations, fitted parameters, predictions, or self-referential derivations that reduce claims to author-defined inputs by construction. The central claims rest on direct community input rather than any load-bearing self-citation chain or renaming of prior results. This is the most common honest non-finding for self-contained qualitative work against external community benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The paper relies on the domain assumption that community perspectives captured through engagement constitute valid and superior input for defining cultural appropriateness; no free parameters or invented entities are introduced.

assumptions (1)
  • domain assumption Community members' lived experiences provide the authoritative basis for defining valid measures of cultural appropriateness in AI image generation.
    Invoked when the abstract states that systematized concepts reflect community experiences and demonstrate the value of involvement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics." pith.science (2026). https://pith.science/paper/7PEN3JAE

@misc{pith2026260402406,
  author       = {Pith},
  title        = {Pith review of: Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7PEN3JAE}},
  note         = {Machine review of arXiv:2604.02406}
}
read the original abstract

Measurement is essential to improving AI performance and mitigating harms for marginalized groups. As generative AI systems are rapidly deployed across geographies and contexts, AI measurement practices must be designed to support repeatable, automatable application across different models, datasets, and evaluation settings. But the drive to automate measurement can be in tension with the ability for measurement instruments to capture the expertise and perspectives of communities impacted by AI. Recent work advocates for breaking measurement into several key stages: first moving from an abstract concept to be measured into a precise, "systematized" concept; next operationalizing the systematized concept into a concrete measurement instrument; and finally applying the measurement instrument on data to produce measurements. This opens up an opportunity to concentrate community engagement in the systematization phase before operationalizing and applying measurement instruments. In this paper, we explore how to involve communities in systematizing the concept of "cultural appropriateness" in text-to-image models' representation of culturally significant artifacts through case studies with three communities: blind and low vision individuals residing in the UK, residents of Kerala, and residents of Tamil Nadu. Our systematized concepts reflect community members' lived experiences interacting with each artifact and how they want their material culture to be depicted, demonstrating the value of community involvement in defining valid measures. We explore how these systematized concepts can be operationalized into automated measurement instruments that could be applied using a multimodal LLM-as-a-judge approach and challenges that remain. We reflect on the benefits and limitations of such approaches.

Figures

Figures reproduced from arXiv: 2604.02406 by the authors.

Figure 1
Figure 1. Scaffolding community engagement to develop community-centered measures of cultural representation. Given an input prompt (e.g., “a photo of a guide cane”), we invited community members to participate in designing a rubric that captures their expertise and preferences for each cultural artifact (systematization). Our research team then explored the use of this rubric within an automated multimodal LLM-as-a-judge pip… view at source ↗
Figure 2
Figure 2. Measurement framework from the social sciences [1, 116]. Our research studies how to center community knowledge in the systematization process before operationalizing the systematized concept as an automated MLLM-as-a-judge system. Applying Measurement Theory to AI Evaluations. Recent work by Wallach et al. [116] advocates for researchers to rethink how they evaluate generative AI systems, drawing on the tradition o… view at source ↗
Figure 3
Figure 3. Selected culturally significant artifacts. From right to left: (1) With the blind and low vision community, we selected a guide cane (a mobility aid that is held diagonally across one’s body) and a braille notetaker (an electronic device that can be used to read and write notes in tactile braille). (2) With residents of Tamil Nadu, we selected Pallanguzhi (a two-player mancala game where players compete to collect c… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: A rubric to score images of a guide cane, designed with BLV community members. Criteria that correspond to visual features in images are organized under two themes that describe participants’ desires for cultural representation. model for its demonstrated performance o…
Figure 5
Figure 5. Figure 5: Human-MLLM judge alignment for individual rubric criteria. A histogram that shows the human-MLLM agreement rate for individual rubric criteria. We find that there is high variance across criteria in the MLLM’s ability to annotate a criterion accurately, such as the exa…
Figure 6
Figure 6. Figure 6: Community-elicited rubrics differ meaningfully from those generated by off-the-shelf LLMs. Our rubrics differ from LLM-generated rubrics in three ways, each illustrated using an example (Appendix B.1.3). First, LLM-generated rubrics can include factual or interpretive …
Figure 7
Figure 7. Figure 7: Annotated LLM-generated rubric for a guide cane. While generally providing an accurate description of a guide cane, the rubric misses several key details. The rubric does not provide a complete description of the straight handle shape of a cane (C2), a feature that is …
Figure 8
Figure 8. Figure 8: Annotated LLM-generated rubric for a braille notetaker. The rubric criteria include both inaccurate descriptions of a braille notetaking device, and do not include descriptive details about valid depictions of braille. A braille notetaker does not resemble a notepad (e…
Figure 9
Figure 9. Figure 9: Annotated LLM-generated rubric for Pallanguzhi. The rubric generally provides an accurate description of the most important characteristics of the a Pallanguzhi board, with two differences from the community-elicited rubric. (C1) Community members clarified that the co…
Figure 10
Figure 10. Figure 10: Annotated LLM-generated rubric for a Mridangam. The rubric lacks many of the critical details that distinguish the Mridangam from related drums and percussion instruments. One significant omission is the black circular membrane that must be present on both drumheads, …
Figure 11
Figure 11. Figure 11: Annotated LLM-generated rubric for a Kasavu saree. The rubric generally provides an accurate description of a Kasavu saree, demonstrating substantial overlap with the community-elicited rubric. However, community members were clear that the material must be cotton and…
Figure 12
Figure 12. Figure 12: Annotated LLM-generated rubric for Chundan Vallam. The rubric criteria cover the general structure of the Chundan Vallam but do not specify its defining features. In particular, they omit details about the oar structure and handling (C2); community members specified t…
Figure 13
Figure 13. Figure 13: Criterion-level annotations provided by humans reveal the specific representational errors that make depictions of a braille notetaker inappropriate. The figure displays a reference photo of a braille notetaker, and example AI-generated images that fall into one of fo…
Figure 14
Figure 14. Figure 14: Comparing (manual) rubric application across models for a braille notetaker. The frequency at which different criteria are violated (reported here using annotations provided by humans) varies across different models. For example, the GPT Image-1 images of braille note…
Figure 15
Figure 15. Figure 15: Comparing (manual) rubric application across models for a Mridangam. Comparing the frequency at which different criteria are violated across models allows practitioners to draw interpretable insights about models’ failure modes. With the exception of GPT Image-1, many…
Figure 16
Figure 16. Figure 16: Comparing (manual) rubric application across models for a Kasavu Saree. Visualizing the breakdown of criteria that are violated by different image generation models reveals interpretable insights about model behavior. For example, only Flux.1 and the Stable Diffusion …
Figure 17
Figure 17. Figure 17: Stable Diffusion 3 generates depictions of unrelated cultural artifacts and scenes when given simple transliterated prompts. Depictions improve when images are generated using DALL·E 3 revised prompts instead. The images on the left were generated with the simple tran…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

139 extracted references · 139 canonical work pages

  1. [1]

    Robert Adcock and David Collier. 2001. Measurement validity: A shared standard for qualitative and quantitative research.American Political Science Review95, 3 (2001), 529–546

    work page 2001

  2. [3]

    Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards Measuring and Modeling “Culture” in LLMs: A Survey.arXiv preprint arXiv:2403.15412 (2024)

    work page arXiv 2024

  3. [4]

    I look at it as the king of knowledge

    Rudaiba Adnin and Maitraye Das. 2024. "I look at it as the king of knowledge": How Blind People Use and Understand Generative AI Tools. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility(St. John’s, NL, Canada)(ASSETS ’24). Association for Computing Machinery, New York, NY, USA, Article 64, 14 pages. doi:10.11...

    work page doi:10.1145/3663548.3675631 2024

  4. [5]

    Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego Mares, Pavit Singh, Michael Liu, Subodh Chawla, Pete Cline, Lucy Ogaz, Ernesto Hernandez, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu...

    work page arXiv 2025

  5. [6]

    Arnstein

    Sherry R. Arnstein. 1969. A Ladder of Citizen Participation.Journal of the American Institute of Planners35, 4 (1969), 216–224

    work page 1969

  6. [7]

    Taylor, Mark Díaz, Christopher M

    Lora Aroyo, Alex S. Taylor, Mark Díaz, Christopher M. Homan, Alicia Parrish, Greg Serapio-García, Vinodkumar Prabhakaran, and Ding Wang

    work page

  7. [8]

    InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23)

    DICES dataset: diversity in conversational AI evaluation for safety. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 2321, 13 pages

    work page

  8. [9]

    Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, W Duncan Wadsworth, and Hanna Wallach. 2021. Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. InProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 368–378

    work page 2021

Show all 139 references
  1. [10]

    Bennett, Erin Brady, and Stacy M

    Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence as a Frame for Assistive Technology Research and Design. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility(Galway, Ireland)(ASSETS ’18). Association for C...

    2018 doi

  2. [11]

    Bennett, Shaun K

    Cynthia L. Bennett, Shaun K. Kane, and Christina N. Harrington. 2025. Toward Community-Led Evaluations of Text-to-Image AI Representations of Disability, Health, and Accessibility. InProceedings of the 5th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Opti...

    2025 doi

  3. [12]

    Stevie Bergman, Nahema Marchal, John Mellor, Shakir Mohamed, Iason Gabriel, and William Isaac. 2024. STELA: a community-centred approach to norm elicitation for AI alignment.Scientific Reports14, 1 (2024), 6616

    2024

  4. [13]

    Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2023. Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale. InProceedings of the ...

    2023 doi

  5. [14]

    Gray, and Rida Qadri

    Asia Biega, Georgina Born, Fernando Diaz, Mary L. Gray, and Rida Qadri. 2025. Towards a Multidisciplinary Vision for Culturally Inclusive Generative AI (Dagstuhl Seminar 25022).Dagstuhl Reports15, 1 (2025), 33–49. doi:10.4230/DagRep.15.1.33

    2025 doi

  6. [15]

    Black Forest Labs. 2024. FLUX. https://github.com/black-forest-labs/flux

    2024

  7. [16]

    Janet Blake. 2000. On Defining the Cultural Heritage.International & Comparative Law Quarterly49, 1 (2000), 61–85

    2000

  8. [17]

    Bogardus

    Emory S. Bogardus. 1942.Fundamentals of Social Psychology(3 ed.). D. Appleton-Century Company, New York and London

    1942

  9. [18]

    Branham and Shaun K

    Stacy M. Branham and Shaun K. Kane. 2015. The Invisible Work of Accessibility: How Blind Employees Manage Accessibility in Mixed-Ability Workplaces. InProceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility(Lisbon, Portugal)(ASSETS ’15). Ass...

    2015 doi

  10. [19]

    Chris Callison-Burch. 2009. Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. InProceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Philipp Koehn and Rada Mihalcea (Eds.). Association for Computational...

    2009

  11. [20]

    Joseph Chee Chang, Saleema Amershi, and Ece Kamar. 2017. Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems(Denver, Colorado, USA)(CHI ’17). Association for Computing Machi...

    2017 doi

  12. [21]

    Kyla Chasalow and Karen Levy. 2021. Representativeness in statistics, politics, and machine learning. InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 77–89

    2021

  13. [22]

    Khaoula Chehbouni, Mohammed Haddou, Jackie Chi Kit Cheung, and Golnoosh Farnadi. 2025. Neither Valid nor Reliable? Investigating the Use of LLMs as Judges. https://arxiv.org/abs/2508.18076

    2025

  14. [23]

    Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat, Koustuv Sinha, Melissa Hall, Michal Drozdzal, and Adriana Romero-Soriano. 2025. Multi- Modal Language Models as Text-to-Image Model Evaluators. https://arxiv.org/abs/2505.00759 Evaluating AI-Generated Images of Cultural Artifac...

    2025

  15. [24]

    Tim Connell. 2008. The Challenge of Assistive Technology and Braille Literacy. https://www.afb.org/aw/9/1/14277 [Online; accessed 6-September- 2025]

    2008

  16. [25]

    Alex Dow, Jean Garcia-Gathright, Nicholas J Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, and Hanna Wallach

    Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alexandra Chouldechova, P. Alex Dow, Jean Garcia-Gathright, Nicholas J Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, and Hanna Wallach. 2025. Taxonomizing Representational Harms using Speech Act Theory. InFindings o...

    2025 doi

  17. [26]

    Amanda Coston, Anna Kawakami, Haiyi Zhu, Ken Holstein, and Hoda Heidari. 2023. A validity perspective on evaluating the justified use of data-driven decision-making algorithms. In2023 IEEE conference on secure and trustworthy machine learning (SaTML). IEEE, 690–704

    2023

  18. [27]

    Lee J Cronbach and Paul E Meehl. 1955. Construct validity in psychological tests.Psychological bulletin52, 4 (1955), 281

    1955

  19. [28]

    inclusion

    Samantha Dalal, Siobhan Mackenzie Hall, and Nari Johnson. 2024. Provocation: Who benefits from "inclusion" in Generative AI? https: //arxiv.org/abs/2411.09102

    2024

  20. [29]

    Maitraye Das, Alexander J Fiannaca, Meredith Ringel Morris, Shaun K Kane, and Cynthia L Bennett. 2024. From provenance to aberrations: Image creator and screen reader user perspectives on alt text for AI-generated images. InProceedings of the 2024 CHI Conference on Human Facto...

    2024

  21. [30]

    It doesn’t win you friends

    Maitraye Das, Darren Gergle, and Anne Marie Piper. 2019. "It doesn’t win you friends": Understanding Accessibility in Collaborative Writing for People with Vision Impairments.Proc. ACM Hum.-Comput. Interact.3, CSCW, Article 191 (Nov. 2019), 26 pages. doi:10.1145/3359293

    2019 doi

  22. [31]

    Nassim Dehouche and Kullathida Dehouche. 2023. What’s in a text-to-image prompt? The potential of stable diffusion in visual arts education. Heliyon9, 6 (2023), e16757. doi:10.1016/j.heliyon.2023.e16757

    2023 doi

  23. [32]

    Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2023. The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice. https://arxiv.org/abs/2310.00907

    2023

  24. [33]

    Sunipa Dev, Vinodkumar Prabhakaran, Rutledge Chin Feman, Aida Davani, Remi Denton, Charu Kalia, Piyawat L Kumjorn, Madhurima Maji, Rida Qadri, Negar Rostamzadeh, Renee Shelby, Romina Stella, Hayk Stepanyan, Erin van Liemt, Aishwarya Verma, Oscar Wahltinez, Edem Wornyo, Andrew ...

    2026

  25. [34]

    Athiya Deviyani and Fernando Diaz. 2025. Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy. https://arxiv.org/abs/2503. 19828

    2025

  26. [35]

    2025.Exploring Black Communities’ Perceptions and Design Approaches for Building Culturally Tailored AI Systems

    Lisa Egede. 2025.Exploring Black Communities’ Perceptions and Design Approaches for Building Culturally Tailored AI Systems. Association for Computing Machinery, New York, NY, USA, 72–76. https://doi.org/10.1145/3715668.3735629

    2025 doi

  27. [36]

    Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. 2025. Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation. https://arxiv.org/abs/2502.06559

    2025

  28. [37]

    Yannick Exner, Jochen Hartmann, Oded Netzer, and Shunyuan Zhang. 2025. AI in Disguise - How AI-Generated Ads’ Visual Cues Shape Consumer Perception and Performance. doi:10.2139/ssrn.5096969

    2025 doi

  29. [38]

    Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. 2009. Describing objects by their attributes. In2009 IEEE Conference on Computer Vision and Pattern Recognition. 1778–1785. doi:10.1109/CVPR.2009.5206772

    2009 doi

  30. [39]

    Sanjana Gautam, Pranav Narayanan Venkit, and Sourojit Ghosh. 2024. From melting pots to misrepresentations: Exploring harms in Generative AI. arXiv preprint arXiv:2403.10776(2024)

    2024

  31. [40]

    Simret Araya Gebreegziabher, Charles Chiang, Zichu Wang, Zahra Ashktorab, Michelle Brachman, Werner Geyer, Toby Jia-Jun Li, and Diego Gómez-Zará. 2025. MetricMate: An Interactive Tool for Generating Evaluation Criteria for LLM-as-a-Judge Workflow. InProceedings of the 4th Annu...

    2025 doi

  32. [41]

    Sourojit Ghosh, Pranav Narayanan Venkit, Sanjana Gautam, Shomir Wilson, and Aylin Caliskan. 2024. Do Generative AI Models Output Harm while Representing Non-Western Cultures: Evidence from A Community-Centered Approach.Proceedings of the AAAI/ACM Conference on AI, Ethics, and ...

    2024 doi

  33. [42]

    Tarleton Gillespie. 2024. Generative AI and the politics of visibility.Big Data & Society11, 2 (2024), 20539517241252131. doi:10.1177/ 20539517241252131

    2024

  34. [43]

    Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, and Alexandra Chouldechova. 2025. Validating LLM-as-a-Judge Systems under Rating Indeterminacy. https://arxiv.org/abs/2503.05965

    2025

  35. [44]

    Kanika Gupta, Monojit Choudhury, and Kalika Bali. 2012. Mining Hindi-English Transliteration Pairs from Online Hindi Lyrics. InProceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), Nicoletta Calzolari, Khalid Choukri, Thierry Decler...

    2012

  36. [45]

    Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2024. Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?arXiv preprint arXiv:2309.07462(2024)

    2024

  37. [46]

    Bell, Candace Ross, Adina Williams, Michal Drozdzal, and Adriana Romero Soriano

    Melissa Hall, Samuel J. Bell, Candace Ross, Adina Williams, Michal Drozdzal, and Adriana Romero Soriano. 2024. Towards Geographic Inclusion in the Evaluation of Text-to-Image Models. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency(Rio de ...

    2024 doi

  38. [47]

    1997.Representation: Cultural Representations and Signifying Practices

    Stuart Hall (Ed.). 1997.Representation: Cultural Representations and Signifying Practices. Sage Publications, London. 20 Johnson et al

    1997

  39. [48]

    Siobhan Mackenzie Hall, Samantha Dalal, Raesetje Sefala, Foutse Yuehgoh, Aisha Alaagib, Imane Hamzaoui, Shu Ishida, Jabez Magomere, Lauren Crais, Aya Salama, et al. 2025. The Human Labour of Data Work: Capturing Cultural Diversity through World Wide Dishes.arXiv preprint arXiv...

    2025

  40. [49]

    Hamna, Gayatri Bhat, Sourabrata Mukherjee, Faisal Lalani, Evan Hadfield, Divya Siddarth, Kalika Bali, and Sunayana Sitaram. 2025. Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settings. https://arxiv.org/abs/2509.24506

    2025

  41. [50]

    Hamna, Deepthi Sudharsan, Agrima Seth, Ritvik Budhiraja, Deepika Khullar, Vyshak Jain, Kalika Bali, Aditya Vashistha, and Sameer Segal. 2025. Kahani: Culturally-Nuanced Visual Storytelling Tool for Non-Western Cultures. InProceedings of the 2025 ACM SIGCAS/SIGCHI Conference on...

    2025 doi

  42. [51]

    Emma Harvey, Emily Sheng, Su Lin Blodgett, Alexandra Chouldechova, Jean Garcia-Gathright, Alexandra Olteanu, and Hanna Wallach. 2025. Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems. https://arxiv.org/abs/ 2506.04482

    2025

  43. [52]

    Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. 2024. LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    2024 doi

  44. [53]

    Huiguo He, Huan Yang, Zixi Tuo, Yuan Zhou, Qiuyue Wang, Yuhang Zhang, Zeyu Liu, Wenhao Huang, Hongyang Chao, and Jian Yin. 2025. Dreamstory: Open-domain story visualization by llm-guided multi-subject consistent diffusion.IEEE Transactions on Pattern Analysis and Machine Intel...

    2025

  45. [54]

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2018. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. https://arxiv.org/abs/1706.08500

    2018 arXiv

  46. [55]

    Rachel Hong, William Agnew, Tadayoshi Kohno, and Jamie Morgenstern. 2024. Who’s in and who’s out? A case study of multimodal CLIP-filtering in DataComp. InProceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 1–17

    2024

  47. [56]

    Sandford

    Chien-Chi Hsu and Brian A. Sandford. 2007. The Delphi technique: Making sense of consensus.Practical Assessment, Research, and Evaluation12, 10 (2007), 1–8. https://openpublishing.library.umass.edu/pare/article/id/1418/ A widely cited methodological overview of the Delphi method

    2007

  48. [57]

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering. https://arxiv.org/abs/2303.11897

    2023

  49. [58]

    Mina Huh, Yi-Hao Peng, and Amy Pavel. 2023. GenAssist: Making image generation accessible. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–17

    2023

  50. [59]

    Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. 2024. Rethinking FID: Towards a Better Evaluation Metric for Image Generation. https://arxiv.org/abs/2401.09603

    2024

  51. [60]

    Akshita Jha, Vinodkumar Prabhakaran, Remi Denton, Sarah Laszlo, Shachi Dave, Rida Qadri, Chandan K Reddy, and Sunipa Dev. 2024. Visage: A global-scale analysis of visual stereotypes in text-to-image generation.arXiv preprint arXiv:2401.06310(2024)

    2024

  52. [61]

    Jiang, Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru

    Harry H. Jiang, Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru

  53. [62]

    InProceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society(Montréal, QC, Canada)(AIES ’23)

    AI Art and its Impact on Artists. InProceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society(Montréal, QC, Canada)(AIES ’23). Association for Computing Machinery, New York, NY, USA, 363–374. doi:10.1145/3600211.3604681

    2023 doi

  54. [63]

    Nari Johnson, Hamna Abid, Deepthi Sudharsan, Theo Holroyd, Samantha Dalal, Siobhan Mackenzie Hall, Jennifer Wortman Vaughan, Daniela Massiceti, and Cecily Morrison. 2025. Position: To Make Text-to-Image Models that Work for Marginalized Communities, We Need New Measurement Pra...

    2025

  55. [64]

    Shivani Kapania, Stephanie Ballard, Alex Kessler, and Jennifer Wortman Vaughan. 2025. Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline. InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency

    2025

  56. [65]

    2025.Translation Tutorial: AI Measurement as a Stakeholder-Engaged Design Practice

    Anna Kawakami, Su Lin Blodgett, Solon Barocas, Alex Chouldechova, Abigail Jacobs, Emily Sheng, Jenn Wortman Vaughan, Hanna Wallach, Amy Winecoff, Angelina Wang, Haiyi Zhu, and Ken Holstein. 2025.Translation Tutorial: AI Measurement as a Stakeholder-Engaged Design Practice. Ret...

    2025

  57. [66]

    Anna Kawakami, Jordan Taylor, Sarah Fox, Haiyi Zhu, and Kenneth Holstein. 2026. AI failure loops in devalued work: The confluence of overconfidence in AI and underconfidence in worker expertise.Big Data & Society13, 1 (2026), 20539517261424164. doi:10.1177/20539517261424164

    2026 doi

  58. [67]

    Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024. The PRISM Alignment Project: What Participatory, Representative and Individualised Human Feedback Reveals About t...

    2024

  59. [68]

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. 2023. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. https://arxiv.org/abs/2305.01569

    2023

  60. [69]

    Kevin Knight and Jonathan Graehl. 1998. Machine Transliteration.Computational Linguistics24, 4 (1998), 599–612. https://aclanthology.org/J98- 4003/

    1998

  61. [70]

    Elisa Kreiss, Cynthia Bennett, Shayan Hooshmand, Eric Zelikman, Meredith Ringel Morris, and Christopher Potts. 2022. Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics.arXiv preprint arXiv:2205.10646(2022). Evaluating AI-G...

    2022

  62. [71]

    Neha Kumar, Naveena Karusala, Azra Ismail, Marisol Wong-Villacres, and Aditya Vishwanath. 2019. Engaging Feminist Solidarity for Comparative Research, Design, and Practice.Proc. ACM Hum.-Comput. Interact.3, CSCW, Article 167 (Nov. 2019), 24 pages. doi:10.1145/3359269

    2019 doi

  63. [72]

    C., Avik Bhattacharyya, Mitesh M

    Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Gokul N. C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages. https://arxiv.org/abs/2005.00085

    2020

  64. [73]

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, and Percy Liang. 2023. Holis...

    2023

  65. [74]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge. https: //arxiv.o...

    2025

  66. [75]

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. Evaluating Text-to-Visual Generation with Image-to-Text Generation. https://arxiv.org/abs/2404.01291

    2024

  67. [76]

    Smith, and Fannie Liu

    Kelly Mack, Rai Ching Ling Hsu, Andrés Monroy-Hernández, Brian A. Smith, and Fannie Liu. 2023. Towards Inclusive Avatars: Disability Representation in Avatar Platforms. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems(Hamburg, Germany)(CHI ’23). A...

    2023 doi

  68. [77]

    They only care to show us the wheelchair

    Kelly Avery Mack, Rida Qadri, Remi Denton, Shaun K Kane, and Cynthia L Bennett. 2024. “They only care to show us the wheelchair”: disability representation in text-to-image AI models. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–23

    2024

  69. [78]

    Jabez Magomere, Shu Ishida, Tejumade Afonja, Aya Salama, Daniel Kochin, Yuehgoh Foutse, Imane Hamzaoui, Raesetje Sefala, Aisha Alaagib, Samantha Dalal, et al . 2025. The World Wide recipe: A community-centred framework for fine-grained data collection and regional bias operati...

    2025

  70. [79]

    Daniela Massiceti, Camilla Longden, Agnieszka Slowik, Samuel Wills, Martin Grayson, and Cecily Morrison. 2024. Explaining CLIP’s performance disparities on data from blind/low vision users. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12...

    2024

  71. [80]

    Nathan Matias and Megan Price

    J. Nathan Matias and Megan Price. 2025. How public involvement can improve the science of AI.Proceedings of the National Academy of Sciences 122, 48 (2025), e2421111122. doi:10.1073/pnas.2421111122

    2025 doi

  72. [81]

    Timothy R McIntosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Dan Xu, Paul Watters, and Malka N Halgamuge. 2025. Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence.IEEE Transactions on Artificial Intelligence(2025), 1–18. doi:10.1...

    2025 doi

  73. [82]

    Annika Muehlbradt and Shaun K. Kane. 2022. What’s in an ALT Tag? Exploring Caption Content Priorities through Collaborative Captioning. ACM Trans. Access. Comput.15, 1, Article 6 (March 2022), 32 pages. doi:10.1145/3507659

    2022 doi

  74. [83]

    1988.A Textbook of Translation

    Peter Newmark. 1988.A Textbook of Translation. Vol. 66. Prentice Hall, New York

    1988

  75. [84]

    OpenAI. 2022. DALL·E now available without waitlist. https://openai.com/index/dall-e-now-available-without-waitlist/ Accessed: 2024-05-28

    2022

  76. [85]

    OpenAI. 2024. DALL·E 3. https://openai.com/index/dall-e-3/. Accessed: 2025-09-11

    2024

  77. [86]

    OpenAI. 2025. GPT-Image-1. https://platform.openai.com/docs/guides/image-generation?image-generation-model=gpt-image-1. Accessed: 2025-09-11

    2025

  78. [87]

    OpenAI and others. 2024. GPT-4o System Card. https://arxiv.org/abs/2410.21276

    2024 arXiv

  79. [88]

    OpenAI Developer Community Forum. 2023. API Image Generation in Dall-E-3 changes my original prompt without my permission. https: //community.openai.com/t/api-image-generation-in-dall-e-3-changes-my-original-prompt-without-my-permission/476355/2 [Online; accessed 6-September-2025]

    2023

  80. [89]

    Iroro Orife, Julia Kreutzer, Blessing Sibanda, Daniel Whitenack, Kathleen Siminyu, Laura Martinus, Jamiil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, Musie Meressa, Espoir Murhabazi, Orevaoghene Ahia, Elan van Biljon, Arshath Ramkilowan, Adewale Akinfaderin, Alp ...

    2020

  81. [90]

    Will Orr and Edward B. Kang. 2024. AI as a Sport: On the Competitive Epistemologies of Benchmarking. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency(Rio de Janeiro, Brazil)(FAccT ’24). Association for Computing Machinery, New York, NY, US...

    2024 doi

  82. [91]

    Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin’ichi Satoh. 2023. Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2023

  83. [92]

    Lawrence Palinkas, Sarah Horwitz, Carla Green, Jennifer Wisdom, Naihua Duan, and Kimberly Hoagwood. 2013. Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research.Administration and policy in mental health42 (11 2013). doi:10.100...

    2013 doi

  84. [93]

    Qian Pan, Zahra Ashktorab, Michael Desmond, Martín Santillán Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. 2024. Human-Centered Design Recommendations for LLM-as-a-judge. InProceedings of the 1st Human-Centered Large Language Modeling Workshop, Nikita So...

    2024 doi

  85. [94]

    Thick Evaluations

    Rida Qadri, Mark Diaz, Ding Wang, and Michael Madaio. 2025. The Case for "Thick Evaluations" of Cultural Representation in AI.arXiv preprint arXiv:2503.19075(2025). 22 Johnson et al

    2025

  86. [95]

    Rida Qadri, Renee Shelby, Cynthia L Bennett, and Remi Denton. 2023. Ai’s regimes of representation: A community-centered study of text-to-image models in south asia. InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 506–517

    2023

  87. [96]

    Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna

    Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021. AI and the Everything in the Whole Wide World Benchmark. https://arxiv.org/abs/2111.15366

    2021

  88. [97]

    David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David L...

    2024

  89. [98]

    Paul Röttger, Fabio Pernisi, Bertie Vidgen, and Dirk Hovy. 2025. SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety. https://arxiv.org/abs/2404.05399

    2025

  90. [99]

    Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. 2025. Measurement to Meaning: A Validity-Centered Framework for AI Evaluation. https://arxiv.org/abs/2505.10573

    2025

  91. [100]

    Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. 2024. Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2). InAdvances in Neural Information Processing Systems, A. Glo...

    2024

  92. [101]

    Agrima Seth, Sanchit Ahuja, Kalika Bali, and Sunayana Sitaram. 2024. Dosa: A dataset of social artifacts from different indian geographical subcultures.arXiv preprint arXiv:2403.14651(2024)

    2024

  93. [102]

    Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo

    Shreya Shankar, J.D. Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. InProceedings of the 37th Annual ACM Symposium on User Interface Software ...

    2024 doi

  94. [103]

    Devinder Singh and Manoj Sharma. 2009. Unfolding the Indian cultural mosaic: a cross-cultural study of four regional cultures.International Journal of Indian Culture and Business Management - Int J Indian Cult Bus Manag2 (01 2009). doi:10.1504/IJICBM.2009.023547

    2009 doi

  95. [104]

    Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2020. Participation is not a Design Fix for Machine Learning. https: //arxiv.org/abs/2007.02423

    2020

  96. [105]

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. A Roadmap to Pluralistic Alignment. https://arxiv.org/abs/2402.05070

    2024 arXiv

  97. [106]

    2024.Announcing the Open Release of Stable Diffusion 3 Medium, Our Most Sophisticated Image Generation Model to Date

    Stability AI. 2024.Announcing the Open Release of Stable Diffusion 3 Medium, Our Most Sophisticated Image Generation Model to Date. https: //stability.ai/news/stable-diffusion-3-medium Accessed: 2025-09-11

    2024

  98. [107]

    2024.Introducing Stable Diffusion 3.5

    Stability AI. 2024.Introducing Stable Diffusion 3.5. https://stability.ai/news/introducing-stable-diffusion-3-5 Accessed: 2025-09-11

    2024

  99. [108]

    Harini Suresh, Divya Shanmugam, Tiffany Chen, Annie G Bryan, Alexander D’Amour, John Guttag, and Arvind Satyanarayan. 2023. Kaleidoscope: Semantically-grounded, context-specific ML model evaluation. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems...

    2023 doi

  100. [109]

    Harini Suresh, Emily Tseng, Meg Young, Mary Gray, Emma Pierson, and Karen Levy. 2024. Participation in the Age of Foundation Models. InThe 2024 ACM Conference on Fairness, Accountability, and Transparency. 1609–1621

    2024

  101. [110]

    Metoyer, and Toby Jia-Jun Li

    Annalisa Szymanski, Simret Araya Gebreegziabher, Oghenemaro Anuyah, Ronald A. Metoyer, and Toby Jia-Jun Li. 2024. Comparing Criteria Development Across Domain Experts, Lay Users, and Models in Large Language Model Evaluation. https://arxiv.org/abs/2410.02054

    2024

  102. [111]

    Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A

    Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer. 2024. Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks. https://arxiv.org/abs/2410.20266

    2024

  103. [112]

    S. J. Tambiah. 1967. Social Change in Modern India. By M. N. Srinivas. University of California Press: Berkeley and Los Angeles, and Cambridge University Press: London, 1966. Pp. xv + 194, 40s.Modern Asian Studies1, 4 (1967), 404–405. doi:10.1017/S0026749X00002687

    1967 doi

  104. [113]

    Brubaker, Sarah E Fox, and Haiyi Zhu

    Jordan Taylor, Ellen Simpson, Anh-Ton Tran, Jed R. Brubaker, Sarah E Fox, and Haiyi Zhu. 2024. Cruising Queer HCI on the DL: A Literature Review of LGBTQ+ People in HCI. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24)....

    2024 doi

  105. [114]

    The Royal National Institute of Blind People. 2025. About us. https://www.rnib.org.uk/about-us/ [Online; accessed 6-September-2025]

    2025

  106. [115]

    Anja Thieme, Rita Faia Marques, Martin Grayson, Sidhika Balachandar, Cameron Tyler Cassidy, Madiha Zahrah Choksi, Camilla Longden, Reeda Shimaz Huda, Nicholas Ileve Kalovwe, Christina Mallon, Courtney Mansperger, Daniela Massiceti, Bhaskar Mitra, Ruth Mueni Nzioka, Ioana Evalu...

    2026

  107. [116]

    Ownership, Not Just Happy Talk

    Emily Tseng, Meg Young, Marianne Aubin Le Quéré, Aimee Rinehart, and Harini Suresh. 2025. "Ownership, Not Just Happy Talk": Co-Designing a Participatory Large Language Model for Journalism. InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency ...

    2025 doi

  108. [117]

    Sruthi Viswanathan, Seray Ibrahim, Ravi Shankar, Reuben Binns, Max Van Kleek, and Petr Slovak. 2025. The Interaction Layer: An Exploration for Co-Designing User-LLM Interactions in Parental Wellbeing Support Systems. InProceedings of the 2025 CHI Conference on Human Factors in...

    2025 doi

  109. [118]

    Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alexandra Chouldechova, Emily Corvi, P

    Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alexandra Chouldechova, Emily Corvi, P. Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nicholas Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaugha...

    2025

  110. [119]

    2003.All the rage: The story of gay visibility in America

    Suzanna Danuta Walters. 2003.All the rage: The story of gay visibility in America. University of Chicago Press

    2003

  111. [120]

    Laura Weidinger, Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Anne Hendricks, Juan Mateos-Garcia, Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, et al. 2023. Sociotechnical Safety Evaluation of Generative AI Systems.arXiv preprint arXiv:2310.11986(2023)

    2023

  112. [121]

    1976.Computer Power and Human Reason: From Judgment to Calculation

    Joseph Weizenbaum. 1976.Computer Power and Human Reason: From Judgment to Calculation. W. H. Freeman & Co., USA

    1976

  113. [122]

    1998.Communities of Practice: Learning, Meaning, and Identity

    Etienne Wenger. 1998.Communities of Practice: Learning, Meaning, and Identity. Cambridge University Press

    1998

  114. [123]

    Kimi Wenzel, Nitya Devireddy, Cam Davison, and Geoff Kaufman. 2023. Can Voice Assistants Be Microaggressors? Cross-Race Psychological Responses to Failures of Automatic Speech Recognition. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg,...

    2023 doi

  115. [124]

    Kimi Wenzel, Avijit Ghosh, Sachin Pendse, Stephanie Milani, Ajeet Singh, Laura Dabbish, and Geoff Kaufman. 2025. Invisible by Design? Generative AI and Mirrors of Misrepresentation. InProceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT) CRAFT...

    2025

  116. [125]

    Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, et al. 2024. WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual ...

    2024

  117. [126]

    Binwei Yao, Ming Jiang, Tara Bobinac, Diyi Yang, and Junjie Hu. 2024. Benchmarking Machine Translation with Cultural Awareness. InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Co...

    2024 doi

  118. [127]

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. https://arxiv.org/abs/2410.02736

    2024 arXiv

  119. [128]

    Meg Young. 2025. Participatory AI? Begin with the Most Affected People.TechPolicy.Press(February 19 2025). https://www.techpolicy.press/ participatory-ai-begin-with-the-most-affected-people/ Essay published as part of the Participatory AI Research & Practice Symposium reflecti...

    2025

  120. [129]

    Meg Young, Uphol Ehsan, Ranjit Singh, Emnet Tafesse, Michele Gilman, Christina Harrington, and Jacob Metcalf. 2024. Participation versus Scale: Tensions in the Practical Demands on Participatory AI.First Monday(2024)

    2024

  121. [130]

    Gray, and Jina Suh

    Alice Qian Zhang, Judith Amores, Hong Shen, Mary Czerwinski, Mary L. Gray, and Jina Suh. 2025. AURA: Amplifying Understanding, Resilience, and Awareness for Responsible AI Content Work.Proc. ACM Hum.-Comput. Interact.9, 2, Article CSCW033 (May 2025), 45 pages. doi:10.1145/3710931

    2025 doi

  122. [131]

    It’s Just Part of Me:

    Kexin Zhang, Elmira Deldari, Zhicong Lu, Yaxing Yao, and Yuhang Zhao. 2022. “It’s Just Part of Me:” Understanding Avatar Diversity and Self-presentation of People with Disabilities in Social Virtual Reality. InProceedings of the 24th International ACM SIGACCESS Conference on C...

    2022 doi

  123. [132]

    Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. 2023. GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks. https://arxiv.org/abs/2311.01361

    2023

  124. [133]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. https://arxiv.org/abs/2306.05685

    2023 arXiv

  125. [134]

    does the guide cane have sections that are white in color?

    Naitian Zhou, David Bamman, and Isaac L Bleaman. 2025. Culture is not trivia: Sociocultural theory for cultural nlp.arXiv preprint arXiv:2502.12057 (2025). 24 Johnson et al. A Systematized Concepts As described in the main text, the systematization of cultural appropriateness ...

    2025

  126. [135]

    A photo of a guide cane

    An image generated from the prompt "A photo of a guide cane"

  127. [136]

    Evaluation Criteria 36 Johnson et al

    A set of evaluation criteria, grouped by theme where each theme includes a description that should inform your interpretation of the criteria within that theme. Evaluation Criteria 36 Johnson et al. Theme1 The object in the image should not be confused with other, more hegemon...

  128. [137]

    Your understanding of a guide cane

  129. [138]

    criteria_evaluation

    The description of the criteria, and the theme it belongs to. For each criterion, respond with: 1 if the image meets the criterion. 0 if the image does not meet the criterion. Return your evaluation as a valid JSON object with the following structure: { "criteria_evaluation": ...

  130. [139]

    One participant (B5) invited a visually impaired friend (VIP5) to participate as their buddy

    Cannot be shown Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics 45 IDs Relationship B’s Age B’s Location Objects discussed B1/SP1 Friends 18–34 Sheffield, England Braille notetaker B2/SP2 Friends 18–34 London Area, England Guide cane, brail...

  131. [140]

    A media company has collected several images of a guide cane, and they need your help to understand which of these images they should show to users

    to understand blind community members not as passive recipients of assistance, but to instead recognize their expertise in creating access throughout the collaboration. When collaborating, we emphasized the unique skills of each participant: the blind community member as the e...

    2024

Pith tools

Reviewed May 21, 2026 · model on record in the stance chip above.