Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context

T0 review · 2 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BharatBBQ, a culturally adapted QA benchmark spanning eight languages, shows that five multilingual LLM families display more social bias in Indian languages than in English, and proposes a stereotypical bias score that surfaces bias standa

desk verdict Useful India-specific bias resource, but the headline claim that models are more biased in Indian languages than English is confounded by language proficiency and needs a control before it can be taken at face value. read the letter →

arxiv 2508.07090 v1 pith:P7ZU33W3 submitted 2025-08-09 cs.CL

Aditya Tomar , Nihar Ranjan Sahoo , Pushpak Bhattacharyya This is my paper
classification cs.CL
keywords socialbiaslanguagemodelsquestionansweringbenchmarkIndianlanguagesstereotypemeasurementmultilingualevaluationcasteBharatBBQ
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BharatBBQ adapts the English-language BBQ question-answering bias benchmark to the Indian context, adding categories such as caste, region, and intersectional religion×gender, and expands each item to seven Indian languages via machine translation with back-translation verification. The paper's central claim is that five multilingual LLM families, evaluated in zero- and few-shot settings, score higher on both bias and stereotypical bias in Hindi, Marathi, Bengali, Tamil, Telugu, Odia, and Assamese than in English. It also introduces a Stereotypical Bias Score (SBS) that separates stereotype-driven answers from generic lexical preferences, and argues this metric reveals bias that the standard BBQ bias score misses. A sympathetic reader would care because, if the claim holds, Western-centric benchmarks understate the stereotypes Indian-language users actually face, making BharatBBQ a practical auditing instrument for models deployed in Indian languages.

What carries the argument

The central object is the BBQ-style QA template: an ambiguous or disambiguated context, a negative or non-negative question, and three options—stereotyped group, non-stereotyped group, and Unknown. BharatBBQ's cultural adaptation combines four operations: removing non-Indian stereotypes, replacing U.S. entities with Indian ones, modifying target groups (for example, Pakistan in terrorism items), and creating new templates for caste, region, and intersectional categories. Multilingual coverage rests on IndicTransv2 translation followed by back-translation cosine-similarity filtering at a threshold of 0.75 using modernBERT embeddings, with manual correction below the threshold. The argumentati

What would settle it

Build a small gold set per language in which professional human translators, blind to the stereotype, render the same English contexts, then compare model bias scores on machine-translated versus human-translated items. If the Indian-language bias premium disappears or shrinks sharply on human-translated items, the paper's central gap is a translation artifact; if it persists, the claim is supported. A second check would run the same five models on items translated into a non-Indian control language to see whether the gap tracks translationese rather than Indian cultural content.

Watch

Extended reading notes

Core claim

The discovery is a measurement: model bias is language-dependent, and existing bias scores understate it. Using ambiguous contexts where 'Unknown' is the correct answer and disambiguated contexts where a stated fact resolves the question, BharatBBQ measures how often models choose a stereotyped versus non-stereotyped answer. The new Stereotypical Bias Score counts, for negative and non-negative questions separately, the choices aligned with the stereotype; this isolates cases where the model links the stereotyped group to the negative trait rather than simply preferring one answer word. Evaluated on five open multilingual LLM families, the benchmark finds persistent bias across all 13 catego

Load-bearing premise

The claim rests on the belief that mechanically translating the English templates into seven Indian languages—keeping only items whose back-translation matches above a cosine threshold—preserves the stereotype's target, valence, and pragmatic force in every language; if translation shifts any of these, the reported English-versus-Indian gap is an artifact of the translator rather than a genuine model bias.

Editorial extensions

If this is right

  • If BharatBBQ measures what it claims, auditing an Indian-language LLM with it will reveal biases that an English-only BBQ evaluation masks.
  • The finding that Indian languages show more bias than English implies that users interacting with models in Hindi, Bengali, Tamil, and the other covered languages face systematically higher stereotype reinforcement, not just lower accuracy.
  • Because SBS and the original bias score diverge, evaluation suites that report only the bias score can miss stereotypical bias; QA-style bias benchmarks should report both metrics.
  • Few-shot examples reduce SBS relative to zero-shot, indicating that in-context exemplars can partly counteract stereotype reliance in Indian languages.
  • Model-scaling effects are non-uniform: larger Gemma models show more bias, while larger Llama and Sarvam models show less, so bias auditing must be per-model rather than assumed from parameter count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the Indian-language gap is rooted in training data, weaker instruction-following in Indian languages, or the translation pipeline itself; if translation is the driver, the gap would shrink under human-translated controls.
  • The SBS logic—separating negative and non-negative questions—could generalize to any QA bias benchmark that uses both question polarities, such as re-analyzing earlier multilingual BBQ variants in other languages.
  • A practical, testable extension is to build a small human-translated gold subset across all seven Indian languages to bound translation-induced measurement error; this would directly test the paper's core comparability assumption.
  • If adopted in practice, the benchmark suggests that fairness assurances based on English evaluations substantially overstate safety for Indian-language users.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces BharatBBQ, a multilingual question-answering bias benchmark adapted from BBQ to the Indian context. It covers 13 social categories, including caste, region, and intersectional axes, and extends 49,108 English examples into seven Indian languages through IndicTransv2 translation, back-translation cosine filtering, and manual correction, for a total of 392,864 examples. The authors evaluate five multilingual LLM families in zero- and few-shot settings using accuracy, the existing bias score (BS), and a newly proposed stereotypical bias score (SBS). The central empirical claim is that models exhibit 'significantly more bias' in Indian languages than in English (§6.7, Abstract, Contribution 2). The paper also reports category-wise, proper-noun vs common-noun, model-size, and BBQ-vs-BharatBBQ comparisons.

Significance. If the headline finding holds, BharatBBQ would be a valuable resource: it is the first large-scale QA bias benchmark targeted at Indian sociocultural contexts, with careful template adaptation, multilingual expansion, and a public release. The proposed SBS metric is a genuinely useful refinement over the standard BBQ bias score because it separates stereotype-consistent answering from generic lexical preference. The paper's strengths include the systematic template validation (Cohen's kappa = 0.83), the human translation-quality assessment, and the breadth of the evaluation across models and languages. However, the central cross-lingual claim is not established to the standard the paper's own language ('significantly more bias') demands, because the reported differences are not accompanied by statistical testing or by a control for the large cross-lingual accuracy gap. The benchmark itself is sound and useful; the empirical headline is conditional.

major comments (2)
  1. [§6.7, Fig. 5, Abstract, Contribution 2] The paper claims models exhibit 'significantly more bias' in Indian languages than in English, but no statistical tests, confidence intervals, or effect sizes are reported; Fig. 5 shows point estimates only. More importantly, the comparison does not control for language proficiency. §6.1 (Fig. 4) shows substantially lower disambiguated-context accuracy in Indian languages; BSD (Eq. 4) and SBSD (Eq. 6) are computed over non-unknown responses and count failures to follow disambiguating cues as stereotypical choices. A model that cannot parse the Telugu or Tamil context can produce higher SBS without any difference in stereotype endorsement. This is exactly the pattern the paper itself acknowledges in §6.7 ('weaker contextual understanding ... increasing SBS'). The Limitations paragraph says causes are not investigated, but that leaves the headline claim unprotected against this confound. P
  2. [§3.4, §A.1, Table 3] Cross-lingual comparability rests on the assumption that the translated items preserve the stereotype target, valence, and pragmatic force of the English templates. The human evaluation samples only 100 sentences per language and scores fluency and adequacy; it does not test whether the translated item still instantiates the intended stereotype (or its negation) with the same strength. The back-translation cosine threshold of 0.75 is a coarse semantic filter and can pass translations that shift a target group or soften a stereotype. Because the central comparison is English vs Indian languages, a systematic translation artifact would produce exactly the reported symptom. Please report per-language and per-category retention rates and conduct a focused annotation study that checks whether the translated item targets the intended group and whether the negative/non-negative question has the
minor comments (5)
  1. [§6.1] The text first states that 'all models show lower accuracy when prompted in target language compared to their English-instruction counterparts' and then says 'accuracy remains nearly invariant between English and native-language instructions.' These statements are contradictory and should be reconciled; the choice to use English instructions for the main analysis depends on this point.
  2. [Table 1] Table 1 is heavily garbled in the manuscript: column headers and row labels are jumbled (e.g., '# of Templates# ofTemplates# ofExamples# ofExamples'). The table needs to be reformatted so that SR, TM/CT, ST, NC, and PN are clearly aligned with the category rows.
  3. [Footnote 1 and Data Availability] The text says 'Dataset and Code' are released, but the footnote does not contain a working URL or repository identifier in the visible text. Please include the actual link or an anonymous access mechanism for review.
  4. [Throughout §6.2, §6.7] Phrases like 'Bengali and Tamil have more BS and SBS among Indian languages' and 'Bloomz ... shows even larger scores' are reported as if they are established differences. Without confidence intervals or pairwise tests, these observations should be phrased as descriptive trends.
  5. [§6.4 and Tables 8–9] The model names are inconsistent: the setup section names 'sarvam-2b-v0.5', but the results discussion refers to 'Sarvam-1' and 'Sarvam-M'. Please define the mapping clearly.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: BharatBBQ is an empirical measurement resource and the cross-lingual bias finding is not a construction artifact.

full rationale

The paper's central claim—that models exhibit more bias in Indian languages than English—is an empirical measurement against a fixed benchmark, not a derived prediction fitted to its own outputs. The bias scores (BS, SBS) are defined from model answer counts and applied to external model families; no equation reduces a prediction to its input. The multilingual extension uses translation and back-translation with a cosine-similarity threshold, but this is a dataset-construction quality filter, not a fitted parameter renamed as a result. The human evaluation in Appendix A.1 validates translation quality on sampled items and does not define the headline outcome. The self-citations (IndiBias, Hollywood Identity Bias) appear in related work and stereotype sourcing; they are not used as load-bearing evidence for the central claim, and no uniqueness theorem is imported from the authors' prior work. The limitation that the paper does not investigate causes of the cross-lingual gap is a validity concern (e.g., proficiency confounds), not circularity. The benchmark is additionally compared against the external BBQ benchmark in Section 6.5, and the model evaluations are self-contained. No step in the derivation chain reduces by construction to its own inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of the translated benchmark and the stereotype seed set; these are domain assumptions rather than derived results. The only hand-chosen numbers are the translation threshold and the stereotype mention cutoff.

free parameters (2)
  • back-translation cosine threshold = 0.75
    Hand-set after manual verification of semantic similarity; items below it are manually corrected, so the threshold defines what counts as an acceptable translation for the multilingual benchmark (Section 3.4).
  • stereotype retention threshold = 3 mentions
    A stereotype concept is kept only if at least three survey respondents mention it; this is a hand-chosen filter for the stereotype seed set (Section 3.5).
assumptions (4)
  • domain assumption Machine-translated and back-translated items with cosine similarity above 0.75 preserve the original stereotype's meaning and social force in all seven target languages
    The cross-lingual comparison of bias scores depends on this; only 100 sampled sentences per language received human fluency/adequacy ratings (Section 3.4, Appendix A.1).
  • domain assumption The stereotype concepts collected from 241 open-form responses and filtered at 3 mentions are representative of Indian social stereotypes
    The form is an uncontrolled convenience sample and the 3-mention cutoff is arbitrary; all templates are built from this seed set (Section 3.5).
  • domain assumption The BBQ-style ambiguous/disambiguated QA structure measures bias equivalently across all eight languages
    The benchmark assumes that 'Unknown' as ground truth for ambiguous contexts and the negative/non-negative question split transfer without language-specific distortion (Section 3).
  • domain assumption Average log-likelihood scoring of the three answer options reflects the model's actual choice behavior in every language
    The evaluation scores options by continuation probability rather than generation; this proxy is standard but not validated per model/language (Appendix C).

how reviews work

0 comments
Cite this review

Pith. "Pith review of BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context." pith.science (2026). https://pith.science/paper/P7ZU33W3

@misc{pith2026250807090,
  author       = {Pith},
  title        = {Pith review of: BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P7ZU33W3}},
  note         = {Machine review of arXiv:2508.07090}
}
read the original abstract

Evaluating social biases in language models (LMs) is crucial for ensuring fairness and minimizing the reinforcement of harmful stereotypes in AI systems. Existing benchmarks, such as the Bias Benchmark for Question Answering (BBQ), primarily focus on Western contexts, limiting their applicability to the Indian context. To address this gap, we introduce BharatBBQ, a culturally adapted benchmark designed to assess biases in Hindi, English, Marathi, Bengali, Tamil, Telugu, Odia, and Assamese. BharatBBQ covers 13 social categories, including 3 intersectional groups, reflecting prevalent biases in the Indian sociocultural landscape. Our dataset contains 49,108 examples in one language that are expanded using translation and verification to 392,864 examples in eight different languages. We evaluate five multilingual LM families across zero and few-shot settings, analyzing their bias and stereotypical bias scores. Our findings highlight persistent biases across languages and social categories and often amplified biases in Indian languages compared to English, demonstrating the necessity of linguistically and culturally grounded benchmarks for bias evaluation.

Figures

Figures reproduced from arXiv: 2508.07090 by the authors.

Figure 1
Figure 1. Examples in BharatBBQ feature ambiguous contexts with "Unknown" as the ground truth and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Expected Answer to different conditions in BharatBBQ [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Constructional adaptations in BharatBBQ. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Zero-Shot Accuracy for English and Target Instructions, averaged across 13 categories. language instructions, indicating its robust perfor￾mance irrespective of the instruction language. In general, accuracy for ambiguous contexts (♦) is lower compared to disambiguated…
Figure 5
Figure 5. Figure 5: Bias and Stereotypical Bias Scores using English instructions in Zero-Shot and Few-Shot settings across 8 languages for 5 models, averaged over 13 categories. The first subgraph represents Bias Scores in Zero-Shot, the second represents Bias Scores in Few-Shot, the thi…
Figure 6
Figure 6. Figure 6: Stereotypical Bias Scores (SBS) for instances using [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Bias Scores (BS) and Stereotypical Bias Scores (SBS) across 13 categories and 8 languages in both Ambiguous and Disambiguated contexts, averaged over 5 models. sistently exhibit lower BS and SBS in English, re￾gardless of context type [PITH_FULL_IMAGE:figures/full_fig…
Figure 8
Figure 8. Figure 8: Zero-Shot Model Size Comparison: Bias and Stereotypical Bias Scores in English Instruction (Discussed in §6.4) (a) Bias Score (b) Stereotypical Bias Score [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Few-Shot Model Size Comparison: Bias and Stereotypical Bias Scores in English Instruction. (Discussed in §6.4) [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Few-Shot Accuracy for English and Target Instructions, averaged across 13 categories (dis￾cussed in §6.1) [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Stereotypical Bias Scores for instances using proper nouns vs common nouns across 4 categories in 8 languages. The line represents the mean Stereotypical Bias Score averaged across all categories and languages in Ambiguous and Disambiguated Context. The shaded region …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Intersectional Fairness in Large Language Models

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    LLMs are more accurate when answers match stereotypes in clear contexts, especially for race-gender combinations, and no tested model shows consistent fairness or reliability across intersectional groups.

Reference graph

Works this paper leans on

38 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Panatchakorn Anantaprayoon, Masahiro Kaneko, and Naoaki Okazaki. 2024. http://arxiv.org/abs/2309.09697 Evaluating gender bias of pre-trained language models in natural language inference by considering all labels

    work page Pith review arXiv 2024

  4. [5]

    Zaheer Baber. 2022 b . https://doi.org/10.1177/0896920521992093 ‘race might be a unicorn, but its horn could draw blood’: Racialisation, class and racism in a non-western context . Critical Sociology, 48(1):151--169

    work page doi:10.1177/0896920521992093 2022

  5. [6]

    Mukul Bhutani, Kevin Robinson, Vinodkumar Prabhakaran, Shachi Dave, and Sunipa Dev. 2024. https://doi.org/10.18653/v1/2024.acl-short.75 S ee GULL multilingual: a dataset of geo-culturally situated stereotypes . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 842--854, Bangkok, Thai...

    doi:10.18653/v1/2024.acl-short.75 2024

  6. [7]

    Business Standard . 2023. https://www.business-standard.com/world-news/india-only-nation-with-no-history-of-antisemitism-diaspora-supports-israel-123102400069_1.html India only nation with no history of antisemitism, diaspora supports israel . Accessed: 2025-06-15

    work page 2023

  7. [8]

    Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. https://aclanthology.org/W07-0718/ (meta-) evaluation of machine translation . In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136--158, Prague, Czech Republic. Association for Computational Linguistics

    work page 2007

  8. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. http://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge

    arXiv 2018

Show all 38 references
  1. [10]

    Jacob Cohen. 1960. https://doi.org/10.1177/001316446002000104 A coefficient of agreement for nominal scales . Educational and Psychological Measurement, 20(1):37--46

    1960 doi

  2. [11]

    CSPS. 2023. https://cspsindia.org/antisemitism-and-perception-of-hitler-in-india Antisemitism and the perception of hitler in india . Accessed: 2025-06-15

    2023

  3. [12]

    Debarati Das, Karin De Langis, Anna Martin-Boyle, Jaehyung Kim, Minhwa Lee, Zae Myung Kim, Shirley Anugrah Hayati, Risako Owan, Bin Hu, Ritik Parkar, Ryan Koo, Jonginn Park, Aahan Tyagi, Libby Ferland, Sanjali Roy, Vincent Liu, and Dongyeop Kang. 2024. http://arxiv.org/abs/240...

    2024 arXiv

  4. [13]

    Sayan Dey. 2021. https://doi.org/10.2478/rsc-2021-0001 Corona-logy: A re-configuration of racial dynamics in contemporary india . Research in Social Change, 13(1):150--157

    2021 doi

  5. [14]

    Jay Gala, Pranjal A Chitale, A K Raghavan, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar M, Janki Atul Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. https://openreview.net/forum?id=vfT4Yu...

    2023

  6. [15]

    Sanjana Gautam and Mukund Srinath. 2024. https://doi.org/10.18653/v1/2024.hcinlp-1.8 Blind spots and biases: Exploring the role of annotator cognitive biases in NLP . In Proceedings of the Third Workshop on Bridging Human--Computer Interaction and Natural Language Processing, ...

    2024 doi

  7. [16]

    Dirk Hovy and Shrimai Prabhumoye. 2021. https://doi.org/https://doi.org/10.1111/lnc3.12432 Five sources of bias in natural language processing . Language and Linguistics Compass, 15(8):e12432

    2021 doi

  8. [17]

    Yufei Huang and Deyi Xiong. 2024. https://aclanthology.org/2024.lrec-main.260/ CBBQ : A C hinese bias benchmark dataset curated with human- AI collaboration for large language models . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lang...

    2024

  9. [18]

    Sophie Jentzsch and Cigdem Turan. 2022. https://doi.org/10.18653/v1/2022.gebnlp-1.20 Gender bias in BERT - measuring and analysing biases through sentiment rating in a realistic downstream classification task . In Proceedings of the 4th Workshop on Gender Bias in Natural Langu...

    2022 doi

  10. [19]

    Akshita Jha, Aida Mostafazadeh Davani, Chandan K Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. 2023. https://doi.org/10.18653/v1/2023.acl-long.548 S ee GULL : A stereotype benchmark with broad geo-cultural coverage leveraging generative models . In Proceedings of...

    2023 doi

  11. [20]

    Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. 2024. https://doi.org/10.1162/tacl_a_00661 K o BBQ : K orean bias benchmark for question answering . Transactions of the Association for Computational Linguistics, 12:507--524

    2024 doi

  12. [21]

    Bean, Hannah Rose Kirk, and Scott A

    Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale. 2024. https://doi.org/10.1145/3677525.3678666 Indian-bhed: A dataset for measuring india-centric biases in large language models . In Proceedings of the 2024 International Conference on Inf...

    2024

  13. [22]

    Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021. https://api.semanticscholar.org/CorpusID:233476528 Mitigating political bias in language models through reinforced calibration . ArXiv, abs/2104.14795

    2021 arXiv

  14. [23]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. https://doi.org/10.18653/v1/2021.acl-long.416 S tereo S et: Measuring stereotypical bias in pretrained language models . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inte...

    2021 doi

  15. [24]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.154 C row S -pairs: A challenge dataset for measuring social biases in masked language models . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  16. [25]

    Vera Neplenbroek, Arianna Bisazza, and Raquel Fernández. 2024. http://arxiv.org/abs/2406.07243 Mbbq: A dataset for cross-lingual comparison of stereotypes in generative llms

    2024 arXiv

  17. [26]

    Shilpi Pandey. 2024. https://doi.org/10.3390/rel15030291 The burden of being a muslim woman in india—the instrumentalisation of muslim women at the intersection of gender, religion, colonialism, and secularism . Religions, 15(3)

    2024 doi

  18. [27]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. https://doi.org/10.18653/v1/2022.findings-acl.165 BBQ : A hand-built bias benchmark for question answering . In Findings of the Association for ...

    2022 doi

  19. [28]

    Ramnarayan S Rawat. 2013. https://doi.org/https://doi.org/10.1111/hic3.12109 Occupation, dignity, and space: The rise of dalit studies . History Compass, 11(12):1059--1067

    2013 doi

  20. [29]

    Nihar Sahoo, Pranamya Kulkarni, Arif Ahmad, Tanu Goyal, Narjis Asad, Aparna Garimella, and Pushpak Bhattacharyya. 2024. https://doi.org/10.18653/v1/2024.naacl-long.487 I ndi B ias: A benchmark dataset to measure social biases in language models for I ndian context . In Proceed...

    2024 doi

  21. [30]

    Sindhuja Sankaran, Maciek Sekerdej, and Ulrich Von Hecker. 2017. The role of indian caste identity and caste inconsistent norms on status representation. Frontiers in psychology, 8:487

    2017

  22. [31]

    Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. https://doi.org/10.18653/v1/D19-1339 The woman worked as a babysitter: On biases in language generation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9...

    2019 doi

  23. [32]

    Sandhya Singh, Prapti Roy, Nihar Sahoo, Niteesh Mallela, Himanshu Gupta, Pushpak Bhattacharyya, Milind Savagaonkar, Nidhi Sultan, Roshni Ramnani, Anutosh Maitra, and Shubhashis Sengupta. 2022. https://aclanthology.org/2022.lrec-1.565/ Hollywood identity bias dataset: A context...

    2022

  24. [33]

    Gopal Sharan Sinha and Ramesh Chandra Sinha. 1967. Exploration in caste stereotypes. Social Forces, 46(1):42--47

    1967

  25. [34]

    Smith, and Luke Zettlemoyer

    Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. https://doi.org/10.18653/v1/P19-1164 Evaluating gender bias in machine translation . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679--1684, Florence, Italy. Ass...

    2019 doi

  26. [35]

    Amit Thorat, Nazar Khalid, Nikhil Srivastav, Payal Hathi, Dean Spears, and Diane Coffey. 2020. Persisting prejudice: Measuring attitudes and outcomes by caste and gender in india. Caste (Waltham, Mass.), 1(2):1

    2020

  27. [36]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. http://arxiv.org/abs/2412.13663 Smarter, better, fa...

    2024 arXiv

  28. [37]

    Craig S Webster, Saana Taylor, Courtney Thomas, and Jennifer M Weller. 2022. Social bias, discrimination and inequity in healthcare: mechanisms, implications and recommendations. BJA education, 22(4):131--137

    2022

  29. [38]

    Jiaxu Zhao, Meng Fang, Shirui Pan, Wenpeng Yin, and Mykola Pechenizkiy. 2023. http://arxiv.org/abs/2312.06315 Gptbias: A comprehensive framework for evaluating bias in large language models

    2023 arXiv

  30. [39]

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. http://arxiv.org/abs/1804.06876 Gender bias in coreference resolution: Evaluation and debiasing methods

    2018 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.