Pith. sign in

REVIEW 5 major objections 5 minor 53 references

GG-BBQ: German Gender Bias Benchmark for Question Answering

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read German-language LLMs exhibit measurable gender bias along and against stereotypes when answering questions from a newly translated German bias benchmark.

desk verdict The dataset is worth having, but the paper overclaims the direction of bias; the uniform sign flip between subsets looks like a label artifact, not a model finding. read the letter →

arxiv 2507.16410 v1 pith:LRFUQRGC submitted 2025-07-22 cs.CL cs.CYcs.LG

Shalaka Satheesh , Katrin Klug , Katharina Beckh , Héctor Allende-Cid , Sebastian Houben , Teena Hassan This is my paper
classification cs.CLcs.CYcs.LG
keywords genderbiasquestionansweringGermanNLPbenchmarktranslationlargelanguagemodelsfairnessevaluationgrammaticalBBQ
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds GG-BBQ, a German-language benchmark for gender bias in question answering, by translating the gender-identity subset of the English BBQ dataset and manually correcting the machine translations with a bilingual language expert. The paper argues that raw machine translation is not adequate for this purpose in German because grammatical gender forces choices that erase or distort gender-neutral wording, and that expert revision fixes this. It then evaluates ten open German-capable LLMs on the benchmark and reports that all of them show measurable bias, both in the direction of social stereotypes and against them. If this is right, German-language model users cannot rely on model size or instruction tuning to remove stereotyped answering.

What carries the argument

The central object is GG-BBQ, a hand-corrected German translation of the gender-identity templates of the BBQ benchmark, split into Subset-I (group terms such as Mann/Frau) and Subset-II (German proper names such as Emma and Matteo). The bias measure is the diff-bias score, computed separately for ambiguous and disambiguated contexts as the difference between the fraction of biased and counter-biased predictions. The dataset follows BBQ's four-condition structure, pairing ambiguous contexts with positive and negative questions and then repeating these with disambiguating information, while expecting 'no answer' in ambiguous settings.

What would settle it

Ask a sample of German speakers to pick the stereotype they associate with each of the 167 rephrased templates; if a majority cannot recover the original association for items like 'Who assisted the board?' as a secretary-related item, then the benchmark's bias scores reflect wording artifacts rather than measured stereotype reliance.

Watch

Extended reading notes

Core claim

The central discovery is that on GG-BBQ, no tested model is bias-free: every model's accuracy and diff-bias scores deviate from the unbiased ideal of accuracy 1.0 and diff-bias 0 on ambiguous contexts, and most models also show strong bias on disambiguated contexts. The bias is not uniform: on group-term items (Subset-I), models show positive bias for ambiguous contexts, while on proper-name items (Subset-II) they show strong negative, counter-stereotypic bias. Larger models that perform extremely well on disambiguated contexts often reach near-maximum bias when the context is ambiguous.

Load-bearing premise

The load-bearing premise is that manually rephrasing gender-neutral English questions into German preserves the same stereotype-relevant meaning and difficulty of each BBQ item.

Editorial extensions

If this is right

  • German QA pipelines built on these open LLMs would inherit stereotype-driven answers whenever the context lacks disambiguating information, because even the least biased models did not reach the no-answer accuracy needed to be bias-free.
  • Model size is not a mitigation: the strongest models on disambiguated items were among the worst on ambiguous items, with bias scores near the theoretical maximum.
  • Instruction tuning has no guaranteed debiasing effect, since the tuned versions of some models improved while the tuned leo-hessianai-13b became more biased.
  • Researchers should not use raw machine-translated bias benchmarks for German without expert review, because grammatical gender turns neutral English questions into gendered German ones.
  • The released dataset gives German NLP a reusable structure with 484 ambiguous and 484 disambiguated samples in Subset-I and 2484 of each in Subset-II for benchmarking gender bias in QA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to have German speakers rate each rephrased item for whether the original stereotype association, such as secretary with woman, is still recoverable; items that fail the rating could be excluded and the bias scores recomputed.
  • The opposite bias directions between Subset-I and Subset-II suggest that name-gender associations, rather than group-term semantics, drive some of the measured bias; ablating to unisex names only would test this.
  • If the same translation pipeline were applied to other grammatical-gender languages, the direction of the translation artifact would likely differ by language, limiting direct cross-lingual bias comparisons.
  • A within-template analysis could reveal whether rephrasing, rather than stereotype direction, is the primary driver of the counter-biased scores on proper-name subsets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces GG-BBQ, a German translation of the gender identity subset of the English BBQ benchmark, created by machine translation followed by manual correction by a single language expert. The dataset is split into Subset-I (group terms such as Mann/Frau) and Subset-II (proper names such as Emma/Matteo). Using the LM Evaluation Harness, the authors evaluate ten German-supporting LLMs under zero-shot multiple-choice QA, computing accuracy and the diff-bias scores of Jin et al. (2024). They report that all evaluated models exhibit gender bias, both aligned with and opposed to existing social stereotypes, and argue that manual revision is essential when translating gender-bias benchmarks into grammatically gendered languages.

Significance. If the translated benchmark is construct-valid, GG-BBQ would be a useful resource for evaluating gender bias in German QA models, a gap that the paper identifies and addresses. The paper also contributes a documented translation pipeline and a transparent bias-score methodology, and it evaluates a broad set of models commonly used for German NLP. The data are released on GitHub, which supports reproducibility. However, the central empirical claim that models exhibit bias 'both along and against' stereotypes rests on the assumption that the translated items preserve the stereotype direction of the original BBQ items; the paper provides no norming evidence for this assumption. The uniform sign flip between Subset-I and Subset-II in Tables 2 and 4 makes this assumption particularly pressing. The contribution is potentially significant, but the current evidence is not sufficient to support the quantitative conclusions.

major comments (5)
  1. [Section 4.1.1, Figure 1] The rephrasing example 'Who is the secretary?' -> 'Wer assistierte dem Vorstand?' changes the question's semantics and its stereotype cue, and no evidence is provided that all rephrased items preserve the intended biased/counter-biased answer structure. Since Equations (1)-(4) derive nab and nac directly from these labels, any label error inverts the sign of diff-bias and can manufacture a 'counter-bias' result. The paper should include an independent validation step, such as norming by multiple German speakers, to confirm each item's stereotype-aligned answer.
  2. [Tables 2 and 4] All ten models have positive diff-bias_amb on Subset-I but all ten have negative diff-bias_amb on Subset-II. This uniform sign flip is implausible as a property of the models, since Subset-II is generated from the same templates by substituting proper names for group terms. The pattern is exactly what would be expected if the name substitution or rephrasing changed which answer is coded as stereotype-aligned. The paper's discussion in Section 7 merely states that this difference 'requires further research'; the claim of bias against stereotypes cannot stand without validating the label direction in Subset-II.
  3. [Section 5.1] The evaluation protocol selects the second prompt 'based on the results' on the same data that is then used for all reported scores, with no held-out prompt development and no reporting of the other four prompts. This is selection on the evaluation set and can materially change the direction and magnitude of bias scores. The paper should either report results for all five prompts or pre-specify a development split before reporting the main tables.
  4. [Tables 2-5] The results are point estimates from a single run with no confidence intervals, bootstrap resampling, or multiple seeds. With sample sizes of 484 (Subset-I) and 2484 (Subset-II), differences such as 'best performance' or 'instruction-tuning varied outcomes' may be within sampling noise. The authors should provide uncertainty estimates or at least a reproducibility statement to support the quantitative comparisons.
  5. [Limitations, Section 4.1.1] The paper acknowledges the reliance on a single language expert, but this is not just a minor limitation: for a new bias benchmark, one annotator's correction of grammar is not sufficient to establish that the items encode stereotypes that German speakers recognize, especially because some stereotypes in the original BBQ are U.S.-specific. Without inter-annotator agreement or a norming study, the construct validity of the dataset remains unverified.
minor comments (5)
  1. [Section 7] The phrase 'seemingly loose this ability' should read 'lose this ability'.
  2. [Section 5.1.1] The formula for Accdisamb should be written with parentheses: Accdisamb = (nbb + ncc) / (nb + nc), since the current inline text is ambiguous.
  3. [References] In the Liang et al. reference, 'Jue W ANG' and 'V olume' contain spurious spacing; please fix.
  4. [Section 4.1] The German example 'befragten {{NAME1}} and {{NAME2}}' uses the English 'and' instead of German 'und'.
  5. [Section 2] The abbreviation 'MINT' should be expanded at first use (Mathematik, Informatik, Naturwissenschaft, Technik).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: bias scores are computed directly from model outputs on an externally sourced translated benchmark.

full rationale

GG-BBQ's central claim—that German LLMs exhibit bias both along and against stereotypes—is computed directly from model outputs on a fixed, externally sourced benchmark (English BBQ templates by Parrish et al. translated to German), using bias scores from Jin et al. (2024) as defined in Equations 1–4. No parameter is fitted to the model outputs, and no reported quantity is defined in terms of the quantity it is supposed to measure. The stereotype-aligned and counter-aligned labels are inherited from the original BBQ templates rather than derived from the models' predictions, so even if those labels are invalid for the German cultural context, that is a validity concern and not a circular derivation. The paper's Limitations section explicitly acknowledges this risk ('there is a possibility that the dataset could not capture some of the differences in the German and the U.S. cultural contexts'), which we weigh as an honest caveat rather than evidence of circularity. The authors cite prior work for the dataset and the metric, but none of the load-bearing citations is authored by the present paper's authors, and none is invoked to forbid alternative approaches. The one selection-like step is the choice of the second prompt from five tested prompts ('Based on the results, we chose the second prompt for subsequent evaluation in Section 6'); this is an experimental-design choice that may affect robustness, but it does not make the bias scores equal to the selection criterion by construction. The derivation chain is therefore self-contained: model predictions on a translated benchmark yield accuracy and bias scores, with no step where an input is renamed as a prediction or where a fitted parameter determines the outcome.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim (models exhibit gender bias) rests on the validity of the translated BBQ templates, the semantic preservation of the rephrasing, and the quality of the single-annotator correction. No free parameters or invented entities are introduced; the analysis relies on the given bias score formulas and the dataset design.

assumptions (3)
  • domain assumption The original English BBQ gender identity subset validly measures social biases in QA.
    The paper adopts the BBQ template design and stereotyping definition from Parrish et al. (2022) without re-validation; translation inherits any validity issues.
  • ad hoc to paper Manual rephrasing to maintain gender neutrality does not alter the bias-evoking semantics of the original templates.
    Section 4.1.1 describes rephrasing e.g. 'Who is the secretary?' to 'Wer assistierte dem Vorstand?', which changes the question content; if this changes what is being measured, bias scores may reflect artifacts.
  • domain assumption The single language expert's corrections are sufficient to remove translation errors introduced by NLLB.
    The paper relies on one bilingual expert's 15+ hours of review (Section 4.1) and acknowledges annotator bias as a limitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GG-BBQ: German Gender Bias Benchmark for Question Answering." pith.science (2026). https://pith.science/paper/LRFUQRGC

@misc{pith2026250716410,
  author       = {Pith},
  title        = {Pith review of: GG-BBQ: German Gender Bias Benchmark for Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LRFUQRGC}},
  note         = {Machine review of arXiv:2507.16410}
}
read the original abstract

Within the context of Natural Language Processing (NLP), fairness evaluation is often associated with the assessment of bias and reduction of associated harm. In this regard, the evaluation is usually carried out by using a benchmark dataset, for a task such as Question Answering, created for the measurement of bias in the model's predictions along various dimensions, including gender identity. In our work, we evaluate gender bias in German Large Language Models (LLMs) using the Bias Benchmark for Question Answering by Parrish et al. (2022) as a reference. Specifically, the templates in the gender identity subset of this English dataset were machine translated into German. The errors in the machine translated templates were then manually reviewed and corrected with the help of a language expert. We find that manual revision of the translation is crucial when creating datasets for gender bias evaluation because of the limitations of machine translation from English to a language such as German with grammatical gender. Our final dataset is comprised of two subsets: Subset-I, which consists of group terms related to gender identity, and Subset-II, where group terms are replaced with proper names. We evaluate several LLMs used for German NLP on this newly created dataset and report the accuracy and bias scores. The results show that all models exhibit bias, both along and against existing social stereotypes.

Figures

Figures reproduced from arXiv: 2507.16410 by the authors.

Figure 1
Figure 1. As in the original BBQ dataset, each sample [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 30 canonical work pages

  1. [1]

    Afra Feyza Aky \"u rek, Muhammed Yusuf Kocyigit, Sejin Paik, and Derry Tanti Wijaya. 2022. https://doi.org/10.18653/v1/2022.gebnlp-1.9 Challenges in measuring bias via open-ended language generation . In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 76--76, Seattle, Washington. Association for Computational ...

    work page doi:10.18653/v1/2022.gebnlp-1.9 2022

  2. [2]

    Andrea Aler Tubella, Dimitri Coelho Mollo, Adam Dahlgren Lindstr\" o m, Hannah Devinney, Virginia Dignum, Petter Ericson, Anna Jonsson, Timotheus Kampik, Tom Lenaerts, Julian Alfredo Mendez, and Juan Carlos Nieves. 2023. https://doi.org/10.1145/3593013.3594059 ACROCP o L is: A Descriptive Framework for Making Sense of Fairness . In Proceedings of the ACM ...

    arXiv 2023

  3. [3]

    Marion Bartl, Malvina Nissim, and Albert Gatt. 2020. https://aclanthology.org/2020.gebnlp-1.1/ Unmasking contextual stereotypes: Measuring and mitigating BERT `s gender bias . In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 1--16, Barcelona, Spain (Online). Association for Computational Linguistics

    work page 2020

  4. [4]

    Emily M. Bender. 2019. https://thegradient.pub/the-benderrule-on-naming-the-languages-we-study-and-why-it-matters/ The \# B ender R ule: On Naming the Languages We Study and Why It Matters . The Gradient

    work page 2019

  5. [5]

    Su Lin Blodgett, Solon Barocas, Hal Daum \'e III, and Hanna Wallach. 2020. https://doi.org/10.18653/v1/2020.acl-main.485 Language (Technology) is Power: A Critical Survey of `` Bias '' in NLP . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454--5476, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.485 2020

  6. [6]

    Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics

    work page 2021

  7. [7]

    Kate Connolly. 2023. https://www.theguardian.com/world/2023/may/22/burgerinnen-backlash-as-berlin-mayor-refuses-to-use-gender-neutral-language Bürger*innen? Backlash as Berlin mayor refuses to use gender-neutral language . The Guardian. Accessed: 20.12.2023

    work page 2023

  8. [8]

    Marta R Costa-Juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. https://arxiv.org/abs/2207.04672 No language left behind: Scaling human-centered machine translation . arXiv preprint arXiv:2207.04672

    arXiv 2022

Show all 53 references
  1. [9]

    Kate Crawford. 2017. The trouble with bias. NeurIPS Invited Talk

    2017

  2. [10]

    Davies, Steven J

    Paul G. Davies, Steven J. Spencer, Diane M. Quinn, and Rebecca Gerhardstein. 2002. https://doi.org/10.1177/014616702237644 Consuming images: How television commercials that elicit stereotype threat can restrain women academically and professionally . Personality and Social Psy...

    2002 doi

  3. [11]

    Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, and Kai-Wei Chang. 2022. https://aclanthology.org/2022.findings-aacl.24 On Measures of Biases and Harms in NLP . In Findings of the Association for ...

    2022

  4. [12]

    Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, Tongshuang Wu, et al

    Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, Tongshuang Wu, et al. 2021. https://arxiv.org/abs/2112.02721 NL - A ugmenter: A Framework for Task-Sensitive Natural La...

    2021

  5. [13]

    Silke Eschert. 2010. https://de.in-mind.org/article/white-men-cant-jump-and-girls-cant-do-math-wenn-stereotype-motivation-und-leistung-bedrohen White men can't jump and girls can't do math? wenn stereotype motivation und leistung bedrohen . In-Mind

    2010

  6. [14]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097--1179

    2024

  7. [15]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  8. [16]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.301 R eal T oxicity P rompts: Evaluating neural toxic degeneration in language models . In Findings of the Association for Computational Linguist...

    2020 doi

  9. [17]

    Dirk Hovy and Shrimai Prabhumoye. 2021. https://doi.org/https://compass.onlinelibrary.wiley.com/doi/full/10.1111/lnc3.12432 Five sources of bias in natural language processing . Language and Linguistics Compass, 15(8):e12432

    2021 doi

  10. [18]

    Yufei Huang and Deyi Xiong. 2024. https://aclanthology.org/2024.lrec-main.260/ CBBQ : A C hinese bias benchmark dataset curated with human- AI collaboration for large language models . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lang...

    2024

  11. [19]

    Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. 2024. https://doi.org/10.1162/tacl_a_00673 Instructed to bias: Instruction-tuned language models exhibit emergent cognitive bias . Transactions of the Association for Computational Linguistics, 12:771--785

    2024 doi

  12. [20]

    Yves Jeanrenaud. 2020. https://www.bmfsfj.de/resource/blob/227404/9c00c907483f6089c13d8040fde223c2/jeanrenaud-yves-mint-warum-nicht-zur-unterrepraesentation-von-frauen-in-mint-speziell-ikt-deren-ursachen-wirksamkeit-bestehender-massnahmen-und-handlungsempfehlungen-data.pdf MIN...

    2020

  13. [21]

    McNamara, and Deming Chen

    Jingru Jia, Zehua Yuan, Junhao Pan, Paul E. McNamara, and Deming Chen. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/cda04d7ea67ea1376bf8c6962d8541e0-Paper-Conference.pdf Decision-making behavior evaluation framework for llms under uncertain context . In Adv...

    2024

  14. [22]

    Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. 2024. https://doi.org/10.1162/tacl_a_00661 K o BBQ : K orean bias benchmark for question answering . Transactions of the Association for Computational Linguistics, 12:507--524

    2024 doi

  15. [23]

    Manuel Lardelli and Dagmar Gromann. 2023. https://doi.org/10.26034/cm.jostrans.2023.531 Translating non-binary coming-out reports: Gender-fair language strategies and use in news articles . The Journal of Specialised Translation, 40:213–240

    2023 doi

  16. [24]

    Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.311 UNQOVER ing Stereotyping Biases via Underspecified Questions . In Findings of the Association for Computational Linguistics: EMNLP, pages 3475-...

    2020 doi

  17. [25]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Aco...

    2023

  18. [26]

    Yuxing Lu, Xukai Zhao, and Jinzhuo Wang. 2024. https://doi.org/10.18653/v1/2024.knowllm-1.6 C linical RAG : Enhancing clinical decision support through heterogeneous knowledge retrieval . In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024...

    2024 doi

  19. [27]

    Simone Madre. 2024. https://www.nordbayern.de/ Die 30 schönsten Unisex-Namen . Accessed: 10.02.2025

    2024

  20. [28]

    Bowman, and Rachel Rudinger

    Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. https://doi.org/10.18653/v1/N19-1063 On measuring social biases in sentence encoders . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational...

    2019 doi

  21. [29]

    Meta. a . Llama-3.1-70B . https://huggingface.co/meta-llama/Llama-3.1-70B

  22. [30]

    Meta. b . Llama-3.1-70B-Instruct . https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct

  23. [31]

    Meta. c . Llama-3.2-3B . https://huggingface.co/meta-llama/Llama-3.2-3B

  24. [32]

    Meta. d . Llama-3.2-3B-Instruct . https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct

  25. [33]

    MistralAI. a . Mistral-7B-Instruct-v0.3 . https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3

  26. [34]

    MistralAI. b . Mistral-7B-v0.3 . https://huggingface.co/mistralai/Mistral-7B-v0.3

  27. [35]

    Sergio Morales, Robert Claris\' o , and Jordi Cabot. 2024. https://doi.org/10.1145/3640310.3674093 A dsl for testing llms for fairness and bias . In Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems, MODELS '24, page 20...

    2024

  28. [36]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.154 C row S -pairs: A challenge dataset for measuring social biases in masked language models . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  29. [37]

    Vera Neplenbroek, Arianna Bisazza, and Raquel Fern \'a ndez. 2024. https://openreview.net/forum?id=X9yV4lFHt4 MBBQ : A dataset for cross-lingual comparison of stereotypes in generative LLM s . In First Conference on Language Modeling

    2024

  30. [38]

    Shangrui Nie, Michael Fromm, Charles Welch, Rebekka G \"o rge, Akbar Karimi, Joan Plepi, Nazia Mowmita, Nicolas Flores-Herr, Mehdi Ali, and Lucie Flek. 2024. https://doi.org/10.18653/v1/2024.c3nlp-1.6 Do multilingual large language models mitigate stereotype bias? In Proceedin...

    2024 doi

  31. [39]

    Nette Nöstlinger. 2021. https://www.politico.eu/article/debate-over-gender-inclusive-neutral-language-divides-germany/ Debate over gender-neutral language divides Germany . Politico. Accessed: 20.12.2023

    2021

  32. [40]

    Melanie Olczyk, Sarah Gentrup, Thorsten Schneider, Anna Volodina, Valentina Perinetti Casoni , Elizabeth Washbrook, Sarah Jiyoon Kwon, and Jane Waldfogel. 2023. https://doi.org/10.1016/j.ssresearch.2023.102938 Teacher judgements and gender achievement gaps in primary education...

    2023

  33. [41]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. https://doi.org/10.18653/v1/2022.findings-acl.165 BBQ : A hand-built bias benchmark for question answering . In Findings of the Association for ...

    2022 doi

  34. [42]

    Björn Plüster, Manuel Brack, Malte Ostendorff, Pedro Ortiz Suarez, Jan-Philipp Harries, Daniel Auras, Christoph Schuhmann, and Jenia Jitsev. a . DiscoResearch/Llama3-DiscoLeo-Instruct-8B-v0.1 . https://huggingface.co/DiscoResearch/Llama3-DiscoLeo-Instruct-8B-v0.1

  35. [43]

    Björn Plüster, Manuel Brack, Malte Ostendorff, Pedro Ortiz Suarez, Jan-Philipp Harries, Daniel Auras, Christoph Schuhmann, and Jenia Jitsev. b . DiscoResearch/Llama3-German-8B . https://huggingface.co/DiscoResearch/Llama3-German-8B

  36. [44]

    Björn Plüster and Christoph Schuhmann. a . leo-hessianai-13b . https://huggingface.co/LeoLM/leo-hessianai-13b

  37. [45]

    Björn Plüster and Christoph Schuhmann. b . leo-hessianai-13b-chat . https://huggingface.co/LeoLM/leo-hessianai-13b-chat

  38. [46]

    Marcelo O. R. Prates, Pedro H. Avelar, and Luís C. Lamb. 2020. Assessing gender bias in machine translation: A case study with google translate. Neural Computing and Applications, 32:6363–81

    2020

  39. [47]

    Zara Siddique, Liam Turner, and Luis Espinosa-Anke. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1035 Who is better at math, jenny or jingzhen? uncovering stereotypes in large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language...

    2024 doi

  40. [48]

    Steele and Joshua Aronson

    Claude M. Steele and Joshua Aronson. 1995. https://doi.org/10.1037/0022-3514.69.5.797 Stereotype threat and the intellectual test performance of african americans . Journal of Personality and Social Psychology, 69(5):797--811

    1995 doi

  41. [49]

    Hitomi Yanaka, Namgi Han, Ryoma Kumon, Jie Lu, Masashi Takeshita, Ryo Sekizawa, Taisei Kato, and Hiromi Arai. 2024. https://arxiv.org/abs/2406.02050 Analyzing social biases in japanese large language models . Preprint, arXiv:2406.02050

    2024 arXiv

  42. [50]

    Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. https://doi.org/10.18653/v1/D19-1531 Examining gender bias in languages with grammatical gender . In Proceedings of the 2019 Conference on Empirical Methods in Natural Langua...

    2019 doi

  43. [51]

    Muitze Zulaika and Xabier Saralegi. 2025. https://aclanthology.org/2025.coling-main.318/ B asq BBQ : A QA benchmark for assessing social biases in LLM s for B asque, a low-resource language . In Proceedings of the 31st International Conference on Computational Linguistics, pag...

    2025

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.