REVIEW 6 cited by
MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Generative large language models (LLMs) have been shown to exhibit harmful biases and stereotypes. While safety fine-tuning typically takes place in English, if at all, these models are being used by speakers of many different languages. There is existing evidence that the performance of these models is inconsistent across languages and that they discriminate based on demographic factors of the user. Motivated by this, we investigate whether the social stereotypes exhibited by LLMs differ as a function of the language used to prompt them, while controlling for cultural differences and task accuracy. To this end, we present MBBQ (Multilingual Bias Benchmark for Question-answering), a carefully curated version of the English BBQ dataset extended to Dutch, Spanish, and Turkish, which measures stereotypes commonly held across these languages. We further complement MBBQ with a parallel control dataset to measure task performance on the question-answering task independently of bias. Our results based on several open-source and proprietary LLMs confirm that some non-English languages suffer from bias more than English, even when controlling for cultural shifts. Moreover, we observe significant cross-lingual differences in bias behaviour for all except the most accurate models. With the release of MBBQ, we hope to encourage further research on bias in multilingual settings. The dataset and code are available at https://github.com/Veranep/MBBQ.
Forward citations
Cited by 6 Pith papers
-
StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs
StereoTales shows that LLMs produce harmful, culturally adapted stereotypes in open-ended multilingual stories, with patterns consistent across providers and aligned human-LLM harm judgments.
-
StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs
StereoTales shows that all tested LLMs emit harmful stereotypes in open-ended stories, with associations adapting to prompt language and targeting locally salient groups rather than transferring uniformly across languages.
-
Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili
Bias in GPT-5.2 and Gemini 2.5 Flash changes rather than transfers between English and Swahili, with GPT-5.2 refusal behavior appearing only in English.
-
QuantiBias: Benchmarking Quantization-Induced Bias in LLMs
Quantization leaves refusal and multiple-choice bias checks flat while open-ended stereotype endorsement remains high (~24–27% under an independent judge), a gap standard safety evaluations miss.
-
BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context
BharatBBQ measures social bias in question-answering models across eight languages and finds that Indian-language examples often elicit more stereotyped answers than English ones.
-
Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protection
Quantization tends to degrade LLM fairness and safety—more in non-English tasks—and preserving top sensitivity-ranked weights in FP16 mostly mitigates the loss.
Discussion (0). Sign in to comment.