Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that language alone changes how LLMs handle sensitive questions, with refusal rates and answer distributions diverging across nearly all language pairs and topics, and that the divergence pattern depends on the model.

desk verdict Useful new dataset and measurement template, but the PERMANOVA analysis is confounded by language identity, so the paper's central claim that ethical bias is widespread is not supported by its own statistics. read the letter →

arxiv 2505.19121 v2 pith:EGIZIBF3 submitted 2025-05-25 cs.CL

Seunguk Yu , Juhwan Choi , Youngbin Kim This is my paper
classification cs.CL
keywords multilingualbiasethicallargelanguagemodelsMSQADMcNemar'stestPERMANOVAcross-lingualevaluationsensitivequestions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that large language models treat ethically charged questions differently depending on the language the question is asked in, even when the underlying content is the same. To test this, the authors build MSQAD, a multilingual dataset of sensitive questions drawn from news coverage of 17 human-rights topics, and generate acceptable and non-acceptable answers in English, Korean, Chinese, Spanish, German, and Hindi. They run two statistical tests: McNemar's paired test on whether the model refuses to answer, and PERMANOVA on the embedding distributions of acceptable answers. The null hypothesis of language-independent behavior is rejected in almost all language-pair and topic combinations, and the pattern of which languages differ shifts across seven models. If the paper is right, the language a user chooses changes both whether a model will engage with a sensitive topic and what content it produces.

What carries the argument

The central object is the MSQAD dataset itself: semantically equivalent sensitive questions in six languages, each paired with a model-generated acceptable answer and a model-generated non-acceptable answer under identical prompt structures. The argument rides on two statistical tests applied to that paired design. McNemar's test compares discordant refusal outcomes between two languages for the same questions and yields a $\chi^2$ statistic; PERMANOVA permutes language labels on the Euclidean-distance matrix of answer embeddings and yields an $F$ statistic. Both tests formalize the paper's main hypothesis: if responses are language-independent, these statistics should not be extreme, and their systematic rejection is the evidence for cross-lingual bias.

What would settle it

Run the same protocol repeatedly with nonzero sampling temperature, say ten or more generations per question-language-prompt cell, and recompute both tests on each sample; if the null hypotheses are accepted in most samples, or within-language variance matches between-language variance, the reported language effects would not be stable.

Watch

Extended reading notes

Core claim

On the paper's own terms, ethical bias is a measurable property of cross-lingual LLM behavior: with prompt configuration, translation service, and evaluation model held constant, changing only the language changes the probability of refusing a sensitive request and changes the distribution of acceptable answers. The authors report that at a 5% significance level, the null hypothesis of equal refusal probability was accepted in only 11 of 255 language-pair/topic cases, while PERMANOVA rejected the 'similar distributions' null for essentially every pair. They further find that Chinese and Hindi show consistently high refusal rates, Spanish and German low rates, and that English answers are usually the most distinct in embedding space; the same experiment across six additional models shows the bias pattern persisting but varying by model family.

Load-bearing premise

The load-bearing premise is that a single generated response per question, language, and prompt type is representative enough to separate language-driven differences from ordinary generation randomness, so the paper can assert that all factors except language were held constant.

Editorial extensions

If this is right

  • A user who asks in Spanish or German is more likely to receive an unsafe or non-acceptable answer than a user who asks in Chinese or Hindi, which tend to be refused.
  • English receives the most distinct acceptable-answer distributions, so evaluations that only use English can miss how the model behaves in other languages.
  • The cross-lingual bias is not fixed by model scaling or family: newer versions in the same model line can show stronger language-specific refusal differences than their predecessors.
  • Any deployment that routes users by language, or translates prompts without re-testing, inherits whatever refusal and content asymmetries the model has.
  • MSQAD gives a reusable benchmark for checking future models: the same two tests can be rerun on new releases and new languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its paired design collapses sampling variance: a testable extension would generate several responses per language-question cell and estimate within-language variance before attributing rejection to language.
  • The consistent high-refusal languages (Chinese, Hindi) and low-refusal languages (Spanish, German) invite a follow-up study correlating refusal rates with the per-language volume or type of safety-tuning data, which the paper does not measure.
  • PERMANOVA's significant $F$ shows distributional divergence, not the direction or quality of bias; the paper's appendix human ratings move partway toward grounding the divergence in informativeness, and future work could scale that annotation to all topics.
  • The dataset can be turned into a training intervention: selectively tuning refusal behavior in low-refusal languages could equalize the refusal surface, which the ethics statement suggests but does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MSQAD, a multilingual dataset of socially sensitive questions and model-generated acceptable/non-acceptable responses in six languages, built from Human Rights Watch articles on 17 topics. Responses were produced with gpt-3.5 and six additional open LLMs. The authors test the hypothesis that responses are consistent across languages by applying McNemar's test to refusal rates and PERMANOVA to embeddings of acceptable responses. They report that the null hypotheses are rejected in almost all language pairs and conclude that ethical biases in LLM responses are widespread across languages and models. The dataset and code are released publicly.

Significance. If the claims hold, the dataset and the hypothesis-testing framework are a useful contribution to multilingual fairness evaluation. The paper is one of the few to apply formal statistical tests to cross-language ethical bias, and the public release of MSQAD is a practical strength. However, the current evidence does not establish that the measured differences are ethical bias rather than language identity, generation stochasticity, or classifier artifacts. The two statistical pillars are both affected by load-bearing confounds, so the core conclusion is not yet supported.

major comments (4)
  1. [Section 4.2 and Appendix B.2] The PERMANOVA test is applied to Euclidean distances between raw bert-base-multilingual-cased embeddings of responses written in six different languages. Multilingual BERT representations are not language-invariant: semantically identical content expressed in different languages occupies systematically different regions of the embedding space. Therefore H_p0, that the two embedding distributions are similar, is false by construction for almost any cross-language pair even if the responses are semantically and ethically identical. The near-universal rejection of H_p0 therefore does not measure ethical bias; it confounds language identity with the construct of interest. The paper's own limitation statement ('we did not fully consider the semantic differences in the responses') concedes exactly this gap. A concrete fix is to repeat the analysis on a language-matched control condition (e.g., translated or back-translated versions of the same English response, or embeddings from a language-alignment model) and to report whether the effect persists on those controls.
  2. [Section 3.2 and Section 4] The experiments generate one response per question, language, and prompt type, and no sampling temperature or repeated runs are reported. McNemar's test treats each question-language pair as a fixed binary outcome, but if generation is stochastic, the pairwise differences in refusal rates and embedding distributions could partly reflect sampling noise. The assertion in Section 4 that 'all factors were held constant except for the language used' is an experimental design claim, not a statistical guarantee; without measuring within-language variance, the test statistics have unknown variability. Please report the temperature and sampling configuration and add repeated sampling (e.g., 5-10 generations per cell) or otherwise bound the within-language variance.
  3. [Appendix B.1] The refusal classifier has no gold labels; the 0.8 probability threshold and the direct-rejection phrase filter were tuned in a pilot test, and the classifier is a multilingual BERT fine-tuned on XNLI that can misclassify negative sentiment toward non-acceptable content as a refusal. Since the classifier is applied with language-specific labels, classification accuracy may differ by language, so the McNemar results in Section 4.1 and Section 5.1 could be an artifact of unequal classifier reliability across languages. Please provide per-language validation of the refusal labels (e.g., a human-annotated gold subset) or otherwise calibrate the classifier separately in each language.
  4. [Section 4.2 and Appendix C.2] The PERMANOVA test measures geometric distributional differences between embedding sets, but the paper defines ethical bias as differences in the 'informativeness and morality' of responses. Distributional distance conflates lexical choice, response length, style, and topic drift with the moral and informative dimensions. The human annotation in Appendix C.2 covers only 3 of 17 topics, only for acceptable responses, and scores 'ethical informativeness' rather than the pairwise distributional differences that PERMANOVA tests; it therefore cannot validate the main statistical claim. To support the construct, the analysis needs a semantic-similarity or human-rated dimension that is actually aligned with the paper's definition.
minor comments (5)
  1. [Section 4.1] The critical value is given as 3.838; the 5% critical value for a chi-square distribution with one degree of freedom is 3.841, so the stated value appears to be a typo.
  2. [Figure 5 caption] The sentence 'While the x-axes are not dependent variables' is unclear; a conventional categorical axis would be easier to interpret.
  3. [Appendix A.2] The data selection description says both 'exclude questions ... with an embedding similarity of 97% or higher with the centroid' and 'removed questions ... whose distance from the centroid was in the bottom 1%'; these criteria should be stated unambiguously, distinguishing similarity from distance and specifying the actual operationalization.
  4. [Appendix D] The observation that some additional LLMs respond in English even when prompted in Korean is acknowledged but not analyzed; this behavior itself could confound language-specific refusal counts and should be measured or controlled.
  5. [Appendix A.3] GEMBA translation-quality checks are reported for only 3 of 17 topics, so the claim that questions are semantically equivalent across all languages is only partially supported.

Circularity Check

1 steps flagged · score 6.0 of 10

The PERMANOVA pillar reduces to the known language-dependence of raw multilingual embeddings, so part of the 'widespread ethical bias' claim is circular; the McNemar refusal-rate analysis is independent.

  1. renaming known result [Section 4.2 (Eqs. 2–5) and Appendix B.2]
    "First, we construct a distance matrix D by pairing the response embeddings of responses within each topic. ... We used the same PLM that was applied in the data selection process to obtain response embeddings. We calculated the distances between these embeddings using Euclidean distance to construct the distance matrix D. ... H p0 is rejected in almost all cases, regardless of the chosen significance level."

    The distance matrix D in Eqs. (2)–(3) is built from Euclidean distances between raw bert-base-multilingual-cased embeddings of responses written in six different languages. Multilingual BERT embeddings are not language-invariant: even semantically identical content in English, Korean, Chinese, Spanish, German, and Hindi lands in systematically different regions of the vector space. H_p0 ('distributions of response embeddings generated between specific language pairs are similar') is therefore false a priori for almost every language pair, independent of whether the responses differ in ethical content or informativeness.

full rationale

The paper's central claim rests on two statistical pillars. The McNemar refusal-rate test (Section 4.1) compares externally generated binary refusal outcomes across languages; it is not circular because the outcomes are measured, not fitted, and no parameter is defined in terms of the conclusion. The PERMANOVA test (Section 4.2), however, computes distances between raw multilingual BERT embeddings of responses in different languages. Because multilingual embeddings encode language identity, the null hypothesis of distributional similarity is false by construction for nearly all language pairs, regardless of ethical content. Interpreting this rejection as evidence of ethical bias renames a known property of multilingual embeddings as a discovered bias. The paper's own limitation statement admits that semantic differences were not considered, which is precisely the gap that makes the embedding test unable to support the claimed construct. Since one of the two main predictive pillars reduces to the input representation, the overall circularity score is 6 rather than lower; the independent McNemar results and the absence of self-citation load-bearing keep it from being 8 or 10.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a stipulated definition of bias, on translation equivalence, and on an unvalidated refusal classifier. The free parameters are thresholds chosen during dataset construction and classification, not fitted to predict the outcome.

free parameters (4)
  • Rejection classification probability threshold = 0.8
    Used in Appendix B.1 to label a response as a refusal; chosen based on a pilot test, not gold labels.
  • Question similarity threshold = 97% (centroid similarity)
    Used in Appendix A.2 to remove duplicate questions during clustering.
  • Outlier removal percentile = bottom 1%
    Removed questions farthest from cluster centroids to eliminate outliers and malformed generations (Appendix A.2).
  • Number of clusters per topic (k) = varies from 4 to 20 per topic
    Selected by inertia-based heuristic in Appendix A.2; affects which questions are retained.
assumptions (5)
  • domain assumption Ethical bias is operationally defined as statistically significant differences in refusal probability and response-embedding distributions across languages.
    Stated in Section 1 footnote 2 and used to frame all hypothesis tests; no external ethical ground truth is involved.
  • domain assumption Language is a proxy for culture and corpus-based cultural content, so differences in model outputs across languages indicate culturally grounded bias.
    Invoked in Section 1 to motivate the study; not empirically verified in the paper.
  • domain assumption The Google Cloud Translation API and the GEMBA checks produce semantically equivalent questions and prompts across the six languages.
    Needed so that observed output differences are attributable to language rather than translation artifacts; only 300 questions from 3 topics were evaluated (Appendix A.3).
  • domain assumption The multilingual BERT classifier identifies response refusals accurately without gold labels.
    Used in Appendix B.1; the authors state gold labels were unavailable and only a pilot test was run.
  • standard math McNemar's test and PERMANOVA are valid for these paired, non-i.i.d. response data.
    The tests assume paired binary outcomes and exchangeability under permutation; the paper applies them per language pair and topic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models." pith.science (2026). https://pith.science/paper/EGIZIBF3

@misc{pith2026250519121,
  author       = {Pith},
  title        = {Pith review of: Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EGIZIBF3}},
  note         = {Machine review of arXiv:2505.19121}
}
read the original abstract

Despite the recent strides in large language models, studies have underscored the existence of social biases within these systems. In this paper, we delve into the validation and comparison of the ethical biases of LLMs concerning globally discussed and potentially sensitive topics, hypothesizing that these biases may arise from language-specific distinctions. Introducing the Multilingual Sensitive Questions & Answers Dataset (MSQAD), we collected news articles from Human Rights Watch covering 17 topics, and generated socially sensitive questions along with corresponding responses in multiple languages. We scrutinized the biases of these responses across languages and topics, employing two statistical hypothesis tests. The results showed that the null hypotheses were rejected in most cases, indicating biases arising from cross-language differences. It demonstrates that ethical biases in responses are widespread across various languages, and notably, these biases were prevalent even among different LLMs. By making the proposed MSQAD openly available, we aim to facilitate future research endeavors focused on examining cross-language biases in LLMs and their variant models.

Figures

Figures reproduced from arXiv: 2505.19121 by the authors.

Figure 1
Figure 1. Results of instructing the same model to re [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Process of constructing the MSQAD and validating [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Heatmaps of McNemar’s statistics whether the response was rejected for each language pair. The large [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Heatmaps of PERMANOVA statistics using the embeddings of acceptable responses on each language [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Rejection rates measured across languages and [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Heatmaps of McNemar’s statistics obtained for specific topics whether the response was rejected for [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Heatmaps of PERMANOVA statistics obtained for specific topics using the embeddings of acceptable [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Word clouds displaying the keywords generated during the intermediate keyword generation task for each [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Relationships between the independent, con [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Heatmaps of McNemar’s statistics obtained [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Heatmaps of PERMANOVA statistics ob￾tained for the remaining topics using the embeddings of acceptable responses on each language pair. Topics English Korean Chinese Spanish German Hindi Children’s Rights 47.50% 11.25% 0.00% 23.75% 20.00% 0.00% Refugees and Migrants 4…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A multilingual two-phase evaluation shows LLMs lean on query language for factual questions and on training-country perspective for territorial and historical disputes.

Reference graph

Works this paper leans on

54 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone . arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Yelaman Abdullin, Diego Molla, Bahadorreza Ofoghi, John Yearwood, and Qingyang Li. 2023. https://aclanthology.org/2023.gem-1.16 Synthetic dialogue dataset generation using LLM agents . In Proceedings of EMNLP 2023 Workshop on Natural Language Generation, Evaluation, and Metrics, pages 181--191

    work page 2023

  3. [3]

    Marti J Anderson. 2001. https://onlinelibrary.wiley.com/action/oidcStart?redirectUri= Austral ecology, 26(1):32--46

    work page 2001

  4. [4]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-family/ Introducing the next generation of claude . Accessed: May 2024

    work page 2024

  5. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of NeurIPS, pages 1877--1901

    work page 2020

  6. [7]

    Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He. 2023. https://aclanthology.org/2023.findings-emnlp.143 Attack prompt generation for red teaming and defending large language models . In Findings of EMNLP, pages 2176--2189

    work page 2023

  7. [8]

    Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. https://aclanthology.org/P18-1128 The hitchhiker ' s guide to testing statistical significance in natural language processing . In Proceedings of ACL, pages 1383--1392

    work page 2018

  8. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 54 references
  1. [10]

    Virginia Felkner, Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. 2023. https://aclanthology.org/2023.acl-long.507 W ino Q ueer: A community-in-the-loop benchmark for anti- LGBTQ + bias in large language models . In Proceedings of ACL, pages 9126--9140

    2023

  2. [11]

    Virginia Felkner, Jennifer Thompson, and Jonathan May. 2024. https://aclanthology.org/2024.acl-long.760 GPT is not an annotator: The necessity of human annotation in fairness benchmark construction . In Proceedings of ACL, pages 14104--14115

    2024

  3. [12]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. https://direct.mit.edu/coli/article/50/3/1097/121961/Bias-and-Fairness-in-Large-Language-Models-A Bias and fairness in large langua...

    2024

  4. [13]

    Google. 2024. https://developers.googleblog.com/en/gemini-15-our-next-generation-model-now-available-for-private-preview-in-google-ai-studio/ Gemini 1.5: Our next-generation model, now available for private preview in google ai studio . Accessed: May 2024

    2024

  5. [14]

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. https://aclanthology.org/2022.acl-long.234 T oxi G en: A large-scale machine-generated dataset for adversarial and implicit hate speech detection . In Proceedings of ACL, pages 3309--3326

    2022

  6. [15]

    Andrew F Hayes and Klaus Krippendorff. 2007. https://www.tandfonline.com/doi/abs/10.1080/19312450709336664?casa_token=tT7VD8BTf5sAAAAA:dj78oUXkW3PQhePPDxMESInWtQpD1VsH0P3vqpfMEUzlHEeuNU6vSjTUu4ZOxHfc7BoaVIQ4MQcy5Rs Answering the call for a standard reliability measure for codi...

    2007 doi

  7. [16]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. https://aclanthology.org/2023.findings-emnlp.826 Not all languages are created equal in LLM s: Improving multilingual capability by cross-lingual-thought prompting . In Findings of EM...

    2023

  8. [17]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  9. [18]

    Wenying Jiang. 2000. https://academic.oup.com/eltj/article-abstract/54/4/328/835293 The relationship between culture and language . ELT journal, 54(4):328--334

    2000

  10. [19]

    Tom Kocmi and Christian Federmann. 2023. https://aclanthology.org/2023.eamt-1.19 Large language models are state-of-the-art evaluators of translation quality . In Proceedings of EAMT

    2023

  11. [20]

    Claire Kramsch. 2014. https://www.jbe-platform.com/content/journals/10.1075/aila.27.02kra Language and culture . AILA review, 27(1):30--55

    2014 doi

  12. [21]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. https://dl.acm.org/doi/10.1145/3600006.3613165 Efficient memory management for large language model serving with pagedattention . In Proceedings...

    2023

  13. [22]

    Hwaran Lee, Seokhee Hong, Joonsuk Park, Takyoung Kim, Meeyoung Cha, Yejin Choi, Byoungpil Kim, Gunhee Kim, Eun-Ju Lee, Yong Lim, Alice Oh, Sangchul Park, and Jung-Woo Ha. 2023. https://aclanthology.org/2023.acl-long.370 SQ u AR e: A large-scale dataset of sensitive questions a...

    2023

  14. [23]

    Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Jose Camacho-Collados, Juho Kim, and Alice Oh. 2024. https://aclanthology.org/2024.naacl-long.236/ Exploring cross-cultural differences in english hate speech annotations: From dataset construction to analysis . In Proceedings of NAACL

    2024

  15. [24]

    Sharon Levy, Neha John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vittorio Castelli, and Dan Roth. 2023. https://aclanthology.org/2023.emnlp-main.634 Comparing biases and the impact of multilingual training across multiple languages . In Proceedi...

    2023

  16. [25]

    Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ninghao Liu, and Mengnan Du. 2024. https://arxiv.org/abs/2404.11553 Quantifying multilingual performance of large language models across languages . arXiv preprint arXiv:2404.11553

    2024 arXiv

  17. [26]

    Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2025. https://aclanthology.org/2025.naacl-long.485/ Is translation all you need? a study on solving multilingual tasks with large language models . In Proceedings of the 2025 Conference of the Nations of th...

    2025

  18. [27]

    James MacQueen et al. 1967. https://projecteuclid.org/ebooks/berkeley-symposium-on-mathematical-statistics-and-probability/Proceedings-of-the-Fifth-Berkeley-Symposium-on-Mathematical-Statistics-and/chapter/Some-methods-for-classification-and-analysis-of-multivariate-observatio...

    1967

  19. [28]

    Quinn McNemar. 1947. https://link.springer.com/article/10.1007/bf02295996 Note on the sampling error of the difference between correlated proportions or percentages . Psychometrika, 12(2):153--157

    1947 doi

  20. [29]

    Edoardo Mosca, Mohamed Hesham Ibrahim Abdalla, Paolo Basso, Margherita Musumeci, and Georg Groh. 2023. https://aclanthology.org/2023.trustnlp-1.17 Distinguishing fact from fiction: A benchmark dataset for identifying machine-generated scientific papers in the LLM era. In Proce...

    2023

  21. [30]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. https://aclanthology.org/2021.acl-long.416 S tereo S et: Measuring stereotypical bias in pretrained language models . In Proceedings of ACL, pages 5356--5371

    2021

  22. [31]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. 2020. https://aclanthology.org/2020.emnlp-main.154/ Crows-pairs: A challenge dataset for measuring social biases in masked language models . In Proceedings of EMNLP, pages 1953--1967

    2020

  23. [32]

    Debora Nozza, Federico Bianchi, Anne Lauscher, and Dirk Hovy. 2022. https://aclanthology.org/2022.ltedi-1.4 Measuring harmful sentence completion in language models for LGBTQIA + individuals . In Proceedings of ACL 2022 Workshop on Language Technology for Equality, Diversity a...

    2022

  24. [33]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    2023 arXiv

  25. [34]

    Sitti Rabiah. 2018. https://osf.io/preprints/marxiv/nw94m Language as a tool for communication and cultural reality discloser

    2018

  26. [35]

    Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. 2021. https://aclanthology.org/2021.findings-acl.80 SOLID : A large-scale semi-supervised dataset for offensive language identification . In Findings of ACL, pages 915--928

    2021

  27. [36]

    Smith, and Yejin Choi

    Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. https://aclanthology.org/2020.acl-main.486 Social bias frames: Reasoning about social and power implications of language . In Proceedings of ACL, pages 5477--5490

    2020

  28. [37]

    Sheikh Shafayat, Eunsu Kim, Juhyun Oh, and Alice Oh. 2024. https://openreview.net/forum?id=lkrH6ovzsj Multi-fact: Assessing multilingual llms' multi-regional knowledge using factscore . In Proceedings of COLM

    2024

  29. [38]

    Farzad Sharifian. 2017. https://benjamins.com/catalog/clscc.8?srsltid=AfmBOopq6xKt0zAagnOVBSOGVLBFfnztWDlgepcgInOrOTWAkDJd-5b7 Cultural Linguistics: Cultural conceptualisations and language , volume 8. John Benjamins Publishing Company

    2017

  30. [39]

    Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.16 Systematic biases in LLM simulations of debates . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 251--267, Mi...

    2024 doi

  31. [40]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. https://arxiv.org/abs/2403.08295 Gemma: Open models based on gemini research and technology . arXiv ...

    2024 arXiv

  32. [41]

    Himanshu Thakur, Atishay Jain, Praneetha Vaddamanu, Paul Pu Liang, and Louis-Philippe Morency. 2023. https://aclanthology.org/2023.acl-short.30 Language models get a gender makeover: Mitigating gender bias with few-shot data interventions . In Proceedings of ACL, pages 340--351

    2023

  33. [42]

    Manuel Tonneau, Diyi Liu, Samuel Fraiberger, Ralph Schroeder, Scott A Hale, and Paul R \"o ttger. 2024. https://aclanthology.org/2024.woah-1.23/ From languages to geographies: Towards evaluating cultural bias in hate speech datasets . In Proceedings of NAACL 2024 Workshop on O...

    2024

  34. [43]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . arXiv preprint arXiv:...

    2023 arXiv

  35. [44]

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023. https://aclanthology.org/2023.findings-emnlp.243 `` kelly is a warm person, joseph is a role model '' : Gender biases in LLM -generated reference letters . In Findings of EMNLP, pages 3730--3748

    2023

  36. [45]

    Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. 2023. https://aclanthology.org/2023.emnlp-main.84 Unveiling the implicit toxicity in large language models . In Proceedings of EMNLP, pages 1322--1338

    2023

  37. [46]

    Weijie Xu, Zicheng Huang, Wenxiang Hu, Xi Fang, Rajesh Cherukuri, Naumaan Nayyar, Lorenzo Malandri, and Srinivasan Sengamedu. 2024. https://aclanthology.org/2024.nlp4hr-1.5 HR - M ulti WOZ : A task oriented dialogue ( TOD ) dataset for HR LLM agent . In Proceedings of EACL 202...

    2024

  38. [47]

    Kai-Ching Yeh, Jou-An Chi, Da-Chen Lian, and Shu-Kai Hsieh. 2023. https://aclanthology.org/2023.rocling-1.37/ Evaluating interfaced llm bias . In Proceedings of ROCLING, pages 292--299

    2023

  39. [48]

    Yue Yu, Rongzhi Zhang, Ran Xu, Jieyu Zhang, Jiaming Shen, and Chao Zhang. 2023. https://aclanthology.org/2023.acl-long.141 Cold-start data selection for better few-shot language model fine-tuning: A prompt-based uncertainty propagation approach . In Proceedings of ACL, pages 2...

    2023

  40. [49]

    Fei Yuan, Shuai Yuan, Zhiyong Wu, and Lei Li. 2024. https://aclanthology.org/2024.findings-acl.721 How vocabulary sharing facilitates multilingualism in llama? In Findings of ACL

    2024

  41. [50]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. https://arxiv.org/abs/2303.18223 A survey of large language models . arXiv preprint arXiv:2303.18223

    2023 arXiv

  42. [51]

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/1bd359b32ab8b2a6bbafa1ed2856cf40-Paper-Conference.pdf How do large language models handle multilingualism? In Advances in Neural Informat...

    2024

  43. [52]

    Peide Zhu and Claudia Hauff. 2022. https://aclanthology.org/2022.findings-naacl.183 Unsupervised domain adaptation for question generation with D omain D ata selection and self-training . In Findings of NAACL, pages 2388--2401

    2022

  44. [53]

    Ran Zmigrod, Tim Vieira, and Ryan Cotterell. 2022. https://aclanthology.org/2022.naacl-main.360 Exact paired-permutation testing for structured test statistics . In Proceedings of NAACL, pages 4894--4902

    2022

  45. [54]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  46. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.