Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Finetuning an LLM in English to reduce bias and toxicity does transfer to other languages, but only as far as the model's pretraining data reaches, and every successful transfer measurably weakens the model's generation in that language.

desk verdict A solid empirical study of cross-lingual transfer of debiasing/detoxification; the transfer-asymmetry finding holds, but the 'predicted by pretraining data' claim is oversold. read the letter →

arxiv 2412.14050 v4 pith:5RKN5SR3 submitted 2024-12-18 cs.CL

Vera Neplenbroek , Arianna Bisazza , Raquel Fernández This is my paper · ORCID
classification cs.CL
keywords cross-lingualtransferdebiasingdetoxificationmultilingualLLMsdirectpreferenceoptimizationsupervisedfinetuninglanguagegenerationabilitypretrainingdatashare
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompted in languages other than English, large language models tend to show stronger social bias and toxicity, and the usual fix—safety finetuning in English—is assumed to carry over. This paper tests that assumption by finetuning seven multilingual LLMs in English with four datasets (two for debiasing, two for detoxification) and evaluating them in up to twenty languages on bias, toxicity, and generation quality. The authors find that transfer is real but uneven: the amount of bias and toxicity mitigation a language receives is predicted, for most models, by that language's share of the model's pretraining data, with low-resource languages gaining the least. They also find that both debiasing and detoxification, whenever they succeed, come at the cost of the model's ability to generate fluent, diverse, correctly language-matched text in that language. The paper concludes that English-only safety tuning is not a sufficient substitute for mitigation done in the target language.

What carries the argument

The machinery is a matched comparison of two finetuning families: supervised finetuning on curated non-harmful sentences (Panda for bias, Jigsaw and DetoxDPO-preferred for toxicity) and direct preference optimization (BiasDPO for bias, DetoxDPO for toxicity), applied in English to seven instruction-tuned LLMs and then evaluated per language. The load-bearing comparison is the correlation between the per-language change in bias or toxicity and the percentage of that language in Common Crawl, a proxy for the model's pretraining data share; this correlation carries the paper's predictive claim. The evaluation stack—CrowS-Pairs, StereoSet, and MBBQ for bias, RTP-LX for toxicity, and language consistency, mT5 fluency, unigram diversity, and Global-MMLU for generation ability—is what converts the finetuning comparison into a cross-lingual claim.

What would settle it

Directly measuring each model's actual pretraining data mix (e.g., through a data-mixture inference attack on the tokenizer) and recomputing the correlation between language share and transfer would settle the predictive claim: if the correlation disappears under true data shares, the paper's predictor is an artifact of the proxy. A second falsifier is a controlled experiment finetuning two identically architected models pretrained with deliberately different language shares; if transfer does not track the manipulated shares, the causal claim fails.

Watch

Extended reading notes

Core claim

The paper's central finding is that English-only safety finetuning is a partial, not free, solution. Supervised finetuning (SFT) on the Panda corpus reliably reduces stereotypical bias scores in non-English languages across the CrowS-Pairs, StereoSet, and MBBQ benchmarks, while direct preference optimization (DPO) on BiasDPO works only for the two Aya models. For toxicity, the situation is sharper: SFT on either Jigsaw or DetoxDPO increases toxicity, and only DPO on DetoxDPO reliably reduces it in both English and non-English languages. When mitigation does transfer, the amount of transfer across languages correlates with the percentage of the language in Common Crawl data, used as a proxy for pretraining data share, and not consistently with typological similarity or subword overlap. And in every case of successful transfer, at least one language-generation metric degrades—language consistency, fluency, diversity, or question-answering accuracy—so the authors argue that relying on transfer from English is especially risky for lower-resource languages.

Load-bearing premise

The load-bearing premise is that Common Crawl language percentages accurately proxy each model's actual pretraining data mix, which is unverified for the Gemma, Llama, and Mistral models; if this proxy is wrong, the predictive claim about transfer loses its main support.

Editorial extensions

If this is right

  • Supervised finetuning on Panda debiases models in non-English languages for most models, but DPO on BiasDPO is effective only for the two Aya models, showing that the winning debiasing recipe depends on dataset size and method.
  • SFT on Jigsaw or DetoxDPO increases toxicity, while DPO on DetoxDPO reduces toxicity in all models and both English and non-English languages, making DPO the only effective detoxification method tested.
  • The degree of toxicity-mitigation transfer correlates negatively with language share in Common Crawl for all models except the Aya models, meaning languages with more pretraining data benefit more from transfer.
  • Every successful debiasing or detoxification transfer is paired with a decrease in at least one language-generation metric, and SFT often reduces language consistency, diversity, and question-answering accuracy.
  • The authors recommend performing bias and toxicity mitigation in the evaluation language rather than relying on English transfer, particularly for lower-resource languages underrepresented in pretraining data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: a deployer could use per-language pretraining share, estimated by tokenizer-based mixture inference rather than Common Crawl proxies, to decide in advance whether English-only safety tuning will transfer to a given language.
  • Editorial extension: the finding that SFT on 'non-toxic' comments increases toxicity suggests that sentiment-level data is not a safe detoxification signal, and that preference-based objectives should be preferred whenever generation quality matters.
  • Editorial extension: the paper's per-language correlation analysis invites a causal test—finetuning two models with identical architecture but deliberately different per-language data shares would confirm whether pretraining share actually drives transfer rather than merely correlating with it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether English-only debiasing and detoxification finetuning transfers to non-English languages in generative LLMs, and whether such transfer comes at the cost of language generation ability. The authors finetune four instruction-tuned models (Aya 23, Aya Expanse, Gemma 2 9B IT, Llama 3.1 Instruct) plus their base variants on four English datasets (Panda, BiasDPO, Jigsaw, DetoxDPO) using SFT and DPO. They measure bias with CrowS-Pairs, StereoSet, and MBBQ; toxicity with RTP-LX; and language generation ability via language consistency, fluency, diversity, and Global-MMLU question answering. The main empirical findings are that SFT on Panda reduces bias, DPO on DetoxDPO reduces toxicity, transfer to non-English languages is uneven and generally weaker for lower-resource languages, and successful transfer is often accompanied by degraded language generation in the target language. The paper also reports correlations suggesting that transfer is predicted by a language's share of Common Crawl data.

Significance. If the empirical pattern holds, the paper makes a useful contribution to the practical question of when English-only safety tuning is sufficient for multilingual deployment. Its strengths include the broad model and language coverage, the systematic comparison of SFT and DPO, the joint evaluation of harmfulness and generation quality, the use of established external benchmarks, and the public release of code. The learning trajectories in Appendix F and the qualitative analysis are also valuable. However, the paper's central predictive claim about pretraining data share is only supported by in-sample correlations with a Common Crawl proxy and is contradicted by the known-mixture results for the Aya models; this weak link needs to be addressed before the conclusion can be accepted as stated.

major comments (4)
  1. [Section 5, Appendix J, Table 11] The only place where the paper has access to a known pretraining mixture is the Aya models against the Aya Collection, and there the correlations between transfer and language data percentage are small and non-significant (-0.19, -0.35, -0.10, 0.08). If pretraining data share were the driver of transfer, the strongest signal should appear where the mixture is actually known, not only in the Common Crawl proxy analysis. The manuscript does not discuss this discrepancy, yet the abstract and conclusion assert that transfer 'can be predicted' by pretraining data share. Please either provide out-of-sample evidence from a model with disclosed per-language pretraining proportions, or explicitly temper the claim to state that transfer correlates with a Common Crawl-based proxy for resource availability.
  2. [Abstract, Section 5, Section 6] The word 'predicted' in the abstract and conclusion implies out-of-sample validity, but the supporting statistics are in-sample Spearman correlations computed on small language samples (8 languages for CrowS-Pairs, 17 for RTP-LX), with no held-out evaluation or cross-validation. A correlation observed on the same languages used to estimate it does not establish predictive power. Please replace 'predicted' with 'correlates with' or add a proper out-of-sample test, such as leave-one-language-out prediction of transfer magnitude.
  3. [Section 5, Table 10] The analysis does not control for plausible confounds. Common Crawl language share is likely collinear with subword overlap, bilingual sentence similarity, and general linguistic resource availability. Indeed, in Table 10 several predictors attain comparable or larger magnitudes than DATA for some models (e.g., RTP-LX Gemma 2 9B base: SIM -0.67 vs DATA -0.60; CrowS-Pairs Gemma 2 9B IT: OVER -0.64 vs DATA 0.40). To support the claim that pretraining data share is the best predictor, please report partial correlations or a regression with nested model comparisons that show DATA adds explanatory power beyond subword overlap and similarity measures. Additionally, the table tests many model-predictor-task combinations without any multiple-comparison correction; only a handful of correlations reach p<0.01, and some are positive (e.g., Aya Expanse RTP-LX SIM 0.47), which should be discussed.
  4. [Appendix J, Table 10, RTP-LX rows] The conclusions about toxicity transfer are inconsistent across model families. For Aya 23 and Aya Expanse, the correlations between transfer and Common Crawl data share are -0.12 and 0.04, respectively, and for Aya Expanse the correlation with bilingual sentence similarity is positive (0.47), opposing the direction reported by Li et al. (2024). The paper notes that these correlations are not significant 'for all except the Aya models,' but the abstract and conclusion still generalize the pretraining-data-share explanation to all models. Please either reconcile these inconsistencies explicitly or restrict the claim to the models for which the correlation actually holds.
minor comments (6)
  1. [Section 5] There is a typo: 'significant modererate correlation' should read 'significant moderate correlation.'
  2. [Tables 2, 9, 12, 13] Several numerical entries contain stray spaces (e.g., '51 .89', '0 .059', '57 .37'), likely from formatting or extraction artifacts; please standardize the table formatting.
  3. [Appendix J] The choice of Common Crawl week 30 of 2024 is clearly motivated by model release timing, but the analysis would be more robust if the authors reported sensitivity to the snapshot, or at least acknowledged that different snapshot choices could change the correlation values.
  4. [Section 4.2, Qualitative Analysis] The qualitative analysis is described as conducted by 'one author and an annotator external to the project.' For reproducibility, please specify the annotation instructions and whether any agreement metric was computed, or explicitly label this part as illustrative rather than systematic.
  5. [Figures 2c and 3b] The 'Mean absolute change in score' panels combine metrics with very different scales (e.g., diversity as a percentage, fluency as negative perplexity, language consistency as a percentage). Consider presenting these on separate subplots or with standardized scales to avoid visual distortions.
  6. [Limitations] The limitations paragraph mentions the black-box nature of Perspective API, but it does not mention that the RTP-LX evaluations only consider completions detected as being in the prompt language; although Appendix E shows minimal differences for the 'all completions' comparison, this filtering decision and its implications for language-confused generations deserve a sentence in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: transfer results are measured on external benchmarks, and the Common Crawl predictor is an explicit proxy rather than a fitted or redefined outcome.

full rationale

The central claims rest on measured changes on external benchmarks (CrowS-Pairs, StereoSet, MBBQ, RTP-LX, Global-MMLU) before and after finetuning, with no fitted parameters or derived constants. The predictive claim is supported by Spearman correlations between transfer and language features (Table 10, Appendix J); Common Crawl percentages are explicitly assumed to approximate pretraining mixtures ('We select the Common Crawl corpus to investigate percentages of language data as we assume models have likely been trained on it, or a similar collection of web crawl data'), which is an untested proxy and external-validity limitation, not a step where the outcome is defined by the input. Self-citations to MBBQ (Neplenbroek et al. 2024) and the subword-overlap computation from Qi et al. (2023) are evaluation or measurement tools, and the main transfer findings also appear on CrowS-Pairs, StereoSet, and RTP-LX; hence they are not load-bearing. No prediction reduces by construction to an input, no parameter is fitted and renamed, and no uniqueness claim is imported from the authors' prior work.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

Free parameters: none; hyperparameters such as learning rate 3e-4, DPO beta 0.1, and QLoRA rank 64 are disclosed standard choices, not fitted to the target result. Axioms: five domain assumptions about benchmark validity, Perspective API reliability, the Common Crawl proxy, QLoRA fidelity, and fluency heuristics. Invented entities: none; MBBQ is a pre-existing benchmark by the authors, not a new theoretical object.

assumptions (5)
  • domain assumption Bias benchmark scores (CrowS-Pairs, StereoSet, MBBQ) validly measure harmful stereotyping in the tested languages.
    Section 3.3 uses these as the main bias outcomes; scores are probability-based and translations may carry errors, as acknowledged in Limitations.
  • domain assumption Perspective API toxicity scores and its language detection are valid across 17 languages.
    Section 3.3 filters generations by Perspective API language detection; Limitations note that the API is a biased, frequently retrained black box.
  • domain assumption Common Crawl language percentages approximate each non-Aya model's pretraining data mix.
    Appendix J uses Common Crawl week 30/2024 as a proxy because actual pretraining data are unknown; this underpins the main predictive claim.
  • domain assumption QLoRA adapters faithfully approximate the effect of full finetuning for these mitigation datasets.
    Section 3.2 trains all models with QLoRA; if adapter training diverges from full finetuning, the magnitude of the reported effects could differ.
  • domain assumption mT5-XL perplexity and unigram diversity measure fluency and diversity meaningfully in non-English languages.
    Section 3.3 computes median negative perplexity and new-unigram proportion; both are heuristics, especially for low-resource languages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation." pith.science (2026). https://pith.science/paper/5RKN5SR3

@misc{pith2026241214050,
  author       = {Pith},
  title        = {Pith review of: Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5RKN5SR3}},
  note         = {Machine review of arXiv:2412.14050}
}
read the original abstract

Recent generative large language models (LLMs) show remarkable performance in non-English languages, but when prompted in those languages they tend to express higher harmful social biases and toxicity levels. Prior work has shown that finetuning on specialized datasets can mitigate this behavior, and doing so in English can transfer to other languages. In this work, we investigate the impact of different finetuning methods on the model's bias and toxicity, but also on its ability to produce fluent and diverse text. We reduce biases by finetuning on curated non-harmful text, but find only direct preference optimization to be effective for mitigating toxicity. The mitigation caused by applying these methods in English also transfers to non-English languages. We find evidence that the extent to which transfer takes place can be predicted by the amount of data in a given language present in the model's pretraining data. However, this transfer of bias and toxicity mitigation often comes at the expense of decreased language generation ability in non-English languages, highlighting the importance of developing language-specific bias and toxicity mitigation methods.

Figures

Figures reproduced from arXiv: 2412.14050 by the authors.

Figure 1
Figure 1. Disclaimer: Potentially sensitive content. An example of a toxic generation by Llama 3.1 Instruct for a German prompt from RTP-LX. After English detox￾ification the generation is no longer toxic, but also no longer in German. To mitigate this behavior, LLMs are finetuned to provide safe responses to harmful user input (Üstün et al., 2024; Allam, 2024). Applying these techniques in each language separately is computa… view at source ↗
Figure 2
Figure 2. Effects of bias mitigation on bias scores, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Effects of toxicity mitigation on toxicity [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Mean absolute change in MBBQ bias score in ambiguous contexts for Llama 3.1 Instruct supervised [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Mean absolute change in bias and toxicity per [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Bias, diversity, perplexity, language consistency and question-answering of Llama 3.1 Instruct finetuned [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Toxicity, diversity, perplexity, language consistency and question-answering of Llama 3.1 Instruct [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Mean absolute change in bias score on the StereoSet benchmark for Llama 3.1 Instruct supervised [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 10
Figure 10. Figure 10: Effects of bias mitigation on bias scores, [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 9
Figure 9. Figure 9: Mean absolute change in bias per language, [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 11
Figure 11. Figure 11: Effects of toxicity mitigation on toxicity [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Effects of bias mitigation on bias scores [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Effects of toxicity mitigation on toxicity [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Effects of bias mitigation on bias scores [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM Harms: A Taxonomy and Discussion

    cs.CY 2025-12 unverdicted novelty 3.0 of 10

    This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.

Reference graph

Works this paper leans on

82 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Aakanksha , Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.671 The multilingual alignment prism: Aligning global and local preferences to reduce harm . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages...

    doi:10.18653/v1/2024.emnlp-main.671 2024

  2. [2]

    Aakanksha, Arash Ahmadian, Seraphina Goldfarb-Tarrant, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. 2024. https://openreview.net/forum?id=L1Hxp8ktiT Mix data or merge models? optimizing for performance and safety in multilingual contexts . In Neurips Safe Generative AI Workshop 2024

    2024

  3. [3]

    AI@Meta. 2024 a . https://github.com/meta-llama/llama-models/blob/main/models/llama3/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    AI@Meta. 2024 b . https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md Llama 3.1 model card

    work page 2024

  5. [5]

    Ahmed Allam. 2024. https://doi.org/10.18653/v1/2024.acl-srw.7 B ias DPO : Mitigating bias in language models through direct preference optimization . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 42--50, Bangkok, Thailand. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.acl-srw.7 2024

  6. [6]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. https://doi.org/10.18653/v1/2020.acl-main.421 On the cross-lingual transferability of monolingual representations . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623--4637, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.421 2020

  7. [7]

    Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. https://arxiv.org/abs/2405.15032 Ay...

    arXiv 2024

  8. [8]

    Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. https://doi.org/10.18653/v1/2021.acl-long.81 Stereotyping N orwegian salmon: An inventory of pitfalls in fairness benchmark datasets . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc...

    doi:10.18653/v1/2021.acl-long.81 2021

Show all 82 references
  1. [9]

    Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. https://proceedings.neurips.cc/paper_files/paper/2016/file/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf Man is to computer programmer as woman is to homemaker? debiasing word embeddings . I...

    2016

  2. [10]

    Rochelle Choenni, Anne Lauscher, and Ekaterina Shutova. 2024. https://doi.org/10.18653/v1/2024.acl-long.803 The echoes of multilinguality: Tracing cultural value shifts during language model fine-tuning . In Proceedings of the 62nd Annual Meeting of the Association for Computa...

    2024 doi

  3. [11]

    Cohere and Cohere for AI . 2024. https://huggingface.co/CohereForAI/c4ai-command-r-v01 Command R Model Card

    2024

  4. [12]

    John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet \"U st \"u n, and Sara Hooker. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.729 RLHF can speak many languages: Unlocking multilingual preference optimization for LLM s . In Proceedings of the 2024 Conferenc...

    2024 doi

  5. [13]

    Adrian de Wynter, Ishaan Watts, Tua Wongsangaroonsri, Minghui Zhang, Noura Farra, Nektar Ege Alt ntoprak, Lena Baur, Samantha Claudet, Pavel Gajdusek, Can G\"oren, Qilong Gu, Anna Kaminska, Tomasz Kaminski, Ruby Kuo, Akiko Kyuba, Jongho Lee, Kartik Mathur, Petter Merok, Ivana ...

    2025

  6. [14]

    Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. 2022. https://doi.org/10.18653/v1/2022.naacl-main.122 Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models . In Proceedings of the 2022 Conference of the N...

    2022 doi

  7. [15]

    Daryna Dementieva, Nikolay Babakov, and Alexander Panchenko. 2024. https://doi.org/10.18653/v1/2024.naacl-short.12 M ulti P ara D etox: Extending text detoxification with parallel data to new languages . In Proceedings of the 2024 Conference of the North American Chapter of th...

    2024 doi

  8. [16]

    Daryna Dementieva, Daniil Moskovskiy, David Dale, and Alexander Panchenko. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-main.70 Exploring methods for cross-lingual text style transfer: The case of text detoxification . In Proceedings of the 13th International Joint Conference...

    2023 doi

  9. [17]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. https://openreview.net/forum?id=OUIFPHEgJU QL o RA : Efficient finetuning of quantized LLM s . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  10. [18]

    Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, and Kai-Wei Chang. 2022. https://doi.org/10.18653/v1/2022.findings-aacl.24 On measures of biases and harms in NLP . In Findings of the Association f...

    2022 doi

  11. [19]

    Beyza Ermis, Luiza Pozzobon, Sara Hooker, and Patrick Lewis. 2024. https://doi.org/10.18653/v1/2024.findings-acl.893 From one to many: Expanding the scope of toxicity mitigation in language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages ...

    2024 doi

  12. [20]

    Zahra Fatemi, Chen Xing, Wenhao Liu, and Caimming Xiong. 2023. https://doi.org/10.18653/v1/2023.acl-short.108 Improving gender fairness of pre-trained language models without catastrophic forgetting . In Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023 doi

  13. [21]

    Schmeisser-Nieto, Emma Raimundo Schulz, Thiziri Saci, Sarah Saidi, Javier Torroba Marchante, Shilin Xie, Sergio E

    Karen Fort, Laura Alonso Alemany, Luciana Benotti, Julien Bezan c on, Claudia Borg, Marthese Borg, Yongjian Chen, Fanny Ducel, Yoann Dupont, Guido Ivetta, Zhijian Li, Margot Mieskes, Marco Naguib, Yuyan Qian, Matteo Radaelli, Wolfgang S. Schmeisser-Nieto, Emma Raimundo Schulz,...

    2024

  14. [22]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.301 R eal T oxicity P rompts: Evaluating neural toxic degeneration in language models . In Findings of the Association for Computational Linguist...

    2020 doi

  15. [23]

    Google. 2024. https://ai.google.dev/gemma/docs/model_card_2 Gemma 2 model card

    2024

  16. [24]

    Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc ' Aurelio Ranzato, Francisco Guzm \'a n, and Angela Fan. 2022. https://doi.org/10.1162/tacl_a_00474 The F lores-101 evaluation benchmark for low-resource and multilingua...

    2022 doi

  17. [25]

    Anthony G Greenwald and Mahzarin R Banaji. 1995. Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychological review, 102(1):4

    1995

  18. [26]

    Francisco Guzm \'a n, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc ' Aurelio Ranzato. 2019. https://doi.org/10.18653/v1/D19-1632 The FLORES evaluation datasets for low-resource machine translation: N epali -- E nglish and S i...

    2019 doi

  19. [27]

    Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A. Smith. 2024. https://arxiv.org/abs/2407.16607 Data mixture inference: What do BPE tokenizers reveal about their training data? Preprint, arXiv:2407.16607

    2024 arXiv

  20. [28]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. https://openreview.net/forum?id=rygGQyrFvH The curious case of neural text degeneration . In International Conference on Learning Representations

    2020

  21. [29]

    Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, and Maarten Sap. 2024. https://openreview.net/forum?id=ootI3ZO6TJ Polyglotoxicityprompts: Multilingual evaluation of neural toxic degeneration in large language models . In First Conference on Languag...

    2024

  22. [30]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  23. [31]

    Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. 2024. https://doi.org/10.1162/tacl_a_00661 K o BBQ : K orean bias benchmark for question answering . Transactions of the Association for Computational Linguistics, 12:507--524

    2024 doi

  24. [32]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. https://aclanthology.org/E17-2068 Bag of tricks for efficient text classification . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume ...

    2017

  25. [33]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. https://doi.org/10.1073/pnas.1...

    2017 doi

  26. [34]

    Anne Lauscher, Tobias Lueken, and Goran Glava s . 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.411 Sustainable modular debiasing of language models . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4782--4797, Punta Cana, Dominican Repu...

    2021 doi

  27. [35]

    Kummerfeld, and Rada Mihalcea

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. 2024. https://arxiv.org/abs/2401.01967 A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity . Preprint, arXiv:2401.01967

    2024 arXiv

  28. [36]

    Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman

    Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. https://doi.org/10.1145/3534678.3539147 A new generation of perspective API : Efficient multilingual character-level transformers . In Proceedings of the 28th ACM SIGKDD C...

    2022

  29. [37]

    Sharon Levy, Neha John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vittorio Castelli, and Dan Roth. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.634 Comparing biases and the impact of multilingual training across multiple languages . In Proce...

    2023 doi

  30. [38]

    Xiaochen Li, Zheng Xin Yong, and Stephen Bach. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.784 Preference tuning for toxicity mitigation generalizes across languages . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13422--13440, Miami...

    2024 doi

  31. [39]

    Sheng Liang, Philipp Dufter, and Hinrich Sch \"u tze. 2020. https://doi.org/10.18653/v1/2020.coling-main.446 Monolingual and multilingual reduction of gender bias in contextualized representations . In Proceedings of the 28th International Conference on Computational Linguisti...

    2020 doi

  32. [40]

    Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.311 T oxic C hat: Unveiling hidden challenges of toxicity detection in real-world user- AI conversation . In Findings of the Associatio...

    2023 doi

  33. [41]

    Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin

    Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. https://aclanthology.org/E17-2002 URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors . In Proceedings of the 15th Conference of ...

    2017

  34. [42]

    Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung. 2021. https://doi.org/10.18653/v1/2021.repl4nlp-1.8 Preserving cross-linguality of pre-trained models via continual learning . In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)...

    2021 doi

  35. [43]

    Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Th \'e o Dehaze, and Sebastian Ruder. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.380 Understanding and mitigating language confusion in LLM s . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language ...

    2024 doi

  36. [44]

    Michael McCloskey and Neal J. Cohen. 1989. https://doi.org/10.1016/S0079-7421(08)60536-8 Catastrophic interference in connectionist networks: The sequential learning problem . volume 24 of Psychology of Learning and Motivation, pages 109--165. Academic Press

    1989 doi

  37. [45]

    Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022. https://doi.org/10.18653/v1/2022.acl-long.132 An empirical survey of the effectiveness of debiasing techniques for pre-trained language models . In Proceedings of the 60th Annual Meeting of the Association for Computati...

    2022 doi

  38. [46]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. https://openreview.net/forum?id=Byj72udxe Pointer sentinel mixture models . In International Conference on Learning Representations

    2017

  39. [47]

    Benjamin Muller, Antonios Anastasopoulos, Beno \^ t Sagot, and Djam \'e Seddah. 2021. https://doi.org/10.18653/v1/2021.naacl-main.38 When being unseen from m BERT is just the beginning: Handling new languages with multilingual language models . In Proceedings of the 2021 Confe...

    2021 doi

  40. [48]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. https://doi.org/10.18653/v1/2021.acl-long.416 S tereo S et: Measuring stereotypical bias in pretrained language models . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inte...

    2021 doi

  41. [49]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.154 C row S -pairs: A challenge dataset for measuring social biases in masked language models . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  42. [50]

    Vera Neplenbroek, Arianna Bisazza, and Raquel Fern \'a ndez. 2024. https://openreview.net/forum?id=X9yV4lFHt4 MBBQ : A dataset for cross-lingual comparison of stereotypes in generative LLM s . In First Conference on Language Modeling

    2024

  43. [51]

    Aur \'e lie N \'e v \'e ol, Yoann Dupont, Julien Bezan c on, and Kar \"e n Fort. 2022. https://doi.org/10.18653/v1/2022.acl-long.583 F rench C row S -pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than E nglish . In...

    2022 doi

  44. [52]

    Ayomide Odumakinde, Daniel D'souza, Pat Verga, Beyza Ermis, and Sara Hooker. 2024. https://arxiv.org/abs/2408.14960 Multilingual arbitrage: Optimizing data pools to accelerate multilingual progress . Preprint, arXiv:2408.14960

    2024 arXiv

  45. [53]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. https://doi.org/10.18653/v1/2022.findings-acl.165 BBQ : A hand-built bias benchmark for question answering . In Findings of the Association for ...

    2022 doi

  46. [54]

    Jonas Pfeiffer, Ivan Vuli \'c , Iryna Gurevych, and Sebastian Ruder. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.800 UNK s everywhere: A dapting multilingual language models to new scripts . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P...

    2021 doi

  47. [55]

    Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. https://doi.org/10.18653/v1/P19-1493 How multilingual is multilingual BERT ? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996--5001, Florence, Italy. Association for Compu...

    2019 doi

  48. [56]

    Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.472 On the challenges of using black-box API s for toxicity evaluation in research . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pr...

    2023 doi

  49. [57]

    Nirmalendu Prakash and Roy Ka-Wei Lee. 2023. https://doi.org/10.18653/v1/2023.blackboxnlp-1.22 Layered bias: Interpreting bias in pretrained large language models . In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 284--2...

    2023 doi

  50. [58]

    Jirui Qi, Raquel Fern \'a ndez, and Arianna Bisazza. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.658 Cross-lingual consistency of factual knowledge in multilingual language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing...

    2023 doi

  51. [59]

    Rebecca Qian, Candace Ross, Jude Fernandes, Eric Michael Smith, Douwe Kiela, and Adina Williams. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.646 Perturbation augmentation for fairer NLP . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Proc...

    2022 doi

  52. [60]

    Leonardo Ranaldi, Elena Sofia Ruzzetti, Davide Venditti, Dario Onorati, and Fabio Massimo Zanzotto. 2024. https://doi.org/10.18653/v1/2024.starsem-1.30 A trip towards fairness: Bias and de-biasing in large language models . In Proceedings of the 13th Joint Conference on Lexica...

    2024 doi

  53. [61]

    Manon Reusens, Philipp Borchert, Margot Mieskes, Jochen De Weerdt, and Bart Baesens. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.175 Investigating bias in multilingual language models: Cross-lingual transfer of debiasing techniques . In Proceedings of the 2023 Conference...

    2023 doi

  54. [62]

    Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. https://doi.org/10.18653/v1/P19-1163 The risk of racial bias in hate speech detection . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668--1678, ...

    2019 doi

  55. [63]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. https://openreview.net/forum?id=fR3wGCk-IXp Language models are multilingual chain-of-thought reasone...

    2023

  56. [64]

    Shivalika Singh, Freddie Vargus, Daniel D ' souza, B \"o rje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O ' Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemi \'n ski, Haki...

    2024

  57. [65]

    Jongyoon Song, Nohil Park, Sangwon Yu, Chehyun Lee, Byunggook Na, Jangho Lee, Siwon Kim, Dongjin Lee, Jiheum Yeom, and Sungroh Yoon. 2021. https://github.com/JongyoonSong/K-StereoSet K-StereoSet v.01 . GitHub repository

    2021

  58. [66]

    NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prang...

    2022 arXiv

  59. [67]

    Ahmet \"U st \"u n, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D ' souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. ht...

    2024 doi

  60. [68]

    Aniket Vashishtha, Kabir Ahuja, and Sunayana Sitaram. 2023. https://doi.org/10.18653/v1/2023.findings-acl.21 On evaluating and mitigating gender biases in multilingual settings . In Findings of the Association for Computational Linguistics: ACL 2023, pages 307--318, Toronto, C...

    2023 doi

  61. [69]

    Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2024 a . https://openreview.net/forum?id=2cRzmWXK9N Beyond reverse KL : Generalizing direct preference optimization with diverse divergence constraints . In The Twelfth International Conference on Learning Repres...

    2024

  62. [70]

    Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.349 All languages matter: On the multilingual safety of LLM s . In Findings of the Association for Computational Linguist...

    2024 doi

  63. [71]

    Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.210 Challenges in detoxifying language models . In Findings...

    2021 doi

  64. [72]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  65. [73]

    Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021. https://doi.org/10.18653/v1/2021.naacl-main.190 Detoxifying language models risks marginalizing minority voices . In Proceedings of the 2021 Conference of the North American Chapter of...

    2021 doi

  66. [74]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

  67. [75]

    Abdelrahman Zayed, Goncalo Mordido, Ioana Baldini, and Sarath Chandar. 2024. https://doi.org/10.18653/v1/2024.acl-long.487 Why don ' t prompt-based fairness metrics correlate? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2024 doi

  68. [76]

    Liang Zhang, Qin Jin, Haoyang Huang, Dongdong Zhang, and Furu Wei. 2024. https://doi.org/10.18653/v1/2024.acl-long.229 Respond in my language: Mitigating language inconsistency in response generation based on large language models . In Proceedings of the 62nd Annual Meeting of...

    2024 doi

  69. [77]

    Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. https://doi.org/10.18653/v1/2020.acl-main.260 Gender bias in multilingual embeddings and cross-lingual transfer . In Proceedings of the 58th Annual Meeting of the Association fo...

    2020 doi

  70. [78]

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. https://doi.org/10.18653/v1/D17-1323 Men also like shopping: Reducing gender bias amplification using corpus-level constraints . In Proceedings of the 2017 Conference on Empirical Methods in Natur...

    2017 doi

  71. [79]

    Gonzalez, Ion Stoica, and Hao Zhang

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. https://openreview.net/forum?id=BOfDKxfwt0 LMSYS -chat-1m: A large-scale real-world LLM convers...

    2024

  72. [80]

    Ibrahim Tolga Öztürk, Rostislav Nedelchev, Christian Heumann, Esteban Garces Arias, Marius Roger, Bernd Bischl, and Matthias Aßenmacher. 2023. https://arxiv.org/abs/2307.07331 How different is stereotypical bias across languages? Preprint, arXiv:2307.07331

    2023 arXiv

  73. [81]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  74. [82]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.