Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read McBE, a 4,077-instance multi-task Chinese bias benchmark, shows larger models are less biased and smaller models' preference scores reflect random selection.

desk verdict A carefully built Chinese bias benchmark whose multi-task protocol is a real contribution, but the headline reversal on model size rests on preference metrics that confound bias with semantic coherence. read the letter →

arxiv 2507.02088 v2 pith:ZZHHP6YQ submitted 2025-07-02 cs.CL

Tian Lan , Xiangdong Su , Xu Liu , Ruirui Wang , Ke Chang , Jiang Li , Guanglai Gao This is my paper
classification cs.CL
keywords Chinesebiasevaluationmulti-taskbenchmarklargelanguagemodelsmeasurementdemographicinstancescenarioselectionnegativelog-likelihood
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that Chinese-language bias in large language models is best measured not by a single task but by five complementary tasks, and that doing so reverses a common conclusion. McBE gathers 4,077 manually annotated bias evaluation instances across 12 categories and 82 subcategories, each structured as a sentence template with a substitution list. Using this benchmark, the paper reports that larger models in the Qwen2.5 series score better on tasks requiring understanding and judgment, while smaller models' high scores on preference and scenario-selection tasks largely reflect inconsistent, near-random choices. If correct, earlier single-task benchmarks such as CBBQ and Rubia reached a one-sided conclusion that bigger models are more biased.

What carries the argument

The organizing unit is the Bias Evaluation Instance (BEI): a context, a sentence template with a placeholder, a substitution list of demographic words, an annotated subcategory, an explanation, and a human bias score. The five tasks are Preference Computation, which measures the variance of negative log-likelihoods across substitution-generated sentences, $$V = \frac{1}{n}\sum_{i=1}^{n}\left(\mathrm{NLL}(s_i)-\overline{\mathrm{NLL}}\right)^2,$$ converted to a 0–100 score by an exponential decay; Subcategory Classification; Scenario Selection, which measures the variance of pairwise selection frequencies; Bias Analysis, judged by an LLM against human reference answers; and Bias Scoring, which compares model-assigned severity scores to human scores. The variance-based tasks are the mechanism that produces the paper's central contrast, because they can be satisfied by random selection.

What would settle it

If a model whose substitution words are reweighted by human plausibility ratings or by corpus frequency still shows the same variance-as-bias ranking across parameter sizes, the random-selection explanation would be supported; if the ranking reverses when such confounds are controlled, the variance-based measure is not a valid bias signal. A reader could test this by taking a subset of McBE substitution lists, having humans rate how natural each substitution is, and recomputing the Preference Computation and Scenario Selection scores on the same Qwen2.5 models.

Watch

Extended reading notes

Core claim

The central claim is that a multi-task benchmark is necessary to measure Chinese bias, and that under such measurement larger models show less bias overall. The key finding is that preference-style tasks, such as negative log-likelihood variance and pairwise scenario selection, reward models that treat all demographic substitutions equally, but a model can earn such a score by choosing inconsistently rather than fairly. Tasks that demand explicit classification, analysis, and severity scoring show the opposite trend: scores rise with parameter size. The paper therefore argues that prior single-task results indicating larger models are more biased conflated random behavior with fairness.

Load-bearing premise

The load-bearing premise is that equal probability and equal selection frequency across substituted demographic words directly measure unbiasedness, even though the words differ in natural frequency, collocation, and plausibility.

Editorial extensions

If this is right

  • Larger Qwen2.5 models (0.5B to 32B) improve on bias classification, analysis, and scoring tasks, with diminishing returns at larger sizes.
  • Smaller models' high Preference Computation and Scenario Selection scores are not evidence of fairness, because their choices are inconsistent.
  • Single-task benchmarks built on question answering or scenario selection can misreport model-size trends in bias.
  • McBE positions itself as a reusable template for building analogous bias benchmarks in other languages and cultures.
  • Black-box models cannot be fully evaluated because Preference Computation requires token-level probabilities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The variance-based definition of bias is likely confounded by substitution-word frequency and plausibility; a model that assigns lower probability to rare or awkward sentences would appear biased even without stereotypes.
  • The claim that larger models are less biased may partly reflect improved instruction-following and test-taking ability rather than a genuine reduction in stereotyped associations, since the data separate understanding tasks from preference tasks but do not fully separate capability from value alignment.
  • A testable extension is to apply McBE's five tasks to a model before and after debiasing interventions: if preference tasks and analysis tasks move in opposite directions, the benchmark would reveal trade-offs that single-task benchmarks hide.
  • The multi-task structure could be extended to intersectional bias by composing substitution lists across two demographic dimensions, whereas McBE currently evaluates one category at a time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces McBE, a Chinese-language multi-task bias evaluation benchmark with 4,077 bias evaluation instances spanning 12 bias categories and 82 subcategories, and proposes five evaluation tasks: Preference Computation, Subcategory Classification, Scenario Selection, Bias Analysis, and Bias Scoring. The authors evaluate eight white-box and three black-box LLMs across these tasks and report that larger models generally show less bias, while smaller models' high scores on Preference Computation and Scenario Selection are attributed to random rather than fair behavior. The paper argues that this multi-task perspective corrects single-task benchmarks such as CBBQ and Rubia that concluded larger models are more biased.

Significance. The benchmark is a substantial and carefully constructed resource for Chinese bias evaluation. The annotation pipeline is a particular strength: 30 annotators with diverse regional and disciplinary backgrounds, explicit bias-education training, majority-vote with expert adjudication, quality review, and reported Fleiss' kappa values around 0.67. The robustness analysis with newly designed prompts, the statistical significance test, and the release of the dataset and code are also commendable. If the proposed scores are valid measures of bias, the multi-task design would be a meaningful advance over existing single-task Chinese bias benchmarks, and the finding that smaller models' high selection-based scores reflect random choice rather than fairness would be an important caution for the community. However, the central empirical claim depends on assumptions about what the PC and SS scores measure and on an unvalidated interpretation of small-model performance, as detailed below.

major comments (4)
  1. [§4.1, Eq. (1); §4.3; Table 1] The null hypothesis that an unbiased model assigns equal probability or equal selection frequency to all substitution sentences is not valid for the constructed templates. In the example in Table 1, the template 'His undergraduate institution is [PLH], so he wasn't that diligent during his graduate studies' is semantically coherent only with substitution words such as 'average', 'bad', or 'low level'; a competent, bias-free model should assign higher likelihood to those continuations than to 'outstanding'. Therefore the variance in Eq. (1) and the analogous variance in Scenario Selection conflate lexical/semantic plausibility and world knowledge with demographic bias. The claim in Section 5.2 that smaller models' high PC and SS scores indicate low bias is unsupported unless the templates are controlled for coherence or the variance is conditioned on semantic plausibility.
  2. [§5.2, Figure 8] The explanation that smaller models' high SS scores are 'largely due to random selection' is asserted rather than measured. Random or inconsistent choices would indeed produce near-uniform selection frequencies and hence high scores, but so would a genuinely fair, uniform selection policy. No chance baseline is provided, no consistency measure across repeated pairwise comparisons is reported, and no test distinguishes random choices from deliberate even-handed choices. The Qwen2.5 SS values cited (87.69, 80.49, 77.82, 77.11 for 0.5B, 1.5B, 7B, 32B) are only consistent with the random-selection story if a random baseline is known to yield scores in that range; without such a baseline, the argument is circular. The authors should compute the expected SS score under a random-choice null model, or add control templates without demographic content, before concluding that small models' high scores reflect incompetence rather than fairness.
  3. [§4.4, §5.2, Appendix E.1] The Bias Analysis task uses GLM4-AIR as the judge, and the reference explanations were consolidated using ChatGLM; both are from the GLM model family, and GLM4-0520 is among the evaluated models. This creates a risk of same-family favoritism that is not adequately addressed. The human-consistency check in Appendix E.1 samples only 10% of BEIs and only validates the winner in pairwise comparisons, all among 7B models; it does not include GLM4-0520 and does not validate the numerical scores assigned to individual responses. The 83.7% average consistency is reassuring but insufficient to rule out a systematic advantage for GLM-family outputs in the BA task. The authors should validate the judge on GLM4-0520 or use a judge from a different model family.
  4. [§4.2, §4.5, §5.2] The Subcategory Classification, Bias Analysis, and Bias Scoring tasks are heavily dependent on general language understanding, instruction following, and knowledge of Chinese cultural conventions. A model with low Chinese proficiency will score poorly on these tasks regardless of whether it holds demographic biases. For example, Llama2-7B-hf has very high PC and SS scores but very low SC and BA scores; the paper interprets this as 'not able to understand biases within the Chinese language context' yet still uses the combined score to conclude that smaller models 'exhibit more bias.' This conflates task competence with bias. The authors should reframe the conclusion as measuring multi-task alignment with human fairness judgments, or analyze the bias-specific component of each task separately, before claiming a reversal of previous single-task findings.
minor comments (5)
  1. [§3.3.1 vs §H.2] Section 3.3.1 states that the final Bias Score is the average of scores from 6 annotators, while Appendix H.2 reports Fleiss' kappa among 'the five annotators'; this inconsistency should be clarified.
  2. [Appendix E.1] The text says GLM4-AIR's selected winners are 'entirely consistent' with human judgments but immediately reports an average consistency of 83.7%; the wording should be revised to avoid the contradiction.
  3. [Abstract] The abstract contains a grammatical error: 'most existing bias evaluation datasets are focus on English' should be 'are focused on English'; similar issues appear elsewhere and the manuscript would benefit from a careful proofread.
  4. [§4.1, Eq. (2)] The decay rate is stated as '2e 3', which is ambiguous and likely a typesetting issue; it should be written as 2×10^3, and the choice of this specific rate should be justified or shown not to affect the ranking.
  5. [Appendix F] The experimental settings say each model is run four times and average results are reported, but the tables report standard deviations; it would be clearer to state explicitly that the deviations are across the four runs and whether the same random seeds or prompts were used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: McBE's ground truth is human-anchored, the five task scores are empirical measurements rather than fitted predictions, and the central size-versus-bias claim has independent content.

full rationale

McBE is anchored in external human annotation: subcategory labels are majority-voted by annotators, explanations are written by annotators and reviewed after LLM consolidation, and Bias Scores are averaged from six annotators with expert adjudication. The five task scores are measurements (NLL variance, classification accuracy, choice variance, judge-assigned analysis quality, and absolute agreement with human severity scores), not parameters fitted to the paper's conclusions. The central claim that larger models show less bias is an interpretation of those measurements, and the Qwen2.5 within-series trend across SC, BA, and BS provides independent evidence not reducible to the benchmark's construction. The two concerns that might resemble circularity are better classified as validity or support issues rather than circular reductions. First, the BA judge GLM4-AIR and the ChatGLM consolidation step overlap with the evaluated GLM4 family; this is a possible judge-independence limitation, but it is not a definitional equivalence, it is partially checked by the reported human-consistency evaluation, and the main trend does not rest on GLM-family scores alone. Second, the Section 5.2 explanation that small models' high SS scores reflect random selection rather than fairness is asserted without a random baseline, but that is a missing-evidence problem, not a circular derivation. There are no author self-citations used as load-bearing support, and no equation in the paper equates an input with the output it is used to predict. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The benchmark relies on human annotations as ground truth and on the assumption that preference variance isolates bias. Hand-set constants and weights shape the reported scores, so their influence should be treated as part of the measurement rather than as discovery output.

free parameters (3)
  • Decay rate r in PC and SS scores = 2000
    Hand-set in Eq. (2); controls mapping from NLL and selection variance to 0-100 scores. No sensitivity analysis is provided, and it affects all PC and SS absolute scores.
  • Bias Analysis weights w_i = 13, 3.5, 1.5, 2.5, 0.5
    Hand-assigned in Eq. (5) and Table 5; determine the BA final score composition without calibration.
  • BS scaling coefficient k = 10
    Coefficient in Eq. (6); converts mean absolute score difference to the 0-100 range. This is a natural scale factor given 0-10 human scores, so low concern.
assumptions (5)
  • domain assumption An unbiased model should show no preference for any demographic group, so the variance of NLLs across substituted sentences measures bias.
    Foundational for the PC task (Section 4.1, Eq. 1); ignores word frequency and collocation confounds.
  • domain assumption Human-annotated Bias Scores and explanations are valid ground truth for evaluating fairness alignment.
    Used as reference in BS and BA tasks (Sections 3.3 and 4.4).
  • domain assumption The GLM4-AIR judge's scores are consistent with human judgments and can stand in for human evaluation.
    Assumed in the BA task (Section 4.4); only pairwise winner consistency is checked in Appendix E.1, not numeric score agreement.
  • domain assumption Correct classification into a predefined bias subcategory reflects a model's understanding of bias and is a proxy for lower bias.
    Defines the SC task score (Section 4.2).
  • domain assumption The 12 categories and 82 subcategories grounded in Chinese laws and social norms adequately cover relevant Chinese biases.
    Basis of dataset coverage (Section 3.2).
invented entities (1)
  • Bias Evaluation Instance (BEI) independent evidence
    purpose: Standard unit of the benchmark combining context, template, substitution list, subcategory, explanation, and bias score.
    The paper releases the dataset as part of McBE; it is a data schema rather than an explanatory construct, and its instances are observable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models." pith.science (2026). https://pith.science/paper/ZZHHP6YQ

@misc{pith2026250702088,
  author       = {Pith},
  title        = {Pith review of: McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZHHP6YQ}},
  note         = {Machine review of arXiv:2507.02088}
}
read the original abstract

As large language models (LLMs) are increasingly applied to various NLP tasks, their inherent biases are gradually disclosed. Therefore, measuring biases in LLMs is crucial to mitigate its ethical risks. However, most existing bias evaluation datasets focus on English and North American culture, and their bias categories are not fully applicable to other cultures. The datasets grounded in the Chinese language and culture are scarce. More importantly, these datasets usually only support single evaluation tasks and cannot evaluate the bias from multiple aspects in LLMs. To address these issues, we present a Multi-task Chinese Bias Evaluation Benchmark (McBE) that includes 4,077 bias evaluation instances, covering 12 single bias categories, 82 subcategories and introducing 5 evaluation tasks, providing extensive category coverage, content diversity, and measuring comprehensiveness. Additionally, we evaluate several popular LLMs from different series and with parameter sizes. In general, all these LLMs demonstrated varying degrees of bias. We conduct an in-depth analysis of results, offering novel insights into bias in LLMs.

Figures

Figures reproduced from arXiv: 2507.02088 by the authors.

Figure 1
Figure 1. Examples in the responses of LLMs, exhibit [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The three limitations of existing bias evalua [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overall structure of McBE. ciation Tests (CAT) to measure language models’ biases and NLU capability, which encompass four categories: gender, occupation, race, and religion. CrowS-Pairs (Nangia et al., 2020) includes nine bias categories, and primarily emphasizes gender and race. BBQ (Parrish et al., 2021) focuses on how biases manifest within QA contexts. CEB (Wang et al., 2024) introduces a systematic bias evalua… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The proportion of each bias category in McBE. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Distribution of average Rouge-L scores be [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The models’ scores across 12 bias categories, [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: The scores of models across 5 tasks averaged [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Word Clouds of All Categories. Quality Review Questions Yes% Does the Context, Sentence Template, and Explanation contain no grammatical errors? 99% Does the Context, Sentence Template, and Explanation avoid ambiguity or misleading expressions? 99% Does each Sentence T…
Figure 10
Figure 10. Figure 10: All 8 white-box models’ scores across all categories. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Scores across all categories for all 3 black [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

    cs.CL 2025-10 conditional novelty 6.0 of 10

    Across nine Asian languages, multilingual LLMs favor Western cultural entities in 30-40% of culturally grounded contexts, with model-specific sentiment biases and extraction accuracy gaps.

Reference graph

Works this paper leans on

53 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Large language models associate muslims with violence. Nature Machine Intelligence, 3(6):461--463

    work page 2021

  4. [4]

    Marzieh Babaeianjelodar, Stephen Lorenz, Josh Gordon, Jeanna Matthews, and Evan Freitag. 2020. Quantifying gender bias in different corpora. In Companion Proceedings of the Web Conference 2020, pages 752--759

    work page 2020

  5. [5]

    Chris Baumann, Andrew R Timming, and Paul J Gollan. 2016. Taboo tattoos? a study of the gendered effects of body art on consumers' attitudes toward visibly tattooed front line staff. Journal of Retailing and Consumer Services, 29:31--39

    work page 2016

  6. [6]

    Aleksandrs Berdi c evskis, Gerlof Bouma, Robin Kurtz, Felix Morger, Joey \"O hman, Yvonne Adesam, Lars Borin, Dana Dann \'e lls, Markus Forsberg, Tim Isbister, et al. 2023. Superlim: A swedish language understanding evaluation benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8137--8153

    work page 2023

  7. [7]

    Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29

    2016

  8. [8]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297

    arXiv 2024

Show all 53 references
  1. [9]

    Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67--73

    2018

  2. [10]

    Virginia K Felkner, Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. 2023. Winoqueer: A community-in-the-loop benchmark for anti-lgbtq+ bias in large language models. arXiv preprint arXiv:2306.15087

    2023 arXiv

  3. [11]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    2024 arXiv

  4. [12]

    Ian Goodfellow. 2016. Deep learning

    2016

  5. [13]

    Veronika Grigoreva, Anastasiia Ivanova, Ilseyar Alimova, and Ekaterina Artemova. 2024. Rubia: A russian language bias detection dataset. arXiv preprint arXiv:2403.17553

    2024 arXiv

  6. [14]

    Reza Hasmath. 2024. How china sees the world in 2024. The China Institute at the University of Alberta

    2024

  7. [15]

    Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Language and linguistics compass, 15(8):e12432

    2021

  8. [16]

    Hsin-Yi Hsieh, Shih-Cheng Huang, and Richard Tsai. 2024. Twbias: A benchmark for assessing social bias in traditional chinese large language models through a taiwan cultural lens. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8688--8704

    2024

  9. [17]

    Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411--4421. PMLR

    2020

  10. [18]

    Shangying Hua, Shuangci Jin, and Shengyi Jiang. 2024. The limitations and ethical considerations of chatgpt. Data intelligence, 6(1):201--239

    2024

  11. [19]

    Yufei Huang and Deyi Xiong. 2023. Cbbq: A chinese bias benchmark dataset curated with human-ai collaboration for large language models. arXiv preprint arXiv:2306.16244

    2023 arXiv

  12. [20]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  13. [21]

    Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. 2024. Kobbq: Korean bias benchmark for question answering. Transactions of the Association for Computational Linguistics, 11:507--524

    2024

  14. [22]

    Masahiro Kaneko and Danushka Bollegala. 2022. Unmasking the mask--evaluating social biases in masked language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11954--11962

    2022

  15. [23]

    Yuri Kuratov and Mikhail Arkhipov. 2019. https://arxiv.org/abs/1905.07213 Adaptation of deep bidirectional multilingual transformers for russian language . Preprint, arXiv:1905.07213

    2019 arXiv

  16. [24]

    Bin Li, Xiaopeng Bai, Siqi Yin, and Jie Xu. 2015. Chinese cogbank: Where to see the cognitive features of chinese words. In Proceedings of the Third Workshop on Metaphor in NLP, pages 77--86

    2015

  17. [25]

    Miaomiao Li, Hao Chen, Yang Wang, Tingyuan Zhu, Weijia Zhang, Kaijie Zhu, Kam-Fai Wong, and Jindong Wang. 2025. Understanding and mitigating the bias inheritance in llm-based data augmentation on downstream tasks. arXiv preprint arXiv:2502.04419

    2025 arXiv

  18. [26]

    Adam Y Liu, Xiaojun Li, and Songying Fang. 2020. What do chinese people think of developed countries? The Diplomat, December, 18

    2020

  19. [27]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  20. [28]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456

    2020 arXiv

  21. [29]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. arXiv preprint arXiv:2010.00133

    2020 arXiv

  22. [30]

    Aur \'e lie N \'e v \'e ol, Yoann Dupont, Julien Bezan c on, and Kar \"e n Fort. 2022. French crows-pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than english. In Proceedings of the 60th Annual Meeting of the Assoc...

    2022

  23. [31]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. 2021. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193

    2021 arXiv

  24. [32]

    Altman Yuzhu Peng. 2021. Amplification of regional discrimination on chinese news portals: an affective critical discourse analysis. Convergence, 27(5):1343--1359

    2021

  25. [33]

    Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301

    2018 arXiv

  26. [34]

    Nihar Ranjan Sahoo, Pranamya Prashant Kulkarni, Narjis Asad, Arif Ahmad, Tanu Goyal, Aparna Garimella, and Pushpak Bhattacharyya. 2024. https://arxiv.org/abs/2403.20147 Indibias: A benchmark dataset to measure social biases in language models for indian context . Preprint, arX...

    2024 arXiv

  27. [35]

    Xabier Saralegi and Muitze Zulaika. 2025. Basqbbq: A qa benchmark for assessing social biases in llms for basque, a low-resource language. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4753--4767

    2025

  28. [36]

    Akash Saravanan, Dhruv Mullick, Habibur Rahman, and Nidhi Hegde. 2023. Finedeb: A debiasing framework for language models. arXiv preprint arXiv:2302.02453

    2023 arXiv

  29. [37]

    Londa Schiebinger. 2014. Scientific research must take gender into account. Nature, 507(7490):9--9

    2014

  30. [38]

    Sandhya Singh, Prapti Roy, Nihar Sahoo, Niteesh Mallela, Himanshu Gupta, Pushpak Bhattacharyya, Milind Savagaonkar, Nidhi Sultan, Roshni Ramnani, Anutosh Maitra, et al. 2022. Hollywood identity bias dataset: A context oriented bias analysis of movie dialogues. In Proceedings o...

    2022

  31. [39]

    Victor Steinborn, Philipp Dufter, Haris Jabbar, and Hinrich Sch \"u tze. 2022. An information-theoretic approach and dataset for probing gender stereotypes in multilingual masked language models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 921--932

    2022

  32. [40]

    Yarden Tal, Inbal Magar, and Roy Schwartz. 2022. Fewer errors, but more stereotypes? the effect of model size on gender bias. arXiv preprint arXiv:2206.09860

    2022 arXiv

  33. [41]

    Qwen Team. 2024. Qwen2. 5: A party of foundation models

    2024

  34. [42]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  35. [43]

    Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao'Kenneth' Huang, and Shomir Wilson. 2023. Nationality bias in text generation. arXiv preprint arXiv:2302.02463

    2023 arXiv

  36. [44]

    Yixin Wan and Kai-Wei Chang. 2024. White men lead, black women help: Uncovering gender, racial, and intersectional bias in language agency. arXiv preprint arXiv:2404.10508

    2024 arXiv

  37. [45]

    kelly is a warm person, joseph is a role model

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023. " kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters. arXiv preprint arXiv:2310.09219

    2023 arXiv

  38. [46]

    Song Wang, Peng Wang, Tong Zhou, Yushun Dong, Zhen Tan, and Jundong Li. 2024. Ceb: Compositional evaluation benchmark for fairness in large language models. arXiv preprint arXiv:2407.02408

    2024 arXiv

  39. [47]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359

    2021 arXiv

  40. [48]

    Hitomi Yanaka, Namgi Han, Ryoma Kumon, Jie Lu, Masashi Takeshita, Ryo Sekizawa, Taisei Kato, and Hiromi Arai. 2024. Analyzing social biases in japanese large language models. arXiv preprint arXiv:2406.02050

    2024 arXiv

  41. [49]

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305

    2023 arXiv

  42. [50]

    Jiaxu Zhao, Meng Fang, Zijing Shi, Yitong Li, Ling Chen, and Mykola Pechenizkiy. 2023. Chbias: Bias evaluation and mitigation of chinese conversational language models. arXiv preprint arXiv:2305.11262

    2023 arXiv

  43. [51]

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876

    2018 arXiv

  44. [52]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  45. [53]

    Markov, Vladislav Mikhailov, and Alena Fenogenova

    Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, and Alena Fenogenova. 2024. https://aclanthology.org/2024.lre...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.