Pith. sign in

REVIEW 3 major objections 4 minor 69 references

Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prompting a large language model in Simplified versus Traditional Chinese changes both whether it answers correctly and which job candidate it would hire, and the two biases point in opposite directions.

desk verdict Solid empirical audit of LLM biases across Simplified/Traditional Chinese, but the name-task headline overstates a script effect that the paper's own control experiments undercut. read the letter →

arxiv 2505.22645 v1 pith:2HEU377Y submitted 2025-05-28 cs.CL cs.CY

Hanjia Lyu , Jiebo Luo , Jian Kang , Allison Koenecke This is my paper
classification cs.CLcs.CY
keywords algorithmicfairnesslargelanguagemodelsSimplifiedChineseTraditionalregionaltermsnameselectionbiastokenizationbenchmarkdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models treat Simplified Chinese and Traditional Chinese—two closely related writing systems—as neutral variants or as markers of region and culture. The authors build two benchmark tasks: a regional term choice task, in which a model must name an object using the word current in Mainland China or in Taiwan, and a regional name choice task, in which a model picks one hire from a list of Mainland Chinese and Taiwanese names. Eleven commercial and open-weight LLMs are prompted in Simplified Chinese, Traditional Chinese, and English. The central result is that bias is task-dependent: most models answer regional term questions significantly more correctly when prompted in Simplified Chinese, and most models choose Taiwanese names significantly more often than Mainland Chinese names in the hiring task. The paper ties the first bias to an imbalance in training-data representation and the second to character preferences and to tokenizers that fragment Traditional Chinese names differently.

What carries the argument

The carrying object is SC-TC-Bench, a paired benchmark holding 110 regional terms with definitions in both scripts and 352 regionally popular three-character names with population counts and gender labels. The primary metrics are the correct, misaligned, and incorrect response shares for the term task, and the Mainland Chinese Name Rate for the name task, with 50% defined as the regionally unbiased reference. Three controlled experiments do the causal work: conditioning on population-based name popularity, balancing gender distributions in the candidate lists, and a same-name-different-script comparison restricted to six names identical except for being written in Simplified or Traditional characters, which flips the majority preference from Taiwanese to Mainland names. Token-generation probabilities for surname characters and per-name token counts round out the mechanism analysis.

What would settle it

Replicate the name-choice task with gender-balanced Mainland and Taiwanese candidate lists of equal size and matched character structure; if the Mainland Chinese Name Rate moves to 50%, the claimed preference for Taiwanese names is an artifact of gender composition rather than a script or regional bias.

Watch

Extended reading notes

Core claim

The discovery, on the paper's own terms, is that LLM behavior is not neutral across Simplified and Traditional Chinese, and the direction of the bias depends on the task rather than on the model's training language. Across 11 models spanning English-, Simplified Chinese-, and Traditional Chinese-oriented training, correct response rates on the regional term task are significantly higher when prompts are in Simplified Chinese, while “misaligned” responses—returning the Mainland term when asked in Traditional Chinese—are significantly more likely under Traditional prompts. On the regional name choice task the pattern reverses: most models select Taiwanese names at rates significantly below the 50% Mainland-share benchmark that an unbiased model would meet. The paper attributes the term bias to overrepresentation of Simplified Chinese regional terms in large text corpora, and the name bias to model preferences for particular characters and to tokenization differences between the two scripts.

Load-bearing premise

The headline Taiwanese-name preference assumes that, after the reported controls, the Mainland and Taiwanese name lists are regionally comparable, so the measured Mainland Chinese name rate isolates script or region rather than gender imbalance—yet the Mainland corpus is heavily female-skewed (134 female against 18 male names) and the direct script-only test uses just six all-female names.

Editorial extensions

If this is right

  • Correct-response gaps on regional terms persist when prompts are rephrased, so the Simplified-Chinese advantage is robust to wording.
  • Terms that models systematically misalign under Traditional prompts appear more often in Simplified form even inside Traditional Chinese-oriented corpora, implicating training-data imbalance as a cause.
  • Holding the name constant and changing only the script flips the majority selection from Taiwanese to Mainland names, isolating script and tokenization as drivers of the name bias.
  • Small single-character edits, such as a surname written 陈 versus 陳, can elicit large swings in which region's names get chosen.
  • A regionally unbiased model should refuse to hire on the basis of a name alone; most tested models instead comply with the prompt and pick a candidate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A gender-balanced replication of the name task, with matched male and female names in both regions, is the natural next audit: the paper's own Mainland corpus is 134 female against 18 male names, and the script-only experiment uses only female names.
  • If tokenization is the mechanism, retraining or re-fragmenting Chinese tokenizers could reduce the name bias more directly than curating additional training data.
  • The matched-prompt design transfers to other Chinese-using regions such as Hong Kong, Macau, and Malaysia, and to other script or dialect pairs, where the bias direction could differ.
  • The finding that English prompts often yield the highest valid-response rates hints that using English as a 'neutral' fallback can import a third, English-flavored bias into Chinese-language decisions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SC-TC-Bench, a new benchmark for auditing large language models (LLMs) when prompted in Simplified versus Traditional Chinese, and applies it to 11 commercial and open-source LLMs. Two tasks are designed: regional term choice, where the model must name an item using the regionally appropriate term (Mainland China versus Taiwan), and regional name choice, where the model must select a job candidate from a mixed list of Mainland Chinese and Taiwanese names. The study reports that most LLMs are significantly more correct in Simplified Chinese on the term task, and that most LLMs select Taiwanese names at a higher rate than Mainland Chinese names across prompt languages. The paper further attempts to explain the name-task bias through name popularity, gender composition, character-level token generation probabilities, a same-name/different-script control, and tokenization analysis. The benchmark dataset is publicly released.

Significance. The term-task result is a solid, well-controlled contribution: the authors use power analysis, Benjamini-Hochberg correction, multiple prompt variants, and corpus-frequency evidence to show a reproducible Simplified-Chinese advantage, and they release the benchmark for future audits. The name-task investigation is also valuable as a real-world hiring audit, particularly the character-preference experiments and the same-name script-control design. However, the headline interpretation of the name task as showing a bias toward Traditional Chinese script is not supported by the paper's own control experiment, which flips the preference toward Simplified forms when name identity is held constant. With a careful reinterpretation that separates regional name content from script effects, and with attention to the gender imbalance in the name corpus, the paper can make a meaningful and publishable contribution to fairness auditing of LLMs across Chinese script variants.

major comments (3)
  1. [§4.6.1, Figure 4; Abstract] The same-name/different-script experiment shows that when only the script differs, the majority of LLMs prefer the Mainland (Simplified) rendering—the opposite of the main result reported in §4.1. This directly contradicts the abstract's claim that LLMs 'surprisingly favored Traditional Chinese names' in the regional name choice task. The observed preference in the main name task is a preference for Taiwanese name content (e.g., given-name characters such as 俊 and 宇, which do not appear in the Mainland corpus), not for Traditional Chinese script. The script effect is real but operates in the opposite direction, so the abstract and the Section 5 statement that 'specific characters and tokenization of written scripts could be a driver of biases favoring Traditional Chinese' conflate two distinct mechanisms. The headline and discussion should be reframed to distinguish regional name identity from script effect.
  2. [§2.2.2, Figure 3, §4.4] The Mainland Chinese name corpus is heavily female-skewed (134 female versus 18 male names) while the Taiwanese corpus is balanced (100 male versus 100 female). Since prior work (Nghiem et al., ref [38]) demonstrates that LLMs exhibit name-based gender bias in employment recommendations, the raw Mainland Chinese name rates in Figure 3 may undercount Mainland names for reasons of gender rather than region. The gender-matched analyses in §4.4 and Tables 24–26 are presented as robustness checks, but the primary experiment is never re-run with balanced gender sampling, and the main-text discussion still reports the unbalanced result as the headline. The authors should either re-run the main name task with gender-balanced candidate lists or report the gender-matched rates as the primary outcome, with the unbalanced result relegated to an auxiliary analysis.
  3. [§4.6.2, Table 37] The tokenization analysis shows that Traditional Chinese names are typically split into more tokens than their Simplified equivalents for most LLMs, a pattern that prior work (Ahia et al., ref [3]) associates with over-fragmentation and degraded processing. This predicts a preference for Simplified forms, and indeed the same-name experiment in §4.6.1 observes exactly that preference. The paper's assertion that tokenization 'likely contributes to the observed name selection biases' favoring Traditional Chinese is therefore internally inconsistent: the tokenization evidence works against the main name-task result and cannot be cited as a partial explanation of it without a more nuanced argument that specifies how fragmentation could produce a preference for the higher-token variant. The authors should either revise this explanatory claim or present the tokenization result as evidence that script-based processing actually counteracts the name-content bias.
minor comments (4)
  1. [Appendix A.2] There is a typo in the description of hyperparameters: 'temprature' should be 'temperature'.
  2. [Figure 4 and Figure 11 captions] The arrow labels 'MC Name Rate Up' and 'MC Name Rate Down' are not self-explanatory; please clarify in the caption that red arrows denote an increase in the selection rate of Mainland Chinese names relative to the corresponding baseline figure.
  3. [Table 37] The rows for GPT-3.5 and Llama-3-8B are numerically identical to those for GPT-4 and Llama-3-70B, respectively; please check whether this duplication is intentional or a reporting error.
  4. [Appendix B.2, Tables 14–17] The full term-level tables are very large and difficult to parse in print; consider supplementing them with a compact visualization or an electronic spreadsheet link in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark claims are measured outcomes against external definitions and official name statistics; explanations are post hoc but not fitted to the target.

full rationale

The paper's central claims are empirical measurements, not derivations from fitted inputs. Regional term correctness is scored against region-specific vocabulary sourced from prior published Cross-Strait vocabulary work and independently reviewed by native speakers (Sections 2.2.1, A.7), and name regional labels come from separate official government reports for Mainland China and Taiwan (Section 2.2.2, Appendix C.3). The headline asymmetry—Simplified-favoring term responses but Traditional-favoring name selections—is read off actual LLM outputs (Figures 2 and 3), so the task-dependence claim does not reduce to the benchmark construction. The explanatory mechanisms (training-data imbalance, character preference, tokenization) are tested with independent measurements: corpus frequencies (Section 3.2.2, Table 18), token-generation probabilities (Sections 4.5.1, Tables 35-36), and tokenizer token counts (Section 4.6.2, Table 37). None of these are fitted to the observed selection rates; they are separate observations that the paper offers as partial, post hoc explanations. The same-name/different-script experiment (Section 4.6.1) is an explicit control that qualifies the name-task claim, and the paper discloses the female-only limitation of that control in Appendix C.6. The only self-citation is [43] in the introduction as background on political sample simulation; it is not load-bearing. The paper is self-contained against external benchmarks and official statistics. The skeptic's concern about the name task conflating name content with script is an interpretation or correctness issue, not circularity: the observed flip under the script-only control weakens the 'favor Traditional Chinese names' headline but does not make any claimed result equivalent to its inputs by construction. No circular step meets the evidentiary bar of quoting a derivation that reduces to its own inputs.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claims rest on the validity of the benchmark tasks and on proxy assumptions about training data and tokenization. No new theoretical entities are introduced. The most consequential assumptions are the comparability of the name corpora and the representativeness of the public text corpora as training data proxies.

free parameters (3)
  • Misalignment threshold = 3 out of 15 trials
    Hand-chosen threshold to define a term as misaligned in Section 3.2.1. Used for per-term counts, not the primary per-response metric; affects the reported average of 37.5 misaligned terms.
  • Number of permutations for name lists = 180
    Chosen after a stability analysis (Appendix A.10) showing results stabilize by 180 permutations. Affects the specific name selection rates but not the direction of the findings.
  • Number of trials for term task = 15
    Set by power analysis to detect a 5% difference with 80% power. Reasonable sample size, but arbitrary in the sense that it is a design choice rather than an externally fixed quantity.
assumptions (6)
  • domain assumption Simplified Chinese maps to Mainland China and Traditional Chinese maps to Taiwan
    The tasks treat Mainland China terms and names as Simplified and Taiwan terms and names as Traditional. The authors restrict scope to Mainland-Taiwan and explicitly exclude Hong Kong, Macau, Malaysia, and Singapore (Section 2.2).
  • domain assumption The official name reports provide representative samples of names in each region
    Used to construct candidate name lists in Section 2.2.2. Each report includes roughly the 200 most popular names, which may not fully reflect the name distributions in the general population.
  • domain assumption Publicly available text corpora are valid proxies for LLM training data
    Section 3.2.2 uses nine Huggingface corpora as proxies because training data are undisclosed. If these corpora do not reflect the actual training distributions of the audited models, the data-imbalance explanation for term bias is unsupported.
  • domain assumption Token generation probabilities reflect character preferences
    Section 4.5 interprets raw and conditioned token probabilities as a measure of preference. This assumes a monotonic relationship between token probability and selection behavior, which is plausible but not proven.
  • standard math Independence of repeated trials from the same LLM for statistical testing
    The t-tests and z-tests assume independent observations. LLM responses to identical prompts are likely correlated, potentially inflating effective sample sizes. This is especially relevant for OpenAI models with temperature set to 0 (Appendix A.2).
  • domain assumption Manual verification by native speakers ensures correctness of definitions and translations
    Relies on a small panel of one Mainland Chinese and three Taiwanese annotators. Annotator agreement and potential bias are not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese." pith.science (2026). https://pith.science/paper/2HEU377Y

@misc{pith2026250522645,
  author       = {Pith},
  title        = {Pith review of: Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2HEU377Y}},
  note         = {Machine review of arXiv:2505.22645}
}
read the original abstract

While the capabilities of Large Language Models (LLMs) have been studied in both Simplified and Traditional Chinese, it is yet unclear whether LLMs exhibit differential performance when prompted in these two variants of written Chinese. This understanding is critical, as disparities in the quality of LLM responses can perpetuate representational harms by ignoring the different cultural contexts underlying Simplified versus Traditional Chinese, and can exacerbate downstream harms in LLM-facilitated decision-making in domains such as education or hiring. To investigate potential LLM performance disparities, we design two benchmark tasks that reflect real-world scenarios: regional term choice (prompting the LLM to name a described item which is referred to differently in Mainland China and Taiwan), and regional name choice (prompting the LLM to choose who to hire from a list of names in both Simplified and Traditional Chinese). For both tasks, we audit the performance of 11 leading commercial LLM services and open-sourced models -- spanning those primarily trained on English, Simplified Chinese, or Traditional Chinese. Our analyses indicate that biases in LLM responses are dependent on both the task and prompting language: while most LLMs disproportionately favored Simplified Chinese responses in the regional term choice task, they surprisingly favored Traditional Chinese names in the regional name choice task. We find that these disparities may arise from differences in training data representation, written character preferences, and tokenization of Simplified and Traditional Chinese. These findings highlight the need for further analysis of LLM biases; as such, we provide an open-sourced benchmark dataset to foster reproducible evaluations of future LLM behavior across Chinese language variants (https://github.com/brucelyu17/SC-TC-Bench).

Figures

Figures reproduced from arXiv: 2505.22645 by the authors.

Figure 1
Figure 1. Examples of a prompt question (asked in Simplified Chinese in the left panel, and in Traditional Chinese in the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. All LLMs (except for Breeze) are significantly more likely to generate correct responses when prompted in Simplified Chinese compared to Traditional Chinese (𝑝 < .05, comparing the two blue shaded bars within each LLM labeled “S” and “T”—referring to the LLM when prompted in Simplified Chinese or Traditional Chinese, respectively). In contrast, LLMs are more likely to generate misaligned responses when prompted in T… view at source ↗
Figure 3
Figure 3. Most LLMs—whether they are English, Simplified Chinese, or Traditional Chinese-oriented—tend to select a valid [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 3
Figure 3. Figure 3: Nearly all LLMs exhibit an increase in Mainland Chinese [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]
Figure 4
Figure 4. Figure 4: The selection bias favoring Taiwanese names is inverted—revealing the majority of LLMs favoring Mainland Chinese [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: We replicate the experiment outlined in Section 3.1, with the only modification being the removal of items whose [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: The number of permutations used in the regional name task experiments (180, at the dotted vertical line) yield results [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: We replicate the experiment outlined in Section 3.1, using the first rephrased version of the original prompt. The [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: We replicate the experiment outlined in Section 3.1, using the second rephrased version of the original prompt. The [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Density plots of the number of individuals bearing [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: The selection rates for Mainland Chinese names by LLMs are overall lower compared to those of Taiwanese names [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: The selection bias favoring Taiwanese names remains (with only 4 of the 11 LLMs yielding majority selection of [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 33 canonical work pages

  1. [38]

    You Gotta be a Doctor, Lin

    Huy Nghiem, John Prindle, Jieyu Zhao, and Hal Daumé Iii. 2024. “You Gotta be a Doctor, Lin” : An Investigation of Name-Based Bias of Large Language Models in Employment Recommendations. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computa...

    doi:10.18653/v1/2024.emnlp-main.413 2024

  2. [3]

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do All Languages Cost the Same? To- kenization in the Era of Commercial Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for ...

    doi:10.18653/v1/2023.emnlp-main.614 2023

  3. [1]

    Name Statistics

    2018. Name Statistics. https://www.ris.gov.tw/documents/data/5/2/107namestat. pdf Accessed: 04-09-2024. Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese FAccT ’25, June 23–26, 2025, Athens, Greece

    work page 2018

  4. [2]

    Anurag Acharya, Kartik Talamadupula, and Mark A Finlayson. 2021. An atlas of cultural commonsense for machine reasoning. InAAAI Conference on Artificial Intelligence

    work page 2021

  5. [4]

    Ai2. 2021. c4. https://huggingface.co/datasets/allenai/c4. Accessed: 2025-04-28

    work page 2021

  6. [5]

    Mohammad Atari, Mona J Xue, Peter S Park, Damián Blasi, and Joseph Henrich

    work page

  7. [6]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report.arXiv preprint arXiv:2309.16609(2023)

    arXiv 2023

  8. [7]

    2023.Fairness and Machine Learning: Limitations and Opportunities

    Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023.Fairness and Machine Learning: Limitations and Opportunities. MIT Press

    2023

Show all 69 references
  1. [8]

    Kaj Bostrom and Greg Durrett. 2020. Byte Pair Encoding is Suboptimal for Language Model Pretraining. InFindings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 4617–462...

    2020 doi

  2. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  3. [10]

    Jianbin Chang. 2023. chinese-c4. https://huggingface.co/datasets/shjwudp/chin ese-c4. Accessed: 2025-04-28

    2023

  4. [11]

    Pokai Chang. 2023. zh-tw-wikipedia. https://huggingface.co/datasets/zetavg/zh- tw-wikipedia. Accessed: 2025-04-28

    2023

  5. [12]

    Jacob Cohen. 1992. Statistical power analysis.Current directions in psychological science1(3) (1992)

    1992

  6. [13]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. (2022), 320–335

    2022

  7. [14]

    Philipp Ennen, Po-Chun Hsu, Chan-Jan Hsu, Chang-Le Liu, Yen-Chen Wu, Yin- Hsiang Liao, Chin-Tung Lin, Da-Shan Shiu, and Wei-Yun Ma. 2023. Extending the pre-training of bloom for improved support of traditional chinese: Models, methods and results.arXiv preprint arXiv:2303.04715(2023)

    2023 arXiv

  8. [15]

    Chengguang Gan, Qinghao Zhang, and Tatsunori Mori. 2024. Application of llm agents in recruitment: A novel framework for resume screening.arXiv preprint arXiv:2401.08315(2024)

    2024 arXiv

  9. [16]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    2025 arXiv

  10. [17]

    Daniil Gurgurov, Mareike Hartmann, and Simon Ostermann. 2024. Adapting Mul- tilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters. InProceedings of the 1st Workshop on Knowledge Graphs and Large Language Mod- els (KaLLM 2024), Russa Biswas, Lucie-Aimée Ka...

    2024 doi

  11. [18]

    Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024. Dialect prejudice predicts AI decisions about people’s character, employability, and criminality.arXiv preprint arXiv:2403.00742(2024)

    2024 arXiv

  12. [19]

    Chan-Jan Hsu, Chang-Le Liu, Feng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, and Da-shan Shiu. 2023. Advancing the Evaluation of Traditional Chinese Lan- guage Models: Towards a Comprehensive Benchmark Suite.arXiv preprint arXiv:2309.08448(2023)

    2023 arXiv

  13. [20]

    Chan-Jan Hsu, Chang-Le Liu, Feng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, and Da-Shan Shiu. 2024. Breeze-7B Technical Report. (2024). arXiv:2403.02712 [cs.CL]

    2024 arXiv

  14. [21]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. 2024. C-eval: A multi- level multi-discipline chinese evaluation suite for foundation models.Advances in Neural Information Processing Systems...

    2024

  15. [22]

    Lee Chak Kei. 2023. OpenOrca-Traditional-Chinese. https://huggingface.co/dat asets/lchakkei/OpenOrca-Traditional-Chinese. Accessed: 2025-04-28

    2023

  16. [23]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask lan- guage understanding in chinese.arXiv preprint arXiv:2306.09212(2023)

    2023 arXiv

  17. [24]

    Sheng-Wei Li. 2024. c4-zhtw. https://huggingface.co/datasets/liswei/c4-zhtw. Accessed: 2025-04-28

    2024

  18. [25]

    Sheng-Wei Li. 2024. common-crawl-zhtw. https://huggingface.co/datasets/liswei /common-crawl-zhtw. Accessed: 2025-04-28

    2024

  19. [26]

    2014.Cross-Strait Common Vocabulary

    Xingjian Li, Zhiqun Qiu, and Fuling Xu. 2014.Cross-Strait Common Vocabulary. Fujian People’s Publishing House

    2014

  20. [27]

    Yizhi Li. 2024. MAP-CC. https://huggingface.co/datasets/m-a-p/MAP-CC. Accessed: 2025-04-28

    2024

  21. [28]

    Yen-Ting Lin. 2024. TaiwanChat. https://huggingface.co/datasets/yentinglin/Tai wanChat. Accessed: 2025-04-28

    2024

  22. [29]

    Yen-Ting Lin and Yun-Nung Chen. 2023. Taiwan llm: Bridging the linguistic divide with a culturally aligned language model.arXiv preprint arXiv:2311.17487 (2023)

    2023 arXiv

  23. [30]

    Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng, Peiyi Zhang, Qingqing Lyu, et al. 2023. M3ke: A massive multi-level multi-subject knowledge evaluation benchmark for chinese large language models.arXiv preprint arXiv:2305.1...

    2023 arXiv

  24. [31]

    Tianyin Liu and Janet Hsiao. 2012. The perception of simplified and traditional Chinese characters in the eye of simplified and traditional Chinese readers. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 34

    2012

  25. [32]

    Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. 2023. Alignbench: Benchmarking chinese alignment of large language models.arXiv preprint arXiv:2311.18743 (2023)

    2023 arXiv

  26. [33]

    Mapull. 2022. Chinese Pinyin Dictionary. https://github.com/mapull/chinese- dictionary Accessed: 04-09-2024

    2022

  27. [34]

    AI Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date.Meta AI.(2024)

    2024

  28. [35]

    Marco Monroy. 2024. Simplified vs. Traditional Chinese: What’s the difference? A guide. https://www.berlitz.com/blog/traditional-vs-simplified-chinese Accessed: 06-15-2024

    2024

  29. [36]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereo- typical bias in pretrained language models. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language ...

    2021 doi

  30. [37]

    New York City Council. 2021. Local Law 144 of 2021. https://www.nyc.gov/asse ts/dca/downloads/pdf/about/Local-Law-144.pdf

    2021

  31. [39]

    Ministry of Public Security (China). 2013. Moat Popular Names. https://web.ar chive.org/web/20160920191749/http://zhaoren.idtag.cn/samename/searchName !pmbyrepeatlist.htm Accessed: 04-09-2024

    2013

  32. [40]

    OpenAI. 2023. GPT-4 Technical Report.CoRRabs/2303.08774 (2023). doi:10.485 50/ARXIV.2303.08774 arXiv:2303.08774

    2023 arXiv

  33. [41]

    OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/ Accessed: 06-12-2024

    2024

  34. [42]

    Anaelia Ovalle, Ninareh Mehrabi, Palash Goyal, Jwala Dhamala, Kai-Wei Chang, Richard Zemel, Aram Galstyan, Yuval Pinter, and Rahul Gupta. 2024. Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies. InFindings of the Association f...

    2024 doi

  35. [43]

    Weihong Qi, Hanjia Lyu, and Jiebo Luo. 2024. Representation bias in political sample simulations with large language models.arXiv preprint arXiv:2407.11409 (2024)

    2024 arXiv

  36. [44]

    Science & Technology Policy Research and Information Center. 2020. Formosa Language Understanding Dataset. https://scidm.nchc.org.tw/dataset/grandchall enge2020 Accessed: 06-12-2024

    2020

  37. [45]

    M Rithani, R Venkatakrishnan, et al. 2024. Empirical Evaluation of Large Language Models in Resume Classification. In2024 Fourth International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT). IEEE, 1–4

    2024

  38. [46]

    Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2018. DRCD: A Chinese machine reading comprehension dataset.arXiv preprint arXiv:1806.00920(2018)

    2018 arXiv

  39. [47]

    Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. 2024. Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense.arXiv preprint arXiv:2405.04655 (2024)

    2024 arXiv

  40. [48]

    Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2023. Sub-Character Tokeniza- tion for Chinese Pretrained Language Models.Transactions of the Association for Computational Linguistics11 (2023), 469–487. d...

    2023 doi

  41. [49]

    Zhi-Rui Tam, Ya-Ting Pai, Yen-Wei Lee, Jun-Da Chen, Wei-Min Chu, Sega Cheng, and Hong-Han Shuai. 2024. An improved traditional chinese evaluation suite for foundation model.arXiv preprint arXiv:2403.01858(2024)

    2024 arXiv

  42. [50]

    David Thissen, Lynne Steinberg, and Daniel Kuang. 2002. Quick and easy imple- mentation of the Benjamini-Hochberg procedure for controlling the false positive rate in multiple comparisons.Journal of educational and behavioral statistics27, FAccT ’25, June 23–26, 2025, Athens, ...

    2002

  43. [51]

    Thanh Tung Tran, Truong Giang Nguyen, Thai Hoa Dang, and Yuta Yoshinaga

  44. [52]

    Dixuan Wang, Yanda Li, Junyuan Jiang, Zepeng Ding, Guochao Jiang, Jiaqing Liang, and Deqing Yang. 2024. Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization.arXiv preprint arXiv:2405.17067 (2024)

    2024 arXiv

  45. [53]

    InInternational Conference on Future Data and Security Engineering

    Improving Human Resources’ Efficiency with a Generative AI-Based Resume Analysis Solution. InInternational Conference on Future Data and Security Engineering. Springer, 352–365

  46. [54]

    Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Hao- nan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. 2023. Superclue: A comprehensive chinese large language model benchmark.arXiv preprint arXiv:2307.15020(2023)

    2023 arXiv

  47. [55]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al

  48. [56]

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large- scale language models.arXiv preprint arXiv:2309.10305(2023)

    2023 arXiv

  49. [57]

    Ruoxiao Yang and William Shi Yuan Wang. 2018. Categorical perception of Chinese characters by simplified and traditional Chinese readers.Reading and Writing31 (2018), 1133–1154

    2018

  50. [58]

    Qinyang Xu. 2023. BaiduBaike-5.63M. https://huggingface.co/datasets/xuqinyan g/BaiduBaike-5.63M. Accessed: 2025-04-28

    2023

  51. [59]

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2023. GLM-130B: An Open Bilingual Pre-trained Model. InThe Eleventh International Conference on Learning Representations

    2023

  52. [60]

    Hui Zeng. 2023. Measuring massive multitask chinese understanding.arXiv preprint arXiv:2304.12986(2023)

    2023 arXiv

  53. [61]

    Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldber...

    2022 doi

  54. [62]

    Yixuan Zhang and Haonan Li. 2023. Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE. InProceedings of the Ancient Language Processing Workshop, Adam Anderson, Shai Gordin, Bin Li, Yudong Liu, and Marco C. Passarotti (Eds.). INCOMA Ltd., Shoumen, ...

    2023

  55. [63]

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang

  56. [64]

    Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023. Don‘t Trust ChatGPT when your Question is not in English: A Study of Mul- tilingual Abilities and Types of LLMs. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing...

    2023 doi

  57. [65]

    MC” refers to “Mainland Chinese

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models.arXiv preprint arXiv:2304.06364 (2023). A Additional Details of Methods A.1 Review of Pre...

    2023 arXiv

  58. [68]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models.arXiv preprint arXiv:2303.18223(2023)

    2023 arXiv

  59. [2018]

    Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. InProceedings of the 2018 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), Marilyn Walker, Heng Ji, and Am...

    2018 doi

  60. [2022]

    Emergent abilities of large language models.arXiv preprint arXiv:2206.07682 (2022)

    2022 arXiv

  61. [2023]

    Which humans? (2023)

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.