Pith. sign in

REVIEW 5 major objections 7 minor 30 references

Disentangling Language and Culture for Evaluating Multilingual Large Language Models

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that multilingual LLMs answer culture-specific questions better when the question is asked in that culture's own language, a counterintuitive effect it names Cultural-Linguistic Synergy, and it traces the effect to a…

desk verdict Useful dual-axis evaluation framework and a likely real synergy effect, but translation-quality baselines and the neuron indicator need tightening before the strong claims can be trusted. read the letter →

arxiv 2505.24635 v1 pith:2VGCSRE5 submitted 2025-05-30 cs.CL

Jiahao Ying , Wei Tang , Yiran Zhao , Yixin Cao , Yu Rong , Wenxuan Zhang This is my paper
classification cs.CL
keywords multilingualevaluationculturalcontextlinguisticmediumCultural-LinguisticSynergylanguage-specificneuronsinterpretabilitycross-culturalbenchmarksLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multilingual language models are typically evaluated by translating English benchmarks, which conflates the language a question is asked in with the culture it asks about. This paper separates those two axes and finds that models answer culture-specific questions better when the question is posed in that culture's own language than in English, even though the models are mostly trained on English. It names this effect Cultural-Linguistic Synergy and reports it across several languages and model families, with the advantage shrinking for low-resource languages like Persian. The paper also proposes an internal explanation: a higher proportion of language-specialized neurons fire in the aligned language-culture setting, and this proportion could serve as a training-time indicator of multilingual capability. If these claims hold, benchmark design for multilingual models should stop treating language and culture as a single dimension.

What carries the argument

The load-bearing object is the Dual Evaluation Framework's four-way question decomposition $Q_{i,j}$, where $i$ indexes the cultural context and $j$ the linguistic medium. Native pairs $Q_{i,i}$ come from localizing BLEnD template questions (country references, phrasing, and answer sets), English translations $Q_{i,\mathrm{en}}$ come from BLEnD, and the remaining cross-lingual versions $Q_{\mathrm{en},i}$ are generated with GPT-4o. Comparing $Q_{i,i}$ with $Q_{i,\mathrm{en}}$ isolates the effect of language on the same cultural content, which is the comparison that produces Cultural-Linguistic Synergy. For the interpretability half, the paper defines Key Neurons as the top-$k$ ($k=5$) most activated neurons in feed-forward layers while the model answers a question, and defines the language-specialized share as the proportion of Key Neurons activated for a target language but not for English; the comparison of $P_{i,i}$ versus $P_{\mathrm{en},i}$ is what connects neuron activation to the behavioral effect.

What would settle it

Have native speakers write matched-difficulty, independently natural versions of the same culture questions in English and the target language, then re-run the $Q_{i,i}$ versus $Q_{i,\mathrm{en}}$ comparisons; if the aligned-language advantage disappears, Cultural-Linguistic Synergy is a translation artifact.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that an LLM's multilingual performance depends on an interaction between the language of the prompt and the cultural content of the question, not on language skill alone. Holding the cultural content fixed, asking in the matching language beats asking in English: Chinese-culture questions scored 8.8 points higher in Chinese than in English on average across eight models, Indonesian-culture questions 15.7 points higher in Indonesian, while the advantage was absent for Persian and Sundanese. Holding the language fixed, English-speaking cultural content is answered better than other cultures' content in the same language, presumably reflecting more English training data. The paper further claims that the aligned-language advantage is accompanied by a higher proportion of language-specialized neurons being activated, and that the proportion is larger for a stronger multilingual model than its weaker predecessor, supporting the use of this proportion as an indicator for multilingual capability during training.

Load-bearing premise

The main comparisons assume the English and native-language versions of each question are matched in naturalness and difficulty, so that score gaps reflect language-culture alignment rather than translation quality; of the 100 human-checked GPT-4o translations, the paper reports high but not perfect agreement.

Editorial extensions

If this is right

  • Multilingual evaluations should report language and cultural context as separate axes, since one number on a translated English benchmark cannot predict performance on culturally grounded questions in other languages.
  • Models primarily trained on English are not uniformly better in English: when the content is culturally non-English, asking in the local language can outperform English by large margins.
  • The proportion of language-specialized neurons activated by aligned language-culture questions is a candidate training-time indicator of multilingual capability, usable before full benchmark evaluation.
  • The total number of key neurons activated for a language correlates with performance on that language's questions, with a Pearson correlation of 0.95 for English, so activation statistics could serve as a performance proxy during development.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Cultural-Linguistic Synergy reflects cultural knowledge being stored with language-specific representations, then fine-tuning on native-language cultural corpora should improve non-English cultural performance more than simply adding more generic target-language text; this is a testable prediction the paper does not run.
  • The same dual-axis decomposition could be extended to generation, dialogue, or reasoning tasks, and to dimensions such as script, dialect, or formality; the paper's own limitation section notes that only one cultural context per language is tested.
  • The neuron-proportion indicator may let developers compare multilingual training runs without full evaluation, but the paper validates it on only two 7-8B models and one same-series comparison, so generalizing it to other architectures and scales is an open inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes a Dual Evaluation Framework that decomposes multilingual evaluation along two axes: linguistic medium (the language of the question) and cultural context (the regional knowledge being tested). Using BLEnD as the base, the authors construct a dataset of question sets Qi,j and evaluate eight LLMs. They report two main findings: (1) models perform best on American-culture questions regardless of the asking language, and (2) a 'Cultural-Linguistic Synergy' effect in which, for several non-English cultures, asking in the corresponding language yields higher scores than asking in English (e.g., +8.8 points for Chinese, +15.7 for Indonesian, Figure 3). The paper then probes feed-forward network neurons, defines 'key neurons' via a top-k threshold, and reports that in aligned language-culture settings a higher proportion of language-specialized neurons is activated, further proposing this proportion as a potential indicator of multilingual capability during training (Section 4.3). The interpretability analysis is validated on Qwen2.5-7B and Llama-3-8B, with an additional Llama-3-vs-Llama-3.1 comparison supporting the indicator idea.

Significance. If the Cultural-Linguistic Synergy effect is robust, it is an important and counterintuitive challenge to the common English-centric evaluation of multilingual LLMs: even when the English version of a culture-specific question is available, models may do better in the native language. The dual-axis decomposition is a clean idea and a useful addition to the benchmarking methodology, and the constructed dataset (derived from BLEnD with added English-to-other-language translations) is a concrete resource. The paper also ships code, which supports reproducibility. The interpretability claims, however, are more provisional than the performance claims: the neuron analysis is correlational, is validated on only two small models, and relies on a threshold-dependent neuron definition. The authors are transparent about several of these limitations in Section 8. The core empirical finding deserves publication after the baseline-difficulty and statistical-significance concerns below are addressed; the interpretability part should be reframed as suggestive rather than explanatory.

major comments (5)
  1. [Section 3.3, Eq. (2), Appendix A.4] The headline gaps Qi,i versus Qi,en rest on the assumption that the English versions are comparable in difficulty and naturalness to the native-language versions. The human evaluation in Appendix A.4 scores only 'content and format consistency' on a 1-3 rubric and checks only 100 GPT-4o-generated cases; it does not assess translationese, linguistic naturalness, or answer-set difficulty. Since the Persian gap is negative (-0.95 in Section 3.3), the effect is not universal, and baseline artifacts in the English translations could directly change the direction or size of the claimed synergy. The authors should validate the English versions with human naturalness/difficulty ratings or back-translation checks, and should report confidence intervals or permutation tests over questions for the Qi,i vs Qi,en differences.
  2. [Section 4.3, Eqs. (6) and (8)] The neuron-based explanation is partly circular: the specialized-neuron proportion p(qen,en,qen,i) is computed from the same question pairs whose score differences define Cultural-Linguistic Synergy, and then compared with those score differences to claim explanation. The Spanish exception is handled post hoc in the final paragraph of Section 4.3 by appealing to English-Spanish structural similarity and neuron overlap, with no independent evidence. To make the interpretability claim load-bearing, the proportions should either predict synergy on held-out language/culture pairs, or be shown to mediate the effect after controlling for translation quality and question difficulty. As written, the neuron analysis is correlational and threshold-dependent.
  3. [Section 4.3.2, Figure 7, Appendix A.2] The claim that the number of activated neurons is 'highly correlated' with performance relies on an aggregate Pearson correlation of 0.95 computed from a small number of points (one per cultural context for one model), with no confidence intervals or significance testing. The aggregate correlation could be confounded by overall cultural-context difficulty. The authors should report the number of data points, provide per-item or per-culture matched analyses, and state confidence intervals, or soften the causal phrasing in the text.
  4. [Section 4.3.1, Figure 6] The validation of Hypothesis 1 uses a single comparison between Llama-3-8B-Instruct and Llama-3.1-8B-Instruct. The paper itself acknowledges in Section 4.3.1 that training-data and architecture differences across model series limit comparability, yet the conclusion that the neuron proportion 'could serve as a potential indicator' leans heavily on this one pair of checkpoints. Additional same-series models or training-time checkpoints with different multilingual capabilities are needed to support the indicator claim.
  5. [Section 4.3 and Section 8] The interpretability experiments are limited to Qwen2.5-7B-Instruct and Llama-3-8B-Instruct, as the Limitation section states, and each language is represented by a single cultural context. Despite this, the text in Section 4.3 generalizes to 'models' without qualifying the scale. The authors should either add at least one larger model (e.g., Llama-3-70B) to the neuron analysis or explicitly limit the interpretability conclusions to small models.
minor comments (7)
  1. [Abstract and Section 3.3] The term 'CulturalLinguistic Synergy' appears without a hyphen in the abstract and in Section 3.3; it should be 'Cultural-Linguistic Synergy' for consistency with the rest of the paper.
  2. [Section 2, Eq. (3)] Equation (3) omits the i≠j restriction stated in the surrounding text; as written it includes the case j=i, which would duplicate Qi,i. Add the condition to the set definition.
  3. [Table 1] The abbreviation 'JB' for West Java is used without expansion; define it in the caption or text (e.g., 'West Java (JB), Indonesia').
  4. [Section 3.3] The sentence reporting the Persian gap is ambiguous: 'in Persian, the performance gap is -0.95 points' followed by 'this advantage diminishes' could mean opposite things. Specify that a negative gap means the English version scores higher.
  5. [Figure 17] The label 'Highest' in Figure 17 is unclear; provide a legend or a caption sentence explaining that it marks the best-performing language condition.
  6. [Appendix A.4] The sentence describing the human evaluators is incomplete: 'trained in advanced' appears to be cut off. Complete the description.
  7. [Section 4.1, Eq. (6) and Appendix A.5] The notation 'threshold function (threshold)' in Eq. (6) is not defined until Appendix A.5, where the actual method is top-k per layer. Align the main-text notation with the appendix definition for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central benchmarking result rests on external BLEnD data and the interpretability analysis is a post-hoc correlation, not a definitional reduction.

full rationale

The paper's central Cultural-Linguistic Synergy finding is an empirical comparison between Q_i,i and Q_i,en, where Q_i,i are localized questions from the external BLEnD benchmark and Q_i,en are BLEnD's English translations; neither is constructed from the model outputs being evaluated. The additional Q_en,i items are produced by GPT-4o, but the main synergy comparisons for Chinese and Indonesian do not rely on these items. No equation defines the neuron proportion in terms of performance; Eq. 8 is a set-difference of key neurons, and the claim that Pi,i exceeds P_en,i in synergy scenarios is an observed correlation rather than an identity. The threshold k=5 is selected by an ablation on masking effects, not by fitting the reported correlations. Self-citations for 'key neurons' and language-specific neurons are methodological conventions with independent provenance, and no uniqueness theorem is imported. The Spanish exception is an ad hoc explanation, which weakens confidence but is not circular. The in-sample nature of the neuron validation and the translation-quality confound are validity concerns, not circularity, under the stated review rules.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The framework adds one threshold parameter for neuron selection and otherwise reuses BLEnD data, GPT-4o translations, and prior neuron-analysis conventions. The main score-based comparisons are not fitted, but the interpretability indicator inherits the threshold choice and depends on several untested domain assumptions about translation equivalence and neuron semantics.

free parameters (1)
  • key neuron threshold k = 5 (top-k per layer)
    Chosen in Section 5 by measuring performance drop on Qen,en when masking key neurons while checking ARC as an out-of-distribution control. All neuron-proportion results in Section 4 depend on this threshold.
assumptions (4)
  • domain assumption BLEnD's localized questions and answer sets faithfully represent everyday cultural knowledge for each region.
    Section 2 adopts BLEnD as the source of Qi,i sets; if the benchmark is unrepresentative or biased, the cultural comparisons lose their meaning.
  • domain assumption The English and native-language versions of each question are comparable in content, difficulty, and answer format.
    The Cultural-Linguistic Synergy comparison in Section 3.3 assumes score gaps come from language-culture alignment rather than translation quality. Appendix A.4 checks only 100 GPT-4o translations for the added Qen,i pairs.
  • domain assumption Highly activated feed-forward neurons are the locus of factual knowledge, so top-k activation sets can be compared across languages.
    Section 4.1 invokes Geva et al. (2021) and related neuron interpretability work; the neuron-based explanation collapses if activation magnitude does not track knowledge use.
  • ad hoc to paper Spanish is an exception because of structural similarity to English, leading to overlapping knowledge-storing neurons.
    Section 4.3 explains away the Spanish mismatch between the neuron-proportion pattern and the synergy pattern with no independent evidence, preserving the proposed explanation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disentangling Language and Culture for Evaluating Multilingual Large Language Models." pith.science (2026). https://pith.science/paper/2VGCSRE5

@misc{pith2026250524635,
  author       = {Pith},
  title        = {Pith review of: Disentangling Language and Culture for Evaluating Multilingual Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2VGCSRE5}},
  note         = {Machine review of arXiv:2505.24635}
}
read the original abstract

This paper introduces a Dual Evaluation Framework to comprehensively assess the multilingual capabilities of LLMs. By decomposing the evaluation along the dimensions of linguistic medium and cultural context, this framework enables a nuanced analysis of LLMs' ability to process questions within both native and cross-cultural contexts cross-lingually. Extensive evaluations are conducted on a wide range of models, revealing a notable "CulturalLinguistic Synergy" phenomenon, where models exhibit better performance when questions are culturally aligned with the language. This phenomenon is further explored through interpretability probing, which shows that a higher proportion of specific neurons are activated in a language's cultural context. This activation proportion could serve as a potential indicator for evaluating multilingual performance during model training. Our findings challenge the prevailing notion that LLMs, primarily trained on English data, perform uniformly across languages and highlight the necessity of culturally and linguistically model evaluations. Our code can be found at https://yingjiahao14. github.io/Dual-Evaluation/.

Figures

Figures reproduced from arXiv: 2505.24635 by the authors.

Figure 1
Figure 1. Dual Evaluation Framework for evaluating [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The performance of the selected models on [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The performance of the selected models on Chinese, Indonesian, and Iranian culture questions when asked [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Workflow for interpreting Cultural-Linguistic Synergy. For every dual-format question pair, we trace [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Comparison of proportion of specialized neurons for Llama-3-8B-instruct and Llama-3.1-8B￾instruct. The result shows that Llama-3.1-8B-instruct, the multilingual extension for Llama-3-8B-instruction, has a higher proportion of specialized neurons. The results shown in …
Figure 8
Figure 8. Figure 8: Performance for Llama-3-8B-Instruction on [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 7
Figure 7. Figure 7: The performance and the number of Key Neurons for the Llama-3-8B on cross-cultural contexts. 5 Ablation Study In our experimental setup (Section 4.2), we select k = 5 as the threshold. The threshold is set to ensure that the selected key neurons accurately rep￾resent t…
Figure 9
Figure 9. Figure 9: The average performance of Claude-3.5-sonnet on Instruction set [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: The average performance of GPT-4o on Instruction set [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: The average performance of Command-r on Instruction set [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: The average performance of Gemma-2-9b-Instruct on Instruction set [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: The average performance of Llama-3-70b-Instruct on Instruction set [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: The average performance of Qwen-2.5-7b-Instruct on Instruction set [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: The average performance of Llama-3-8b-Instruct on Instruction set [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: The average performance of Bloomz-7b on Instruction set [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: The performance of the selected models on the American culture question when asked in the other six [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: The performance and the number of Key Neurons for the Llama-3-8B-Instruction on cross-cultural [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 19
Figure 19. Figure 19: The performance and the number of Key Neurons for the Qwen-2.5-7B-Instruction on cross-cultural [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 6 canonical work pages

  1. [1]

    Anthropic. 2024. https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf Claude 3.5 sonnet model card addendum

    2024

  2. [2]

    Yixin Cao, Shibo Hong, Xinze Li, Jiahao Ying, Yubo Ma, Haiyuan Liang, Yantao Liu, Zijun Yao, Xiaozhi Wang, Dan Huang, Wenxuan Zhang, Lifu Huang, Muhao Chen, Lei Hou, Qianru Sun, Xingjun Ma, Zuxuan Wu, Min-Yen Kan, David Lo, Qi Zhang, Heng Ji, Jing Jiang, Juanzi Li, Aixin Sun, Xuanjing Huang, Tat-Seng Chua, and Yu-Gang Jiang. 2025 a . http://arxiv.org/abs/...

    arXiv 2025

  3. [3]

    Yixin Cao, Jiahao Ying, Yaoning Wang, Xipeng Qiu, Xuanjing Huang, and Yugang Jiang. 2025 b . http://arxiv.org/abs/2504.07440 Model utility law: Evaluating llms beyond performance through mechanism interpretable metric

    arXiv 2025

  4. [4]

    Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024. http://arxiv.org/abs/2410.02677 Culturalbench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of llms

    arXiv 2024

  5. [5]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    arXiv 2018

  6. [6]

    Cohere. 2024. https://docs.cohere.com/docs/command-r The command r model (details and application)

    work page 2024

  7. [7]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484--5495

    2021

  8. [8]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    arXiv 2024

Show all 30 references
  1. [9]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021

  2. [10]

    Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. 2024. http://arxiv.org/abs/2406.11614 Intrinsic evaluation of unlearning using parametric knowledge traces

    2024 arXiv

  3. [11]

    Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. 2024. On the multilingual ability of decoder-based pre-trained language models: Finding and controlling language-specific neurons. In Proceedings of the 2024 Conference of the North American Chap...

    2024

  4. [12]

    Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, and William Chandra Tjhi. 2023. http://arxiv.org/abs/2309.06085 Bhasa: A holistic southeast asian linguistic and cultural evaluation suite for large language models

    2023 arXiv

  5. [13]

    Chaoqun Liu, Wenxuan Zhang, Jiahao Ying, Mahani Aljunied, Anh Tuan Luu, and Lidong Bing. 2025. http://arxiv.org/abs/2502.06298 Seaexam and seabench: Benchmarking llms with local multilingual questions in southeast asia

    2025 arXiv

  6. [14]

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786

    2022 arXiv

  7. [15]

    Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Victor Gutierrez Basulto, Yazmin Ibanez-Garcia, Hwaran Lee, Shamsuddeen H Muhammad, Kiwoong Park, Anar Rzayev, Nina White, Seid M...

    2024

  8. [16]

    Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024. https://doi.org/10.18653/v1/2024.acl-demos....

    2024 doi

  9. [17]

    OpenAI. 2024. https://huggingface.co/datasets/openai/MMMLU Multilingual massive multitask language understanding

    2024

  10. [18]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  11. [19]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  12. [20]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. http://arxiv.org/abs/2210.03057 Language models are multilingual chain-of-thought reasoners

    2022 arXiv

  13. [22]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024 b . http://arxiv.org/abs/2402.16438 Language-specific neurons: The key to multilingual capabilities in large language models

    2024 arXiv

  14. [23]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  15. [24]

    Teng Wang, Zhenqi He, Wing-Yin Yu, Xiaojin Fu, and Xiongwei Han. 2024 a . Large language models are good multi-lingual learners: When llms meet cross-lingual prompts. CoRR

    2024

  16. [25]

    Weixuan Wang, Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch. 2024 b . http://arxiv.org/abs/2406.09265 Sharing matters: Analysing neurons across languages and tasks in llms

    2024

  17. [26]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. http://arxiv.org/abs/2402.10588 Do llamas work in english? on the latent language of multilingual transformers

    2024 arXiv

  18. [27]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

  19. [28]

    Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. https://openreview.net/forum?id=hJPATsBb3l M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models . In Thirty-seventh Conference on Neural Information Processi...

    2023

  20. [29]

    Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, Xin Li, and Lidong Bing. 2024. Seallms 3: Open foundation and chat multilingual large language models for southeast asian languages. CoRR, abs/2...

    2024 arXiv

  21. [30]

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? arXiv preprint arXiv:2402.18815

    2024 arXiv

  22. [31]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. http://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.