MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlish

Ai Ti Aw; Bin Wang; Minh Duc Pham; Tarun Kumar Vangani; Xin Huang; Xunlong Zou; Zhengyuan Liu

REVIEW 5 major objections 4 minor 1 cited by

MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlish

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Continued pretraining on 125B tokens of English, Indonesian, and Chinese, followed by weight merging with the instruction-tuned checkpoint, produces an 8B model that the paper reports as beating Meta-Llama-3.1-8B-Instruct on Cross-MMLU…

desk verdict Useful Indonesian/Chinese checkpoint; the Malay and Singlish headline claim is undercut by the paper's own tables. read the letter →

arxiv 2501.08335 v3 pith:6U62F2PG submitted 2024-12-21 cs.CL cs.AI

Xin Huang , Tarun Kumar Vangani , Minh Duc Pham , Xunlong Zou , Bin Wang , Zhengyuan Liu , Ai Ti Aw This is my paper · ORCID

classification cs.CLcs.AI

keywords multilinguallargelanguagemodelscontinuedpretrainingweightmergingcross-lingualunderstandingIndonesianChineseMalaySinglish

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports an open-weights 8B model built by continued pretraining of Llama-3.1-8B-Base on roughly 125B tokens (English 38B, Indonesian 45B, Chinese 42B) and then merging the adapted weights with Llama-3.1-8B-Instruct. The central claim is that this recipe beats the official Llama-3.1-8B-Instruct on the model's headline benchmarks: Cross-MMLU average, IndoMMLU, and CN-Eval, while keeping English strong. The intended significance is that a relatively cheap two-stage recipe, continued pretraining plus weight merging, can adapt a strong open base model to regional languages without full retraining or extensive instruction tuning. The paper positions the approach as a template for expanding coverage to other underrepresented languages.

What carries the argument

The machinery is a two-stage pipeline: continued pretraining on a balanced multilingual corpus, then weight merging between the adapted base and the official instruct checkpoint. The corpus is split into English (38B tokens), Indonesian (45B tokens), and Chinese (42B tokens), with domain classification, hyperparameter optimization, and replay techniques used to mitigate catastrophic forgetting. Weight merging combines the parameters of the continued-pretrained base with Llama-3.1-8B-Instruct so that the adapted knowledge and instruction-following abilities coexist in one set of weights. The paper's evidence for the mechanism is the contrast between a standalone instruction-tuned model, which underperforms the baseline, and the merged model, which outperforms it.

What would settle it

Run the released model on held-out native Malay and Singlish benchmarks (for example, a Malay MMLU and a Singlish conversational or cultural-knowledge set) and compare with Meta-Llama-3.1-8B-Instruct; if the model does not exceed the baseline there, the title's cross-lingual claim for those languages fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the merged model LLaMA-3-MERaLiON-8B-Instruct outperforms Meta-Llama-3.1-8B-Instruct on Cross-MMLU (average 0.717 vs 0.690), IndoMMLU (0.576 vs 0.548), and CN-Eval (0.514 vs 0.457). The pattern the authors emphasize is that the gains come from continued pretraining on a balanced multilingual corpus, and that weight merging preserves those gains while restoring instruction-following behavior that a standalone instruction-tuned model lacked. The paper's own tables show the gains are concentrated in English, Chinese, and Indonesian on the general-knowledge benchmark, with a mixed pattern on the logic benchmark.

Load-bearing premise

The load-bearing assumption is that continued pretraining on English, Indonesian, and Chinese transfers to Malay and Singlish, since no Malay or Singlish training corpus is described and Singlish is not evaluated anywhere in the paper.

Editorial extensions

If this is right

The released checkpoint gives an 8B open model with stronger Chinese and Indonesian knowledge and general-knowledge performance than Meta-Llama-3.1-8B-Instruct.
Continued pretraining plus weight merging is demonstrated as a resource-efficient alternative to full retraining or large-scale instruction tuning for multilingual adaptation.
Balanced multilingual pretraining on English, Indonesian, and Chinese does not sacrifice English ability on Cross-MMLU, suggesting the recipe can be applied without an English regression.
The reported gains on IndoMMLU and CN-Eval show the approach transfers to domain-specific national knowledge benchmarks, not just translated general-knowledge tests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the training data described in the paper contains no Malay or Singlish corpus, the title's promise for those languages rests on transfer from English, Indonesian, and Chinese; the right test is a held-out native benchmark in each language.
The same recipe could be applied to other underserved languages by swapping the target corpus and re-merging with the official instruct checkpoint, which would test whether the reported gains are specific to this language trio or a general property of the method.
The improvements in the paper's tables are clearest on knowledge-heavy benchmarks; a reader should not extrapolate them to reasoning-heavy benchmarks without checking the per-benchmark columns, since the Cross-LogiQA average is slightly below the baseline.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing to know: this is a useful checkpoint for Indonesian and Chinese, and the paper's headline claim about Malay and Singlish does not survive its own tables. The model, LLaMA-3-MERaLiON-8B-Instruct, is Llama-3.1-8B continued-pretrained on 125B tokens of English, Indonesian, and Chinese, then merged with instruct weights. The authors publish the weights and describe the data mix, which is more than many such reports do.

What actually works: the model beats Meta-Llama-3.1-8B-Instruct on IndoMMLU (0.576 vs 0.548) and CN-Eval (0.514 vs 0.457), and improves Cross-MMLU for English, Chinese, and Indonesian. That is a credible narrow result and a genuine resource for the SEA NLP community.

The problems start with the abstract. It claims the model exceeds "the capabilities of the official Llama-3 models" across these languages. Against the paper's own primary baseline, Meta-Llama-3.1-8B-Instruct, the model is worse on Malay Cross-MMLU (0.613 vs 0.647), worse on Chinese and Malay Cross-LogiQA (0.528 vs 0.585 and 0.489 vs 0.523), and worse on average Cross-LogiQA (0.526 vs 0.537). The only way the headline holds is if the intended baseline is the older Llama-3-8B-Instruct, which the paper never says.

The scope problem is just as serious. The title promises Malay and Singlish, but the training data contains no Malay or Singlish tokens—only English, Indonesian, and Chinese. Singlish is never evaluated anywhere. Malay is tested only on Cross-MMLU and Cross-LogiQA, both from SeaEval, which four of the present authors co-authored. That self-citation is not disqualifying, but the numbers should be read with that in mind.

There are also no error bars, significance tests, or evaluation protocol details (prompt, few-shot, temperature). The reported gains are plausible but not rigorously established.

For a reader working on Indonesian or Chinese LLM adaptation, this checkpoint is worth a look. For a reader who cares about Malay or Singlish, the paper has nothing to offer yet. If it goes to a venue, it needs major revisions: soften the abstract, evaluate or drop Malay and Singlish, and add evaluation details. It deserves review because the artifact is real and the Indonesian/Chinese results are likely useful. I would not accept as is.

Referee Report

5 major / 4 minor

Summary. This manuscript describes MERaLiON-TextLLM, a continued-pretraining and weight-merging recipe applied to Llama-3.1-8B-Base. The released LLaMA-3-MERaLiON-8B-Instruct model is trained on roughly 125B tokens distributed across English (38B), Indonesian (45B), and Chinese (42B), followed by instruction tuning on a ~3M-pair multilingual corpus and merging with Llama-3.1-8B-Instruct weights to preserve instruction following. The paper evaluates the model on Cross-MMLU, Cross-LogiQA, IndoMMLU, and CN-Eval, reporting per-language and aggregate scores against Meta-Llama-3.1-8B-Instruct and several other 7B-9B models, and releases the checkpoint on Hugging Face. The central claim, as stated in the abstract, is that the approach achieves performance improvements across benchmarks in Chinese, Indonesian, Malay, and Singlish, exceeding the official Llama-3 models; the body, however, supports a narrower claim for Indonesian and Chinese on some benchmarks and does not evaluate Singlish at all.

Significance. If the narrow claims survive scrutiny, the work is a useful empirical data point: continued pretraining on ~125B tokens followed by weight merging yields an open 8B model that improves over Llama-3.1-8B-Instruct on IndoMMLU (0.576 vs 0.548) and CN-Eval (0.514 vs 0.457), and on several Cross-MMLU and Cross-LogiQA rows for English, Chinese, and Indonesian. The released checkpoint and the per-language result tables are concrete assets. However, the headline four-language claim is not established by the reported evidence: Singlish appears nowhere in the evaluation, Malay is tested only on two benchmarks with mixed results, the baseline used in the text is outperformed on several rows, and the evaluation protocol is unspecified. The contribution can be credited only after the claims are re-scoped and the protocol is documented.

major comments (5)

[Abstract and Section 4.1, Tables 2-3] The abstract's core claim of "performance improvements across benchmarks in these languages, exceeding the capabilities of the official Llama-3 models" is contradicted by the paper's own stated baseline. Against Meta-Llama-3.1-8B-Instruct, which Section 3 and Section 4.1 explicitly identify as the baseline, Table 2 reports a Malay Cross-MMLU drop (0.613 vs 0.647) and Table 3 reports a lower average Cross-LogiQA (0.526 vs 0.537), with Chinese (0.528 vs 0.585) and Malay (0.489 vs 0.523) individually lower. The manuscript must either name Meta-Llama-3-8B-Instruct as the intended official baseline, in which case the claim mostly holds, or qualify the abstract and Section 4.1 to the languages and benchmarks on which the improvement is actually observed.
[Section 2 and Section 4] The title and abstract promise cross-lingual capability in Chinese, Indonesian, Malay, and Singlish, but Section 2 lists training tokens only for English (38B), Indonesian (45B), and Chinese (42B), and Section 4 contains no Singlish evaluation at all. Malay is evaluated only on Cross-MMLU and Cross-LogiQA, and even there the results are mixed relative to the stated baseline. The manuscript needs either to add Malay and Singlish data and evaluation, or to narrow the title, abstract, and conclusion to the languages actually trained and tested.
[Section 3, Table 1] The text states that the merged model "consistently outperformed both the baseline Llama-3.1-8B-Instruct and our standalone instruction-tuned variant across English, Chinese, and Indonesian test sets," but Table 1 shows a Chinese Cross-LogiQA score of 0.528 against the baseline's 0.585. This internal contradiction must be corrected, and the corresponding claim in Section 4.1 about Cross-LogiQA improvements should be restricted to Indonesian and English.
[Section 4, Tables 1-5] No evaluation protocol is reported: the manuscript does not specify the prompt template, few-shot count, decoding parameters, or answer-extraction method used for Cross-MMLU, Cross-LogiQA, IndoMMLU, or CN-Eval, and no variance, confidence interval, or significance test is provided. Several reported advantages are small (for example, English Cross-LogiQA is 0.591 vs 0.585, and Indonesian Cross-LogiQA is 0.494 vs 0.455), so the quantitative comparisons are not reproducible and may not be robust. A protocol subsection and uncertainty estimates are needed before the comparative claims can be accepted.
[Section 4, benchmark descriptions] The two headline benchmarks, Cross-MMLU and Cross-LogiQA, are taken from Wang et al. (2024), which is co-authored by four of the present authors (Xin Huang, Bin Wang, Zhengyuan Liu, and Ai Ti Aw). This does not by itself invalidate the measurements, but because the main multilingual claim rests on these two benchmarks, the overlap should be disclosed explicitly and at least one external multilingual benchmark should be added to guard against inadvertent alignment with the authors' own evaluation suite.

minor comments (4)

[Abstract and Section 3] The abstract says the model is built on Llama-3-8B-Base, while Section 3 says Llama-3.1-8B-Base; the model name also alternates between "MERaLiON-LLaMA-3.1-8B-Instruct" and "LLaMA-3-MERaLiON-8B-Instruct" and should be standardized throughout.
[Section 4, Cross-LogiQA bullet] The benchmark description says Cross-LogiQA provides parallel question sets in English, Chinese, and Indonesian, but Table 3 includes a Malay column; the description and the table should be reconciled.
[Tables 2-5] The table captions and column headers are inconsistent (for example, the header "Model Series Model" in Tables 2-5), and the captions should identify the source of each number and the exact evaluation setting.
[Title and Section 5] The paper refers to a "series" and to future directions, but only one model checkpoint is described; the manuscript should clarify whether this is a technical report announcing a first release and which additional variants are planned.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; the central claim is empirical and measured, with only a minor non-load-bearing self-citation of the authors' own SeaEval benchmarks.

full rationale

The paper's derivation chain is empirical rather than deductive: continued pretraining on 125B tokens of English, Indonesian, and Chinese, followed by weight merging with Llama-3.1-8B-Instruct, then evaluation on fixed benchmarks. No parameter is fitted to a benchmark and then renamed as a prediction; no equation defines a result in terms of itself. The headline benchmarks Cross-MMLU and Cross-LogiQA come from Wang et al. (2024), which shares four of the seven authors, but the benchmarks are externally published fixed test sets and the model's scores are measured, not constructed by the benchmark definition. This self-citation is not load-bearing for the method's validity. The abstract's claim of 'exceeding the capabilities of the official Llama-3 models' is contradicted by the paper's own Tables 2 and 3 when the baseline is Meta-Llama-3.1-8B-Instruct (Malay Cross-MMLU 0.613 vs 0.647; average Cross-LogiQA 0.526 vs 0.537), and Singlish is never evaluated despite being in the title; these are evidence-coverage and consistency problems, not circularity. Because the central claim is a measured outcome rather than a derived one, and the only circularity-adjacent issue is a minor self-citation, the score is 2.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on training data choices and a merging procedure that are not fully specified, and on cross-lingual transfer to Malay and Singlish that is assumed but not demonstrated. No mathematical axioms are invoked; the free parameters are data allocations and the unspecified merging coefficients.

free parameters (3)

Token allocation = 38B English / 45B Indonesian / 42B Chinese
Chosen by hand to balance language coverage; no search procedure or justification for these exact proportions is given in Section 2.
Instruction-tuning corpus size = ~3M pairs
Described in Section 3 as 'approximately 3M pairs' without source or curation details.
Merging weights / algorithm = unspecified
Section 3 says 'carefully merging the weight sets' but does not state the merging method (e.g., SLERP, task arithmetic) or coefficients. The result depends on these unspecified choices.

assumptions (3)

domain assumption Cross-MMLU and Cross-LogiQA benchmarks are valid measures of cross-lingual understanding
The paper uses these benchmarks to support its central claim, but they are from a paper by co-authors (Wang et al. 2024) and no independent validation of their reliability for Malay is given.
domain assumption Continued pretraining on English, Indonesian, and Chinese transfers to Malay and Singlish
The title and abstract claim Malay and Singlish coverage, but the training data described in Section 2 only includes English, Indonesian, and Chinese; the paper assumes cross-lingual transfer.
domain assumption Weight merging combines the capabilities of base and instruct models without catastrophic interference
Section 3 assumes that 'carefully merging the weight sets of Llama-3.1-8B-Base and Llama-3.1-8B-Instruct' yields improved instruction-following and domain knowledge; no proof or ablation is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlish." pith.science (2026). https://pith.science/paper/6U62F2PG

@misc{pith2026250108335,
  author       = {Pith},
  title        = {Pith review of: MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlish},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6U62F2PG}},
  note         = {Machine review of arXiv:2501.08335}
}

read the original abstract

Multilingual large language models (MLLMs) have shown impressive capabilities across a variety of languages. However, efficacy can differ greatly between different language families, especially for those with limited linguistic resources. This report presents MERaLiON-TextLLM, a series of open-source language models specifically tailored to improve understanding and generation in Chinese, Indonesian, Malay, and Singlish. The initial released model is built on Llama-3-8B-Base and refined through a meticulously crafted process of continued pre-training and weight merging. Our approach achieves performance improvements across benchmarks in these languages, exceeding the capabilities of the official Llama-3 models. We provide the model checkpoints as a resource to support further research and development in cross-lingual language understanding.

Figures

Figures reproduced from arXiv: 2501.08335 by the authors.

**Figure 1.** Distribution of Tokens These performance metrics were achieved through optimized sharding, checkpoint strategies, and the selection of optimal batch sizes, to ensure efficient and effective model training. We conducted multiple runs to achieve optimal performance and sampling strategies. The pre-training data was allocated across three primary languages: English, Indonesian, and Chinese. Specifically, 38 billion to… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It
cs.CL 2025-05 accept novelty 6.0 of 10

LLM safety research at ACL venues from 2020 to 2024 is predominantly English-only, and the language gap is growing over time.

Reference graph

Works this paper leans on

10 extracted references · 3 canonical work pages · cited by 1 Pith paper

[1]

Maxtext: A simple, performant, and scalable jax llm, 2024

AI-Hypercomputer. Maxtext: A simple, performant, and scalable jax llm, 2024. URL https://github.com/AI-Hypercomputer/maxtext. Accessed: 2024-12-16

work page 2024
[2]

Dubey, A

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024
[3]

Huang, Y

Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, Y. Fu, M. Sun, and J. He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models, 2023. URL https://arxiv.org/abs/2305.08322

arXiv 2023
[4]

F. Koto, N. Aisyah, H. Li, and T. Baldwin. Large language models only pass primary school exams in indonesia: A comprehensive test on indommlu, 2023. URL https://arxiv.org/abs/2310.04928

work page Pith review arXiv 2023
[5]

H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2024. URL https://arxiv.org/abs/2306.09212

arXiv 2024
[6]

Singapore

A. Singapore. Sea-lion (southeast asian languages in one network): A family of large language models for southeast asia. https://github.com/aisingapore/sealion, 2024

work page 2024
[7]

G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thakoor, J.-B. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Sev...

arXiv 2024
[8]

B. Wang, Z. Liu, X. Huang, F. Jiao, Y. Ding, A. Aw, and N. Chen. S ea E val for multilingual foundation models: From cross-lingual alignment to cultural reasoning. In K. Duh, H. Gomez, and S. Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol...

doi:10.18653/v1/2024.naacl-long 2024

Show all 10 references

[9]

A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. ...

2024 arXiv
[10]

Zhang, H

W. Zhang, H. P. Chan, Y. Zhao, M. Aljunied, J. Wang, C. Liu, Y. Deng, Z. Hu, W. Xu, Y. K. Chia, X. Li, and L. Bing. Seallms 3: Open foundation and chat multilingual large language models for southeast asian languages, 2024. URL https://arxiv.org/abs/2407.19672

2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Maxtext: A simple, performant, and scalable jax llm, 2024

AI-Hypercomputer. Maxtext: A simple, performant, and scalable jax llm, 2024. URL https://github.com/AI-Hypercomputer/maxtext. Accessed: 2024-12-16

work page 2024

[2] [2]

Dubey, A

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024

[3] [3]

Huang, Y

Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, Y. Fu, M. Sun, and J. He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models, 2023. URL https://arxiv.org/abs/2305.08322

arXiv 2023

[4] [4]

F. Koto, N. Aisyah, H. Li, and T. Baldwin. Large language models only pass primary school exams in indonesia: A comprehensive test on indommlu, 2023. URL https://arxiv.org/abs/2310.04928

work page Pith review arXiv 2023

[5] [5]

H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2024. URL https://arxiv.org/abs/2306.09212

arXiv 2024

[6] [6]

Singapore

A. Singapore. Sea-lion (southeast asian languages in one network): A family of large language models for southeast asia. https://github.com/aisingapore/sealion, 2024

work page 2024

[7] [7]

G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, P. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thakoor, J.-B. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Sev...

arXiv 2024

[8] [8]

B. Wang, Z. Liu, X. Huang, F. Jiao, Y. Ding, A. Aw, and N. Chen. S ea E val for multilingual foundation models: From cross-lingual alignment to cultural reasoning. In K. Duh, H. Gomez, and S. Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol...

doi:10.18653/v1/2024.naacl-long 2024