REVIEW 4 major objections 7 minor 16 references
Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History
T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read GPT-4o outperformed every other tested model on Lithuanian and general history questions in all nine languages, and Nordic-tuned models did not beat general multilingual models, against the paper's initial hypothesis.
desk verdict Useful dataset and a plausible model ranking, but the cross-lingual conclusions are not yet secure because translation quality and statistical power are weak. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the translated multiple-choice history exam: 550 Lithuanian questions drawn from the EXAMS dataset, machine-translated into nine languages, with Estonian and Latvian translated directly with DeepL and the other languages produced through an English pivot, plus a native-speaker quality check on 100 samples per language. Evaluation follows a fixed protocol: an English system prompt, four worked examples in the target language, the question with four lettered choices, and a parser that accepts a single capital letter as the answer. Comparing accuracy across the Nordic, Baltic, and multilingual groups is what turns the dataset into a test of whether cultural and historical alignment matters.
What would settle it
Retranslate the same 550 questions from Lithuanian directly into each of the nine languages with a human translator or an independent system, run the identical evaluation, and compare per-language accuracy; if the Baltic-language deficits and Nordic advantages shrink or vanish, the paper's cross-language conclusions are translation artifacts rather than model-knowledge effects.
Extended reading notes
Core claim
The paper's central claim is that on this translated exam, GPT-4o consistently outperformed all other models across every evaluated language and language group, with its best average accuracy on Baltic and Nordic versions of the Lithuanian-related questions and a slightly lower score on the multilingual group. Its second claim is negative: the Nordic fine-tuned models, GPT-SW3 and a Nordic-adapted LLaMa3 8b, performed consistently across their supported languages but failed to surpass the general multilingual models, which rejects the authors' initial hypothesis that cultural or historical alignment improves accuracy. The paper also reports that smaller open models show notable accuracy gaps on Lithuanian-related questions in Baltic languages, even when asked in Lithuanian.
Load-bearing premise
The cross-language comparisons stand or fall on the machine-translated versions preserving each question's meaning, difficulty, and correct answer; translation quality varied by language pair, with annotator agreement as low as 55% for Estonian.
Editorial extensions
If this is right
- The largest open-weight models, QWEN2.5 72b and LLaMa3.1 70b, form a clear second tier behind GPT-4o in every language group.
- For smaller open models, Lithuanian-related history questions are hardest when asked in Baltic languages, including Lithuanian itself.
- Nordic fine-tuned models perform consistently on Swedish and Danish but still trail general multilingual models, so region-specific pretraining did not add historical knowledge.
- The paper's closing recommendation is that closing these gaps will require targeted datasets and fine-tuning strategies for lower-resourced languages such as the Baltic group.
Reading between the lines
- Because GPT-4o produced most of the translated test set, part of its measured advantage may come from having generated the very questions on which it is later scored; an independently translated version would separate model knowledge from data-generation familiarity.
- The translation pipeline is inconsistent across languages, with Estonian and Latvian translated directly from Lithuanian and other languages routed through English, and annotator agreement is as low as 55% for Estonian, so some cross-language accuracy gaps may be translation artifacts rather than knowledge differences.
- A parallel benchmark on history exams from other Baltic and Nordic countries, for example Estonian, Latvian, or Finnish history, would show whether the pattern is specific to Lithuanian content or generalizes across the language groups.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper constructs a multilingual multiple-choice QA benchmark by translating 550 Lithuanian history exam questions from the EXAMS dataset into nine languages, grouped into Baltic (Lithuanian, Estonian, Latvian), Nordic (Danish, Finnish, Swedish) and multilingual (English, Ukrainian, Arabic) language groups. It evaluates a range of open and closed models, including GPT-4o, LLaMa 3/3.1/3.2, QWEN2.5, Mistral Nemo, and Nordic fine-tuned models (GPT-SW3 variants and NRD LLaMa3). The central claims are that GPT-4o consistently outperforms all other models across languages; larger open models (QWEN2.5 72b, LLaMa3.1 70b) perform well but show weaker Baltic alignment; smaller models show notable gaps on Lithuania-related questions in Baltic languages; and Nordic fine-tuned models do not surpass general multilingual models, which the authors interpret as evidence that shared cultural or historical context alone does not guarantee better performance.
Significance. If the findings are robust, the paper would be a useful contribution to multilingual and low-resource language evaluation, particularly for history-domain knowledge and for Baltic/Nordic language groups, which are underrepresented in LLM benchmarks. The authors provide a new translated dataset, a manual quality-validation effort with native speakers, and a broad model comparison including region-specific fine-tuned models. The central ranking (GPT-4o best, larger open models second) is plausible and consistent with prior work. However, the cross-lingual claims currently rest on translation comparability and single-run evaluations that are not sufficiently demonstrated, and one of the headline conclusions conflates model comparison with the language-alignment hypothesis. These issues are fixable within the manuscript's scope, so the work merits revision rather than rejection.
major comments (4)
- [§3, Appendix A, Table 3] The central cross-lingual comparison presupposes that the translated questions preserve meaning, difficulty, and answer-letter alignment equally across languages, but the evidence provided does not establish this. The translation pipeline is inconsistent: Danish, Finnish, Swedish, Ukrainian, and Arabic were translated via English (Lithuanian→English→target, with GPT-4o and DeepL), while Estonian and Latvian were translated directly from Lithuanian with DeepL. The manual validation shows widely varying inter-annotator agreement, with only 55% intersection for Estonian and 65% for Arabic, and no adjudication or third-annotator resolution is reported. Since the dataset has only 550 questions and each language group aggregates three languages, a systematic translation problem in one language could shift a group average enough to change the reported ordering (e.g., Baltic vs. Nordic for smaller models). The authors should use a single translation pipeline for all languages, or report and adjudicate per-language agreement and show that translation quality is comparable across groups, or explicitly restrict claims to differences that are robust to translation error.
- [§4, Tables 1–2 and Figures 3–5] All accuracy numbers come from a single generation per model-language pair (with seed set to 2 for open models and no sampling control for GPT-4o), and no confidence intervals or significance tests are reported. Several central differences are small: for GPT-4o, the LT-related average is 0.88 (Baltic), 0.87 (Nordic), and 0.84 (multilingual), and for QWEN2.5 72b the Baltic/Nordic gap on LT-related questions is 0.05. Given the per-language sample size of 550 and the stochastic nature of decoding, these differences are likely within sampling noise. The authors should report multiple runs with variance, or at least binomial confidence intervals or a paired significance test across languages, and temper claims about 'better performance' for language groups that do not survive such an analysis.
- [§3 (Data Preparation)] GPT-4o was used to translate the test questions into most languages, and the resulting test set is then used to evaluate GPT-4o itself. This is a confound rather than a proof of superior historical knowledge: if the machine translations contain systematic model-specific phrasing, or if answer choices are altered in a way that favors models trained on similar text, the higher GPT-4o scores could be an artifact of test construction. The manual validation addresses meaning preservation but does not check for model-specific bias. I recommend adding a control condition, such as evaluating all models on a human-translated subset or on the original Lithuanian questions with English prompts, to demonstrate that the reported ranking is not driven by the translation engine.
- [§4 (Nordic fine-tuned models, Table 2) and §5] The conclusion that 'shared cultural or historical context alone does not guarantee better performance' is not supported by the Nordic fine-tuned model comparison. The hypothesis stated in §3 is about the effect of prompt language on the same model, whereas the comparison of GPT-SW3 and NRD LLaMa3 against general multilingual models confounds model architecture, scale, and training data with cultural alignment. The GPT-SW3 models are much smaller (up to 6.7b) than the multilingual comparison models (up to 70b), and NRD LLaMa3 is a fine-tuned LLaMa3 8b. To test the alignment hypothesis, the same model should be evaluated on the same questions in different languages, which is done for the other models but not for the Nordic-specific models. The rejection of the initial hypothesis should be rephrased or supported by a matched comparison.
minor comments (7)
- [Table 2] The text states that NRD LLaMa3 was fine-tuned for Swedish, Norwegian, and Danish, but Table 2 reports only Swedish and Danish; please clarify whether Norwegian was evaluated and, if not, why it was omitted.
- [§1 and §3] Estonian is grouped under 'Baltic' for the geographic language-group analysis, but Estonian is not a Baltic language in the linguistic sense; a brief explanation of the geographic grouping would avoid confusion for readers.
- [Appendix A, Table 3] The agreement metric 'Intersect, %' is ambiguous: it is not stated whether this is raw percentage agreement on the 20 shared items, Cohen's kappa, or something else, and the denominator is not given; please specify the metric and the number of shared items.
- [§3 (Models benchmarking)] For GPT-4o, the paper does not specify the sampling temperature or other decoding parameters; since GPT-4o is evaluated only once, reporting these settings is important for reproducibility.
- [§4 (Results)] The sentence 'Mistral Nemo 12b scored the smallest scores comparing to other, even smaller (7-8b, 3b) models' is strongly worded; please verify that this holds for all language groups and question types, as Table 1 shows several cases where Mistral Nemo 12b is comparable to or better than LLaMa3.2 3b and LLaMa3 8b.
- [Throughout] Minor language issues: 'did not performed at par' (§4), 'from- and to-English' (§3), and inconsistent capitalization of model names (LLaMa, Llama, LLaMa3) should be corrected.
- [Appendix A] The GitHub repository is hidden for review and no dataset or code is released; for reproducibility, please provide a public link to the translated dataset, the annotation guidelines, and the evaluation scripts upon publication.
Circularity Check
No circularity: the evaluation is a direct empirical benchmark against an external dataset, with no fitted parameter or derivation whose output is fixed by an input.
full rationale
This paper reports an empirical evaluation: a fixed 550-question Lithuanian history subset of the EXAMS benchmark was machine-translated into nine languages and presented to ten model families; accuracy is measured directly from model outputs. There is no mathematical derivation, no parameter fitted to the evaluation data, and no 'prediction' that is constructed from the outcome it claims to explain. The use of GPT-4o (and DeepL) to produce translations is a potential confound for cross-linguistic comparability, since the translated instruments are not independently certified and annotator agreement is low for some languages (e.g., 55% for Estonian in Table 3), but this is a question of measurement validity, not circularity: even a biased translation does not make GPT-4o's ranking equal to the translation procedure by construction. Likewise, the result that Nordic fine-tuned models do not surpass multilingual models is an observed comparison against an external benchmark, not an artifact of the paper's definitions. No load-bearing self-citation or imported uniqueness claim is present. The central empirical claim is self-contained: it depends on the external EXAMS dataset and the recorded model answers, so circularity score is 0.
Assumptions & free parameters
assumptions (6)
- domain assumption The EXAMS Lithuanian history subset is a valid measure of Lithuanian history knowledge.
- domain assumption Machine-translated questions preserve semantics and answer ordering.
- domain assumption Estonian is treated as a Baltic language for grouping.
- ad hoc to paper A single generation run per model-language pair is representative.
- domain assumption The answer parsing rule (capital letters A-D) reflects model choices.
- domain assumption The manual translation check on 100 samples justifies the full dataset.
Cite this review
Pith. "Pith review of Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History." pith.science (2026). https://pith.science/paper/OLD7LJQ7
@misc{pith2026250109154,
author = {Pith},
title = {Pith review of: Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian History},
year = {2026},
howpublished = {\url{https://pith.science/paper/OLD7LJQ7}},
note = {Machine review of arXiv:2501.09154}
}
read the original abstract
In this work, we evaluated Lithuanian and general history knowledge of multilingual Large Language Models (LLMs) on a multiple-choice question-answering task. The models were tested on a dataset of Lithuanian national and general history questions translated into Baltic, Nordic, and other languages (English, Ukrainian, Arabic) to assess the knowledge sharing from culturally and historically connected groups. We evaluated GPT-4o, LLaMa3.1 8b and 70b, QWEN2.5 7b and 72b, Mistral Nemo 12b, LLaMa3 8b, Mistral 7b, LLaMa3.2 3b, and Nordic fine-tuned models (GPT-SW3 and LLaMa3 8b). Our results show that GPT-4o consistently outperformed all other models across language groups, with slightly better results for Baltic and Nordic languages. Larger open-source models like QWEN2.5 72b and LLaMa3.1 70b performed well but showed weaker alignment with Baltic languages. Smaller models (Mistral Nemo 12b, LLaMa3.2 3b, QWEN 7B, LLaMa3.1 8B, and LLaMa3 8b) demonstrated gaps with LT-related alignment with Baltic languages while performing better on Nordic and other languages. The Nordic fine-tuned models did not surpass multilingual models, indicating that shared cultural or historical context alone does not guarantee better performance.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[4]
arXiv preprint arXiv:2302.09210
How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210. Maxim Ifergan, Leshem Choshen, Roee Aharoni, Idan Szpektor, and Omri Abend
-
[5]
arXiv preprint arXiv:2408.10646
Beneath the sur- face of consistency: Exploring cross-lingual knowl- edge representation sharing in llms. arXiv preprint arXiv:2408.10646. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave
-
[6]
arXiv preprint arXiv:2301.08745
Is chatgpt a good translator? yes with gpt-4 as the en- gine. arXiv preprint arXiv:2301.08745. Tom Kocmi, Eleftherios Avramidis, Rachel Baw- den, Ond ˇrej Bojar, Anton Dvorkovich, Chris- tian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Had- dow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Mur...
-
[7]
In Proceed- ings of the Eighth Conference on Machine Transla- tion, pages 1–42, Singapore
Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In Proceed- ings of the Eighth Conference on Machine Transla- tion, pages 1–42, Singapore. Association for Com- putational Linguistics. Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen
work page 2023
-
[8]
Okapi: Instruction-tuned large lan- guage models in multiple languages with reinforce- ment learning from human feedback. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pages 318–327, Singapore. Association for Computational Linguistics. Li Linlin
work page 2023
-
[9]
arXiv preprint arXiv:2310.10378
Cross-lingual consistency of factual knowledge in multilingual language models. arXiv preprint arXiv:2310.10378. Rajesh Ranjan, Shailja Gupta, and Surya Narayan Singh
-
[10]
arXiv preprint arXiv:2410.08928
Towards multilingual llm evaluation for european languages. arXiv preprint arXiv:2410.08928. Jiaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, Jiarong Xu, and Fandong Meng
-
[11]
arXiv preprint arXiv:2309.08952
Cross- lingual knowledge editing in large language models. arXiv preprint arXiv:2309.08952. Jingjing Wang
Show all 16 references
-
[12]
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations , pages 38–45, Online
Transformers: State-of-the-art natural language pro- cessing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations , pages 38–45, Online. Association for Computational Linguistics. An Yang, Baosong Yang, Binyuan Hui...
2020
-
[13]
arXiv preprint arXiv:2407.10671
Qwen2 technical report. arXiv preprint arXiv:2407.10671. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge
-
[15]
arXiv preprint arXiv:2403.05189
Tracing the roots of facts in multilingual lan- guage models: Independent, shared, and transferred knowledge. arXiv preprint arXiv:2403.05189. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang
-
[16]
ArXiv, abs/2304.04675
Multilingual machine translation with large language models: Empirical results and analy- sis. ArXiv, abs/2304.04675. A Manual Translation Quality Evaluation The annotation guidelines and examples can be found in our GitHub repository
-
[2019]
PAWS-X: A cross-lingual ad- versarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) , pages 3687– 3692,...
2019
-
[2020]
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , EMNLP ’20, pages 5427–5444, Online
EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , EMNLP ’20, pages 5427–5444, Online. Association for Computational Linguist...
2020
-
[2023]
arXiv preprint arXiv:2305.12987
Gpt-sw3: An autoregressive language model for the nordic languages. arXiv preprint arXiv:2305.12987. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Marc’Aurelio Ranzato, Francisco Guzm ´an, and Angela Fan
-
[2024]
CoRR, abs/2406.18676
Un- derstand what LLM needs: Dual preference align- ment for retrieval-augmented generation. CoRR, abs/2406.18676. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Let- man, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goya...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.