REVIEW 4 major objections 7 minor 22 references
Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training
T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A 750M-parameter proxy model selects translated Turkish training corpora that, after fine-tuning and linear merging of Llama-3-8B, yield the highest-scoring open-source Turkish models in the 7-8B parameter range.
desk verdict Useful empirical recipe for Turkish LLMs, but the headline accuracy claim is undercut by overlapping selection and evaluation benchmarks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the proxy-transfer rule: train a 750M-parameter Turkish model briefly on a candidate corpus and keep the corpus only if the proxy's few-shot accuracy rises. The transfer target is Llama3-8B, trained with 8-bit AdamW at learning rate 1e-6 over one epoch, and the final component is linear weight merging (interpolating fine-tuned and original weights) to combine instruction-following and corpus-specific strengths. The evaluation machinery is a set of translated few-shot benchmarks plus an Elo-style human-vote comparison with eight judges and 3000 votes.
What would settle it
Run the same selection procedure but fine-tune Llama3-8B on the datasets that hurt the proxy (AutoMath, Web1, Web2) and on the selected ones, then compare on ARC and the other benchmark sets. If the proxy-rejected corpora match or beat the proxy-selected ones at large scale, the transfer hypothesis is false. A cheaper check is to list small-model rankings and large-model rankings across all candidate corpora and measure the rank correlation.
Extended reading notes
Core claim
The central claim is that a dataset that improves a small Turkish model also improves a much larger one, and that this transfer can be exploited to build state-of-the-art open Turkish models. Concretely, the authors fine-tune turkish-gpt2-large on each candidate subset and keep only SKWO (Stanford, Khan Academy, WikiHow, OpenStax), Stories, and OpenOrca based on average accuracy over COPA, XStoryCloze, ARC, and HellaSwag. They then full-fine-tune Llama3-8B and Llama3-8B-Instruct on those selected corpora, track only ARC during training, and linearly merge each fine-tuned model with its original. The reported result is 49.15% accuracy on ARC for the instruction model, with average few-shot scores of 52.70 and a merged-base average of 53.09 across ARC, HellaSwag, GSM8K, MMLU, TruthfulQA, and Winogrande — the best open-source Turkish results the authors find for 7-8B models.
Load-bearing premise
The argument stands or falls on the assumption that what helps a 750M-parameter Turkish model also helps a 7-8B model, since the large model's corpus choices are made entirely by the small proxy.
Editorial extensions
If this is right
- A corpus can be vetted for a large model at roughly one percent of the parameter cost, and only the survivors need expensive large-scale training runs.
- Translated English corpora plus synthetic textbook-style data can measurably improve a low-resource language model, not just a high-resource one.
- Linear merging of a fine-tuned model with its original base can push accuracy above either model alone, giving a no-retraining performance gain.
- Human preference for the merged instruction model over existing Turkish baselines suggests that corpus selection affects not only benchmark accuracy but perceived answer quality.
- The same pipeline could be repeated for other languages with limited data, using the same proxy-selection and merge steps.
Reading between the lines
- The proxy-transfer rule is tested here on only one large model family; if small-model rankings are not stable across model families, the rule may need per-family recalibration.
- The large-model training runs tracked only ARC, so the selected corpora could be chosen to fit that single benchmark; a follow-up with held-out monitoring on HellaSwag, GSM8K, MMLU, TruthfulQA, and Winogrande during training would test whether the choice is robust.
- Machine-translation artifacts in the selected corpora may act either as data augmentation or as noise; measuring performance on natively written Turkish data versus translated data would separate these effects.
- The method is language-agnostic in principle, so applying it to another under-resourced language with a strong multilingual base model would show whether the gains generalize beyond Turkish.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a pipeline for improving Turkish large language models: (i) translate selected English corpora (Cosmopedia subsets and OpenOrca) into Turkish via Google Translate; (ii) select corpora by fine-tuning a 750M Turkish GPT-2 model and measuring average few-shot accuracy on COPA, XStoryCloze, ARC Easy, ARC, and HellaSwag, choosing SKWO, Stories, and OpenOrca; (iii) full-fine-tune Llama3-8B and Llama3-Instruct on the selected corpora while tracking only ARC; (iv) linearly merge the trained models with the base Llama3 models; and (v) evaluate on ARC, HellaSwag, GSM8K, MMLU, TruthfulQA, and Winogrande, plus a human ELO voting study. The paper claims that the resulting Base, Instruct, and Merged models are the highest-performing open-source Turkish models in the 7-8B parameter range.
Significance. If the central claim were supported, the paper would offer a useful low-resource recipe: small-model proxy selection on translated and synthetic corpora, followed by full fine-tuning and linear merging, could improve multilingual models for under-resourced languages. The paper clearly describes a concrete corpus-selection procedure, gives training hyperparameters, and includes a human ELO evaluation with permutation-based confidence intervals. However, as reported, the headline result is not independently established: the final few-shot evaluation reuses benchmarks that appear in the corpus-selection step and in the large-model training monitor, and no statistical significance testing accompanies the few-shot comparisons.
major comments (4)
- [§II and §V-B (Tables II and IV)] The central claim that the proposed models are the highest-performing open-source Turkish 7-8B models rests on Table IV, but two of the six benchmarks in Table IV (ARC and HellaSwag) are exactly the benchmarks used in Section II to select the training corpora (Table II), and ARC was the only metric tracked during Llama3-8B training in Section III. The corpora were therefore selected, and the large-model checkpoint was effectively chosen, on signals from ARC/HellaSwag that later appear in the headline evaluation. The baseline models were not given this selection advantage, so the comparison is not controlled. To support the claim, the authors should report results on a held-out set of Turkish benchmarks that were never used for corpus selection or training monitoring, or should otherwise control for selection effects.
- [§II (proxy-transfer hypothesis) and §III] The corpus selection relies on the hypothesis that dataset improvements measured on a 750M Turkish GPT-2 transfer to a 7-8B model. The only large-model evidence presented is the ARC accuracy tracked during training, so the paper does not establish transfer for SKWO, Stories, and OpenOrca on the other benchmarks in Table IV. Without a large-model ablation that trains on each selected corpus individually, or at least reports large-model scores on the full evaluation suite for models trained with and without each corpus, the selection step could be selecting corpora that are beneficial to the small model but neutral or harmful to the larger model. The conclusion that 'enhancements made in small-scale models are reflected in large-scale models' (Section VI) is therefore supported only for ARC.
- [§V-B, Table IV] The few-shot evaluation protocol is under-specified and the headline differences are not tested statistically. The paper does not state the number of shots, prompt templates, answer extraction method, or metric computation used for GSM8K, MMLU, TruthfulQA, and Winogrande; without these details the numbers in Table IV cannot be reproduced. Several comparisons are within one point (e.g., ARC 48.72 vs. 49.15; MMLU 50.91 vs. 51.99), and no confidence intervals or significance tests are given. The phrase 'significantly outperform' in Section V-B is not justified by the reported evidence.
- [§V-A, Table V] The human evaluation is described too loosely to support the strength of the claims. The V dataset's composition, source, and question count are not described; the paper reports 'eight judges' but Figure 2 labels ten raters (R1-R10); and no inter-annotator agreement is reported. The ELO confidence intervals are computed only by permuting matchup order (as stated in Section V-A), which does not reflect judge sampling or question sampling variability. The authors should report the V dataset, the number of questions, the judge instructions, and a more appropriate uncertainty estimate.
minor comments (7)
- [§III] The sentence 'As a result, our base model, with an accuracy of 48.72% on the instruction dataset, achieved 49.15% on the ARC dataset' is confusing: it is unclear what 'the instruction dataset' refers to, and the wording should be corrected.
- [§V-B, Table IV] The column header 'Hella Swag' should be 'HellaSwag' for consistency with the text and references.
- [§II] There are several grammatical slips, e.g., 'Accuracy metric were used' and 'we evaluated the data set as follows'; these should be corrected.
- [§IV] The phrase 'the combined model can be more successful than the 2 models' is informal; it should be rephrased as 'more successful than either individual model'.
- [§V-A] The WinPct formula uses 'both' but this term is not defined in the text; please clarify whether it denotes votes where the judge marks both responses as equal.
- [§V-D] The correlation matrices in Figures 2-4 report coefficients without stating the correlation method (Pearson or Spearman), sample sizes, or significance levels; the axes of the matrices are also not labeled.
- [General] The paper does not state whether the trained model weights and translated corpora will be released; providing these resources would greatly aid reproducibility and community use.
Circularity Check
No circular derivation; the ARC/HellaSwag overlap is a validity caveat, not a reduction, and self-citations are not load-bearing.
full rationale
The paper is an empirical training study rather than a derivation chain, so there is no equation or fitted parameter that is renamed as a prediction. Corpus selection in Section II tests candidate translated corpora on turkish-gpt2-large using COPA, XStoryCloze, ARC Easy, ARC, and HellaSwag; Section III then transfers the selected corpora to Llama3-8B and says 'we only tracked the performance on the ARC dataset' during training. The final comparison in Section V.B reuses ARC and HellaSwag among its six benchmarks, so the reported gains on those two datasets are not fully independent evidence for the 'highest-performing' claim. However, this is benchmark selection leakage rather than construction-level circularity: the corpora were selected on a smaller model and then actually retrained on Llama3-8B, so the final scores are measurements rather than identities. Table IV also includes unselected benchmarks (GSM8K, MMLU, TruthfulQA, Winogrande) with mixed but partly positive results, and the human-judge comparison provides separate evidence. The self-citations ([8] for the small Turkish model and [19] for baseline selection) are used as empirical tools and are not invoked to forbid alternatives or to define the target result. Consequently, no load-bearing step reduces, by the paper's own equations or by self-citation, to its inputs; the main risk is external validity, not circularity.
Assumptions & free parameters
free parameters (3)
- learning_rate =
1e-6
- gradient_accumulation_steps =
512
- merge_coefficient =
not reported
assumptions (4)
- domain assumption Improvements on a 750M parameter model proxy transfer to a 7-8B parameter model.
- domain assumption Google Translate translation of English datasets preserves enough task-relevant meaning for training and evaluation in Turkish.
- domain assumption The three comparison models (SambaLingo, Trendyol, Turkcell) are the most successful Turkish models, as claimed in [19].
- domain assumption The adapted Turkish evaluation benchmarks retain the properties of their English originals.
Cite this review
Pith. "Pith review of Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training." pith.science (2026). https://pith.science/paper/72GNJHZQ
@misc{pith2026241202775,
author = {Pith},
title = {Pith review of: Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/72GNJHZQ}},
note = {Machine review of arXiv:2412.02775}
}
read the original abstract
In this study, we develop and assess new corpus selection and training methodologies to improve the effectiveness of Turkish language models. Specifically, we adapted Large Language Model generated datasets and translated English datasets into Turkish, integrating these resources into the training process. This approach led to substantial enhancements in model accuracy for both few-shot and zero-shot learning scenarios. Furthermore, the merging of these adapted models was found to markedly improve their performance. Human evaluative metrics, including task-specific performance assessments, further demonstrated that these adapted models possess a greater aptitude for comprehending the Turkish language and addressing logic-based queries. This research underscores the importance of refining corpus selection strategies to optimize the performance of multilingual models, particularly for under-resourced languages like Turkish.
Figures
Reference graph
Works this paper leans on
-
[1]
Multilingual large language model: A survey of resources, taxonomy and frontiers,
L. Qin, Q. Chen, Y . Zhou, Z. Chen, Y . Li, L. Liao, M. Li, W. Che, and P. S. Yu, “Multilingual large language model: A survey of resources, taxonomy and frontiers,” arXiv preprint arXiv:2404.04925 , 2024
arXiv 2024
-
[2]
Llama beyond english: An empirical study on language capability transfer,
J. Zhao, Z. Zhang, Q. Zhang, T. Gui, and X. Huang, “Llama beyond english: An empirical study on language capability transfer,” arXiv preprint arXiv:2401.01055, 2024
arXiv 2024
-
[3]
Hel- laswag: Can a machine really finish your sentence?,
R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hel- laswag: Can a machine really finish your sentence?,” arXiv preprint arXiv:1905.07830, 2019
arXiv 1905
-
[4]
Think you have solved question answering? try arc, the ai2 reasoning challenge,
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457 , 2018
arXiv 2018
-
[5]
Cosmopedia,
L. Ben Allal, A. Lozhkov, G. Penedo, T. Wolf, and L. von Werra, “Cosmopedia,” 2024
2024
-
[6]
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023
arXiv 2023
-
[7]
Orca: Progressive learning from complex explanation traces of gpt-4,
S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah, “Orca: Progressive learning from complex explanation traces of gpt-4,” 2023
2023
-
[8]
Introducing cosmosGPT: Monolingual Training for Turkish Language Models
H. T. Kesgin, M. K. Yuce, E. Dogan, M. E. Uzun, A. Uz, H. E. Seyrek, A. Zeer, and M. F. Amasyali, “Introducing cosmosgpt: Monolingual training for turkish language models,” arXiv preprint arXiv:2404.17336, 2024
work page Pith review arXiv 2024
Show all 22 references
-
[9]
Xcopa: A multilingual dataset for causal commonsense reasoning,
E. M. Ponti, G. Glava ˇs, O. Majewska, Q. Liu, I. Vuli´c, and A. Korhonen, “Xcopa: A multilingual dataset for causal commonsense reasoning,” arXiv preprint arXiv:2005.00333 , 2020
2005 arXiv
-
[10]
Few-shot learning with multilingual generative language models,
X. V . Lin, T. Mihaylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, et al., “Few-shot learning with multilingual generative language models,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp. 9019–...
2022
-
[11]
Llama 3 model card,
AI@Meta, “Llama 3 model card,” 2024
2024
-
[12]
Decoupled weight decay regularization,
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017
2017 arXiv
-
[13]
Arcee’s mergekit: A toolkit for merging large language models,
C. Goddard, S. Siriwardhana, M. Ehghaghi, L. Meyers, V . Karpukhin, B. Benedict, M. McQuade, and J. Solawetz, “Arcee’s mergekit: A toolkit for merging large language models,” arXiv preprint arXiv:2403.13257 , 2024
2024 arXiv
-
[14]
Chatbot arena: An open platform for evaluating llms by human preference,
W.-L. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, et al., “Chatbot arena: An open platform for evaluating llms by human preference,” arXiv preprint arXiv:2403.04132, 2024
2024 arXiv
-
[15]
Training verifiers to solve math word problems,
K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schul- man, “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021
-
[16]
Measuring massive multitask language understanding,
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” Proceedings of the International Conference on Learning Representa- tions (ICLR), 2021
2021
-
[17]
Truthfulqa: Measuring how models mimic human falsehoods,
S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” 2021
2021
-
[18]
Winogrande: An adversarial winograd schema challenge at scale,
“Winogrande: An adversarial winograd schema challenge at scale,” 2019
2019
-
[19]
T ¨urkc ¸e dil mod- ellerinin performans kars ¸ılas ¸tırması performance comparison of turkish language models,
E. Dogan, M. Egemen Uzun, A. Uz, H. E. Seyrek, A. Zeer, E. Sevi, H. Toprak Kesgin, M. K. Yuce, and M. F. Amasyali, “T ¨urkc ¸e dil mod- ellerinin performans kars ¸ılas ¸tırması performance comparison of turkish language models,” arXiv e-prints, pp. arXiv–2404, 2024
2024
-
[20]
Trendyol/trendyol-llm-7b-chat-v0.1
TrendyolGroup, “Trendyol/trendyol-llm-7b-chat-v0.1.” https: //huggingface.co/Trendyol/Trendyol-LLM-7b-chat-v0.1, 2024
2024
-
[21]
Turkcell/turkcell-llm-7b-v1
Turkcell, “Turkcell/turkcell-llm-7b-v1.” https://huggingface.co/ TURKCELL/Turkcell-LLM-7b-v1, 2024
2024
-
[22]
sambanovasystems/sambalingo-turkish-chat
SambanovaSystems, “sambanovasystems/sambalingo-turkish-chat.” https://huggingface.co/sambanovasystems/SambaLingo-Turkish-Chat, 2024
2024
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.