{"id":"266aebfb-8c1b-4c99-a754-c63422516184","arxiv_id":"2412.02775","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning Llama3-8B on a small-model-selected mix of translated and synthetic Turkish corpora improves few-shot benchmark scores and human preference ratings.","lead":"The authors translated English AI training data into Turkish, used a small model to pick the most helpful datasets, then fine-tuned much larger Turkish LLMs, reporting accuracy gains and better human-vote scores. The work shows a practical recipe for improving LLMs in under-resourced languages by combining translated and synthetic corpora with model merging.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Final evaluation reuses the same ARC/HellaSwag benchmarks used to select corpora and to monitor 8B training; the 'highest-performing' claim therefore lacks independent evidence.","rationale":"Agreement with the reader is partial: the reader's selected weakest assumption is the proxy-transfer hypothesis, which is real and undefended, but the most load-bearing threat to the headline 'best model' claim is the overlap between the datasets used for selection and early stopping and the datasets used for final comparison. This overlap is directly visible in Sections II, III, and V.B, and it does not require the proxy-transfer step to fail; even a perfect small-to-large transfer would leave the evaluation contaminated. The concern is actionable rather than fatal: the directional gains may survive a clean held-out evaluation. I therefore keep the reader's CONDITIONAL verdict unchanged, with the explicit added condition that an independent Turkish benchmark suite be reported.","tokens_in":7982,"tokens_out":7445,"duration_ms":80872,"concrete_test":"Pre-register a held-out evaluation protocol: remove ARC and HellaSwag from all selection and monitoring decisions; run the Section II small-model selection using only COPA, XStoryCloze, and ARC Easy (with a disjoint validation split of ARC/HellaSwag), train the 8B model, and select checkpoints on that validation split. Then evaluate all models once on the untouched official ARC and HellaSwag tests plus one never-used Turkish benchmark such as Turkish XCOPA or XNLI. If the model trained under this protocol no longer beats Llama3 and the baselines, the original 'best' claim is a test-set-selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the trained models are the highest-performing open-source Turkish 7-8B models is supported mainly by Table IV, yet that table includes ARC and HellaSwag. These exact benchmarks were used earlier as selection criteria in Section II (Table II includes COPA, XStoryCloze, ARC Easy, ARC, HellaSwag) and as the only tracked metric during Llama3-8B training in Section III ('we only tracked the performance on the ARC dataset'). Thus the corpora were chosen, and the final checkpoints were effectively tuned, on datasets that later appear in the headline evaluation. The three baselines in Table IV were not given the same test-set-dependent selection or early stopping, so the comparison is not controlled. This does not make the results false, but it means the current evidence does not establish a general Turkish-language advantage; the reported gain could be selection on the evaluation set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for improving Turkish large language models: (i) translate selected English corpora (Cosmopedia subsets and OpenOrca) into Turkish via Google Translate; (ii) select corpora by fine-tuning a 750M Turkish GPT-2 model and measuring average few-shot accuracy on COPA, XStoryCloze, ARC Easy, ARC, and HellaSwag, choosing SKWO, Stories, and OpenOrca; (iii) full-fine-tune Llama3-8B and Llama3-Instruct on the selected corpora while tracking only ARC; (iv) linearly merge the trained models with the base Llama3 models; and (v) evaluate on ARC, HellaSwag, GSM8K, MMLU, TruthfulQA, and Winogrande, plus a human ELO voting study. The paper claims that the resulting Base, Instruct, and Merged models are the highest-performing open-source Turkish models in the 7-8B parameter range.","tokens_in":8153,"tokens_out":6076,"duration_ms":59231,"significance":"If the central claim were supported, the paper would offer a useful low-resource recipe: small-model proxy selection on translated and synthetic corpora, followed by full fine-tuning and linear merging, could improve multilingual models for under-resourced languages. The paper clearly describes a concrete corpus-selection procedure, gives training hyperparameters, and includes a human ELO evaluation with permutation-based confidence intervals. However, as reported, the headline result is not independently established: the final few-shot evaluation reuses benchmarks that appear in the corpus-selection step and in the large-model training monitor, and no statistical significance testing accompanies the few-shot comparisons.","major_comments":[{"comment":"The central claim that the proposed models are the highest-performing open-source Turkish 7-8B models rests on Table IV, but two of the six benchmarks in Table IV (ARC and HellaSwag) are exactly the benchmarks used in Section II to select the training corpora (Table II), and ARC was the only metric tracked during Llama3-8B training in Section III. The corpora were therefore selected, and the large-model checkpoint was effectively chosen, on signals from ARC/HellaSwag that later appear in the headline evaluation. The baseline models were not given this selection advantage, so the comparison is not controlled. To support the claim, the authors should report results on a held-out set of Turkish benchmarks that were never used for corpus selection or training monitoring, or should otherwise control for selection effects.","section":"§II and §V-B (Tables II and IV)"},{"comment":"The corpus selection relies on the hypothesis that dataset improvements measured on a 750M Turkish GPT-2 transfer to a 7-8B model. The only large-model evidence presented is the ARC accuracy tracked during training, so the paper does not establish transfer for SKWO, Stories, and OpenOrca on the other benchmarks in Table IV. Without a large-model ablation that trains on each selected corpus individually, or at least reports large-model scores on the full evaluation suite for models trained with and without each corpus, the selection step could be selecting corpora that are beneficial to the small model but neutral or harmful to the larger model. The conclusion that 'enhancements made in small-scale models are reflected in large-scale models' (Section VI) is therefore supported only for ARC.","section":"§II (proxy-transfer hypothesis) and §III"},{"comment":"The few-shot evaluation protocol is under-specified and the headline differences are not tested statistically. The paper does not state the number of shots, prompt templates, answer extraction method, or metric computation used for GSM8K, MMLU, TruthfulQA, and Winogrande; without these details the numbers in Table IV cannot be reproduced. Several comparisons are within one point (e.g., ARC 48.72 vs. 49.15; MMLU 50.91 vs. 51.99), and no confidence intervals or significance tests are given. The phrase 'significantly outperform' in Section V-B is not justified by the reported evidence.","section":"§V-B, Table IV"},{"comment":"The human evaluation is described too loosely to support the strength of the claims. The V dataset's composition, source, and question count are not described; the paper reports 'eight judges' but Figure 2 labels ten raters (R1-R10); and no inter-annotator agreement is reported. The ELO confidence intervals are computed only by permuting matchup order (as stated in Section V-A), which does not reflect judge sampling or question sampling variability. The authors should report the V dataset, the number of questions, the judge instructions, and a more appropriate uncertainty estimate.","section":"§V-A, Table V"}],"minor_comments":[{"comment":"The sentence 'As a result, our base model, with an accuracy of 48.72% on the instruction dataset, achieved 49.15% on the ARC dataset' is confusing: it is unclear what 'the instruction dataset' refers to, and the wording should be corrected.","section":"§III"},{"comment":"The column header 'Hella Swag' should be 'HellaSwag' for consistency with the text and references.","section":"§V-B, Table IV"},{"comment":"There are several grammatical slips, e.g., 'Accuracy metric were used' and 'we evaluated the data set as follows'; these should be corrected.","section":"§II"},{"comment":"The phrase 'the combined model can be more successful than the 2 models' is informal; it should be rephrased as 'more successful than either individual model'.","section":"§IV"},{"comment":"The WinPct formula uses 'both' but this term is not defined in the text; please clarify whether it denotes votes where the judge marks both responses as equal.","section":"§V-A"},{"comment":"The correlation matrices in Figures 2-4 report coefficients without stating the correlation method (Pearson or Spearman), sample sizes, or significance levels; the axes of the matrices are also not labeled.","section":"§V-D"},{"comment":"The paper does not state whether the trained model weights and translated corpora will be released; providing these resources would greatly aid reproducibility and community use.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a plausible engineering recipe, and the authors should be encouraged to continue. The decisive issue is the overlap between corpus selection, training monitoring, and final evaluation: ARC and HellaSwag appear in both the selection step and the headline Table IV, so the current evidence does not establish a general Turkish-language advantage. A well-designed revision with held-out benchmarks and full evaluation-protocol details could turn this into a solid contribution. I do not see the flaws as irreparable within the manuscript's scope, so major revision is more appropriate than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful empirical recipe for Turkish LLMs, not a breakthrough. The small-model proxy selection is a sensible cost-saving trick, and the translated Cosmopedia/OpenOrca subsets are a concrete reusable resource. But the 'highest-performing' claim rests partly on benchmarks that were also used to pick the training data, so treat the numbers as suggestive rather than conclusive.\n\nWhat's new: they translate Cosmopedia subsets and OpenOrca into Turkish, use turkish-gpt2-large (750M) to select which subsets help few-shot scores, then fine-tune Llama3-8B on those subsets and linearly merge weights with the base/instruct models. The merge step gives consistent gains on top of fine-tuning. The human evaluation is more careful than most: 3000 votes, 8 judges, ELO with confidence intervals, and correlation matrices. That part is independent of the automatic benchmarks and shows a real preference for their models.\n\nSoft spots: the automatic evaluation lacks error bars and protocol details (prompt format, answer extraction, number of runs). Several table differences are tiny and could be noise. The proxy-transfer hypothesis is stated but never tested directly; they only track ARC during the 8B training, so we do not actually know whether the other selected corpora transfer. The selection-on-evaluation concern is real but partial: ARC and HellaSwag appear in both the selection table and the final comparison, so those two columns are not independent evidence. The non-overlapping benchmarks (GSM8K, MMLU, TruthfulQA, Winogrande) mostly improve, which helps, but the baselines are only three Turkish models and no significance tests are reported. The claim of being 'highest performing' is therefore not established as written. Self-citation to their own prior work is appropriate here, since the cosmos models and the Turkish model survey are the natural baselines.\n\nWho this is for: people training LLMs for Turkish or other low-resource languages will find the recipe useful, especially the small-model proxy selection and the merge step. It is a workshop-level empirical paper with a clear methodology and honest limitations. It deserves a serious referee to tighten the evaluation and ask for code and data.\n\nRecommendation: send it to peer review. A competent referee can ask for confidence intervals, protocol details, and a held-out evaluation set. The core idea is plausible and the artifacts are useful.","headline":"Useful empirical recipe for Turkish LLMs, but the headline accuracy claim is undercut by overlapping selection and evaluation benchmarks.","tokens_in":8722,"tokens_out":2310,"would_cite":false,"duration_ms":24305,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 750M-parameter proxy model selects translated Turkish training corpora that, after fine-tuning and linear merging of Llama-3-8B, yield the highest-scoring open-source Turkish models in the 7-8B parameter range.","keywords":["Turkish language models","corpus selection","cross-lingual transfer","synthetic datasets","model merging","few-shot evaluation","low-resource NLP","Llama-3 fine-tuning"],"falsifier":"Run the same selection procedure but fine-tune Llama3-8B on the datasets that hurt the proxy (AutoMath, Web1, Web2) and on the selected ones, then compare on ARC and the other benchmark sets. If the proxy-rejected corpora match or beat the proxy-selected ones at large scale, the transfer hypothesis is false. A cheaper check is to list small-model rankings and large-model rankings across all candidate corpora and measure the rank correlation.","tokens_in":7807,"feed_emoji":"🇹🇷","tokens_out":4191,"duration_ms":40058,"temperature":0.7,"pith_summary":"This paper claims that a cheap, small-model-driven corpus selection pipeline can make a large multilingual model substantially better at Turkish. The authors rank candidate translated and synthetic corpora by how much they improve a 750M-parameter Turkish GPT-2, then fine-tune Llama3-8B on the winners for one epoch and linearly merge the result with the original model. They report that the resulting base and instruction models outperform existing open-source Turkish models in the 7-8B range on a set of few-shot benchmarks, and that human judges prefer the merged instruction model over the compared Turkish baselines. If correct, the result would give lower-resource languages a low-cost recipe: translate existing English corpora, select by small-model proxy, fine-tune, and merge.","feed_headline":"Proxy-chosen Turkish corpus lifts 8B model to top of open-source ranks","feed_subtitle":"A 750M-parameter model picks translated training data; fine-tuning and merging Llama-3-8B beats existing Turkish LLMs.","key_machinery":"The load-bearing mechanism is the proxy-transfer rule: train a 750M-parameter Turkish model briefly on a candidate corpus and keep the corpus only if the proxy's few-shot accuracy rises. The transfer target is Llama3-8B, trained with 8-bit AdamW at learning rate 1e-6 over one epoch, and the final component is linear weight merging (interpolating fine-tuned and original weights) to combine instruction-following and corpus-specific strengths. The evaluation machinery is a set of translated few-shot benchmarks plus an Elo-style human-vote comparison with eight judges and 3000 votes.","core_discovery":"The central claim is that a dataset that improves a small Turkish model also improves a much larger one, and that this transfer can be exploited to build state-of-the-art open Turkish models. Concretely, the authors fine-tune turkish-gpt2-large on each candidate subset and keep only SKWO (Stanford, Khan Academy, WikiHow, OpenStax), Stories, and OpenOrca based on average accuracy over COPA, XStoryCloze, ARC, and HellaSwag. They then full-fine-tune Llama3-8B and Llama3-8B-Instruct on those selected corpora, track only ARC during training, and linearly merge each fine-tuned model with its original. The reported result is 49.15% accuracy on ARC for the instruction model, with average few-shot scores of 52.70 and a merged-base average of 53.09 across ARC, HellaSwag, GSM8K, MMLU, TruthfulQA, and Winogrande — the best open-source Turkish results the authors find for 7-8B models.","pith_inferences":["The proxy-transfer rule is tested here on only one large model family; if small-model rankings are not stable across model families, the rule may need per-family recalibration.","The large-model training runs tracked only ARC, so the selected corpora could be chosen to fit that single benchmark; a follow-up with held-out monitoring on HellaSwag, GSM8K, MMLU, TruthfulQA, and Winogrande during training would test whether the choice is robust.","Machine-translation artifacts in the selected corpora may act either as data augmentation or as noise; measuring performance on natively written Turkish data versus translated data would separate these effects.","The method is language-agnostic in principle, so applying it to another under-resourced language with a strong multilingual base model would show whether the gains generalize beyond Turkish."],"forward_implications":["A corpus can be vetted for a large model at roughly one percent of the parameter cost, and only the survivors need expensive large-scale training runs.","Translated English corpora plus synthetic textbook-style data can measurably improve a low-resource language model, not just a high-resource one.","Linear merging of a fine-tuned model with its original base can push accuracy above either model alone, giving a no-retraining performance gain.","Human preference for the merged instruction model over existing Turkish baselines suggests that corpus selection affects not only benchmark accuracy but perceived answer quality.","The same pipeline could be repeated for other languages with limited data, using the same proxy-selection and merge steps."],"supporting_citations":[{"why":"Supplies the 750M-parameter Turkish proxy model on which all corpus-selection decisions are made.","marker":"[8]"},{"why":"Provides the Cosmopedia subset sources that are translated into Turkish and used as training candidates.","marker":"[5]"},{"why":"Provides the OpenOrca instruction-completion corpus that is translated and selected for large-model training.","marker":"[7]"},{"why":"Provides the Llama3-8B base and instruct models that are fine-tuned and merged.","marker":"[11]"},{"why":"Provides the ARC benchmark used to track large-model training progress and to report final accuracy.","marker":"[4]"},{"why":"Provides HellaSwag, a few-shot evaluation dataset used in corpus selection and final comparison.","marker":"[3]"},{"why":"Supplies the model-merging toolkit and linear merging method used to combine trained models with their bases.","marker":"[13]"},{"why":"Identifies the existing successful Turkish models that the paper compares against in final evaluation.","marker":"[19]"}],"fun_headline_variants":["Small model picks Turkish data; merged Llama-3-8B tops open ranks","Corpus selection via proxy lifts Turkish Llama-3-8B to SOTA","Turkish LLM recipe: 750M selector, 8B merger, top scores","Proxy-based corpus pick sets Turkish 8B open benchmark","Fine-tune, merge, repeat: Turkish Llama tops 7-8B open rank"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands or falls on the assumption that what helps a 750M-parameter Turkish model also helps a 7-8B model, since the large model's corpus choices are made entirely by the small proxy.","fun_headline_variants_meta":{"raw":{"variants":["Small model picks Turkish data; merged Llama-3-8B tops open ranks","Corpus selection via proxy lifts Turkish Llama-3-8B to SOTA","Turkish LLM recipe: 750M selector, 8B merger, top scores","Proxy-based corpus pick sets Turkish 8B open benchmark","Fine-tune, merge, repeat: Turkish Llama tops 7-8B open rank"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1269,"prompt_tokens":894,"completion_tokens":375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":268}},"tokens_in":510,"tokens_out":375,"duration_ms":4446,"temperature":1.0,"reasoning_tokens":268,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:06:22.851693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same selection procedure but fine-tune Llama3-8B on the datasets that hurt the proxy (AutoMath, Web1, Web2) and on the selected ones, then compare on ARC and the other benchmark sets. If the proxy-rejected corpora match or beat the proxy-selected ones at large scale, the transfer hypothesis is false. A cheaper check is to list small-model rankings and large-model rankings across all candidate corpora and measure the rank correlation.","supporting_citations":[{"cited_title":"Introducing cosmosGPT: Monolingual Training for Turkish Language Models","cited_arxiv_id":"2404.17336","evidence_quote":"Supplies the 750M-parameter Turkish proxy model on which all corpus-selection decisions are made."},{"cited_title":"T ¨urkc ¸e dil mod- ellerinin performans kars ¸ılas ¸tırması performance comparison of turkish language models,","cited_arxiv_id":null,"evidence_quote":"Identifies the existing successful Turkish models that the paper compares against in final evaluation."}],"review_version":1}