{"id":"27013be7-e1b8-4458-87ff-ad0707b9a2f1","arxiv_id":"2509.03972","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A 102B Korean-English model, expanded from Llama 3 70B with LlamaPro and Masked Structure Growth and trained on 194B tokens, scores 64.74 on KMMLU and 83.34 on KorMedMCQA, roughly matching GPT-4 on Korean benchmarks.","lead":"The paper describes Llama-3-Motif, a 102 billion parameter Korean-English model grown from the open Llama 3 70B checkpoint by adding layers and widening the network, then trained on 194 billion tokens with a 9 to 1 Korean to English mix. The authors report scores on Korean benchmarks that match or slightly beat GPT-4, though the model, data, and code are not released and the English-side claims are not evaluated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No decontamination analysis for the 194B Korean corpus vs KMMLU/KorMedMCQA; the sub-1-point GPT-4 margins could reflect training-data overlap rather than capability.","rationale":"The reader's weakest assumption—that the evaluation is clean and free of training-data overlap—is also the most load-bearing issue in my read. The Korean benchmark numbers are the entire quantitative evidence for the headline 'comparable to GPT-4' claim, and the paper itself describes a web-crawled specialized corpus whose sources overlap thematically and, plausibly, textually with public exam-based benchmarks. Without a decontamination check, the claimed margins are too small to distinguish genuine capability from memorization. I also note the absence of any English-retention evaluation despite the abstract's 'retaining strong performance in English' claim, but that affects only half of the claim and is not what the headline comparison rests on. Since the reader already assigned CONDITIONAL on essentially this basis, my stress-test does not shift the verdict; it reinforces that the condition—clean evaluation and release/audit of the corpus—must be met before the claim is accepted.","tokens_in":6298,"tokens_out":3061,"duration_ms":31480,"concrete_test":"Perform a contamination audit on the actual pretraining corpus (or a representative released sample): normalise questions/answers and compute 8-gram and 13-gram overlap between the corpus and all KMMLU and KorMedMCQA evaluation items. Flag any near-duplicate items, remove them, and recompute Llama-3-Motif scores on the non-overlapping subset, comparing against the same GPT-4 baselines measured under identical prompting. If the margins disappear or fall within sampling error, the headline is an artifact of training-data overlap. If the full corpus cannot be released, require a decontamination report with exact-match statistics and benchmark-release/split dates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on Tables 2 and 3 being clean capability comparisons. The paper reports a 194B-token web-crawled Korean/English corpus (Sec. 4.2) that 'included a substantial proportion of specialized documents, such as domestic academic papers, research reports, and patents' (Sec. 6.1), and explicitly credits this corpus for KMMLU gains. KMMLU questions are sourced from Korean standardized exams, and KorMedMCQA is built from public Korean licensing-exam questions (Secs. 6.1, 6.2). No decontamination step, substring/n-gram overlap check, or benchmark-exclusion list is reported anywhere in Sections 4-6. The reported advantages over GPT-4 are 0.63 points on KMMLU-direct (64.74 vs 64.11) and 0.28/0.73 points on KorMedMCQA average/Doctor (83.34 vs 83.06; 77.19 vs 76.49). Even a small fraction of memorized benchmark items could account for all of these margins. The baseline scores also come from heterogeneous sources (community reports, author measurements, different instruction-tuned variants), but the more fundamental issue is that no evidence rules out direct overlap between the training corpus and the evaluation sets. Therefore the 'comparable to GPT-4' claim is not currently supported as a capability comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Llama-3-Motif, a 102B-parameter dense Transformer built by expanding Llama-3-70B through depth (LlamaPro) and width (Masked Structure Growth) expansion, followed by continual pretraining on about 194B tokens with a 9:1 Korean:English ratio, SFT with NEFTune, and preference alignment with KTO. The authors report KMMLU-direct 5-shot score of 64.74 and KorMedMCQA scores of 77.19 (Doctor) / 83.34 (average), claiming the model outperforms existing Korean-specialized models and is comparable to GPT-4, while 'retaining strong performance in English.' The central evidence is Tables 2 and 3, which compare against a mix of community-reported and author-measured baselines. No English evaluation is reported, and no decontamination analysis is described for the training corpus versus the benchmark test sets.","tokens_in":6584,"tokens_out":3336,"duration_ms":33456,"significance":"If the empirical claims are reliable, the paper would demonstrate a practical recipe for producing a Korean-specialized 102B model from an existing 70B checkpoint using relatively few additional tokens, and it would provide useful engineering evidence for LlamaPro and MSG at scale. The reported 102B parameter count is consistent with the architecture in Table 1 (96 layers, dim 9,216, FFN 30,720, vocab 128k), and the use of external benchmarks (KMMLU, KorMedMCQA) is appropriate. The main contributions are not machine-checked proofs or released code/weights; the value is in the empirical scaling and data-curation result. However, the headline comparison to GPT-4 rests entirely on small margins (0.63 points on KMMLU-direct; 0.28/0.73 points on KorMedMCQA) that are vulnerable to training-data contamination and uncontrolled evaluation protocols. The absence of any English benchmark also leaves the 'retaining strong performance in English' claim unsupported.","major_comments":[{"comment":"No decontamination analysis is reported for the 194B-token web-crawled corpus against the KMMLU and KorMedMCQA test sets. Section 6.1 states that KMMLU questions come from Korean standardized exams and that the model's strong performance is 'largely attributed to the composition of our training dataset,' which included 'domestic academic papers, research reports, and patents.' KorMedMCQA is built from public Korean licensing-exam questions. Because the reported margins over GPT-4 are sub-point (0.63 KMMLU, 0.28/0.73 KorMedMCQA), even minor overlap/memorization could account for the advantage. The paper needs to report an n-gram/substring overlap analysis, exclude any overlapping items, and re-run the evaluation before 'comparable to GPT-4' can be accepted as a capability claim.","section":"§4.2-4.3, §6.1-6.2"},{"comment":"The central claim includes 'retaining strong performance in English,' but Section 6 contains only Korean benchmarks. No English evaluation (e.g., MMLU, HellaSwag, ARC, or any held-out English task) is reported. Without a comparison to the original Llama-3-70B or to other English-strong models, the claim of retained English capability is unsupported. Please add standard English benchmark results, ideally with the same evaluation protocol used for baselines.","section":"Abstract, §1, §6"},{"comment":"The baseline scores are not produced under a single comparable protocol: Table 2 mixes community reports (†), author measurements (‡), and different instruction-tuned/base variants; Table 3 uses a report from [9] for GPT-4-base and author measurements for other models. No decoding parameters, prompt templates, or variance/confidence intervals are given. The 0.63-point KMMLU gap and 0.28-point KorMedMCQA-average gap are small enough that protocol differences or sampling noise could reverse the comparison. A controlled re-evaluation of all models with identical 5-shot prompts and greedy decoding, or at least a sensitivity analysis, is needed to support the 'comparable to GPT-4' statement.","section":"Tables 2 and 3"},{"comment":"The paper states that four preliminary experiments on Qwen 1.8B were conducted and that LlamaPro was selected 'based on our preliminary results,' but no results, tables, or quantitative comparisons are provided. Since the model-scaling methodology is a listed contribution, the absence of these experimental data makes the choice of depth/width expansion strategy unverifiable. Please report at least a summary table of the four methods' performance and the criterion used for selection.","section":"§2"}],"minor_comments":[{"comment":"Typo: 'validaters' should be 'validators.'","section":"§5"},{"comment":"'Direct Policy Optimization (DPO)' should be 'Direct Preference Optimization (DPO).'","section":"§5"},{"comment":"The final KTO hyperparameters (batch size 128, LR 1e-6, NEFT alpha 0, KTO lambda 1.375/1) are stated, but the SFT/NEFTune setup lacks training length, LR, and dataset size; please add these for reproducibility.","section":"§5"},{"comment":"The 'Community report' reference [4] is a LinkedIn post; please replace with an archival or citable source, or describe the evaluation protocol used for those entries.","section":"Table 2"},{"comment":"The 9:1 Korean:English ratio is stated, but the final token counts per language (approximately 174.6B Korean and 19.4B English if the total is 194B) could be made explicit to clarify the balance.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like an industry technical report. The core capability claim is plausible, but the evidence as presented is too thin for the claimed parity with GPT-4: no decontamination, no English evaluation, and heterogeneous baselines. I would not reject outright because the missing analyses are within scope to add, but the authors need to either provide the decontamination and controlled-evaluation evidence or substantially soften the claims. Also note the reference list includes a LinkedIn post as evidence for baseline scores, which is not appropriate for a journal-level evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a credible engineering report with a plausibly new recipe—LlamaPro depth expansion plus MSG width growth to turn Llama 3 70B into a 102B model, then continual pretraining on 194B tokens at a 9:1 Korean:English ratio, then NEFTune and KTO. I haven't seen that exact combination for non-English expansion, and the architecture table is internally consistent (96 layers, dim 9,216, FFN 30,720 ≈ 102B). The parameter count checks out, and the authors are transparent about which baselines they measured themselves versus took from community reports.\n\nThe real problem is the headline comparison. The margins over GPT-4 are tiny: 0.63 points on KMMLU-direct, 0.28 points on KorMedMCQA average. The training corpus reportedly includes domestic academic papers, research reports, and patents, while KMMLU is built from Korean standardized exams and KorMedMCQA from public licensing exams. No decontamination step, n-gram overlap check, or exclusion list is reported anywhere. With margins that thin, even a small amount of memorized benchmark items would erase the claimed advantage. The stress-test is right: this is currently a training-data re-measurement risk, not a clean capability comparison.\n\nSecond, the English-retention half of the central claim is asserted but never evaluated. There is no English benchmark table, not even MMLU. If the model loses English ability, the 'retaining strong performance' claim is unsupported. Third, the abstract's numbers don't match the body: the 9-40% and 4.9-16.6 point ranges don't align with the table deltas, and 'balanced ratio' is actually 9:1. Fourth, despite the open-source framing, no model, code, or data is released, and the key ablations (LlamaPro vs other depth methods, KTO vs DPO/PPO) are described but not shown.\n\nThese are fixable problems. The scaling methodology itself is sound enough to deserve referee time, and the Korean benchmark numbers, if decontamination can be demonstrated, would be a meaningful result for multilingual model growth. I'd send this to a serious referee with instructions to focus on decontamination and English evaluation. If the authors can rule out overlap and show English retention, this becomes a solid contribution. Without that, treat the GPT-4 comparison as unverified.","headline":"A plausibly novel scaling recipe for Korean LLMs, but the GPT-4-comparable claim currently rests on un-decontaminated benchmark scores and no English evaluation.","tokens_in":7265,"tokens_out":2163,"would_cite":false,"duration_ms":20926,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Llama-3-Motif is a 102B model grown from Llama 3 70B by depth and width expansion and continually pre-trained on 194B tokens at a 9:1 Korean-to-English ratio; the paper reports it matches or exceeds GPT-4 on the KMMLU and KorMedMCQA Korean","keywords":["Llama-3-Motif","Korean language modeling","continual pretraining","depth expansion","width expansion","KMMLU","KorMedMCQA","KTO alignment"],"falsifier":"Take the 35,030 KMMLU questions and the KorMedMCQA exam items, run exact and fuzzy string matching against the 194B-token training corpus after the paper's deduplication pipeline; any nontrivial match count means the benchmark gap is partly memorization. Then re-administer both benchmarks on a newly released Korean exam that postdates the training data cutoff and compare Llama-3-Motif against GPT-4o and GPT-4-base under identical 5-shot conditions.","tokens_in":6132,"feed_emoji":"🇰🇷","tokens_out":6776,"duration_ms":61203,"temperature":0.7,"pith_summary":"Llama-3-Motif is a 102-billion-parameter model built by expanding Llama 3 70B to 96 layers and a hidden size of 9,216, then continually pre-trained on about 194 billion tokens with a 9:1 Korean-to-English ratio. The paper reports that this model scores 64.74 on KMMLU-direct (5-shot), above the authors' measurement of GPT-4o (64.11), and 77.19/83.34 on KorMedMCQA Doctor/average, above GPT-4-base (76.49/83.06), while claiming English performance is retained. If true, the work shows a practical recipe for raising a less-resourced language's capability from an existing open checkpoint with relatively modest additional training data, using depth/width expansion plus targeted post-training. The authors position it as a step toward closing the gap for under-resourced languages, with a medical consultation service as one application. The main caveat is the absence of reported decontamination between the web-crawled corpus and benchmark items.","feed_headline":"102B Korean model edges GPT-4o on KMMLU at 64.74","feed_subtitle":"Grown from Llama 3 70B and trained on 194B mostly-Korean tokens, it also edges GPT-4 on Korean medical questions.","key_machinery":"The load-bearing mechanism is structured parameter growth with initialization preservation. Depth expansion (LlamaPro) inserts layers initialized to copy existing blocks, so the model's behavior is initially unchanged; width expansion (Masked Structure Growth) adds hidden and FFN units whose outputs are masked at first and unmasked gradually each training step. This lets the enlarged model absorb new Korean-domain knowledge during 194B tokens of continual pretraining (9:1 Korean:English) while retaining the base model's English and general abilities. Post-training uses NEFTune (noisy embeddings during SFT) and KTO (unpaired preference optimization) to align the model at lower data-collection","core_discovery":"The central claim is that a 70B English-centric checkpoint can be grown into a 102B model that is competitive with GPT-4 on Korean understanding without abandoning the Transformer architecture or starting from scratch. Growth is done in two stages: LlamaPro adds 20% more layers, and Masked Structure Growth widens hidden and FFN dimensions, with new neurons initially masked and progressively unmasked during continual pretraining. All weights are then tuned on a Korean-heavy corpus of 194B tokens, followed by NEFTune-based instruction fine-tuning and KTO preference optimization. The paper reports the resulting model outperforms existing Korean-specialized open models by 9–40% on KMMLU and 4.9–","pith_inferences":["Editorial inference: no decontamination step is reported, so the headline comparison should be treated as provisional until the 194B-token corpus is checked for overlap with KMMLU and KorMedMCQA test items.","Editorial inference: the 9:1 Korean:English ratio is likely language-specific; other languages may need a different balance between new-language exposure and preservation of English general knowledge, and sweeping this ratio would be a natural extension.","Editorial inference: because the reported parity rests on two benchmarks and 5-shot prompting, the paper's more durable contribution may be the engineering recipe (structured growth plus continual pretraining plus low-cost alignment) rather than a broad claim about GPT-4-level Korean ability."],"forward_implications":["The expansion-plus-continual-pretraining recipe can be repeated from an existing open checkpoint, so other languages with scarce high-quality data could get a foundation model tuned to their domains without training from scratch.","A 194B-token, 9:1 Korean:English mix appears sufficient to lift KMMLU-direct from about 54.5 (reported for Llama-3-70B-Instruct) to 64.74.","The KorMedMCQA results suggest the model can serve as a base for Korean medical consultation services, the use case the authors state they are developing.","The 0.63-point edge over GPT-4o on KMMLU and the 0.28/0.70-point edges over GPT-4-base on KorMedMCQA are best read as parity claims, not dominance claims, given the mixed provenance of the baseline numbers."],"supporting_citations":[{"why":"Supplies the Llama 3 70B base model whose weights are expanded and whose architecture is inherited.","marker":"[5]"},{"why":"Supplies the depth-expansion method (LlamaPro) chosen after preliminary experiments, inserting layers initialized to copy existing blocks.","marker":"[15]"},{"why":"Supplies the width-expansion mechanism (Masked Structure Growth) that masks new neurons and progressively unmask them.","marker":"[16]"},{"why":"Defines the KMMLU benchmark, the primary Korean general-knowledge test set with its 35,030 questions.","marker":"[13]"},{"why":"Defines the KorMedMCQA benchmark and supplies the GPT-4-base baseline scores used in the medical comparison.","marker":"[9]"},{"why":"Supplies the NEFTune noisy-embedding method used during supervised fine-tuning.","marker":"[7]"},{"why":"Supplies the KTO unpaired preference-optimization method used for human alignment.","marker":"[6]"},{"why":"Supplies one of the alternative depth up-scaling approaches compared in the preliminary experiments.","marker":"[8]"}],"fun_headline_variants":["102B Korean LLM beats GPT-4 on KMMLU","From 70B to 102B: Korean model tops GPT-4","Korean-specialized 102B model surpasses GPT-4","Llama 3 grown to 102B excels in Korean, rivals GPT-4","KMMLU: 102B Korean model edges GPT-4"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The headline comparison to GPT-4 holds only if the 194B-token crawl that trained the model never overlapped the KMMLU or KorMedMCQA test questions and every baseline was measured under the same prompt-and-scoring protocol; the paper reports no decontamination check and mixes community reports with author-run numbers.","fun_headline_variants_meta":{"raw":{"variants":["102B Korean LLM beats GPT-4 on KMMLU","From 70B to 102B: Korean model tops GPT-4","Korean-specialized 102B model surpasses GPT-4","Llama 3 grown to 102B excels in Korean, rivals GPT-4","KMMLU: 102B Korean model edges GPT-4"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1274,"prompt_tokens":668,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":412,"completion_tokens_details":{"reasoning_tokens":509}},"tokens_in":412,"tokens_out":606,"duration_ms":5253,"temperature":1.0,"reasoning_tokens":509,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:30:38.882429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 35,030 KMMLU questions and the KorMedMCQA exam items, run exact and fuzzy string matching against the 194B-token training corpus after the paper's deduplication pipeline; any nontrivial match count means the benchmark gap is partly memorization. Then re-administer both benchmarks on a newly released Korean exam that postdates the training data cutoff and compare Llama-3-Motif against GPT-4o and GPT-4-base under identical 5-shot conditions.","supporting_citations":[{"cited_title":"KMMLU: Measuring Massive Multitask Language Understanding in Korean","cited_arxiv_id":"2402.11548","evidence_quote":"Defines the KMMLU benchmark, the primary Korean general-knowledge test set with its 35,030 questions."}],"review_version":1}