{"id":"4116ec51-2f8e-4f71-aa3f-5ee6494cd939","arxiv_id":"2506.21595","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A cost-effective recipe consisting of tokenizer extension, continual pretraining, FP8 training, and SFT/DPO post-training yields Korean-English bilingual 8B models with top Korean benchmark scores.","lead":"The authors adapted an existing English LLM (Llama 3.1 8B) to Korean using about 100 billion tokens of mostly web-crawled text, a tokenizer extension, and selective FP8 training, producing two bilingual models. The paper reports top Korean benchmark scores among 8B models and shares the full recipe, code, and six new Korean benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Korean 'best' claim rests on two train-leaked benchmarks; excluding KoBEST-HellaSwag and KMMLU flips Thunder-LLM-Ins behind EXAONE by ~0.7 points.","rationale":"The reader's weakest_assumption is the load-bearing one. The abstract and Section 6.2 explicitly claim best-in-class Korean performance, and Table 5 is the only evidence. Two of the nine Korean benchmarks in that table (KoBEST-HellaSwag, KMMLU) have their train splits included in the post-training data (Table 10), and these two produce the largest positive deltas over the strongest competitor, EXAONE. The paper itself notes in §6.2 that performance improves more on benchmarks whose train sets were used. Recomputing the average without those two columns flips the ranking: EXAONE leads 67.2 to 66.4. Thus the headline claim depends on distribution overfitting. This is not a claim of misconduct; it is an evaluation-design flaw that the authors partially acknowledge and could fix by reporting uncontaminated averages. I also flag Appendix D.2.3/D.2.6's use of test-split few-shot examples as a separate validity issue, but that affects all models roughly equally and is not the deciding factor. I checked other candidate concerns—the FP8 'no accuracy loss' claim lacks a direct BF16-vs-FP8 accuracy table, and tokenizer details are under review—but neither is as directly load-bearing as the benchmark contamination, because even if true they would not rescue the 'best Korean' claim. Since the reader already conditions acceptance on addressing this, I recommend UNCHANGED.","tokens_in":32994,"tokens_out":8387,"duration_ms":85640,"concrete_test":"From Table 5, recompute the average Korean score for each model excluding KoBEST-HellaSwag and KMMLU (the two Korean benchmarks whose train splits are used in post-training per Table 10). If the corrected average leaves Thunder-LLM-Ins behind EXAONE (predicted: 66.4 vs 67.2), the 'best Korean performance' claim is unsupported; if Thunder-LLM-Ins still leads, the contamination concern is quantitatively minor. Additionally, re-run Ko-ARC and Ko-GSM8K with few-shot demonstrations drawn from a designated training split rather than the test split to confirm the reported numbers are not inflated by test-set leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—Thunder-LLM-Ins is best-in-class for Korean—is supported by a 1.1-point average margin over EXAONE-3.5-7.8B-Instruct in Table 5. The margin is entirely driven by two benchmarks whose training splits were used in post-training: KoBEST-HellaSwag (+12.4 over EXAONE) and KMMLU (+2.5), per Appendix B3 and Table 10. §6.2 concedes that post-training yields larger gains on benchmarks whose training sets were included. Recomputing Table 5 excluding just these two columns gives Thunder-LLM-Ins 66.4 vs EXAONE 67.2 on the remaining seven Korean benchmarks; Thunder is no longer first. The non-leaked Ko-IFEval and KR-HumanEval deltas are strongly negative (−16.4 and −4.3), so the 'best Korean' headline is a distribution-overfitting artifact rather than demonstrated general Korean competence. A secondary methodological flaw compounds this: Ko-ARC and Ko-GSM8K draw few-shot demonstrations from the test split itself (Appendix D.2.3, D.2.6), an invalid evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes an end-to-end pipeline for adapting Llama-3.1-8B to Korean: crawling about 3 TB of Korean web text, applying rule-based/deduplication/KenLM filtering, extending the Llama tokenizer with 72K Korean tokens via a Unigram-based method, continual pre-training on 102B tokens with a 1:1 Korean-to-English ratio, and SFT+DPO post-training on benchmark training sets plus synthetic data. The authors also introduce six new Korean benchmarks, analyze which Transformer components can be trained in FP8, and report that the resulting Thunder-LLM-Ins achieves the best average Korean score among 8B-scale models (65.0 vs. 63.9 for EXAONE-3.5-7.8B-Instruct), comparable English performance, a 1.43x training speedup from FP8, and an 18% inference speedup from the tokenizer extension.","tokens_in":33181,"tokens_out":5513,"duration_ms":60643,"significance":"If the performance and efficiency claims held, this would be a valuable case study for low-resource language adaptation: it is unusually complete in documenting data collection, preprocessing, training, and evaluation, and it contributes new Korean benchmarks plus a simple layer-wise FP8 stability analysis. The concrete cost accounting (48 H100s, 3,150 hours for continual pre-training) and the commitment to release code are strengths. However, the headline 'best Korean performance' claim is not currently supported because the evaluation is partly circular, two Korean benchmarks draw few-shot examples from the test split, the FP8 accuracy claim is not demonstrated, and the key margin rests on single-run results with no variance estimates. The underlying pipeline is salvageable, but the evidence as presented does not establish the central claim.","major_comments":[{"comment":"The evaluation of the central claim is partly circular. Table 10 shows that the post-training dataset includes the training splits of KoBEST-HellaSwag, KMMLU, HellaSwag, WinoGrande, OBQA, MMLU, GSM8K, ARC-Easy, and ARC-Challenge, and Section 6.2 concedes that gains are largest on benchmarks whose training sets were included. In Table 5, the 1.1-point average margin of Thunder-LLM-Ins (65.0) over EXAONE-3.5-7.8B-Instruct (63.9) depends on KoBEST-HellaSwag (+12.4) and KMMLU (+2.5). Excluding those two columns, the average over the remaining seven Korean benchmarks is 66.4 for Thunder-LLM-Ins versus 67.2 for EXAONE, so the 'best Korean performance' claim is not supported by the current comparison. The paper should report held-out benchmarks separately and compare against baselines that also post-train on the same benchmark training splits, or remove the overlapping benchmarks from the headline average.","section":"Section 3.3 / Table 10 / Section 6.2 / Table 5"},{"comment":"The evaluation protocols for Ko-ARC-Easy/Challenge and Ko-GSM8K explicitly sample few-shot demonstrations from the test split. This is invalid regardless of any training-set overlap, because the model is given answer-annotated instances from the evaluation set when computing the score. These two benchmarks contribute to the averages in Table 5, so the reported numbers are inflated. The authors should re-evaluate with demonstrations sampled from a train or dev split, or use zero-shot evaluation.","section":"Appendix D.2.3 and D.2.6"},{"comment":"The claim that FP8 training provides a 1.43x speedup 'without sacrificing model accuracy' is not supported by the evidence shown. Table 7 reports only tokens per second; no accuracy comparison between BF16 and FP8 training is presented anywhere in the paper. The stability test in Table 4 shows that a 360M model's loss converges for certain FP8 component configurations, but that is not the same as demonstrating accuracy equivalence for the final 8B model. The authors should report final benchmark scores (or at least training loss curves) for BF16 and FP8 runs under the same configuration.","section":"Section 6.3 / Table 7 / Section 4.4"},{"comment":"Appendix C3 states that all model training was performed once due to computational constraints. Given that the headline Korean advantage over EXAONE-3.5-7.8B-Instruct is an average margin of about 1.1 points, single-run results without error bars or multiple evaluation seeds are not sufficient to establish superiority. At minimum, the paper should report variance over multiple few-shot sampling seeds and evaluation runs, and should soften any claim that the model is 'best' if the margin is within noise.","section":"Appendix C3 / Table 5"}],"minor_comments":[{"comment":"The heading 'Evalution' should be 'Evaluation'.","section":"Section 6 heading"},{"comment":"The word 'desribed' is a typo for 'described'.","section":"Table 8 caption"},{"comment":"The abstract and conclusion claim 'superior Korean performance compared to state-of-the-art models', while the Limitations section states 'We do not aim to build a state-of-the-art LLM.' This tension should be resolved by either revising the headline claim or clarifying the intended contribution.","section":"Abstract / Section 7 / Limitations"},{"comment":"Please specify exactly which components (Attention, Linear, LM Head) were run in FP8 for the final Thunder-LLM training runs whose speed is reported in Table 7, so that the 1.43x speedup is reproducible.","section":"Section 4.4 / Table 7"},{"comment":"The paper states that the evaluation pipeline 'most of which originated from lm-evaluation-harness' was modified, but the exact modifications (beyond import paths) are not described. Listing the modifications would improve reproducibility.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful and unusually complete pipeline description, but the headline performance claim is not adequately supported because of benchmark overlap and test-set few-shot sampling. I would ask the authors to re-analyze on held-out benchmarks, fix the evaluation protocols, and either provide a BF16-vs-FP8 accuracy comparison or remove the no-accuracy-drop claim. The contribution is salvageable, so I am not recommending rejection, but the revision needs to address the load-bearing evaluation issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful engineering report, but the central \"best Korean model\" claim does not survive contact with its own evaluation setup. The recipe itself—crawling/filtering, tokenizer extension with branching entropy, selective FP8, SFT/DPO—is the real contribution, and the six Korean benchmarks, especially the redesigned Ko-LAMBADA, are a service to the community. The authors are unusually transparent about what they did: Appendix B3 lists which training splits went into post-training, and Appendix C3 admits single-run results. Credit where due: the non-leaked benchmarks (Ko-IFEval, KR-HumanEval, Ko-LAMBADA, Ko-WinoGrande) show real gains over Llama-3.1-8B, and the English regression is mild. The FP8 stability selection is simple and believable; the 1.43x speedup claim is specific and testable, though the \"no accuracy drop\" claim needs the actual comparison table, which is missing.\n\nSoft spots, in order of importance. First, the headline margin over EXAONE-3.5-7.8B-Instruct in Table 5 is driven by KoBEST-HellaSwag (+12.4) and KMMLU (+2.5), both of whose training splits were used for SFT/DPO. Exclude those two columns and Thunder-LLM-Ins lands around 66.4 versus EXAONE's 67.2 on the remaining seven Korean benchmarks. That does not disprove the recipe, but it does mean the abstract's \"superior Korean performance\" should be softened to \"competitive.\" Second, the few-shot evaluations for Ko-ARC and Ko-GSM8K (Appendix D.2.3 and D.2.6) sample demonstrations from the test split. That is an invalid protocol regardless of which model it helps; it needs to be redone with held-out dev/train splits. Third, there are no error bars anywhere, and the tokenizer method is under review and uncited, so the 72k-token claim is not independently checkable yet. Fourth, code and models are promised but not released; the datasets themselves cannot be shared for legal reasons, which is fair but limits reproducibility.\n\nNone of this is fatal to the paper's practical value. The recipe section is concrete enough to replicate, and the benchmark translations appear carefully done with cross-review. The paper deserves a serious peer review, but the authors should be asked to recompute the comparison on uncontaminated benchmarks, fix the few-shot protocol, and provide the FP8 accuracy comparison before acceptance.","headline":"Useful low-budget Korean adaptation recipe and six new benchmarks, but the headline 'best Korean performance' claim rests on leaked training splits and a flawed few-shot evaluation protocol.","tokens_in":33810,"tokens_out":2594,"would_cite":true,"duration_ms":28970,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An English-trained Llama 3.1 8B can be adapted to Korean with ~100B tokens and beat comparable 8B models in Korean benchmarks, with a 1.4x FP8 training speedup at no accuracy cost","keywords":["Korean LLM adaptation","continual pre-training","tokenizer extension","FP8 precision","low-resource language modeling","Korean benchmarks","bilingual LLM","post-training"],"falsifier":"Take a newly written Korean evaluation set — for instance a reading-comprehension and reasoning test built after Thunder-LLM-Ins was released, covering domains absent from its post-training data (fresh news, legal or medical text, or new common-sense questions) — and run Thunder-LLM-Ins, EXAONE-3.5-7.8B-Instruct, and Qwen2.5-7B-Instruct through it with identical prompting. If Thunder-LLM-Ins no longer leads that held-out Korean benchmark, the paper's central performance claim is not supported.","tokens_in":32736,"feed_emoji":"🇰🇷","tokens_out":11214,"duration_ms":106685,"temperature":0.7,"pith_summary":"This paper claims that a capable Korean-English bilingual LLM can be built from an English model on a low budget. The authors start from Llama 3.1 8B, add roughly 72,000 Korean tokens to its tokenizer, continually pre-train on about 102B tokens (half Korean, half English), and then apply SFT and DPO with a few million post-training examples. Their resulting models, Thunder-LLM and Thunder-LLM-Ins, post the highest average Korean benchmark score among the compared 8B-scale models while staying within a few points of the best English scores, and they do this with far less data than the multi-trillion-token training of the comparison models. They also report a 1.43x end-to-end training speedup from applying FP8 precision only to the layers that were verified to remain stable, with no measured accuracy loss. The paper's aim is a reproducible recipe that other researchers can copy for Korean or adapt to other low-resource languages.","feed_headline":"Korean 8B model tops similar-scale rivals on far less data","feed_subtitle":"Continual pre-training on 102B tokens plus tokenizer extension and FP8 speedups yields best-in-class Korean scores.","key_machinery":"The load-bearing mechanism is the combination of a tokenizer extension and a data-efficient training schedule. The tokenizer extension builds a Korean-optimized Unigram vocabulary of 72,000 tokens, appends them to Llama's original 128,000 tokens, and initializes each new embedding as the average of the sub-token embeddings from the base tokenizer, preserving English behavior while roughly halving the token count of Korean text. Continual pre-training on a 1:1 Korean/English mixture of about 102B tokens transfers Korean competence without destroying English skills, and post-training with SFT and DPO on a few million examples — benchmark training sets plus synthetic responses generated by larger models — supplies instruction-following and reasoning. The FP8 scheme is the speed mechanism: a stability screen over layer types shows attention matrix multiplications cannot tolerate FP8, while linear layers and the LM head can, and keeping FP8 to the stable layers yields a 1.43x training speedup with no measured accuracy change.","core_discovery":"The central discovery is that each stage of the pipeline contributes a measurable gain and together they close most of the Korean gap at 8B scale. Extending the Llama 3.1 tokenizer with 72,000 Korean Unigram tokens (embeddings initialized by averaging the base tokenizer's sub-token embeddings) nearly halves the number of tokens required for Korean text and cuts Korean inference time by 18%. Continual pre-training on 102B tokens of roughly balanced Korean and English text raises the average Korean benchmark score from 48.7 to 52.7, while leaving English scores largely intact. Selective FP8 training — FP8 for linear layers and the LM head but not for attention, which the authors found unstable — achieves a 1.43x end-to-end training speedup with no accuracy drop. Post-training with SFT and DPO on benchmark training sets plus synthetic data lifts the average Korean score to 65.0 and the English average to 74.7, putting Thunder-LLM-Ins at the top of the Korean suite and second on the English suite among the compared 8B models. The authors emphasize that they spent about 3,150 H100 hours on continual pre-training plus roughly 61 H100 hours on post-training, versus the several-trillion-token budgets behind the compared models.","pith_inferences":["The paper's own Section 6.2 observation — that gains are largest on benchmarks whose training sets were used in post-training — implies the true general Korean advantage is probably smaller than the headline 65.0 average; a held-out Korean test set would be needed to measure it.","The recipe's portability to other low-resource languages is plausible but untested; languages with very different scripts or little token overlap with English would not get the same benefit from the embedding-averaging initialization, since the base tokenizer's sub-tokens would be less informative.","A cheap ablation the authors do not run — varying the English share in continual pre-training while holding total tokens fixed — would reveal how much of the admitted English regression is caused by the 1:1 Korean/English mix, which directly informs the transfer of this recipe."],"forward_implications":["Korean adaptation of an 8B English model is feasible with about 102B tokens of mixed continual pre-training plus a few million post-training examples, i.e., roughly 3,200 H100 hours rather than a from-scratch multi-trillion-token budget.","The extended Unigram tokenizer roughly halves Korean token counts, giving an 18% Korean inference speedup, with the cost of a small English slowdown that shrinks at larger model sizes.","Applying FP8 only to layers proven stable (linear layers and the LM head, not attention) gives a 1.43x end-to-end training speedup with no measured accuracy loss, a cheaper route than architecture-level FP8 stabilization.","The six new Korean benchmarks — five localized translations plus Ko-LAMBADA, redesigned because Korean sentences end in verbs — give future Korean LLM work a common evaluation yardstick.","Post-training on benchmark training sets and synthetic data lifts Korean average scores by roughly 12 points (from 52.7 to 65.0) and English by roughly 9 points (from 65.4 to 74.7), while English scores remain second among the compared models."],"supporting_citations":[{"why":"Supplies the Llama 3.1 8B base model and the model family that the whole adaptation starts from.","marker":"Grattafiori et al., 2024"},{"why":"Provides the WECHSEL-style sub-token embedding averaging used to initialize the 72K new Korean token embeddings.","marker":"Minixhofer et al., 2021"},{"why":"Establishes the continual-training approach for bilingual adaptation from a monolingual base, the strategy the paper follows for Korean.","marker":"Gosal et al., 2024"},{"why":"Defines the supervised fine-tuning step used for the first phase of post-training.","marker":"Ouyang et al., 2022"},{"why":"Supplies the SimPO preference-optimization method used in the paper's DPO stage.","marker":"Meng et al., 2024"},{"why":"Prior FP8 LLM training approach that the paper's selective-FP8 stability screening contrasts with.","marker":"Peng et al., 2023"},{"why":"Provides the Transformer Engine FP8 kernels that deliver the measured 1.43x training speedup.","marker":"NVIDIA, 2025"},{"why":"GPU-based document deduplication framework used to clean the crawled Korean corpus.","marker":"Son et al., 2025b"},{"why":"KoBEST is the source of KoBEST-HellaSwag, a Korean benchmark used both for post-training data and evaluation.","marker":"Jang et al., 2022"},{"why":"KMMLU supplies the largest Korean post-training data source and the KMMLU evaluation benchmark.","marker":"Son et al., 2024"}],"fun_headline_variants":["8B Korean LLM tops similar-scale rivals with minimal data and compute","Thunder-LLM: Low-budget Korean adaptation at 8B scale","Token extension and FP8 yield faster, cheaper Korean LLM training","From English to Korean: Thunder-LLM's minimal-resource recipe"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that Thunder-LLM-Ins is the best Korean model of its scale depends on treating test splits of benchmarks whose training splits were used in post-training as a fair measure, and the paper itself reports the largest gains on exactly those benchmarks (Section 6.2), so the advantage may reflect overfitting to benchmark distributions rather than general Korean ability.","fun_headline_variants_meta":{"raw":{"variants":["8B Korean LLM tops similar-scale rivals with minimal data and compute","Thunder-LLM: Low-budget Korean adaptation at 8B scale","Token extension and FP8 yield faster, cheaper Korean LLM training","From English to Korean: Thunder-LLM's minimal-resource recipe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00112,"raw_usage":{"total_tokens":4684,"prompt_tokens":989,"completion_tokens":3695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":3615}},"tokens_in":605,"tokens_out":3695,"duration_ms":32367,"temperature":1.0,"reasoning_tokens":3615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:51:44.515701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a newly written Korean evaluation set — for instance a reading-comprehension and reasoning test built after Thunder-LLM-Ins was released, covering domains absent from its post-training data (fresh news, legal or medical text, or new common-sense questions) — and run Thunder-LLM-Ins, EXAONE-3.5-7.8B-Instruct, and Qwen2.5-7B-Instruct through it with identical prompting. If Thunder-LLM-Ins no longer leads that held-out Korean benchmark, the paper's central performance claim is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SimPO preference-optimization method used in the paper's DPO stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Transformer Engine FP8 kernels that deliver the measured 1.43x training speedup."}],"review_version":1}