{"id":"c503543c-9eaf-4e21-aff0-e7e796b844a2","arxiv_id":"2502.09056","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Thai 70B model trained with an SFT-plus-DARE-merge recipe matches DeepSeek R1 on reasoning benchmarks while retaining most Thai language quality.","lead":"This paper shows that fine-tuning a Thai-language model on a small translated reasoning dataset and then merging it with DeepSeek R1's 70B model can give the Thai model near-R1 reasoning while keeping its Thai. The recipe is open and costs about $120 in GPU time, which could make advanced reasoning models practical for low-resource languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out evaluation: merge ratio and SFT mixture were selected on the exact benchmarks used to claim DeepSeek-R1 parity, so the reported parity may be in-sample overfitting.","rationale":"The reader's weakest assumption (unvalidated GPT-4o Thai translations) is a legitimate concern, but it is not the most load-bearing one. It threatens only the Thai subset of reasoning scores. A more fundamental problem is that the final model is the argmax of a small hyperparameter and data-mixture search evaluated on the same benchmarks that are later used to assert DeepSeek-R1 parity and language preservation. This selection-on-the-test-set issue undermines the reliability of all reported numbers, not just the Thai numbers, and it is not addressed by the Sea-Lion transfer, which fixes the recipe before evaluation. AIME's small sample size (30 problems per language) makes the reported single-run parity margins especially fragile. The paper is still promising: the recipe is transparent, the gains are large, and the Sea-Lion result provides some out-of-model evidence. But the quantitative parity claim needs a held-out benchmark evaluation and variance reporting before it can be fully accepted. Hence the reader's CONDITIONAL verdict remains appropriate, and no verdict change is needed.","tokens_in":17431,"tokens_out":10562,"duration_ms":97356,"concrete_test":"Fix the final recipe (SFT-v3 + M3) and evaluate Typhoon2-R1-70B and DeepSeek R1 70B Distill on a reasoning benchmark not used anywhere in the configuration search (e.g., AIME 2025, GPQA, or a new LiveCodeBench split), plus repeat the Thai AIME/MATH-500/LiveCodeBench runs across at least 3 sampling seeds and report mean±std. If the merged model no longer stays within the claimed ~4% language gap or the reasoning parity margin, the current numbers are an in-sample selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim rests on a configuration search whose objective is the same benchmark table used for the final comparison. In §4.2, merge ratios M1/M2/M3 are chosen by comparing end-to-end scores on AIME, MATH-500, LiveCodeBench, IFEval, MT-Bench, and Language Accuracy (Tables 3 and 5). In §4.3, the SFT data mixture (SFT-v1 through SFT-v4) is selected by optimizing the same benchmark averages (Table 7). Table 10 then reports the best-found configuration on those same benchmarks, with no held-out split, no repeated runs, and no variance estimates. AIME has only 30 problems per language, so a few answers shift percentages by several points. The Sea-Lion transfer (§4.7) is a helpful out-of-model check, but the recipe was fixed before applying it, and it uses the same benchmark suite; it does not validate the absolute 'within 4%' and 'comparable' margins for Typhoon. The Thai-translation concern flagged by the reader is real, but it affects only the Thai reasoning subset; the selection issue affects every number in the headline claim, including English reasoning and language preservation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a recipe for equipping a Thai-specialized 70B LLM (Typhoon2 70B Instruct) with reasoning abilities comparable to DeepSeek R1 70B Distill through a two-stage pipeline: (1) supervised fine-tuning of the language model on a bilingual reasoning dataset, and (2) layer-wise DARE-based merging between the SFT model and DeepSeek R1. The authors report that the resulting Typhoon2-R1-70B stays within about 4% of Typhoon2 on language tasks and matches DeepSeek R1 on reasoning benchmarks, at a claimed compute cost of $120. They also transfer the recipe to Sea-lion v3 70B as an external check and release data, merge configurations, and model weights.","tokens_in":17697,"tokens_out":10421,"duration_ms":85576,"significance":"If the results are correct, this is a highly practical, low-cost recipe for adding reasoning abilities to low-resource language models. The transfer to Sea-lion v3 70B, which was developed by a different organization with a different CPT/SFT recipe, provides a useful out-of-model validation. The open release of the data, merge configurations, and weights is a clear strength that supports reproducibility. However, the evidence is weakened by the use of the same benchmarks for configuration selection and final reporting, by unvalidated GPT-4o translations for the Thai reasoning benchmarks, and by the absence of repeated runs or variance estimates. These issues are addressable with additional experiments and clarifications.","major_comments":[{"comment":"The final model's performance is reported on the same benchmarks that were used to select the merge configuration and the SFT mixture. Section 4.2.2 selects M3 based on Tables 3 and 5, and Section 4.3 selects SFT-v3 based on Table 7; Table 10 then presents the best-found configuration on those identical benchmarks. Because AIME consists of only 30 problems per language, one response changes the score by about 3.3 percentage points, and the paper reports no repeated runs, confidence intervals, or error bars. The Sea-lion experiment (§4.7) is a helpful external check, but it uses the same benchmark suite and a recipe that was fixed beforehand, so it does not validate the absolute margins ('within 4%', 'comparable') claimed for Typhoon. I recommend reporting results on a held-out split of the benchmarks, or at minimum providing error bars and an explicit statement of the in-sample nature of the reported numbers.","section":"§4.2, §4.3, Table 10"},{"comment":"The Thai reasoning scores for AIME, MATH-500, and LiveCodeBench are obtained by translating the English queries with GPT-4o, with no validation of translation quality, difficulty preservation, or answerability. If the translations are flawed, the Thai reasoning numbers do not measure actual Thai-language reasoning ability, and the central claim that the merged model matches DeepSeek R1 on Thai reasoning is unsupported. I recommend adding a validation step, such as back-translation agreement, human evaluation, or a control experiment comparing a strong multilingual model on original versus translated problems, and reporting the results.","section":"§3.2"},{"comment":"The computational-budget claim is internally inconsistent. Section 3.1 states that each experiment is run on 4×H100 GPUs for up to 15 hours, while footnote 1 computes $120 as '14 H100 GPUs × 15 hours × $2 per hour'. At the stated rate, 4×H100×15h is $120, whereas 14×H100×15h is $420. Please clarify the actual hardware configuration and correct the cost calculation, since the $120 figure is a headline claim in the abstract and introduction.","section":"§3.1 vs footnote 1"},{"comment":"The averaging procedure used for the headline 'within 4%' and 'comparable' claims differs from the 'simple arithmetic mean' described in Section 3.2. Footnotes 12 and 13 first average the English and Thai scores within each benchmark and then average across benchmarks, while the Avg column in Table 10 is a simple mean over all cells (with MT-Bench multiplied by 10). This inconsistency makes the reported margins non-reproducible. For example, on English-only reasoning, the gap between Typhoon2-R1-70B and DeepSeek R1 is about 8 percentage points, whereas the two-step average makes the two models appear nearly equal. Please use a single, explicitly defined aggregation for all reported averages, or report both aggregations separately.","section":"§4.6, footnotes 12-13"}],"minor_comments":[{"comment":"The think-accuracy pseudocode uses the condition 'if len(think_content.strip()) >= 0:', which is always true, meaning empty thoughts would be counted as valid. This should be '> 0' to enforce the non-empty requirement, and the affected Think accuracy values should be recomputed if this was implemented as written.","section":"Appendix A.3, Listing 2"},{"comment":"The caption does not state that MT-Bench scores are multiplied by 10 before inclusion in the Avg column. Please add this detail to the caption, since the aggregation rule is essential for interpreting the reported averages.","section":"Table 10 caption"},{"comment":"The description 'translating the question and solution components into Thai while retaining the original English reasoning traces' is ambiguous. It is unclear whether the final answer (the solution) is translated into Thai or left in English, and whether 'solution components' includes the chain-of-thought. Please clarify the exact construction of the translated data.","section":"§2.2"},{"comment":"Figure 5's caption says 'Example demonstrate code-switching / language accuracy problem in DeepSeek R1 70B Distill', but the figure appears to show a successful Thai response from the proposed model. The caption should be corrected to describe the actual example shown.","section":"Figure 5 caption"}],"recommendation":"major_revision","confidential_remarks":"The authors are the developers of Typhoon2, the language-specific model used in the main experiments, so there is a natural conflict of interest in evaluating their own model. The Sea-lion transfer partially mitigates this, as that model originates from a different organization. The paper is written in a workshop-recipe style; for a full journal, the contribution would need stronger validation, but the open release and practical recipe are of value to the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful open recipe for injecting reasoning into a language-specific LLM, and it's reproducible. The Sealion transfer is a real plus. But the headline numbers are in-sample: the merge ratios and SFT mixtures were chosen by looking at the same benchmark table that later appears as the final comparison. Until there is a held-out evaluation or repeated runs, I'd treat the \"within 4%\" and \"comparable\" claims as optimistic.\n\nWhat's new: the bilingual SFT alignment step that pairs Thai questions with English reasoning traces, the layer-wise finding that later layers should favor the language model while earlier layers favor the reasoning model, and the open release of data, merge configs, and weights. The combination is not something I've seen published, and the cost is genuinely low if you count one run.\n\nWhere it's soft: the evaluation is the main issue. No held-out split, no error bars, single runs. AIME has 30 problems per language, so a couple of answers shift percentages several points. The Thai reasoning numbers rest on GPT-4o translations with no validation of difficulty preservation or answerability. The $120 figure undercounts the compute because it's the cost of one 15-hour run, but the paper reports many experiments. There's also a small bug in Think Accuracy: the pseudo-code has `if len(think_content.strip()) >= 0`, which is always true, so that metric only checks token presence, not non-empty thought. That one is minor and not load-bearing, but worth fixing if they revise.\n\nThe central recipe itself holds up well enough. The direction is plausible and the transfer to Sealion, a different CPT/SFT model, is the best evidence that the method generalizes. The limitation section is honest, and the artifacts are released.\n\nWho this is for: people building or serving language-specific LLMs in low-resource settings, and anyone working on model merging. It's not a deep technical breakthrough, but it's a practical, well-documented recipe.\n\nRecommendation: this deserves a serious referee. The right outcome is a moderate revision with a held-out or out-of-distribution evaluation, repeated runs, and validation of the Thai translations. I'd accept it with those changes; without them, the quantitative claims should be downgraded.","headline":"Useful open recipe for injecting reasoning into a language-specific LLM, but the headline numbers are partly fitted on the same benchmarks used for the final comparison.","tokens_in":18216,"tokens_out":2908,"would_cite":true,"duration_ms":26521,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a one-day, $120 pipeline—SFT on bilingual reasoning traces plus a layer-aware DARE merge—transplants DeepSeek R1-level reasoning into a Thai-specialized 70B model while keeping Thai quality within roughly 4%.","keywords":["model merging","reasoning distillation","low-resource languages","Thai","DARE","supervised fine-tuning","representation alignment","language-specific LLMs"],"falsifier":"Have native Thai speakers produce or approve a second translation of the AIME, MATH-500, and LiveCodeBench questions used in the paper, then re-run Typhoon2-R1-70B on those human-verified Thai prompts under the same sampling settings; if the model's Thai reasoning average falls clearly below DeepSeek R1 70B Distill's English average, the claim of R1-level reasoning in Thai is not supported.","tokens_in":17285,"feed_emoji":"🧠","tokens_out":13039,"duration_ms":113474,"temperature":0.7,"pith_summary":"The paper claims that a language-specific LLM can acquire DeepSeek R1-level long-thought reasoning in about one day and for roughly $120 of GPU time, without giving up the target language. The recipe has two stages: fine-tune the Thai-specialized model on a small bilingual reasoning set (Thai questions and solutions matched with English reasoning traces), then merge its weights with DeepSeek R1 70B Distill using a layer-aware sparse merge that lets the language model dominate the final output layers. The resulting Typhoon2-R1-70B stays within about 4% of Typhoon2 70B Instruct on language tasks, scores comparably to DeepSeek R1 on AIME, MATH-500, and LiveCodeBench, and raises the average across all tasks by 41.6% over the Thai model. The same recipe transfers to a second Llama-3.1-based Southeast Asian model. If the results hold, low-resource reasoning becomes a translation-and-merging problem rather than a large-scale distillation problem.","feed_headline":"One day and $120 turn a Thai LLM into an R1-grade reasoner","feed_subtitle":"SFT alignment plus a layer-aware DARE merge keeps Thai within 4% while matching R1 on math and code","key_machinery":"The machinery is a two-stage transfer pipeline. Stage one is representation alignment: a LoRA supervised fine-tuning of the language-specialized model on a bilingual dataset made from a public 17K reasoning-distillation set, a 6.5K Thai translation of it, and 500 DeepSeek-distilled Thai long-thought prompts; the purpose is to bring the language model's hidden states into the reasoning model's representation space. Stage two is ability-aware merging: a DARE merge, a sparse operation that drops a fraction of each model's weight deltas and rescales the survivors, with per-layer-block mixing weights. The tuned schedule gives DeepSeek 75% weight in layers 0-53 and a linear decay to 12.5% at layer 80, with Typhoon taking the complement; this layer schedule is the mechanism that keeps reasoning in early and middle layers while preserving Thai fluency in late layers.","core_discovery":"The central claim is that representation alignment plus ability-aware weight merging is sufficient to combine two specializations that neither SFT alone nor merging alone can combine. Starting from Typhoon2 70B Instruct and DeepSeek R1 70B Distill, both built on Llama 3.1 70B, the paper shows that a short LoRA SFT on translated reasoning traces makes the language model's parameter space compatible enough for a DARE merge to transfer reasoning behavior, and that the merge ratio must be layer-dependent: the early and middle layers should draw 75% of their weight from the reasoning model, decaying to 12.5% in the final layer so Thai output fluency survives. The final model lands within roughly 4% of Typhoon2 70B Instruct on average language-task performance and essentially at parity with DeepSeek R1 70B Distill on the three reasoning benchmarks (66.85 vs 66.31 average), with an all-task average of 76.5.","pith_inferences":["A full reproduction cost includes more than the $120 SFT GPU bill: GPT-4o translation calls and the DeepSeek R1 distillation runs that produced the Thai training traces are additional line items, so the true end-to-end cost of the recipe is higher than the headline number.","The layer-weighted result suggests a testable design rule for multilingual merging: late layers carry output-language identity and early-to-middle layers carry reasoning, so any language pair on a shared backbone should favor the language model in late layers; swapping the schedule on a non-Thai language would test this directly.","Because the Thai reasoning numbers come from machine-translated benchmarks, an adopting team should treat a human-verified translated benchmark suite as part of the recipe; without it, the Thai reasoning scores inherit translation risk."],"forward_implications":["Typhoon2-R1-70B comes within about 4% of Typhoon2 70B Instruct on average language-task performance and near parity with DeepSeek R1 70B Distill on AIME, MATH-500, and LiveCodeBench, while lifting the all-task average 41.6% over the Thai model and 12.8% over the reasoning model.","The two stages are jointly necessary: merging without the SFT alignment step drops the all-task average to 63.9, and SFT without merging drops it to 59.0, against 76.5 for the full recipe.","Giving Typhoon more weight in later layers is what protects the language: raising its late-layer share from 25% to 87.5% improves Thai response-language accuracy from 59.8 to 87.6 while reasoning scores stay roughly unchanged.","The same recipe applied to Sealion 70B Instruct raises its all-task average from 52.8 to 74.6 and brings its reasoning to DeepSeek R1 level, indicating the method transfers across different language-specialization recipes.","The best data mixture is the SFT-v3 set (17K English reasoning traces, 6.5K Thai translations, and 500 distilled Thai general thoughts); adding 20K general instruction examples does not improve end-to-end performance."],"supporting_citations":[{"why":"Defines the long-thought reasoning model and the 600K-plus-200K distillation recipe whose reasoning behavior the paper aims to transfer.","marker":"(DeepSeek-AI et al., 2025)"},{"why":"Provides Typhoon2 70B Instruct, the Thai-specialized model whose language ability the merge preserves, and the language-accuracy metric and Thai instruction data.","marker":"(Pipatanakul et al., 2024)"},{"why":"Supplies the 17K public reasoning-distillation dataset that the paper translates into Thai for the SFT alignment stage.","marker":"(Labs, 2025)"},{"why":"Introduces the DARE sparse-merge operation and the density and weight parameters that the paper tunes by layer.","marker":"(Yu et al., 2024)"},{"why":"Implements the merge operation used to combine the two models with per-layer-block densities and weights.","marker":"(Goddard et al., 2025)"},{"why":"Establishes the shared Llama 3.1 70B backbone that makes the two models' representations compatible for merging.","marker":"(Grattafiori et al., 2024)"},{"why":"Supplies the low-cost distillation baseline and the reasoning evaluation setup (AIME, MATH-500, LiveCodeBench, IFEval) that structures the paper's experiments.","marker":"(Team, 2025)"},{"why":"Provides Sealion v3 70B Instruct, the second language-specific model used to show the recipe transfers.","marker":"(Singapore, 2024)"},{"why":"Provides MATH-500, one of the three reasoning benchmarks used to measure the merged model's math capability.","marker":"(Lightman et al., 2023)"},{"why":"Provides LiveCodeBench, the coding benchmark used to measure the merged model's code-reasoning capability.","marker":"(Jain et al., 2024)"}],"fun_headline_variants":["One-day $120 merge: Thai LLM gains R1 reasoning, keeps Thai","Layer-wise DARE merge upgrades Thai LLM to R1-level reasoning","Open recipe: turn a Thai LLM into an R1-grade reasoner in a day","$120 and a day: Thai LLM matches R1 reasoning, retains language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o Thai translations of AIME, MATH-500, and LiveCodeBench preserve the difficulty and answerability of the original problems; if those translations are lossy or unanswerable, the Thai reasoning scores do not measure Thai-language reasoning.","fun_headline_variants_meta":{"raw":{"variants":["One-day $120 merge: Thai LLM gains R1 reasoning, keeps Thai","Layer-wise DARE merge upgrades Thai LLM to R1-level reasoning","Open recipe: turn a Thai LLM into an R1-grade reasoner in a day","$120 and a day: Thai LLM matches R1 reasoning, retains language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000398,"raw_usage":{"total_tokens":2083,"prompt_tokens":945,"completion_tokens":1138,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":1052}},"tokens_in":561,"tokens_out":1138,"duration_ms":9518,"temperature":1.0,"reasoning_tokens":1052,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:45:59.652610+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have native Thai speakers produce or approve a second translation of the AIME, MATH-500, and LiveCodeBench questions used in the paper, then re-run Typhoon2-R1-70B on those human-verified Thai prompts under the same sampling settings; if the model's Thai reasoning average falls clearly below DeepSeek R1 70B Distill's English average, the claim of R1-level reasoning in Thai is not supported.","supporting_citations":[{"cited_title":"Bespoke-Stratos: The unreasonable effectiveness of reasoning distillation","cited_arxiv_id":null,"evidence_quote":"Supplies the 17K public reasoning-distillation dataset that the paper translates into Thai for the SFT alignment stage."},{"cited_title":"Sky-T1: Train your own O1 preview model within \\ 450","cited_arxiv_id":null,"evidence_quote":"Supplies the low-cost distillation baseline and the reasoning evaluation setup (AIME, MATH-500, LiveCodeBench, IFEval) that structures the paper's experiments."},{"cited_title":"SEA-LION (Southeast Asian Languages In One Network): A Family of Large Language Models for Southeast Asia","cited_arxiv_id":null,"evidence_quote":"Provides Sealion v3 70B Instruct, the second language-specific model used to show the recipe transfers."}],"review_version":1}