{"id":"a1a6d982-1f91-4b76-8e03-6acda5bcc41b","arxiv_id":"2412.13702","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Typhoon 2 improves Thai LLM performance through continual pre-training on curated Thai data and post-training, releasing text, vision, audio, and safety models.","lead":"Typhoon 2 is a family of open Thai large language models covering text, vision, and audio, built by adapting Llama 3 and Qwen2.5 with Thai-focused data. The release includes models from 1B to 70B parameters plus a safety classifier, with reported gains on Thai benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ThaiExam/M3Exam serve as both development signal and headline evidence; the paper itself concedes contamination/overfitting risk, so the text SOTA claim is not settled until a held-out Thai evaluation confirms it.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: ThaiExam and M3Exam are acknowledged to be contaminated/saturated, yet they drive both development decisions and headline results. I agree, and would sharpen the mechanism: this is selection-on-the-test, not merely incidental contamination. Because pretraining data mixtures are chosen by improvements on these exact benchmarks, any overlap between training corpora and exam items can inflate the final numbers without improving genuine Thai competence. A held-out evaluation is therefore the minimum condition for the paper's central text-model claim. Other issues I noticed are real but secondary to this: Table 7 appears to contain a copy-paste error (Typhoon2-Llama-8B and Qwen2.5-7B rows are identical in the code-evaluation table), the vision section overclaims relative to its own OCRBench results for the released Qwen-based model, and the audio section's 'competitive quality' claim is strained given Table 39's low UTMOS and high English WER. These affect specific sections but do not by themselves invalidate the primary text contribution; the evaluation-validity problem does. The paper still merits a conditional verdict rather than rejection: the authors release weights, describe reproducible pipelines, and explicitly flag the ThaiExam/M3Exam limitation, which is exactly why the claims should be held to a fresh evaluation before being taken as established.","tokens_in":48038,"tokens_out":6209,"duration_ms":57675,"concrete_test":"Construct a held-out Thai evaluation set from sources published after the September 2023 training-data cutoff, or from human-authored questions not publicly available on the web, and verify that no exact or near-duplicate n-grams overlap with the pretraining corpus. Then evaluate Typhoon2 base and instruct models against Llama-3.1-8B and Qwen2.5-7B on this uncontaminated set. If Typhoon2's ThaiExam/M3Exam advantage shrinks or reverses on the held-out set, the SOTA claim is an artifact of benchmark contamination or selection; if the advantage persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central text-model claim depends on benchmarks that are both the optimization target and the measurement instrument. Section 2.3 states that each pretraining data source and mixture was selected by checking improvements on M3Exam and/or ThaiExam, and Section 2.5 explicitly concedes that scores on these datasets are 'highly above the average level of a typical Thai person' and 'can be attributed to contamination and saturation due to overfitting.' Table 2 then uses the same ThaiExam and M3Exam numbers as the headline evidence for Typhoon2-Text superiority over base models. If the pretraining corpus overlaps with exam items, or if mixture choices were selected on the final test set, the reported gains (e.g., Table 2 Typhoon2-Llama-8B-base ThaiExam 51.20 vs Llama3.1-8B 45.80) may reflect memorization or selection rather than general Thai language ability. The same pattern appears downstream: post-training decisions are made on IFEval, MT-Bench, and BFCL, with Thai versions created by in-house translation and no held-out, independently collected Thai benchmark reported anywhere. The conclusion's claim of 'superior performance across a majority of evaluated tasks' therefore rests on measurement instruments whose validity for real-world Thai use is explicitly called into question by the authors themselves.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This technical report describes Typhoon 2, a family of Thai-language LLMs built by continual pre-training of Llama and Qwen backbones. It covers Thai corpus construction and data-mixture selection, post-training for instruction following, math/code, long context, function calling, distillation and model merging, a Thai safety classifier, a vision model for Thai documents, and an end-to-end audio/speech model. The central claims are that Typhoon2-Text models reach state-of-the-art Thai exam and instruction-following performance while preserving base-model abilities, that Typhoon2-Safety provides strong Thai-specific content moderation, that Typhoon2-Vision improves Thai OCR and document VQA, and that Typhoon2-Audio enables simultaneous text and speech output in an end-to-end speech-to-speech system. The paper also releases model weights and hyperparameters.","tokens_in":48274,"tokens_out":5550,"duration_ms":49856,"significance":"If the headline results hold, the paper is a valuable resource for Thai NLP: it publicly releases a full model family across 1B-70B sizes, documents data-filtering and post-training recipes in unusual detail, and includes a genuinely interesting small safety classifier with high F1 on Thai-sensitive topics. The audio section is also substantial, with a concrete recipe for adapting an SALMONN-style encoder and Llama-Omni-style decoder to Thai. The authors are candid about several limitations, including possible contamination of ThaiExam/M3Exam and the English weakness of the audio model. However, the central text-model claim is currently supported mainly by benchmarks that the authors themselves used as development signals and that they concede may be contaminated, so the significance will be much stronger once a held-out Thai evaluation is provided.","major_comments":[{"comment":"The primary evidence for Typhoon2-Text superiority is circular. Section 2.3 states that each pretraining data source and mixture was selected by checking improvements on M3Exam and/or ThaiExam, and Section 2.5 explicitly concedes that scores on these datasets are 'highly above the average level of a typical Thai person' and 'can be attributed to contamination and saturation due to overfitting.' Table 2 then uses the same ThaiExam and M3Exam numbers as the headline evidence that Typhoon2-Llama-8B-base (ThaiExam 51.20) beats Llama3.1-8B (45.80). Because the model has been optimized on these exact instruments, these gains cannot be interpreted as general Thai language ability. Please add a genuinely held-out Thai evaluation (e.g., a Thai exam or Thai benchmark not used during development, with an n-gram overlap analysis against the pretraining corpus) and present it alongside Table 2.","section":"Sections 2.3, 2.5, and Table 2"},{"comment":"The code evaluation table contains a duplicated row that makes the comparison invalid as printed. The rows for Typhoon2-Llama3.1-8B-Instruct and Qwen2.5-7B-Instruct are identical across HumanEval-TH, HumanEval-EN, MBPP-TH, and MBPP-EN (58.5, 68.9, 60.8, 63.0 in each column). This cannot be correct for two different models unless the table was copied from one row to another. Please correct the table, rerun the evaluation if necessary, and report the actual numbers.","section":"Table 7"},{"comment":"The claim that 'Typhoon2-Qwen2-VL excels in key areas... compared to other competitive models' is selective and not supported by the full table. On OCRBench, Typhoon2-Qwen2-VL scores ROUGE-L/Accuracy of 64.38/49.60, which is below both Llama-3.2-11B (72.84/51.10) and Typhoon2-Llama-3.2-11B (81.20/71.70). Its average ROUGE-L across the eight benchmarks (62.77) is also lower than the Typhoon2-Llama-3.2 prototype (64.16). The Qwen-based model may still be the right release choice given its size and specific Thai OCR accuracy, but the stated rationale should be revised to name the metrics where it actually wins and to acknowledge the losses on OCRBench and average ROUGE-L.","section":"Section 4.5 and Table 25"},{"comment":"The conclusion that 'our evaluation demonstrates superior performance across a majority of evaluated tasks' is not supported by the full evaluation tables. In Table 16, Qwen2.5-72B-Instruct outperforms Typhoon2-70B-Instruct on MTBench-EN (9.28 vs 8.85), GSM8K-EN (94.6 vs 93.4), HumanEval-EN (87.2 vs 83.5), MBPP-EN (90.5 vs 84.9), and FC-EN (77.9 vs 65.7). Similar gaps appear at 7-8B scale in Table 15, for example HumanEval-EN 81.1 for Qwen2.5-7B vs 68.9 for Typhoon2-Llama-8B. The conclusion should be scoped to the specific tasks and languages where Typhoon2 actually leads, with the trade-offs stated explicitly.","section":"Section 6 and Tables 15-16"}],"minor_comments":[{"comment":"The 'FuncCall' column for the 1B and 3B models is marked with '?' while Tables 8-9 report BFCL results for these sizes. Please clarify whether these models were trained on function-calling data and evaluated, or evaluated without that training stage.","section":"Table 12"},{"comment":"The Thai math and code test sets are described as translations made with an early Typhoon2 model and GPT-4o, but these translated test sets are not released. Reporting construction details and releasing the sets would improve reproducibility and allow readers to assess translation quality.","section":"Section 3.2.2"},{"comment":"The notation 'Accuracy x' is not explained in the table caption, and the 'Average(Accuracy)' row appears to ignore cells marked 'x'. Please clarify how missing values are handled in the average.","section":"Table 25"},{"comment":"The code-switching metric counts non-Thai characters and would penalize legitimate loanwords or code-mixed Thai-English usage that is natural in context; please state this limitation explicitly.","section":"Section 3.1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a technical report with genuinely useful released artifacts, and the authors are unusually open about limitations. The main blocker is not the absence of a mathematical proof but the reliance on ThaiExam/M3Exam as both development and headline evaluation instruments despite the authors' own contamination caveat. A held-out Thai evaluation plus correction of the duplicated Table 7 row would address most of my concerns. The vision and conclusion overclaims are also easy to fix by tightening the language."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper is a 55-page technical report that releases a family of open Thai models: text 1B-70B (base and instruct), vision, audio, and a safety classifier. The main contribution is the artifacts, not a new method. The central text claim—state-of-the-art on Thai exams—is plausible but not fully settled because the evaluation uses M3Exam and ThaiExam as both the development signal and the headline evidence, and the authors themselves concede those scores may reflect contamination and overfitting.\n\nWhat is actually new: the model weights are on HuggingFace; the report documents useful pipelines, including TyphoonIF (Thai instruction data), an agentic OCR curation framework for Thai documents, and a speech-to-speech architecture that combines SALMONN's encoder with Llama-Omni's decoder. The ablations on data mixture, distillation, and model merging are detailed and honest, and the audio section reports clear negative results (gender classification drop, lower UTMOS) instead of hiding them.\n\nThe soft spots are real but manageable. First, the evaluation circularity: Section 2.3 says each pretraining data source was selected by checking M3Exam/ThaiExam improvements, and Section 2.5 warns those scores are 'highly above the average level of a typical Thai person' and may be contaminated. Using the same benchmarks in Table 2 as proof of superiority is therefore shaky; a held-out Thai benchmark would settle it. Second, Table 7 has a suspicious duplicated row: Qwen2.5-7B-Instruct shows exactly the same numbers as Typhoon2-Llama3.1-8B-Instruct, likely a copy-paste error. Third, no error bars anywhere, which matters for the smaller differences. Minor: the vision findings bullet says the Qwen-based model 'excels' on OCRBench, but its OCRBench accuracy is actually below Qwen2-VL's. The overclaim is small but worth fixing.\n\nWho is this for: anyone working on Thai NLP, low-resource language adaptation, or multimodal Thai systems. The report deserves a serious referee because the released artifacts are valuable and widely reusable, and the training details are unusually complete. I'd conditionally accept it, with a requested revision: add one held-out Thai evaluation, fix the duplicate row, and recalibrate the 'excels' language. The paper is honest; it just needs the evidence to match the claims.","headline":"Open Thai model family worth knowing about, but the headline benchmark story is circular.","tokens_in":48844,"tokens_out":3729,"would_cite":true,"duration_ms":31750,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that a single open recipe—curated Thai-English continued pretraining plus staged post-training—brings Thai text, vision, and audio models to the top of Thai-language benchmarks while preserving the base…","keywords":["Thai language models","continual pre-training","instruction tuning","function calling","long context","multimodal LLM","speech-to-speech","AI safety classifier"],"falsifier":"Write a new Thai national-exam-style test with questions dated after the model's training cutoff, run the released Typhoon2-Text models and their base models on it, and compare; if the Typhoon2 advantage over the base models shrinks to near zero, the benchmark-driven improvement is largely an artifact of overlap with training data.","tokens_in":47863,"feed_emoji":"🇹🇭","tokens_out":8153,"duration_ms":64400,"temperature":0.7,"pith_summary":"This paper reports a family of Thai language models, from 1 to 70 billion parameters, built by continuing to train capable open base models on a mixture of Thai and English text and then applying a staged post-training recipe. The authors claim the resulting text models reach the best published results on Thai exams and Thai instruction-following while keeping English, math, coding, and long-context abilities largely intact. The same project adds a Thai vision model focused on document OCR and chart questions, an end-to-end audio model that can listen to speech and answer in parallel text and speech, and a lightweight safety classifier tuned to Thai cultural sensitivities. The significance, if the evaluations hold, is that a low-resource language can be brought to competitive model quality with an open, reproducible recipe rather than proprietary data.","feed_headline":"Open Thai model family tops Thai exams at every size from 1B to 70B","feed_subtitle":"Continued training plus post-training lifts Thai text, vision, OCR, and audio performance, with open weights.","key_machinery":"The load-bearing mechanism is the Typhoon2 training recipe. It starts with continual pre-training, where an already-trained English-centric base model is trained further on roughly 12 billion high-quality Thai tokens plus a 50 percent English mix to avoid forgetting; the Thai tokens are selected by four filters—a cultural-relevance classifier, a Thai quality fastText classifier, synthetic textbook-style augmentation, and high-educational-content filtering. Post-training then layers general SFT with a 3:7 Thai-to-English ratio, math and code SFT with a Thai-translated subset, long-context data at 15 percent of the mix, function-calling data at 5–10 percent, top-k logits distillation for 1B/3B models, and a DARE+linear merge with a newer instruction model for the 70B model. For audio, the machinery is an encoder-adapter-LLM front end (Whisper-type speech encoder plus audio-event encoder, aligned through a Q-Former) feeding a non-autoregressive speech decoder trained with CTC to output discrete speech units, which a unit vocoder turns into waveform; this lets text and speech be decoded in parallel from the same LLM hidden states.","core_discovery":"On the paper's own terms, the central claim is that a single adaptable recipe—domain-filtered continual pre-training on a Thai-English mix, followed by general and domain-specific supervised fine-tuning, long-context adaptation, function-calling data, logit distillation for small models, and DARE-plus-linear merging for the 70B model—produces Thai LLMs that outperform their base models and existing Thai-focused baselines across exams (ThaiExam, M3Exam), instruction following (IFEval-TH, MT-Bench-TH), math, coding, and function calling. The authors further claim the same approach transfers to other modalities: a vision model built on Qwen2-VL improves Thai OCR and chart QA without losing general captioning, and an audio model built on the Typhoon2 text model achieves end-to-end speech-to-speech interaction that beats an open baseline and is competitive with a proprietary system on Thai. The conclusion states that evaluation demonstrates superior performance across a majority of evaluated tasks, including math and reasoning.","pith_inferences":["Editorial inference: if ThaiExam and M3Exam contain contamination—a concern the paper itself raises—then the headline exam gains may overstate real Thai ability; a held-out, post-cutoff Thai exam would settle this.","Editorial inference: the same CPT-and-post-training recipe could plausibly transfer to other low-resource Southeast Asian languages, but the paper only demonstrates it for Thai, so the transfer claim remains untested.","Editorial inference: the audio architecture's parallel text-and-speech decoding should lower time-to-first-speech-token latency in conversational assistants; the paper does not report this latency directly, making it a measurable but unverified benefit."],"forward_implications":["Typhoon2-Text models from 1B to 70B improve Thai exam and instruction-following performance over their base models, with the Qwen-based 7B variant reaching the top overall BFCL function-calling score (79.08% English, 75.12% Thai).","The 70B model's DARE+linear merge with a newer Llama instruction model raises IFEval and MT-Bench scores beyond the unmerged SFT model.","Llama-based Typhoon2 models hold longer contexts up to roughly 90,000 tokens and the Qwen-based model reaches 128,000 tokens.","Typhoon2-Vision improves Thai OCR and ChartQA over Qwen2-VL and prior Thai vision models, while the audio model generates text and speech in parallel and performs competitively with a proprietary system on Thai speech-to-speech.","Typhoon2-Safety, a small mDeBERTa-based classifier, reaches higher F1 than larger Llama-based guards on Thai-sensitive topics (about 88.5–88.7) and comparable English safety performance."],"supporting_citations":[{"why":"Supplies the previous Typhoon model, the Typhoon1-Corpus, and the ThaiExam benchmark used as the development signal throughout pre-training.","marker":"(Pipatanakul et al., 2023)"},{"why":"Provides the Llama 3 base models used for the 1B, 3B, 8B, and 70B text variants and for comparison baselines.","marker":"(Grattafiori et al., 2024)"},{"why":"Provides the Qwen2.5 base models used for the 7B text variant and for comparison baselines.","marker":"(Yang et al., 2024a)"},{"why":"FineWeb educational-filter approach that the paper adapts to build cultural-relevance and high-education Thai classifiers.","marker":"(Penedo et al., 2024)"},{"why":"DCLM-style high-quality web filtering that motivates the Thai fastText quality classifier and the English data mixture selection.","marker":"(Li et al., 2024a)"},{"why":"SALMONN supplies the speech-encoding architecture (Whisper/BEATs encoders plus Q-Former) that Typhoon2-Audio builds on.","marker":"(Tang et al., 2024a)"},{"why":"Llama-Omni supplies the non-autoregressive speech decoder and unit-vocoder design used for parallel text and speech generation.","marker":"(Fang et al., 2024)"},{"why":"MT-Bench provides the LLM-as-a-judge protocol used for instruction-following and spoken-assistant evaluation.","marker":"(Zheng et al., 2023)"}],"fun_headline_variants":["Typhoon 2: open Thai models, 1B-70B, top Thai benchmarks","Open Thai AI family: Typhoon 2 beats baselines in text, vision, audio","Thai LLM family from 1B to 70B: open weights, top exam scores","Typhoon 2: open Thai models excel in exams, vision, and speech","Thai-language AI: Typhoon 2 family dominates from 1B to 70B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ThaiExam and M3Exam scores used to guide every pre-training and post-training decision are honest measures of Thai language ability; the paper itself notes these scores sit far above typical human performance and may reflect contamination or overfitting, and if that is true the claimed gains may not appear in real Thai text.","fun_headline_variants_meta":{"raw":{"variants":["Typhoon 2: open Thai models, 1B-70B, top Thai benchmarks","Open Thai AI family: Typhoon 2 beats baselines in text, vision, audio","Thai LLM family from 1B to 70B: open weights, top exam scores","Typhoon 2: open Thai models excel in exams, vision, and speech","Thai-language AI: Typhoon 2 family dominates from 1B to 70B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1522,"prompt_tokens":930,"completion_tokens":592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":474}},"tokens_in":546,"tokens_out":592,"duration_ms":5145,"temperature":1.0,"reasoning_tokens":474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:51:43.190948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Write a new Thai national-exam-style test with questions dated after the model's training cutoff, run the released Typhoon2-Text models and their base models on it, and compare; if the Typhoon2 advantage over the base models shrinks to near zero, the benchmark-driven improvement is largely an artifact of overlap with training data.","supporting_citations":[],"review_version":1}