{"id":"484b0e9d-72ad-47d4-9b49-6bdfaa4653a8","arxiv_id":"2502.09042","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Structured long-thinking SFT turns a 3B Thai instruct model into a reasoning model that improves on several English benchmarks and can think in Thai, with a fully open recipe.","lead":"This paper introduces Typhoon T1, an open 3B-parameter Thai reasoning model trained by supervised fine-tuning on about 55,700 long-thinking examples built from public datasets. It also introduces a structured thinking format with XML tags, and releases the dataset, weights, and training configuration.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never validates the correctness of the LLM-generated thinking traces; if many traces are incorrect or format-mimicking, the SFT gains on GSM8K/HumanEval+/GPQA may not reflect genuine reasoning.","rationale":"The reader's weakest_assumption correctly identifies the untested quality of the synthetic thinking traces as the most load-bearing condition for the central claim. If the traces are wrong, the SFT recipe does not demonstrably teach reasoning, and the modest benchmark gains (e.g., GSM8K 57.32 to 62.09, GPQA 25.00 to 30.80) could be selection artifacts or format effects. Other concerns are real but secondary: the dataset-size selection on the same benchmarks inflates the reported numbers, but the qualitative improvement over baseline appears across most sizes; the format comparison is partially controlled by deriving semi/unstructured from structured via tag removal, though it still overclaims a clear winner; and the absence of error bars limits confidence but does not invalidate the recipe. The proposed audit is cheap, direct, and would settle whether the central mechanism is genuine reasoning or trace mimicry. Since the reader already conditioned the verdict on this and the paper provides open artifacts that make the audit feasible, the verdict remains CONDITIONAL with the stated requirement for trace-quality validation.","tokens_in":22599,"tokens_out":12183,"duration_ms":116184,"concrete_test":"Audit a stratified random sample of 200 generated training records across the five domains (math, instruction following, coding, safety, finance). For each record, (1) verify that the final answer in <response> exactly matches the original ground truth, and (2) have two independent annotators label whether every intermediate step is logically valid, counting intentional PRM800K errors as valid only if they are followed by a correct correction. Compute the fraction of fully correct traces. If more than 10% of traces have an incorrect final answer, or more than 20% contain an unresolved step error, the data-quality assumption fails and the benchmark gains should be reattributed to format or length effects rather than to genuine reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the assumption that the 55,677 synthetic long-thinking records (Sec. 2.2.1) actually contain correct reasoning that teaches the model to reason. The pipeline (Sec. 2.2.2) uses GPT-4o-mini with three few-shot exemplars to transform ground-truth answers into structured thoughts, then Qwen2.5-32B-Instruct to 'refine' them. No quantitative validation of trace correctness is reported: no human eval, no automatic check that the final answer in <response> still matches the original ground truth, and no measurement of the fraction of steps that are logically valid. The PRM800K conversion deliberately selects incorrect steps and appends a correction (Sec. 2.2.1), so a nontrivial part of the training data contains wrong intermediate reasoning by design; if the correction step is malformed or the resampled step is also wrong, the trace teaches error patterns. If a large fraction of traces are incorrect or merely format-mimicking, the observed improvements on GSM8K/HumanEval+/GPQA could come from longer outputs and the base model's latent capabilities rather than from genuine reasoning learned via SFT. This is load-bearing because the paper's contribution is a generalizable SFT-only recipe for reasoning models; a recipe that depends on unvalidated, possibly noisy data does not establish that claim. The paper's own Table 2 shows zero-shot CoT on the base model produces long but unhelpful traces (0.00 on HumanEval+), so length alone is not sufficient, but no evidence isolates correct reasoning as the active ingredient.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Typhoon T1, an open Thai reasoning model built by supervised fine-tuning Typhoon 2 3B Instruct on about 55,677 synthetic long-thinking records. The authors introduce a 'structured thinking' format using XML tags (<plan>, <step>, <title>, <scratch_pad>, <summary>, <next_step>) and a two-stage transformation-and-refinement pipeline (GPT-4o-mini few-shot transformation followed by Qwen2.5-32B-Instruct refinement) that converts existing ground-truth answers into long reasoning traces. The paper reports ablations on thinking format, dataset size, domain mixture, and Thai-translated data, claiming that structured thinking is best overall, that 75% of the full dataset is optimal, that safety data matters most, and that adding a small amount of Thai-translated data enables Thai reasoning traces at some cost to English benchmarks. All datasets, pipeline details, training configurations, and model weights are released.","tokens_in":23020,"tokens_out":4664,"duration_ms":46341,"significance":"If the central claim holds, the paper provides a valuable, reproducible, and cost-effective SFT-only recipe for building reasoning models in low-resource languages, an area where open documentation is scarce. The explicit release of the data mixture, training configuration, model weights, and benchmark evaluation setup is a genuine strength, as is the systematic ablation over thinking formats, data quantity, and domain composition. The structured-thinking idea is clearly described and could be useful beyond the Thai setting. However, the significance is currently limited by two load-bearing gaps: the correctness of the generated thinking traces is not validated, and the choice of the 75% dataset size is made on the same benchmarks used to report the final model's performance. These issues need to be addressed before the recipe can be taken as a reliable guide for future work.","major_comments":[{"comment":"The load-bearing assumption of the entire paper is that the transformation-and-refinement pipeline turns correct ground truths into correct long thinking traces that teach genuine reasoning. This is not validated. The paper reports no automatic check that the refined <response> still matches the original ground truth, no human evaluation of the logical validity of the generated steps, and no analysis of the PRM800K conversions where incorrect steps are deliberately included and then 'corrected.' If a substantial fraction of traces are incorrect or merely format-mimicking, the benchmark gains in Section 3.1 could reflect output length or dataset overlap rather than improved reasoning. Please add a validation study: for example, exact-match or LLM-judge consistency between the final answer and the original ground truth on a random sample, human ratings of step-level correctness, and a quantitative analysis of how often the PRM800K self-correction step actually corrects the deliberately wrong step.","section":"Section 2.2.1 and Section 2.2.2"},{"comment":"The 75% dataset size is selected by comparing model performance on the same six benchmarks that are later used to report the final Typhoon T1-EN results in Tables 3 and 8. Since the selection is made on the evaluation benchmarks themselves, the reported scores are optimistically biased. This is a load-bearing issue for the paper's claim about the optimal data quantity and for the final model's benchmark numbers. Please either evaluate the selected configuration on a held-out set that was not used for model selection, or explicitly frame the procedure as model selection and report the selected model's performance on a separate test set.","section":"Section 3.2, Figure 3, and Table 7"},{"comment":"The section title and text overclaim the benefits of structured thinking. The title says structured thinking improves mathematics and coding, but Table 2 shows semi-structured thinking outperforming structured thinking on HumanEval+ (72.87 vs. 69.76) and IFEval (55.27 vs. 53.60), while structured thinking is best only on GSM8K, GPQA, and MMLU Pro. The coding claim is therefore not supported by the table. In addition, all numbers come from single runs without error bars or multiple seeds; differences of 2-3 points on some benchmarks (e.g., GSM8K 62.02 vs. 57.24, GPQA 27.23 vs. 27.68) may be within run-to-run noise. Please either revise the title and text to a more accurate summary, or provide additional evidence, such as confidence intervals or repeated runs, before concluding that structured thinking is the best format.","section":"Section 3.1 and Table 2"},{"comment":"The conclusion that 'the safety domain has the most impact' is only partially supported by Table 8. Removing safety yields the largest drop in GSM8K, HumanEval+, IFEval, and ThaiExam, but removing coding also produces large drops (e.g., IFEval 41.96 vs. 41.59; ThaiExam 19.57 vs. 17.19), and on GPQA and MMLU Pro removing safety actually improves or matches the baseline. The differences are small in several columns (e.g., GSM8K 56.71 vs. 56.94), and without variance estimates these rankings may be unstable. Please report confidence intervals, multiple seeds, or a more nuanced interpretation that acknowledges the role of coding data and the mixed effect on multiple-choice benchmarks.","section":"Section 3.3 and Table 8"}],"minor_comments":[{"comment":"The title contains a typo: 'A N OPEN' should be 'AN OPEN'.","section":"Title"},{"comment":"The caption reads 'This figures show domain distribution'; it should be 'This figure shows' or 'These figures show.'","section":"Figure 5 caption"},{"comment":"The text says 'The exact prompt used for this purpose is provided bekiw'; 'bekiw' should be 'below.'","section":"Appendix A.10"},{"comment":"The figure caption refers to 'Typhoon T1 3B Instruct' as the baseline, but the rest of the paper consistently calls the baseline 'Typhoon 2 3B Instruct.' Please correct this inconsistency.","section":"Figure 4"},{"comment":"The three few-shot exemplars are hosted on pastes.io, which is ephemeral and not a stable archival location. They should be included in the paper or in the project's Hugging Face repository so that the data-generation pipeline is actually reproducible.","section":"Appendix A.5"},{"comment":"The '+ 1.5k, CSFT' row shows a dramatic drop in MMLU Pro from 27.39 to 4.30, which is more than a 'significant reduction' and deserves a dedicated explanation; this pattern is not discussed in the text.","section":"Table 3"},{"comment":"The sentence about average response length says Typhoon T models generate longer responses 'except when prompting Typhoon 2 with zero-shot chain-of-thought reasoning'; as written this is confusing because the exception refers to the baseline model, not the Typhoon T models. Please rephrase.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"This is a solid workshop-level contribution with genuinely open artifacts, and the core recipe is plausible. My recommendation of major_revision is driven by two fixable but load-bearing gaps: the lack of any validation of the synthetic thinking-trace correctness, and the selection of the 75% dataset size on the same benchmarks used for the final claims. The structured-vs-semi-structured comparison also needs more careful wording and ideally variance estimates. None of these issues suggest misconduct; they are standard scientific robustness concerns. The paper's use of ephemeral pastes.io links for the few-shot exemplars is an archival problem that should be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the Typhoon T1 paper. Bottom line: it is a real, open, useful empirical contribution, and the caveat the reader flagged is exactly right—nobody checked whether the synthetic reasoning traces the model trains on are actually correct.\n\nWhat is new: this is the first fully open Thai reasoning model, with data, pipeline, configs, and weights all released. The structured-thinking format (plan/step/scratchpad/summary XML tags) is a genuine new variant, and the comparison across unstructured/semi-structured/structured formats is clean because they derive all three from the same transformed examples, so information content is equivalent. The data-size and leave-one-out ablations are useful, and the Thai-translation result—1,565 records giving Thai thinking traces without destroying English performance—is a concrete, practical finding. The paper is honest about limitations.\n\nSoft spots, in order. The biggest is trace validation. The pipeline uses GPT-4o-mini to transform ground truth into structured thoughts and Qwen2.5-32B-Instruct to refine; there is no human or automatic check that the final answer still matches the ground truth, no measure of step validity, and no analysis of diversity. PRM800K is deliberately converted with wrong steps plus corrections; if the correction or resample is bad, that teaches error patterns. Since the paper claims a general SFT-only recipe, this is load-bearing. The release of the dataset partially mitigates it because others can check, but the paper should have done at least a sample audit.\n\nNext: 75% data was selected on the same benchmarks used for final claims, with no error bars. That is a real but common weakness; it does not invalidate the recipe, but it means the headline numbers are optimistic. Also, Section 3.1's title says structured thinking improves coding, but semi-structured scores higher on HumanEval+ (72.87 vs 69.76). That is a modest overclaim. The IFEval and ThaiExam drops are honestly reported, and the forced-language experiment showing degradation is interesting and well-done.\n\nThe central finding—SFT on structured long-thinking data produces a small reasoning model that beats its instruct base on math/code/GPQA—looks robust enough. The mechanism is not proven, and the data-quality gap is real, but the paper earns a serious referee. I would send it to review and ask for trace validation, error bars, and a reworded title.\n\nWho should read it: anyone working on low-resource reasoning models or SFT-only reasoning recipes. I would cite it for the open recipe and the thinking-format comparison.","headline":"Open Thai reasoning recipe with real artifacts and a clean thinking-format comparison, but the unvalidated synthetic traces are the main thing to fix.","tokens_in":23492,"tokens_out":2722,"would_cite":true,"duration_ms":27345,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Supervised fine-tuning alone, on 42k structured long-thinking records, converts a 3B Thai instruct model into a reasoning model that improves on math, code, and science benchmarks and can think in Thai.","keywords":["reasoning models","supervised fine-tuning","Thai language","structured thinking","long chain-of-thought","low-resource language","open-source LLM","synthetic data"],"falsifier":"Run the same SFT recipe on traces whose intermediate steps have been independently verified correct, and compare against traces generated without verification; if benchmark gains vanish, the effect came from format or dataset overlap rather than reasoning quality.","tokens_in":22415,"feed_emoji":"🧠","tokens_out":6069,"duration_ms":49695,"temperature":0.7,"pith_summary":"The paper claims that supervised fine-tuning on roughly 42,000 machine-generated long-thinking records, formatted with XML structural tags, converts a 3B Thai instruct model into a reasoning model that beats its base on GSM8K, HumanEval+, and GPQA, and can produce Thai thinking traces after adding 1,565 translated records. It argues that a structured 'plan-then-step' thinking format outperforms semi-structured and unstructured formats, and that safety data is the most influential domain in the mixture. If correct, it establishes a reproducible, low-cost SFT-only recipe for building small reasoning models in low-resource languages without distillation from an existing reasoning model.","feed_headline":"A small Thai model learns to reason from 42k structured thoughts","feed_subtitle":"An open SFT-only recipe improves math, code, and science scores and adds Thai thinking traces.","key_machinery":"The load-bearing mechanism is the transformation-and-refinement data pipeline: starting from open datasets with ground-truth answers, a frontier language model (GPT-4o mini) converts each answer into a long structured-thinking trace using three hand-curated few-shot exemplars, then a larger open instructor model (Qwen2.5-32B-Instruct) refines the trace for structural and factual correctness. The structured thinking format itself—with explicit planning, per-step scratchpads, summaries, and next-step decisions—is the syntactic object that carries the reasoning behavior, and it can be reduced to semi-structured or unstructured formats by tag removal so that the formats are compared with equivalent information.","core_discovery":"The central discovery is that long-chain reasoning behavior in a 3B model can be elicited by supervised fine-tuning alone on synthetically transformed open datasets, without reinforcement learning or knowledge distillation. The paper introduces structured thinking—a format with XML tags such as <plan>, <step>, <scratch_pad>, <summary>, and <next_step> inside <thoughts>—and shows it improves performance over semi-structured and unstructured formats, especially on math and code. It further finds that 75% of the full dataset is better than 100%, that safety data is the most impactful domain, and that training on a small number of Thai-translated records lets the model reason in Thai without a catastrophic drop on English tasks.","pith_inferences":["The paper's result suggests that the value of RL for reasoning models may be largely in generating high-quality long thoughts, which can instead be curated once and distilled via SFT; a testable extension is to compare a model trained on these traces against an RL-trained baseline on the same base model.","The structured thinking format could be viewed as a lightweight inductive bias that encourages planning and self-checking; one could test this by ablating each tag (plan, scratch_pad, summary, next_step) to see which contributes the gains.","The leave-one-out results hint that the 'safety' domain in this mixture may proxy for general helpfulness and response formatting diversity rather than safety per se; a dataset-diversity ablation with matched sizes would separate those effects.","The Thai-translation experiment opens a low-cost path for other low-resource languages: translate a small subset of a high-quality thinking dataset and SFT, rather than building reasoning data from scratch."],"forward_implications":["An open, fully documented SFT recipe can produce a reasoning model with about 42k records and no RL, making reasoning-model training accessible outside large labs.","Structured thinking with XML scratchpads yields better math/code scores and more token-efficient reasoning than free-form or simply delimited thought.","Training with too much long-thinking data can hurt performance; the sweet spot in this setup was 75% of the mixture.","Adding a small set of Thai-translated traces (around 1.5k records) is enough to switch the model's thinking language to Thai while preserving most English benchmark performance.","Forcing the model to reason in one language degrades accuracy, so multilingual reasoning models should be allowed to pick their own thinking language."],"supporting_citations":[{"why":"Provides the reasoning-model behavior baseline (long traces, self-correction) and the RL approach that the paper contrasts with SFT.","marker":"(DeepSeek-AI, 2025)"},{"why":"Provides an SFT-based reasoning model that the paper's approach extends by avoiding distillation from an existing reasoning model.","marker":"(Muennighoff et al., 2025)"},{"why":"Supplies PRM800K, whose step choices are converted into self-correction examples in the training set.","marker":"(Lightman et al., 2024)"},{"why":"Supplies plan-and-solve prompting, the direct inspiration for structured thinking's plan-then-step design.","marker":"(Wang et al., 2023a)"},{"why":"Supplies the scratchpad mechanism for intermediate output, used in the structured thinking tags.","marker":"(Nye et al., 2021)"},{"why":"The Llama 3.2 base underlying Typhoon 2 3B Instruct.","marker":"(Grattafiori et al., 2024)"},{"why":"The Typhoon 2 3B Instruct base model and the source of its Thai language enhancements.","marker":"(Pipatanakul et al., 2024)"},{"why":"GSM8K is the math word-problem benchmark used to evaluate mathematical performance.","marker":"(Cobbe et al., 2021)"},{"why":"HumanEval+ is the code-generation benchmark used to evaluate coding performance.","marker":"(Liu et al., 2023; 2024)"},{"why":"GPQA is the graduate-level science QA benchmark used to evaluate scientific reasoning.","marker":"(Rein et al., 2024)"}],"fun_headline_variants":["SFT alone, no RL: 3B Thai model reasons","Open 3B Thai model learns to reason from 42k structured thoughts","Structured SFT thoughts give 3B Thai model reasoning","No RL needed: SFT builds reasoning in 3B Thai model","Thai reasoning from 3B SFT: no RL required"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the LLM-generated thinking traces correctly reflect the ground truth and teach genuine multi-step reasoning; if they merely mimic the format or contain incorrect steps, the benchmark improvements would not demonstrate real reasoning.","fun_headline_variants_meta":{"raw":{"variants":["SFT alone, no RL: 3B Thai model reasons","Open 3B Thai model learns to reason from 42k structured thoughts","Structured SFT thoughts give 3B Thai model reasoning","No RL needed: SFT builds reasoning in 3B Thai model","Thai reasoning from 3B SFT: no RL required"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001313,"raw_usage":{"total_tokens":5306,"prompt_tokens":858,"completion_tokens":4448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":4356}},"tokens_in":474,"tokens_out":4448,"duration_ms":32071,"temperature":1.0,"reasoning_tokens":4356,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:47:37.857053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SFT recipe on traces whose intermediate steps have been independently verified correct, and compare against traces generated without verification; if benchmark gains vanish, the effect came from format or dataset overlap rather than reasoning quality.","supporting_citations":[{"cited_title":"Let's Verify Step by Step","cited_arxiv_id":null,"evidence_quote":"Supplies PRM800K, whose step choices are converted into self-correction examples in the training set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPQA is the graduate-level science QA benchmark used to evaluate scientific reasoning."}],"review_version":1}