{"id":"31d5bb61-c0a9-4c95-b92f-b009fe01456b","arxiv_id":"2608.09893","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Training a single model on both concise and long-chain-of-thought math data creates a trade-off: more concise-answer supervision degrades long-reasoning accuracy, and the optimal training order depends on the data ratio.","lead":"This paper fine-tunes an AI language model with two response styles: short direct answers and long step-by-step reasoning. It varies the mix and training order of the two styles and finds they trade off against each other, with more short-answer training hurting long-reasoning performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Thinking-mode training/evaluation is length-truncated (5,120-token cap vs 11,877 mean trace), so the reported ratio trade-off may be an artifact of truncation rather than TMF dynamics.","rationale":"Good faith reading: the paper is a systematic TMF study, and the central claim requires that thinking-mode supervision is actually present and that thinking accuracy is measured on complete responses. The reported length caps contradict the data statistics: training caps at 5,120 tokens while thinking traces average 11,877; evaluation caps generated tokens at 5,120 while test references average 11,992. The table of generated lengths even exceeds the stated cap, so the length handling is inconsistent. If the model is trained on truncated thinking traces, it never learns to emit the final boxed answer for most examples; if evaluation truncates before the answer, accuracy is not accuracy. The ratio trend (thinking accuracy 0.199 at 1:4 to 0.224 at 4:1) and the negative correlation could be explained by how truncation interacts with the amount of non-thinking data, rather than by an inherent mode tension. The reader's dataset-confounding concern is also valid, but the truncation issue is more fundamental and verifiable from the released artifacts. A conditional verdict remains appropriate, but the required control should include fixing the context/generation window, not only matching datasets and total steps.","tokens_in":17487,"tokens_out":13286,"duration_ms":122197,"concrete_test":"Inspect the released Fusion Bench training code and data to count how many of the 1,500 thinking training examples exceed 5,120 tokens and whether the truncated sequences contain the \\boxed{} answer. Then re-run one full ratio sweep (e.g., all 7 ratios under the Mix schedule) with max_seq_length=32,768 and max_new_tokens=32,768 (or filter thinking data to examples whose full answer fits, keeping the same ratio structure), and recompute the thinking-accuracy trend and the non-thinking/thinking correlation. If the monotonic improvement with T:NT weakens or reverses, the central claim is an artifact of truncation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing flaw is that the thinking-mode training and evaluation are length-truncated in a way that invalidates the measured 'thinking accuracy'. Appendix A.6 fixes max_seq_length=5,120, while Table 1 reports the 1,500 thinking training examples have mean length 11,877 tokens (max 54,786). Standard SFT truncation cuts the long CoT before the \\boxed{} answer, so the model is never trained to produce the final answer for most thinking examples. Similarly, A.7 caps thinking-mode generation at max_new_tokens=5,120 while Table 2's thinking test references have mean length 11,992, so many generated responses are cut before an answer can be extracted; Table 3's reported thinking Len values (e.g., 7,027.6) contradict a 5,120 cap, indicating an undocumented or inconsistent length limit. With the thinking mode incompletely trained and evaluated, the observed ratio-dependent degradation (Table 3 thinking Acc from 0.199 at 1:4 to 0.224 at 4:1) and the r=-0.58 correlation cannot be attributed to TMF dynamics; they may reflect differential truncation behavior across conditions. This undermines the central 'inherent tension' claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies Thinking Mode Fusion (TMF) in Qwen3-4B via LoRA-based supervised fine-tuning, varying the ratio of thinking-mode data (OpenMathReasoning long chain-of-thought traces) to non-thinking-mode data (GSM8K short solutions) across seven ratios (T:NT from 1:4 to 4:1) and three training schedules (T-NT, NT-T, Mix). It reports that increasing non-thinking supervision degrades thinking-mode accuracy, that the optimal schedule depends on the data ratio, and that the two modes exhibit a negative correlation (r = -0.58, R^2 = 0.34), which it interprets as an inherent tension. The authors release a benchmark, Fusion Bench, with code and data.","tokens_in":17617,"tokens_out":4560,"duration_ms":42072,"significance":"If the empirical claims were valid, this would be a useful practical study of an under-explored post-training design choice, with clear implications for how to schedule and mix thinking and non-thinking supervision in hybrid reasoning models. The paper has notable strengths: it releases code and data, uses a full factorial design across ratios and schedules, averages over three runs, and formulates concrete research questions. However, the central quantitative claims rest on experimental comparisons that are confounded in ways the manuscript does not address. The length-truncation issue in particular calls into question whether the thinking mode was trained and evaluated in a way that measures long-form reasoning at all. The paper is therefore not yet ready for publication; the claims need re-analysis or re-experimentation before they can support the stated conclusions.","major_comments":[{"comment":"This is the most load-bearing issue and needs to be fixed before the central claim can be assessed.","section":"Appendix A.6/A.7; Tables 1-3"},{"comment":"","section":"Section 3; Tables 1-2"},{"comment":"","section":"Appendix A.1/A.6; Eq. (6)"},{"comment":"","section":"Section 5.5; Figure 5; Table 3"}],"minor_comments":[{"comment":"","section":"Section 5.4; Table 3"},{"comment":"","section":"Table 3 caption and Appendix A.7"},{"comment":"","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid structure and a reproducible benchmark, but the three confounds identified in the major comments (truncation, dataset difficulty, and step count) are intertwined with the central claim. The truncation issue is the most serious and should be addressed first; if the re-run with adequate lengths changes the qualitative pattern, the paper may not be salvageable in its current scope. I would also encourage the editor to ensure the code release actually reflects the stated max_length settings, as the current text is internally inconsistent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The first thing you should know: this is the first systematic look at how data ratio and training schedule interact in Qwen3-style Thinking Mode Fusion post-training, and the released Fusion Bench is a real resource. The second thing: I would not trust the central “inherent tension” claim yet, because the thinking mode is trained and evaluated under a 5,120-token cap while the thinking traces average around 12k tokens.\n\nWhat is genuinely new and good: prior work introduced TMF and analyzed inference-side mode separation, but nobody had varied the data ratio and schedule in a controlled factorial way. Fixing T=1500, sweeping ratios from 1:4 to 4:1, comparing T-NT, NT-T, and Mix across three runs, and releasing code and data is a sensible way to ask the question. The qualitative finding that Mix is robust across ratios and that the optimal schedule depends on the ratio is plausible and worth pursuing.\n\nNow the soft spots, in proportion. The stress-test concern about truncation is real and load-bearing. Appendix A.6 sets max_seq_length=5120 while Table 1 reports thinking training examples with mean length 11,877 tokens. Standard SFT truncation cuts the long CoT before the boxed answer, so for most thinking examples the model is never trained to produce the final answer. Evaluation is worse: A.7 caps generation at max_new_tokens=5120, but thinking test references average 11,992 tokens, so most generated responses are cut before an answer can be extracted. Table 3 reports thinking Len values above 5,120 (e.g., 7,027.6), which contradicts the stated cap and suggests an undocumented or inconsistent length limit. With the thinking mode incompletely trained and evaluated, the observed ratio-dependent degradation and the r=-0.58 correlation cannot cleanly be attributed to TMF dynamics.\n\nThere are also confounds the reader flagged. The two modes are instantiated with different datasets (OpenMathReasoning vs GSM8K) that differ in difficulty and answer style, so mode is confounded with dataset. All conditions train for one epoch, so total gradient steps vary by a factor of 4 across the ratio sweep, conflating mixture proportion with training volume. The negative correlation is a descriptive fit to outcome data, not evidence of an inherent tension, and no confidence intervals are reported.\n\nWho is this for? Practitioners deciding how to mix thinking and non-thinking data in post-training. It deserves a serious referee, not a desk reject, because the question is timely and the design is mostly clean. But it needs major revision: fix truncation (either train/eval with longer limits or filter/truncate consistently and report exclusion rates), add a same-problem dual-mode control, hold total gradient steps fixed, and report variability. I would not cite the headline result in its current form, but I would send it to review with a request for those controls.","headline":"A useful, timely empirical map of TMF training dynamics, but the headline trade-off is undermined by a 5,120-token truncation cap that cuts most thinking traces and test outputs before the answer.","tokens_in":18279,"tokens_out":4164,"would_cite":false,"duration_ms":39889,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training one model to answer both concisely and with long chain-of-thought carries a measurable cost: the more short-answer data it sees, the worse its long-form math reasoning becomes.","keywords":["thinking mode fusion","data ratio","training schedule","chain-of-thought","mathematical reasoning","supervised fine-tuning","mode interference","large language models"],"falsifier":"Train on one fixed set of math problems answered in both formats (empty think block with a short solution, and full chain of thought), holding difficulty and distribution constant while sweeping the data ratio. If thinking-mode accuracy no longer falls as non-thinking data rises, the trade-off is a dataset effect rather than an inherent property of mode fusion; conversely, if the negative slope persists on identical problems, the paper's central claim is confirmed.","tokens_in":17156,"feed_emoji":"🧮","tokens_out":11017,"duration_ms":86849,"temperature":0.7,"pith_summary":"Thinking Mode Fusion (TMF) is a post-training recipe for giving one language model both a concise-answering mode and a long chain-of-thought reasoning mode, switched by chat template. The authors ask whether how much of each mode's data is used, and the order in which the modes are trained, changes what such a model can do. By sweeping seven thinking-to-non-thinking data ratios and three training schedules on math word problems, they find the interaction is asymmetric: adding non-thinking supervision degrades thinking-mode accuracy, while the reverse direction costs less. The paper also reports a negative correlation between the two modes' accuracies and shows that the best training schedule depends on the data ratio, with interleaved training the strongest average choice.","feed_headline":"Short-answer data erodes an LLM's long-form math reasoning","feed_subtitle":"Fusion training benchmark shows the two modes trade off (r = -0.58) and the best schedule depends on the data mix.","key_machinery":"The object under study is Thinking Mode Fusion (TMF): a single model whose chat template either leaves the <think></think> block empty and gives a short solution (non-thinking mode) or fills it with a long chain of thought (thinking mode). The experimental machinery is a full factorial sweep over seven values of the ratio $\\rho$ (thinking to non-thinking) from 4:1 to 1:4 and three schedules — T-NT (all thinking examples first), NT-T (all non-thinking first), and Mix (interleaved to the target ratio) — with exact-match accuracy and response length as the measured outputs.","core_discovery":"On a benchmark built from short-solution grade-school problems and long chain-of-thought Olympiad-style problems, the paper's central claim is that the two TMF modes compete inside one model. Holding thinking data fixed at 1,500 examples and varying non-thinking data from 375 to 6,000, thinking-mode accuracy falls as the non-thinking share grows; across all configurations, non-thinking and thinking accuracy are negatively correlated ($r = -0.58$, $R^2 = 0.34$), with a slope of $-0.91$, meaning a 0.01 gain in concise-answer accuracy is associated with about a 0.0091 loss in long-form reasoning accuracy. The schedule matters too: the interleaved Mix schedule gives the best average accuracy on both modes, but no single schedule is optimal at every data ratio.","pith_inferences":["Because the non-thinking and thinking datasets differ in both answer length and problem difficulty, the measured \"mode\" trade-off may partly be a difficulty or domain effect; a same-problem, two-format control would separate these.","With $R^2 = 0.34$, most of the variance in the accuracies is not explained by the aggregate negative correlation, so schedule choice and per-ratio effects are doing real work; modeling the joint dependence on ratio and schedule could tighten the picture.","A natural next experiment is to keep the same problem set and vary only response format; if thinking accuracy stops falling when the underlying problems are identical, the claimed inherent tension would need to be reframed as data-distribution interference.","The slope of $-0.91$ gives practitioners a rough exchange rate: a one-point gain in concise accuracy costs nearly one point of reasoning accuracy, so the economically sensible ratio depends on the actual proportion of simple versus hard queries in production."],"forward_implications":["Adding cheap short-answer data to a reasoning model will trade away some long-form reasoning accuracy; the trade-off can be modulated but not eliminated by choosing a different schedule.","Interleaved (Mix) training is the safest default when the deployment mix of simple and hard queries is unknown, because it ranks best on average across ratios.","The optimal training order is ratio-dependent, so a deployment that knows its query mix can pick a schedule to favor one mode.","Response length tracks the trade-off: ratios and schedules that favor thinking accuracy also produce longer outputs, consistent with length reflecting reasoning effort.","Evaluations of fused models should report both modes; reporting only one can conceal the cost paid by the other."],"supporting_citations":[{"why":"Supplies the OpenMathReasoning corpus of 3.2 million long chain-of-thought solutions, from which the thinking-mode training and test sets are sampled.","marker":"(Moshkov et al., 2025)"},{"why":"Supplies the GSM8K grade-school math problems that instantiate the non-thinking mode and its evaluation set.","marker":"(Cobbe et al., 2021)"},{"why":"Introduces Thinking Mode Fusion and the thinking/non-thinking chat templates that define the two modes under study.","marker":"(Yang et al., 2025)"},{"why":"Provides the DeepSeek-R1 generated long chain-of-thought traces used as high-quality thinking-mode supervision.","marker":"(DeepSeek-AI, 2025)"},{"why":"Provides the LoRA low-rank adaptation method used for all fine-tuning runs.","marker":"(Hu et al., 2022)"}],"fun_headline_variants":["Short-answer data erodes LLM long-form math skill","Concise answers, weaker reasoning: fusion training trade-off","LLM fusion: boosting short answers hurts deep math thinking","Data ratio shift: more short answers, less long reasoning","Optimal LLM training schedule depends on data mix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that short GSM8K answers and long OpenMathReasoning chains cleanly instantiate the two modes, so the results are attributed to mode rather than to differences in problem difficulty, dataset distribution, or answer style.","fun_headline_variants_meta":{"raw":{"variants":["Short-answer data erodes LLM long-form math skill","Concise answers, weaker reasoning: fusion training trade-off","LLM fusion: boosting short answers hurts deep math thinking","Data ratio shift: more short answers, less long reasoning","Optimal LLM training schedule depends on data mix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3204,"prompt_tokens":936,"completion_tokens":2268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":2187}},"tokens_in":552,"tokens_out":2268,"duration_ms":15400,"temperature":1.0,"reasoning_tokens":2187,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:48:16.407656+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train on one fixed set of math problems answered in both formats (empty think block with a short solution, and full chain of thought), holding difficulty and distribution constant while sweeping the data ratio. If thinking-mode accuracy no longer falls as non-thinking data rises, the trade-off is a dataset effect rather than an inherent property of mode fusion; conversely, if the negative slope persists on identical problems, the paper's central claim is confirmed.","supporting_citations":[],"review_version":1}