{"id":"86260c3f-8515-468a-9d34-b68fc922b8be","arxiv_id":"2507.01887","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Training small language models on intermediate-length reasoning chains from a merged mid-sized teacher assistant improves their math reasoning scores over direct distillation from a large teacher.","lead":"Small AI models often fail to learn from long, step-by-step reasoning traces produced by much larger models. This paper shows that using a mid-sized 'teacher assistant' to generate shorter reasoning steps, then training the small model on those, improves math scores by 3 to 4 points on benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central mechanism rests on unverified properties of D_MiCoTA: no answer filter, no quantitative length/accuracy stats for the merged TA, so the claimed intermediate-length bridge is not established.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the merged teacher assistant's outputs are not verified for correctness or intermediate length, even though the strong teacher's data is filtered in Eq. (1). I agree with this assessment and do not see a more fundamental flaw. The controlled comparisons in Table 2 are internally consistent, the baselines are reasonable, and the gains over Strong Teacher CoT are large enough that they cannot be dismissed as trivial noise, though the absence of error bars remains a secondary limitation. The most direct threat to the central claim is that D_MiCoTA could be shorter but less correct, making the apparent improvement an artifact of training on easier, more base-like data rather than evidence for the proposed capacity-and-length bridge. The concrete test of measuring answer accuracy and length distribution on the released pipeline would settle this concern. The reader's CONDITIONAL verdict is appropriate; my read does not move it.","tokens_in":13799,"tokens_out":5699,"duration_ms":68019,"concrete_test":"Using the released code or dataset, regenerate or inspect D_MiCoTA for the 12K prompts and apply the same rule-based answer extraction used in evaluation; report per-example answer accuracy and token-length distribution (mean, median, sd) for D_MiCoTA, Dstrong (Eq. 1), and Half-size CoT. If the merged TA's median length is not roughly half of Dstrong's, or if its answer accuracy is substantially below the unmerged TA or Strong Teacher, the intermediate-length and correctness premises of the central claim are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that SLMs improve because the merged teacher assistant provides intermediate-length CoT traces that bridge both the capacity and length gaps. This requires D_MiCoTA to be (a) roughly half the token length of Dstrong and (b) correct enough to serve as reliable distillation targets. Neither property is demonstrated. Eq. (1) filters Dstrong by answer correctness (Ai=true) and length (Li<=Lmax), but Sec. 3.3 defines D_MiCoTA with no analogous filter. Figure 2 is illustrative and reports no length distribution, mean/median token counts, or answer accuracy for the merged 14B model. Because the merged model is a Dare/TIES blend of the base and fine-tuned TA, it could plausibly produce shorter traces primarily because the base component's System-1 style truncates reasoning, lowering answer accuracy. If D_MiCoTA contains many wrong or incomplete solutions, the student gains could reflect learning a simpler, more base-like distribution rather than the proposed intermediate-length bridge. The BPC results in Table 4 are consistent with this alternative: lower BPC on MiCoTA data just means the data is closer to the base distribution, which a shorter or noisier dataset would also produce. Without quantitative length and accuracy statistics for D_MiCoTA, the mechanism and even the interpretation of the reported gains are not fully established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MiCoTA, a distillation framework for transferring long chain-of-thought (CoT) reasoning from a 32B strong teacher to small Qwen instruct models. The method fine-tunes a 14B teacher assistant on long CoT traces filtered by answer correctness and length (Eq. 1), merges the fine-tuned assistant with its base instruct checkpoint via DARE/TIES, uses this merged 'Mid-CoT Teacher Assistant' to generate a dataset D_MiCoTA of putatively intermediate-length traces, and then SFTs 1.5B/3B/7B students on this dataset. The central empirical claim is that MiCoTA beats direct long-CoT distillation and the instruct baselines on AIME2024, AMC, OlympiadBench, MATH-500, and GSM8K, with ablations against an unmerged 14B assistant and a reproduced Mix-Long baseline. The paper supplements these results with a BPC distributional analysis and a case study, and releases code and data.","tokens_in":14072,"tokens_out":8620,"duration_ms":85589,"significance":"If substantiated, MiCoTA is a practical and inexpensive recipe: model merging is used to shorten teacher traces instead of training a separate short-CoT generator, and the reported average gains are consistent across three model scales. The paper is commendable for including a reproduced baseline, ablations that separate the assistant-size and trace-length effects, and a public code/dataset link. However, the central mechanism is not yet fully established: the properties of D_MiCoTA (length distribution, correctness, completeness) are not measured, all headline numbers are unseeded point estimates, and the BPC analysis is a post-hoc explanation computed on base models. The contributions are therefore promising but require additional evidence before the mechanism and the reported gains can be taken at face value.","major_comments":[{"comment":"The stress-test concern is well-founded. Dstrong is filtered by answer correctness and length in Eq. (1), but D_MiCoTA in Section 3.3 has no analogous filter, and Figure 2 supports the 'half-length, no performance drop' claim only illustratively. Table 2 shows the merged 14B assistant achieves 56.40 average versus 72.83 for the strong teacher, so the quality of the generated traces cannot be assumed. Please report the mean/median token length, the answer-accuracy rate, and the fraction of truncated or incomplete traces for D_MiCoTA; if no filter is applied, the improvement over Strong Teacher CoT could be due to shorter or noisier data rather than to the proposed bridge.","section":"Section 3.3, Eq. (1), Figure 2"},{"comment":"All results are single-run point estimates without error bars, standard deviations, or significance tests. The reported gains are not uniformly positive per benchmark (e.g., Qwen2.5-7B MiCoTA scores 70.40 on MATH-500 versus 72.60 for Instruct), so the average improvements of 3.47 and 3.93 need variance estimates or multiple seeds to be interpretable. Please add at least repeated-seed results for the main comparisons or bootstrap confidence intervals.","section":"Section 4.3, Table 2"},{"comment":"The BPC argument is computed on the base instruct models, not on the fine-tuned students whose learnability is the subject of the paper, and lower BPC on MiCoTA data is also what one would expect if the data were shorter or closer to the base instruct distribution. It therefore does not discriminate between the proposed length-bridging mechanism and the alternative that the dataset is simply easier or more base-like. A stronger test would compute BPC of the trained student models on held-out reasoning traces, or control D_MiCoTA for length and answer correctness.","section":"Section 4.5, Eq. (3), Table 4"}],"minor_comments":[{"comment":"The framework is named 'MiCoTAl' in the abstract and 'MiCoTA' elsewhere; please standardize the acronym.","section":"Abstract"},{"comment":"The caption refers to 'Owen2.5-7B'; this should be Qwen2.5-7B. In addition, the 'medium length' conclusion is qualitative; please include the numeric length statistics (mean/median tokens) that support it.","section":"Section 4.6, Figure 3"},{"comment":"Please state the base of the logarithm and clarify whether the denominator counts UTF-8 bytes or Unicode code points, since 'len-utf-8(T)' is not formally defined in the text.","section":"Section 4.5, Eq. (3)"},{"comment":"The rows for Qwen2.5-14B-Instruct '+Teacher' correspond to the same model later used as the intermediate teacher assistant; this reuse should be stated explicitly to avoid confusion.","section":"Section 4.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest and the method is reproducible in principle, but the missing D_MiCoTA statistics and the lack of variance estimates are the main barriers. I recommend major revision rather than rejection because both issues are addressable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a practical distillation recipe for long-CoT reasoning into small models, and the headline result is credible: training Qwen2.5-1.5B/3B/7B on CoT generated by a merged \"mid-size\" teacher assistant gives consistent gains over direct long-CoT distillation and over the un-merged intermediate teacher. The comparison is clean, the arithmetic checks out, and the ablation isolating Half-size CoT from MiCoTA is genuinely informative. The main new thing here is the specific combination — intermediate teacher + DARE/TIES model merging to shorten the CoT before distilling — not any single component, and the paper is honest about the precedents.\n\nThe soft spot is the load-bearing one: the paper never shows that D_MiCoTA is actually intermediate-length and correct. Equation (1) filters D_strong by answer correctness and length, but D_MiCoTA is defined with no such filter, and Figure 2 is illustrative, not quantitative. The authors say the merged TA produces \"approximately half\" the tokens, but there are no length statistics, no answer-accuracy numbers, and no seed-to-seed variance for any of the main results. This matters because the BPC analysis in Table 4 is consistent with a simpler alternative: MiCoTA data is just closer to the base SLM distribution, which could be because it is shorter, noisier, or both. The mechanism claim (intermediate-length bridging) is therefore not established, even though the empirical gains may well hold up.\n\nThat said, I don't think this is fatal. The recipe is concrete, the code and data are promised, and the gains are large enough that even a partially-confounded explanation is practically interesting. The paper's own limitation section only mentions domain generality, not the missing data statistics, so I'd ask the authors to add: length distribution and accuracy of D_MiCoTA, merge hyperparameters, and multiple-seed results. With those, this would become a solid paper, and I'd support sending it to a serious referee as a borderline paper.","headline":"A practical distillation recipe with credible gains, but the central mechanism is under-supported because the paper never quantifies the length or accuracy of the mid-CoT training data.","tokens_in":14596,"tokens_out":3713,"would_cite":true,"duration_ms":40512,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that distilling small models on intermediate-length reasoning traces from a merged mid-size teacher assistant beats both direct long-chain-of-thought distillation and the instruct baseline on math benchmarks.","keywords":["knowledge distillation","chain-of-thought reasoning","small language models","teacher assistant","model merging","long-to-short reasoning","math reasoning","learnability gap"],"falsifier":"Measure the token-length distribution and answer-verification accuracy of the merged teacher assistant's generated data; if the median trace length is not close to half that of the strong teacher's traces, or if a substantial share of traces are incomplete or incorrect, the proposed bridge loses empirical support.","tokens_in":13608,"feed_emoji":"🧮","tokens_out":6305,"duration_ms":61733,"temperature":0.7,"pith_summary":"The paper proposes that small language models fail when trained on long chain-of-thought traces from large reasoners not just because the teacher is too big but because the reasoning traces are too long. It claims this 'learnability gap' can be bridged by training a mid-sized teacher assistant on the long traces, merging it with its own base weights to make it produce reasoning of about half the length, and then distilling those medium-length traces into the small student. On five math benchmarks this raises Qwen2.5-7B-Instruct from 40.54 to 49.36 average score and Qwen2.5-3B-Instruct from 31.92 to 43.29, beating both the instruct baseline and a long-plus-short CoT mix. The paper also reports that the medium-length data has lower bits-per-character for the student models, which it takes as evidence that the data is closer to what small models can absorb.","feed_headline":"Half-length reasoning traces lift small-model math scores by 11 points","feed_subtitle":"Distilling from a merged half-length teacher beats direct long-CoT training on five math benchmarks.","key_machinery":"The central object is the Mid-CoT Teacher Assistant, a mid-sized model (Qwen2.5-14B-Instruct) that is first fine-tuned on long CoT generated by a 32B strong teacher and then merged with its pre-fine-tuning weights. The merge, carried out with DARE pruning and TIES sign consensus, is the mechanism that shortens output length to about half while keeping reasoning quality, yielding the intermediate-length CoT dataset used to fine-tune student models. The argument also leans on bits-per-character as a probe: lower values for MiCoTA data are read as showing the traces are closer to the student's own distribution and therefore easier to learn.","core_discovery":"The central claim is that the length of the reasoning trace, not only the size of the teacher, determines whether a small model can learn from distillation. The paper's discovery is that a merged intermediate model—a teacher assistant fine-tuned on the strong teacher's long CoT and then merged with its pre-fine-tuning version using DARE sparsification with TIES sign consensus—produces CoT traces about half as long as the teacher's without a performance drop. Training students on these Mid-CoT traces outperforms direct long-CoT distillation across Qwen2.5-1.5B, 3B, and 7B models, and the student distributions align better with the MiCoTA data as measured by bits-per-character.","pith_inferences":["Editorial inference: if distribution alignment is the active mechanism, then filtering the merged assistant's traces by answer correctness—which the paper does for the strong teacher's data but not for D_MiCoTA—could widen the observed gains.","Editorial inference: the same merge-then-distill recipe could be tested on non-math reasoning domains such as code generation or planning, where long traces also overwhelm small models; the paper only evaluates math benchmarks.","Editorial inference: bits-per-character on candidate distillation data could serve as a cheap pre-training screen, letting practitioners choose between teachers without running full distillation.","Editorial inference: the half-length property was demonstrated on Qwen models, so it is an open question whether the merge-induced shortening and its benefit persist across model families and optimization objectives."],"forward_implications":["Direct long-CoT distillation can degrade small models below their instruct baseline, while intermediate-length CoT data from a merged assistant improves them on the same tasks.","Model merging can act as a training-free length control, converting a long-CoT reasoner into a mid-CoT generator.","The learnability gap in distillation has a length component, so data design should match the trace length to the student's capacity.","The proposed recipe transfers across student scales from 1.5B to 7B within the same model family."],"supporting_citations":[{"why":"Supplies the strong teacher (R1-Distill-Qwen-32B) whose long CoT traces are the distillation source.","marker":"[4]"},{"why":"Documents the learnability gap and provides the Mix-Long baseline that mixes long and short CoT.","marker":"[14]"},{"why":"Introduces the teacher-assistant paradigm that MiCoTA extends to the CoT-length dimension.","marker":"[16]"},{"why":"Shows model merging shortens long reasoning to about half length without performance drop, the mechanism behind the Mid-CoT assistant.","marker":"[38]"},{"why":"Provides the DARE sparsification algorithm used in the model merge.","marker":"[46]"},{"why":"Supplies the model-merging toolkit used in the merge step.","marker":"[7]"},{"why":"Provides the Qwen2.5 base models used as students and as the teacher assistant.","marker":"[42]"},{"why":"Supplies the adapted bits-per-character metric used to measure distribution alignment.","marker":"[54]"}],"fun_headline_variants":["Half-length traces from merged teachers boost small-model math","Shorter CoT distillation from teacher assistants narrows the gap","Mid-CoT teacher assistants unlock long reasoning for small models","Intermediate-length traces improve small-model math reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the merged teacher assistant's traces are correct, complete, and genuinely about half as long as the strong teacher's traces; if the traces are noisy or not actually intermediate in length, the reported gains could come from simpler data rather than from the proposed bridge.","fun_headline_variants_meta":{"raw":{"variants":["Half-length traces from merged teachers boost small-model math","Shorter CoT distillation from teacher assistants narrows the gap","Mid-CoT teacher assistants unlock long reasoning for small models","Intermediate-length traces improve small-model math reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0011,"raw_usage":{"total_tokens":4597,"prompt_tokens":957,"completion_tokens":3640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":3576}},"tokens_in":573,"tokens_out":3640,"duration_ms":27960,"temperature":1.0,"reasoning_tokens":3576,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:41:00.440341+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the token-length distribution and answer-verification accuracy of the merged teacher assistant's generated data; if the median trace length is not close to half that of the strong teacher's traces, or if a substantial share of traces are incomplete or incorrect, the proposed bridge loses empirical support.","supporting_citations":[{"cited_title":"Improved knowledge distillation via teacher assistant","cited_arxiv_id":null,"evidence_quote":"Introduces the teacher-assistant paradigm that MiCoTA extends to the CoT-length dimension."}],"review_version":1}