{"id":"bd2d8b80-3989-4581-b8e7-71ad02d43613","arxiv_id":"2502.07374","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Long chain-of-thought reasoning can be distilled into Qwen2.5-32B-Instruct with 17k samples and LoRA, and performance degrades far more when reasoning steps are shuffled or deleted than when step content is corrupted.","lead":"Fine-tuning a large language model on just 17,000 long reasoning traces from DeepSeek R1, with full or low-rank updates, sharply improves math and coding benchmarks and approaches OpenAI's o1-preview. Controlled corruption experiments suggest the global order and logical consistency of reasoning steps matter more than the correctness of individual steps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own 100% digit-corruption row contradicts 'content has minimal impact': a content-only perturbation causes the largest accuracy drop in Table 2.","rationale":"The reader's weakest_assumption focuses on the perturbation pipeline (imperfect step segmentation, topic shift from inserted steps) and on benchmark noise. These are legitimate but secondary. The most load-bearing concern is internal to the paper's own results: the 100% digit-corruption experiment, classified as a content perturbation, produces the single largest accuracy drop in Table 2. This directly contradicts the paper's headline claim that content has minimal impact. The paper's explanation—that 100% corruption is 'severe'—does not rescue the universal claim; it merely narrows the conclusion to 'content matters little unless it is totally destroyed.' That is a significant qualification that should appear in the abstract and conclusion. The rest of the paper's contributions (data-efficient SFT/LoRA distillation, ablation across models, comparison to Best-of-N) remain valuable and are not undermined by this issue. The reader's CONDITIONAL verdict already anticipates the need for revision; our concern strengthens the condition but does not change the overall disposition. Hence UNCHANGED. This is not a rejection because the empirical distillation results are solid, but acceptance should require revising the overbroad content-vs-structure claim and/or adding a caveat about the 100% corruption regime. A concrete computational check—rerunning the 100% digit-corruption condition with controls—would settle whether the collapse is due to content per se or to the loss of arithmetic coherence, which is the key confound.","tokens_in":17500,"tokens_out":3815,"duration_ms":37687,"concrete_test":"Retrain the 4.6k QwQ traces with 100% digit corruption under two controls: (1) digits replaced by consistent random values (same digit always maps to the same random digit within a trace) to preserve arithmetic coherence, and (2) digits replaced by random values exactly as in the paper, but with 3 independent seeds. Evaluate on AIME 2024 and report mean and standard deviation. If both controls still collapse to near-zero accuracy, the 'content has minimal impact' claim is falsified; if the consistent-replacement control recovers meaningfully, the original collapse is due to incoherent arithmetic rather than content per se. Also report average output token length and reasoning-keyword counts for the 100%-corruption model to verify whether it still produces structurally long CoTs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'the content of individual reasoning steps has minimal impact' is directly contradicted by Table 2's 100% corrupted-digits condition. This content-only perturbation (digits replaced with random values, structure untouched) collapses average accuracy from 66.3 (Correct) to 2.7—a 63.6-point drop, the largest in the entire table. For comparison, the most destructive structural perturbation (100% inserted steps) drops only 21.8 points (to 44.5). The paper dismisses the 100% case as 'severe corruption' that leads to nonsensical statements, but the claim as stated is universal: if content truly had minimal impact, entirely wrong numeric content should not destroy learning. The finding suggests that the 'structure not content' conclusion is an artifact of comparing modest content perturbations (20–70% corruption, keyword removal, wrong answers that still preserve local coherence) against structural perturbations applied at 33–100%. At 100% content corruption, the training signal becomes meaningless, and the model cannot even imitate the output format. This is an internal inconsistency, not merely an unvalidated assumption about perturbation cleanliness: the paper's own strongest content perturbation shows content can dominate. The abstract and conclusion therefore overstate the result and need qualification (e.g., 'content matters little when traces remain locally coherent').","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports that supervised fine-tuning (SFT) of Qwen2.5-32B-Instruct on 17k Long CoT traces distilled from DeepSeek R1 produces substantial gains on math and coding benchmarks, and that LoRA fine-tuning achieves comparable results. The central scientific claim is that the global structure of Long CoT traces—the ordering and logical coherence of reasoning steps—is what matters for learning, while the local content of individual steps has minimal impact. This claim is supported by perturbation experiments on QwQ-32B-Preview traces: replacing final answers with wrong ones, corrupting 20-70% of digits, and removing reasoning keywords cause small accuracy drops, whereas shuffling, deleting, or inserting whole reasoning steps causes larger drops. Additional ablations cover data scale, student model architectures, non-reasoning benchmarks, and comparison to Best-of-N sampling.","tokens_in":17732,"tokens_out":4794,"duration_ms":39179,"significance":"If the structure-versus-content claim is valid, it has practical implications: data collection for reasoning distillation could focus on logical coherence rather than correctness of every intermediate step, and parameter-efficient fine-tuning could be more widely used. The paper contributes an open, reproducible study with released code and a clear experimental design. However, the central claim is currently overstated and internally contradicted by the 100% digit-corruption condition, and the absence of uncertainty quantification makes the quantitative comparisons hard to evaluate. With a careful re-scoping of the claim and additional statistical support, this would be a useful empirical contribution to the understanding of reasoning distillation.","major_comments":[{"comment":"The 100% corrupted-digits condition is a content-only perturbation that reduces average accuracy from 66.3 (Correct) to 2.7, a 63.6-point drop that is the largest in the entire table. Yet the abstract and Section 4.1 conclude that 'the content of individual reasoning steps has minimal impact.' This is an internal inconsistency in the manuscript's central claim. The claim should be re-scoped to apply to locally coherent content (e.g., 20-70% digit corruption, wrong final answers, keyword removal) and the exceptional 100% case should be explicitly reconciled with the stated conclusion.","section":"4.1, Table 2"},{"comment":"All perturbation results come from single training runs with no confidence intervals, repeated seeds, or significance tests. Because AIME 2024 contains only 30 problems, the key differences around 3-10 percentage points (e.g., Correct 40.0 vs Wrong Answers 30.0, or Correct 40.0 vs 70% corrupted digits 30.0) are within the sampling error of roughly ±9 percentage points at 95% confidence. The 'little impact' and 'significantly degrade' conclusions need uncertainty estimates, multiple seeds, or a larger evaluation set to be convincing.","section":"4, Table 2"},{"comment":"The structural perturbation pipeline relies on Llama-3.3-70B-Instruct to segment reasoning traces into distinct steps, but the paper provides no validation of segmentation quality. If step boundaries are inaccurate, the delete/insert/shuffle operations may also alter local content or introduce distributional shifts (e.g., cross-domain topic changes in the insertion condition) rather than purely global ordering. The shuffle condition is the cleanest test, but its 100% drop (to 56.1) is still much smaller than the 100% digit-corruption drop (to 2.7), so the paper needs to either validate the segmentation or address the confound before concluding that global structure, not content, is the dominant factor.","section":"4.2"}],"minor_comments":[{"comment":"The phrase 'difficulty higher than Leval 3' contains a typo; it should read 'Level 3'.","section":"3.1"},{"comment":"Model names are inconsistent: 'Qwen2.5-7-Base' vs 'Qwen2.5-7B-Base' and 'Llama-8b-Base' vs 'Llama-3.1-8B' appear in different places; please standardize them.","section":"5.2, Figure 4"},{"comment":"The o1-preview AMC 2023 score is reported as 87.5 in Table 1 but as 85.0 in Figure 1b; these values should be reconciled.","section":"Table 1 vs Figure 1b"},{"comment":"'Table 4 summaries the statistics' should read 'summarizes the statistics'.","section":"Table 4 caption"},{"comment":"'the modelcan achieve' contains a missing space between 'model' and 'can'.","section":"1, paragraph 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is the academic write-up of the authors' Sky-T1-32B-Preview release, and the experiments are largely self-contained. The main obstacle is the internal contradiction in the central claim: the 100% digit-corruption result directly undercuts the 'content has minimal impact' statement, and the single-run benchmark numbers on a 30-problem test set are too noisy to support the fine-grained comparisons. These issues are fixable within the manuscript's scope by re-scoping the claim, adding uncertainty quantification, and possibly adding a segmentation validation. The paper's empirical findings on data- and parameter-efficient reasoning distillation are of interest to the community, so I recommend a major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's genuinely new bit is the controlled perturbation study: train on traces where you corrupt content (wrong answers, random digits, removed keywords) versus structure (shuffle, delete, insert steps). The finding that moderate content corruption is tolerated while structural corruption hurts is interesting, and the paper supports it reasonably well. The SFT/LoRA distillation recipe itself overlaps with concurrent work (Min et al., Huang et al.) and the authors say so; the contribution is the analysis angle plus the scale of ablations.\n\nWhat the paper does well: the results are clear and the code/data are public. Showing that 17k traces can push Qwen2.5-32B-Instruct to 56.7% on AIME 2024 and that LoRA (r=256) gets there too is practically useful. The ablations across model sizes and teacher models are solid. The appendix gives the system prompt and keyword lists, which helps reproducibility.\n\nThe soft spot is the headline claim. Table 2's 100% digit-corruption row collapses average accuracy from 66.3 to 2.7 — the largest drop in the table, bigger than any structural perturbation. The paper mentions this but dismisses it as 'severe corruption.' That's not enough: if content truly had minimal impact, entirely wrong numbers shouldn't destroy learning. The honest conclusion is that content matters little when steps remain locally coherent; at 100% corruption the local coherence is gone, so the content/structure dichotomy isn't clean. The abstract and conclusion need this qualification.\n\nOther issues: single runs, no confidence intervals, and AIME 2024 has only 30 problems, so 3-10 point gaps are likely within noise. The step segmentation via Llama-3.3-70B is not validated, and inserted steps from other problems can shift topic distribution, so the structural perturbations aren't purely structural. These are addressable and don't break the central distillation result.\n\nWho this is for: anyone doing SFT-based reasoning distillation or data filtering for reasoning models. The recipe is cheap to reproduce. I'd send it to peer review, but require the authors to qualify the structure-not-content claim and to add error bars or multiple seeds if feasible.\n\nRegards.","headline":"A practical distillation recipe with a fresh perturbation study, but the 'structure not content' claim is overstated by the paper's own 100% digit-corruption result.","tokens_in":18295,"tokens_out":2805,"would_cite":true,"duration_ms":23701,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A large language model can learn long reasoning from a few thousand demonstrations, and what transfers is the structure of the reasoning trace, not the correctness of its individual steps.","keywords":["long chain-of-thought","reasoning distillation","supervised fine-tuning","LoRA","reasoning structure vs content","math reasoning","code reasoning","data-efficient training"],"falsifier":"Run the same 17k-trace fine-tune with traces whose steps are fully shuffled, but give the model an auxiliary training objective that lets it reconstruct the original ordering; if accuracy recovers to the correct-order level, the shuffle loss is an artifact of input format rather than evidence about learned structure.","tokens_in":17259,"feed_emoji":"🧠","tokens_out":5606,"duration_ms":45513,"temperature":0.7,"pith_summary":"The paper argues that the ability to produce long, self-reflective chains of thought—the kind behind the strongest reasoning models—can be distilled into an ordinary instruction-tuned LLM with very little data and very few updated parameters. It further claims that what transfers is not the content of individual reasoning steps but their global logical structure: training on traces with wrong answers or corrupted numbers barely hurts, while shuffling, deleting, or inserting steps sharply degrades downstream accuracy. If right, this means reasoning skill is largely a structural pattern that can be installed cheaply, and that data curation for reasoning models should focus on preserving coherent multi-step trajectories rather than on step-level correctness. The paper supports this with a 17k-sample fine-tune of Qwen2.5-32B-Instruct reaching 56.7% on AIME 2024 and 57.0% on LiveCodeBench, competitive with the proprietary o1-preview model.","feed_headline":"17,000 traces turn a 32B LLM into an o1-class reasoner","feed_subtitle":"Data-efficient fine-tuning matches o1-preview on math and code; step order matters more than step correctness.","key_machinery":"The central object is the Long CoT trace: a response with an explicit thought block containing reflection, backtracking, and self-validation steps followed by a concise solution block. The controlled study splits each trace into reasoning steps using an LLM segmenter, then applies three structural perturbations (shuffle, delete, insert) and three content perturbations (wrong answers, digit corruption, keyword removal). The argument's engine is the contrast between these perturbation families: content perturbations leave accuracy almost intact while structural perturbations degrade it monotonically with the fraction of damaged steps. This contrast is what carries the claim that structure, not content, is what the model learns.","core_discovery":"The paper's central claim is that a standard instruction-tuned LLM can acquire long-form reasoning by supervised fine-tuning on a modest set of long chain-of-thought traces, and that what it acquires is a global reasoning structure rather than the local content of individual steps. On the efficiency side, 17k traces from a strong reasoning teacher push Qwen2.5-32B-Instruct to 56.7% on AIME 2024 (from 16.7%) and 57.0% on LiveCodeBench, and LoRA with a rank of 64 matches full SFT. On the structure side, training on traces whose final answers are wrong loses only 3.2% average accuracy, corrupting up to 70% of digits costs 4.3%, and removing all reasoning keywords costs 3.3%, while shuffling 67% of steps costs 13.3% on AIME 2024 and deleting all steps drops accuracy to near the untrained base model. The author's interpretation is that the logical consistency of the multi-step trajectory—reflection, backtracking, self-validation—is the carrier of the learned capability.","pith_inferences":["If structure is what transfers, synthetic data generators could deliberately produce traces with wrong turns and corrections, even fabricated errors, as long as the correction structure stays coherent, making data curation far cheaper.","The same perturbation logic could serve as a diagnostic: measuring how much a model's performance drops under step-shuffling estimates how 'structural' its reasoning skill is.","Content-insensitivity hints at a potential failure mode: models may imitate the choreography of reflection without acquiring deeper computational content, which would surface on out-of-distribution problems requiring genuine step-level novelty.","A testable extension is to vary the granularity of step boundaries to find the minimal structural unit that must be preserved for successful transfer."],"forward_implications":["A reasoning model can be built from a standard instruction-tuned LLM with roughly 17k demonstrations, without reinforcement learning or large proprietary pipelines.","Parameter-efficient LoRA, updating fewer than 5% of parameters, suffices to match full fine-tuning, implying the acquired capability is largely structural rather than knowledge-intensive.","Data collection for reasoning models should prioritize global coherence of traces over step-level correctness: wrong answers and noisy digits are tolerable, while step order and completeness are not.","Distilled models retain most non-reasoning capabilities such as MMLU, ARC-C, and IEval, avoiding the drastic forgetting seen in specialized reasoning models.","Long CoT fine-tuning approximates the benefit of Best-of-16 sampling at inference time, offering a cheap substitute for test-time scaling."],"supporting_citations":[{"why":"Supplies DeepSeek-R1, the teacher model whose traces form the 17k distillation dataset.","marker":"Guo et al., 2025"},{"why":"Provides QwQ-32B-Preview, the alternative teacher used to generate the controlled perturbation datasets and additional distillation data.","marker":"Team, Q., 2024"},{"why":"Introduces Qwen2.5-32B-Instruct, the base student model fine-tuned throughout the paper.","marker":"Yang et al., 2024"},{"why":"Provides the LoRA method that enables parameter-efficient fine-tuning with rank 64 or 256.","marker":"Hu et al., 2021"},{"why":"Prior demonstration of data-efficient reasoning distillation and the source of the Long CoT system prompt adopted here.","marker":"Min et al., 2024"},{"why":"Documents OpenAI o1-preview, the proprietary baseline whose scores are compared against the fine-tuned models.","marker":"Jaech et al., 2024"},{"why":"Provides Llama-3.3-70B-Instruct, the model used to segment reasoning traces into steps for structural perturbations.","marker":"Dubey et al., 2024"},{"why":"Supplies the Numina-Math dataset from which the difficult math prompts for distillation are selected.","marker":"LI et al., 2024"}],"fun_headline_variants":["Structure, not content, teaches LLMs to reason","Step order beats step accuracy in LLM reasoning","17K traces give 32B LLM o1-class reasoning","Wrong answers fine - step structure drives LLM reasoning","LLM reasoning: structure matters, content barely does"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments assume that each perturbation changes only the intended axis—that splitting traces into steps is faithful, that inserted steps come from comparable distributions, and that single-run accuracy differences of a few points on 30-problem benchmarks are real signal rather than noise.","fun_headline_variants_meta":{"raw":{"variants":["Structure, not content, teaches LLMs to reason","Step order beats step accuracy in LLM reasoning","17K traces give 32B LLM o1-class reasoning","Wrong answers fine - step structure drives LLM reasoning","LLM reasoning: structure matters, content barely does"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000695,"raw_usage":{"total_tokens":3233,"prompt_tokens":1124,"completion_tokens":2109,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":740,"completion_tokens_details":{"reasoning_tokens":2031}},"tokens_in":740,"tokens_out":2109,"duration_ms":13756,"temperature":1.0,"reasoning_tokens":2031,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:56:43.913510+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 17k-trace fine-tune with traces whose steps are fully shuffled, but give the model an auxiliary training objective that lets it reconstruct the original ordering; if accuracy recovers to the correct-order level, the shuffle loss is an artifact of input format rather than evidence about learned structure.","supporting_citations":[],"review_version":1}