{"id":"43b079e4-3f84-4ce5-8f0d-479f71272d16","arxiv_id":"2507.01921","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Selecting difficult, strategy-diverse reasoning traces from DeepSeek-R1 on NaturalReasoning questions improves supervised distillation of reasoning into 7B-70B student models.","lead":"This paper studies which kinds of AI reasoning examples, drawn from a large pool of questions, best teach smaller models to reason when fine-tuned on a larger model's step-by-step traces. It finds that simply using more randomly selected examples is a strong baseline, and that picking harder questions that require a wider variety of reasoning strategies works even better per sample.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never checks overlap between NaturalReasoning and the evaluation benchmarks; if training traces contain near-duplicates of GPQA/MMLU-Pro/SuperGPQA questions, the reported transfer gains and the selection-method ranking are inflated. This is the load-bearing gap.","rationale":"Reviewing the full manuscript, the weakest step is exactly where the reader placed it: the unexamined overlap between the NaturalReasoning question pool and the three evaluation benchmarks. The abstract and Section 5.1 claim a general transfer of reasoning skills and a sample-efficiency ordering of selection methods. If NaturalReasoning contains questions drawn from the same sources as GPQA-Diamond, MMLU-Pro, or SuperGPQA, the SFT gains in Tables 1-3 could be explained by memorization. This would not only reduce the absolute numbers; it could flip the selection-method ranking if contamination is concentrated in the 'difficult' or 'diverse' subsets. The paper does not mention even a basic decontamination heuristic, and no code or data is released, so the check cannot be performed by readers. This gap is more fundamental than the N=1 evaluation on two benchmarks, which affects confidence intervals but not the structural validity of the comparison. I therefore agree with the reader's weakest assumption. The conditional verdict remains appropriate: the empirical claims are plausible and internally consistent, but the transfer conclusion should not be accepted as definitive until the overlap analysis is performed. No change to the reader's verdict is needed.","tokens_in":25415,"tokens_out":3457,"duration_ms":42132,"concrete_test":"Ask the authors to release NaturalThoughts (or run internally) and perform near-duplicate containment against GPQA-Diamond, MMLU-Pro, and SuperGPQA: exact/substring 8-gram matching plus embedding retrieval (e.g., cosine similarity above 0.9) between each evaluation question and the training pool used in Tables 1-3. Report the number of evaluation items with a near-duplicate training example per subset (Random, Long, Models Disagree, Reasoning Strategies at 10k, 100k, and 500k). If any hits exist, rerun the main scaling and selection tables after removing overlapping training examples; the central claim survives only if the comparative gains and absolute improvements persist after deletion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that the NaturalThoughts subsets transfer reasoning skill to students, with selection by difficulty/diversity being more sample-efficient than random. This claim presumes that gains come from learning general reasoning rather than from memorizing benchmark-like items. NaturalReasoning, the 2.8M-question web-sourced pool from which all training traces are generated (Section 3.1), is not checked for overlap with the evaluation benchmarks GPQA-Diamond, MMLU-Pro, and SuperGPQA; the paper reports no decontamination or containment analysis in Sections 3, 4, or 5. Because training on 100k-500k examples yields large gains (e.g., Llama-3.1-8B GPQA-D accuracy rising from 29.0 to 48.8 at 500k), even a small fraction of near-duplicate evaluation questions could materially inflate the absolute numbers. More importantly, the selection-method ordering could be confounded: if the 'Models Disagree' or 'Long' subsets preferentially include contaminated hard questions, the paper's main conclusion about which selection criterion transfers reasoning would not follow. The absence of data release and overlap analysis leaves this unverifiable by readers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NaturalThoughts, a collection of reasoning traces generated by DeepSeek-R1 on questions from the authors' NaturalReasoning pool, and uses it to study data-selection strategies for supervised-finetuning distillation of reasoning capabilities. It reports three main findings: (i) scaling the training set size with random sampling yields steady performance gains, contrary to the 'Less is More' hypothesis; (ii) selecting examples that are difficult and that exercise diverse reasoning strategies is more sample-efficient than random selection, especially at small and intermediate scales; and (iii) mixing System-1 (final-answer-only) and System-2 (full-chain-of-thought) distillation, with difficulty-based assignment, improves inference-time accuracy-efficiency tradeoffs. The claims are evaluated with Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Llama-3.3-70B-Instruct student models on GPQA-Diamond, MATH-500, MMLU-Pro, and SuperGPQA.","tokens_in":25696,"tokens_out":6384,"duration_ms":70429,"significance":"If the results are robust, this is a useful systematic study of reasoning-data curation, providing practical guidance on scale, difficulty, diversity, and efficiency. The paper is thorough in several respects: it compares multiple selection criteria, includes ablations of clustering methods and strategy-count thresholds in the appendices, trains three student model families, and compares against existing datasets such as OpenThoughts, LIMO, and S1K. The mixed System-1/System-2 method is simple and could be broadly applicable. The main caveat is the absence of any contamination analysis between the web-sourced NaturalReasoning pool and the evaluation benchmarks, which is load-bearing for the central transfer-learning claims.","major_comments":[{"comment":"The paper never checks for overlap between NaturalReasoning, the 2.8M-question source pool, and the evaluation benchmarks GPQA-Diamond, MMLU-Pro, and SuperGPQA. The central claim is that the training traces transfer general reasoning skill rather than memorized content, and the selection-method ordering could be confounded if, for example, the 'Models Disagree' or 'Long' subsets preferentially contain near-duplicates of benchmark questions. Since the source pool is web-sourced, this is not a remote risk. Please report a decontamination analysis (e.g., exact match, n-gram containment, and embedding similarity) between each training subset and the evaluation sets, and show that the main conclusions hold after removing near-duplicates or that the contamination rate is negligible.","section":"§3.1, §4, §5"},{"comment":"The claim that difficulty- and strategy-based selection is more sample-efficient than random is supported at 10k and 100k, but the ordering reverses or ties at 500k: on GPQA-D with Llama, 'Models Disagree' scores 45.2 vs. 48.3 for random, and 'Reasoning Strategies' scores 48.6 vs. 48.3; with Qwen, 'Reasoning Strategies' at 500k scores 48.3 vs. 48.6 for random. The text says 'the gap becomes smaller when scaling up data size,' but at 500k it actually becomes negative or negligible. The paper should discuss this crossover explicitly and clarify whether the sample-efficiency claim is intended only for smaller budgets; as written, the abstract and Section 5.1 overstate the advantage at scale.","section":"§5.1, Tables 1 and 2"},{"comment":"MMLU-Pro and SuperGPQA are reported with N=1, so no error bars are available for two of the three benchmarks named in the abstract. Several key comparisons on these benchmarks involve differences of less than one point (e.g., Table 1, MMLU-Pro at 100k: 59.8 vs. 61.2; SuperGPQA at 100k: 31.2 vs. 31.8 vs. 32.2). Without repeated-seed estimates or confidence intervals, it is not possible to assess whether the reported ordering is significant. Please provide variance estimates for the main comparisons on these benchmarks, or restrict the strong comparative claims to GPQA-D and MATH-500.","section":"§4, Tables 1 and 2"}],"minor_comments":[{"comment":"Section 3.1 states the taxonomy includes 13 top-level domains, while Section 3.2 says sampling is 'uniformly across all the 12 topic domains'; please correct the inconsistency.","section":"§3.1 vs. §3.2"},{"comment":"The text uses 'GQPA-D' in the sentence about Llama-3.3-70B; this is a typo for GPQA-D.","section":"§5.2"},{"comment":"The sentence 'Models Disagree subset leads to the best average performance across all 10K filtered subsets' is correct only for the 10k scale; given the reversal at 500k, this phrasing should be qualified to avoid overgeneralization.","section":"§5.1, Table 1"},{"comment":"The length-sampling parameters C and τ are chosen without sensitivity analysis; Appendix C ablates the reasoning-strategy count but not the length-sampling hyperparameters. A short sensitivity study (or a justification that the conclusions are robust to C and τ) would strengthen the difficulty-selection claims.","section":"§3.2 and Appendix C"},{"comment":"The paper does not include a data or code availability statement. Since the decontamination question is central, readers would benefit from at least a description of whether NaturalThoughts and the filtering code will be released.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The decontamination omission is the most serious issue. Because the source pool is the authors' own web-scraped dataset and the evaluation benchmarks are public, the absence of any overlap analysis makes the transfer claims unverifiable. I would ask the authors to add a containment analysis and, if any overlap is found, re-run the key experiments on cleaned training data. The reversal of the selection-method ordering at 500k should also be addressed before the paper is accepted. The paper is otherwise well-structured and the experiments are extensive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a solid empirical study of distillation data selection, and I think the main findings will hold up—but the missing decontamination analysis is a genuine gap that needs to be fixed before the headline numbers can be trusted.\n\nWhat's actually new: unlike LIMO and S1K, which argue less is more, this paper shows that scaling up random sampling from NaturalThoughts continues to improve GPQA, MMLU-Pro, and SuperGPQA performance even at 500k examples, and that selecting for reasoning-strategy diversity beats question diversity. That's a real result, not just a combination of known tricks. The systematic comparison across multiple selection axes—random, length, verbosity, model agreement, semantic clusters—is the paper's strength. The mixed System-1/System-2 distillation section is also a useful practical contribution: difficulty-based mixing gives a better accuracy-efficiency tradeoff than random mixing, and it's a clean experiment.\n\nThe soft spots are real but not fatal on their own. Two of the four benchmarks are reported with N=1 and no error bars, which matters because some of the selection-method differences are small. The free parameters—length sampling constants, verbosity thresholds, strategy-count cutoffs—are set post hoc on the dataset, so the paper is better read as an empirical recipe than a theory. And there's no data release or code, which makes replication hard.\n\nBut the load-bearing gap is the complete absence of a decontamination analysis. NaturalReasoning is a 2.8M-question web-sourced pool, and the evaluation benchmarks—GPQA, MMLU-Pro, SuperGPQA—are also web-sourced. The paper never checks for overlap. If even a small fraction of training traces contain near-duplicates of evaluation questions, the scaling gains and the selection ranking would be inflated. This isn't a minor omission; it's central to the claim that the model is learning transferable reasoning rather than memorizing benchmark items. The authors need to report a containment analysis, ideally with exact/near-dup matching, and show the main results after removing overlapping items.\n\nIf that analysis comes back clean, this is a strong paper worth citing. If not, the conclusions need to be substantially revised. As it stands, I'd send it to review with the decontamination check as a required revision, not a desk reject. It's a serious piece of empirical work that the community will engage with.","headline":"Scaling and diversity findings are likely real, but the missing decontamination check is a load-bearing gap that must be fixed before the headline numbers can be trusted.","tokens_in":26232,"tokens_out":2350,"would_cite":true,"duration_ms":25989,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Choosing hard, strategy-diverse reasoning traces distills smaller AI models best, the paper argues, with 500k examples lifting Llama-3.1-8B from 29.0% to 48.8% on GPQA-Diamond.","keywords":["reasoning distillation","data selection","supervised fine-tuning","chain-of-thought","sample efficiency","system-1 system-2 mixing","STEM reasoning benchmarks","NaturalThoughts"],"falsifier":"A containment analysis matching NaturalThoughts training prompts (or nearest-neighbor variants, e.g. 13-gram or embedding similarity) against GPQA-Diamond, MMLU-Pro, and SuperGPQA evaluation items; if a non-trivial fraction of benchmark questions appear in near-duplicate form in the training pool, the reported gains from selection and scaling would be inflated by memorization rather than reasoning transfer.","tokens_in":25249,"feed_emoji":"🧠","tokens_out":3915,"duration_ms":33736,"temperature":0.7,"pith_summary":"The paper sets out to determine which reasoning demonstrations from a strong teacher model most effectively transfer reasoning ability to a smaller student model through supervised fine-tuning. It builds NaturalThoughts, a pool of roughly 2.8M question-to-reasoning-trace pairs generated by DeepSeek-R1 from the NaturalReasoning question set, and systematically compares selection strategies: random sampling, difficulty proxies like trace length and teacher-model disagreement, and diversity proxies based on question topics, semantic embeddings, and annotated reasoning strategies. The central claim is that difficult examples that require more diverse reasoning strategies are more sample-efficient for transferring reasoning skills, and that scaling up to 100k–500k examples with such selection outperforms existing reasoning datasets like OpenThoughts and LIMO on STEM benchmarks. A sympathetic reader would care because this reverses the prevailing 'less is more' finding from math-focused distillation work and offers a scalable recipe for building small reasoning models from large ones.","feed_headline":"Hard, diverse reasoning traces beat 'less is more' for distillation","feed_subtitle":"A 500k-example pool of teacher traces lifts Llama-3.1-8B to 48.8% on GPQA-Diamond—if the benchmark isn't in the training data.","key_machinery":"The load-bearing mechanism is the selection of SFT examples from a large pool of teacher-generated traces, annotated along three dimensions: question topic (from a 13-discipline taxonomy), meta-reasoning strategies (self-verification, backtracking, exploration, etc., identified by Llama-3.1-70B-Instruct), and verbosity (an integer 0–10 score). Difficulty proxies are (i) trace length, sampled with probability $p = (l/C)^{\\tau}$ with $C=5000$ and $\\tau=2.5$, and (ii) disagreement between DeepSeek-R1 and Llama-3.3-70B answers, judged by Llama-3.1-8B-Instruct. Diversity proxies are uniform sampling across topics, HDBSCAN clustering of Llama-3.1-8B-Instruct question embeddings, and filtering to examples with 4–8 unique reasoning strategies. The mixed System-1/System-2 distillation uses the disagreement signal to route hard questions to full-trace training and easy ones to final-answer-only training, with inference-time instruction prompts ('Think carefully...' vs 'Answer directly...') to steer the student's token budget.","core_discovery":"The paper claims that when distilling reasoning from a strong teacher into a smaller student, the choice of training examples matters along two axes—difficulty and diversity—and that both improve on random selection, which itself is a surprisingly strong baseline. Concretely, filtering for long reasoning chains, for disagreement between two teacher models (a proxy for hard questions), and for traces annotated as using a moderate number of distinct meta-reasoning strategies each outperforms random selection at 10k examples; the reasoning-strategy and disagreement filters remain competitive at 500k, where Llama-3.1-8B-Instruct reaches 48.8% on GPQA-Diamond versus 29.0% untrained and 46.3% for the DeepSeek-R1-Distill-Llama-8B reference model. The paper also claims that scaling to 500k examples keeps improving performance across GPQA-Diamond, MMLU-Pro, and SuperGPQA, contradicting the 'less is more' hypothesis from math-only distillation; this holds for Llama-3.1-8B, Qwen-2.5-7B, and Llama-3.3-70B student models, with the 70B student at 100k outperforming DeepSeek-R1-Distill-Llama-70B on three of four benchmarks. A separate contribution is a mixed System-1/System-2 distillation method that trains on final answers for easy questions and full reasoning traces for hard ones, enabling the student to trade accuracy against inference-time token budget by switching between thinking and non-thinking modes.","pith_inferences":["The disagreement-based difficulty signal may generalize beyond the two specific teacher models used here: any pair of models with different reasoning styles could serve as a difficulty oracle for distillation.","The reasoning-strategy annotation (done by Llama-3.1-70B-Instruct) could itself be a bottleneck; a cheaper automatic classifier trained on those annotations would make the selection recipe reproducible at larger scale.","The paper's claim that diversity of reasoning strategies matters more than diversity of questions suggests that future datasets could be built by steering the teacher to explore diverse strategies per question, rather than by covering more questions.","If the reported gains partially reflect memorization (see the missing decontamination analysis), the central claim about what transfers reasoning would need to be re-tested on fresh benchmarks; a public release of the training/benchmark overlap analysis would settle this."],"forward_implications":["Reasoning-data curation should track the teacher's reasoning strategies, not just the question source or topic diversity.","The 'less is more' conclusion from math-only distillation datasets (LIMO, S1K) does not transfer to broad STEM reasoning; scaling diverse traces to 500k gives consistent gains.","Difficulty-based mixing of full traces and final answers gives students a controllable accuracy-efficiency trade-off at inference time, improving on both pure System-1 and pure System-2 distillation.","A 70B student trained on 100k randomly selected NaturalThoughts traces can match or exceed a distilled 70B model trained on 800k non-public traces on general STEM benchmarks.","Fine-grained control over which reasoning primitives are distilled could complement RL-based post-training recipes for small reasoning models."],"supporting_citations":[{"why":"Supplies the NaturalReasoning pool of 2.8M questions used as the prompt source for teacher-generated traces.","marker":"Yuan et al., 2025"},{"why":"Supplies the DeepSeek-R1 teacher model and the finding that distillation outperforms RL alone, which motivates the whole study.","marker":"Guo et al., 2025"},{"why":"LIMO is the 'less is more' baseline that the paper compares against and contradicts at scale.","marker":"Ye et al., 2025"},{"why":"S1K is the other small-scale curated distillation baseline and the source of the diversity-selection idea.","marker":"Muennighoff et al., 2025"},{"why":"OpenThoughts is the large-scale reasoning dataset baseline that NaturalThoughts is claimed to outperform at 100k–500k.","marker":"Guha et al., 2025"},{"why":"Provides the GPQA-Diamond benchmark, the primary evaluation of reasoning transfer.","marker":"Rein et al., 2024"},{"why":"Provides MMLU-Pro, one of the three STEM benchmarks used for evaluation.","marker":"Wang et al., 2024"},{"why":"Provides SuperGPQA and the question taxonomy used for topic annotation and evaluation.","marker":"Du et al., 2025"},{"why":"Supplies the method of distilling System 2 reasoning into System 1 responses, which the mixed distillation extends.","marker":"Yu et al., 2024a"},{"why":"Establishes the 'long is more' baseline in alignment data selection that the paper validates and then goes beyond for reasoning data.","marker":"Zhao et al., 2024"}],"fun_headline_variants":["Hard diverse reasoning traces beat random picks for distillation","Scale beats 'less is more' in reasoning-trace distillation","Selecting difficult, diverse teacher traces boosts student reasoning","Distill smarter: pick hard, varied reasoning examples","Why 'less is more' fails: scaling reasoning traces helps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation benchmarks are assumed to be uncontaminated: the training pool (NaturalReasoning) and the generated traces must not substantially contain the actual GPQA-Diamond, MMLU-Pro, or SuperGPQA questions, because the paper reports no decontamination check.","fun_headline_variants_meta":{"raw":{"variants":["Hard diverse reasoning traces beat random picks for distillation","Scale beats 'less is more' in reasoning-trace distillation","Selecting difficult, diverse teacher traces boosts student reasoning","Distill smarter: pick hard, varied reasoning examples","Why 'less is more' fails: scaling reasoning traces helps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1866,"prompt_tokens":1103,"completion_tokens":763,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":719,"completion_tokens_details":{"reasoning_tokens":684}},"tokens_in":719,"tokens_out":763,"duration_ms":9699,"temperature":1.0,"reasoning_tokens":684,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:39:28.728871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A containment analysis matching NaturalThoughts training prompts (or nearest-neighbor variants, e.g. 13-gram or embedding similarity) against GPQA-Diamond, MMLU-Pro, and SuperGPQA evaluation items; if a non-trivial fraction of benchmark questions appear in near-duplicate form in the training pool, the reported gains from selection and scaling would be inflated by memorization rather than reasoning transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GPQA-Diamond benchmark, the primary evaluation of reasoning transfer."}],"review_version":1}