Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Choosing hard, strategy-diverse reasoning traces distills smaller AI models best, the paper argues, with 500k examples lifting Llama-3.1-8B from 29.0% to 48.8% on GPQA-Diamond.

desk verdict Scaling and diversity findings are likely real, but the missing decontamination check is a load-bearing gap that must be fixed before the headline numbers can be trusted. read the letter →

arxiv 2507.01921 v1 pith:7QYJMUUV submitted 2025-07-02 cs.CL

Yang Li , Youssef Emad , Karthik Padthe , Jack Lanchantin , Weizhe Yuan , Thao Nguyen , Jason Weston , Shang-Wen Li
show 3 more authors
Dong Wang Ilia Kulikov Xian Li
This is my paper · ORCID
classification cs.CL
keywords reasoningdistillationdataselectionsupervisedfine-tuningchain-of-thoughtsampleefficiencysystem-1system-2mixingSTEMbenchmarksNaturalThoughts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to determine which reasoning demonstrations from a strong teacher model most effectively transfer reasoning ability to a smaller student model through supervised fine-tuning. It builds NaturalThoughts, a pool of roughly 2.8M question-to-reasoning-trace pairs generated by DeepSeek-R1 from the NaturalReasoning question set, and systematically compares selection strategies: random sampling, difficulty proxies like trace length and teacher-model disagreement, and diversity proxies based on question topics, semantic embeddings, and annotated reasoning strategies. The central claim is that difficult examples that require more diverse reasoning strategies are more sample-efficient for transferring reasoning skills, and that scaling up to 100k–500k examples with such selection outperforms existing reasoning datasets like OpenThoughts and LIMO on STEM benchmarks. A sympathetic reader would care because this reverses the prevailing 'less is more' finding from math-focused distillation work and offers a scalable recipe for building small reasoning models from large ones.

What carries the argument

The load-bearing mechanism is the selection of SFT examples from a large pool of teacher-generated traces, annotated along three dimensions: question topic (from a 13-discipline taxonomy), meta-reasoning strategies (self-verification, backtracking, exploration, etc., identified by Llama-3.1-70B-Instruct), and verbosity (an integer 0–10 score). Difficulty proxies are (i) trace length, sampled with probability $p = (l/C)^{\tau}$ with $C=5000$ and $\tau=2.5$, and (ii) disagreement between DeepSeek-R1 and Llama-3.3-70B answers, judged by Llama-3.1-8B-Instruct. Diversity proxies are uniform sampling across topics, HDBSCAN clustering of Llama-3.1-8B-Instruct question embeddings, and filtering to examples with 4–8 unique reasoning strategies. The mixed System-1/System-2 distillation uses the disagreement signal to route hard questions to full-trace training and easy ones to final-answer-only training, with inference-time instruction prompts ('Think carefully...' vs 'Answer directly...') to steer the student's token budget.

What would settle it

A containment analysis matching NaturalThoughts training prompts (or nearest-neighbor variants, e.g. 13-gram or embedding similarity) against GPQA-Diamond, MMLU-Pro, and SuperGPQA evaluation items; if a non-trivial fraction of benchmark questions appear in near-duplicate form in the training pool, the reported gains from selection and scaling would be inflated by memorization rather than reasoning transfer.

Watch

Extended reading notes

Core claim

The paper claims that when distilling reasoning from a strong teacher into a smaller student, the choice of training examples matters along two axes—difficulty and diversity—and that both improve on random selection, which itself is a surprisingly strong baseline. Concretely, filtering for long reasoning chains, for disagreement between two teacher models (a proxy for hard questions), and for traces annotated as using a moderate number of distinct meta-reasoning strategies each outperforms random selection at 10k examples; the reasoning-strategy and disagreement filters remain competitive at 500k, where Llama-3.1-8B-Instruct reaches 48.8% on GPQA-Diamond versus 29.0% untrained and 46.3% for the DeepSeek-R1-Distill-Llama-8B reference model. The paper also claims that scaling to 500k examples keeps improving performance across GPQA-Diamond, MMLU-Pro, and SuperGPQA, contradicting the 'less is more' hypothesis from math-only distillation; this holds for Llama-3.1-8B, Qwen-2.5-7B, and Llama-3.3-70B student models, with the 70B student at 100k outperforming DeepSeek-R1-Distill-Llama-70B on three of four benchmarks. A separate contribution is a mixed System-1/System-2 distillation method that trains on final answers for easy questions and full reasoning traces for hard ones, enabling the student to trade accuracy against inference-time token budget by switching between thinking and non-thinking modes.

Load-bearing premise

The evaluation benchmarks are assumed to be uncontaminated: the training pool (NaturalReasoning) and the generated traces must not substantially contain the actual GPQA-Diamond, MMLU-Pro, or SuperGPQA questions, because the paper reports no decontamination check.

Editorial extensions

If this is right

  • Reasoning-data curation should track the teacher's reasoning strategies, not just the question source or topic diversity.
  • The 'less is more' conclusion from math-only distillation datasets (LIMO, S1K) does not transfer to broad STEM reasoning; scaling diverse traces to 500k gives consistent gains.
  • Difficulty-based mixing of full traces and final answers gives students a controllable accuracy-efficiency trade-off at inference time, improving on both pure System-1 and pure System-2 distillation.
  • A 70B student trained on 100k randomly selected NaturalThoughts traces can match or exceed a distilled 70B model trained on 800k non-public traces on general STEM benchmarks.
  • Fine-grained control over which reasoning primitives are distilled could complement RL-based post-training recipes for small reasoning models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The disagreement-based difficulty signal may generalize beyond the two specific teacher models used here: any pair of models with different reasoning styles could serve as a difficulty oracle for distillation.
  • The reasoning-strategy annotation (done by Llama-3.1-70B-Instruct) could itself be a bottleneck; a cheaper automatic classifier trained on those annotations would make the selection recipe reproducible at larger scale.
  • The paper's claim that diversity of reasoning strategies matters more than diversity of questions suggests that future datasets could be built by steering the teacher to explore diverse strategies per question, rather than by covering more questions.
  • If the reported gains partially reflect memorization (see the missing decontamination analysis), the central claim about what transfers reasoning would need to be re-tested on fresh benchmarks; a public release of the training/benchmark overlap analysis would settle this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces NaturalThoughts, a collection of reasoning traces generated by DeepSeek-R1 on questions from the authors' NaturalReasoning pool, and uses it to study data-selection strategies for supervised-finetuning distillation of reasoning capabilities. It reports three main findings: (i) scaling the training set size with random sampling yields steady performance gains, contrary to the 'Less is More' hypothesis; (ii) selecting examples that are difficult and that exercise diverse reasoning strategies is more sample-efficient than random selection, especially at small and intermediate scales; and (iii) mixing System-1 (final-answer-only) and System-2 (full-chain-of-thought) distillation, with difficulty-based assignment, improves inference-time accuracy-efficiency tradeoffs. The claims are evaluated with Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Llama-3.3-70B-Instruct student models on GPQA-Diamond, MATH-500, MMLU-Pro, and SuperGPQA.

Significance. If the results are robust, this is a useful systematic study of reasoning-data curation, providing practical guidance on scale, difficulty, diversity, and efficiency. The paper is thorough in several respects: it compares multiple selection criteria, includes ablations of clustering methods and strategy-count thresholds in the appendices, trains three student model families, and compares against existing datasets such as OpenThoughts, LIMO, and S1K. The mixed System-1/System-2 method is simple and could be broadly applicable. The main caveat is the absence of any contamination analysis between the web-sourced NaturalReasoning pool and the evaluation benchmarks, which is load-bearing for the central transfer-learning claims.

major comments (3)
  1. [§3.1, §4, §5] The paper never checks for overlap between NaturalReasoning, the 2.8M-question source pool, and the evaluation benchmarks GPQA-Diamond, MMLU-Pro, and SuperGPQA. The central claim is that the training traces transfer general reasoning skill rather than memorized content, and the selection-method ordering could be confounded if, for example, the 'Models Disagree' or 'Long' subsets preferentially contain near-duplicates of benchmark questions. Since the source pool is web-sourced, this is not a remote risk. Please report a decontamination analysis (e.g., exact match, n-gram containment, and embedding similarity) between each training subset and the evaluation sets, and show that the main conclusions hold after removing near-duplicates or that the contamination rate is negligible.
  2. [§5.1, Tables 1 and 2] The claim that difficulty- and strategy-based selection is more sample-efficient than random is supported at 10k and 100k, but the ordering reverses or ties at 500k: on GPQA-D with Llama, 'Models Disagree' scores 45.2 vs. 48.3 for random, and 'Reasoning Strategies' scores 48.6 vs. 48.3; with Qwen, 'Reasoning Strategies' at 500k scores 48.3 vs. 48.6 for random. The text says 'the gap becomes smaller when scaling up data size,' but at 500k it actually becomes negative or negligible. The paper should discuss this crossover explicitly and clarify whether the sample-efficiency claim is intended only for smaller budgets; as written, the abstract and Section 5.1 overstate the advantage at scale.
  3. [§4, Tables 1 and 2] MMLU-Pro and SuperGPQA are reported with N=1, so no error bars are available for two of the three benchmarks named in the abstract. Several key comparisons on these benchmarks involve differences of less than one point (e.g., Table 1, MMLU-Pro at 100k: 59.8 vs. 61.2; SuperGPQA at 100k: 31.2 vs. 31.8 vs. 32.2). Without repeated-seed estimates or confidence intervals, it is not possible to assess whether the reported ordering is significant. Please provide variance estimates for the main comparisons on these benchmarks, or restrict the strong comparative claims to GPQA-D and MATH-500.
minor comments (5)
  1. [§3.1 vs. §3.2] Section 3.1 states the taxonomy includes 13 top-level domains, while Section 3.2 says sampling is 'uniformly across all the 12 topic domains'; please correct the inconsistency.
  2. [§5.2] The text uses 'GQPA-D' in the sentence about Llama-3.3-70B; this is a typo for GPQA-D.
  3. [§5.1, Table 1] The sentence 'Models Disagree subset leads to the best average performance across all 10K filtered subsets' is correct only for the 10k scale; given the reversal at 500k, this phrasing should be qualified to avoid overgeneralization.
  4. [§3.2 and Appendix C] The length-sampling parameters C and τ are chosen without sensitivity analysis; Appendix C ablates the reasoning-strategy count but not the length-sampling hyperparameters. A short sensitivity study (or a justification that the conclusions are robust to C and τ) would strengthen the difficulty-selection claims.
  5. [General] The paper does not include a data or code availability statement. Since the decontamination question is central, readers would benefit from at least a description of whether NaturalThoughts and the filtering code will be released.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: selection-method comparisons rest on external benchmarks and independent baselines; the missing decontamination analysis is a containment risk, not a by-construction circularity.

full rationale

The paper's central claim—that selecting difficult examples with diverse reasoning strategies is more sample-efficient than random selection for distillation—is supported by controlled comparisons on fixed external benchmarks (GPQA-Diamond, MMLU-Pro, SuperGPQA, MATH-500). The selection criteria are defined independently of those benchmarks: difficulty is proxied by response length, model disagreement, and verbosity annotations; diversity is proxied by reasoning-strategy counts, topic taxonomy, and embedding clusters. None of these proxies is defined in terms of the reported evaluation numbers, so the observed performance ordering is not enforced by construction. The 'Models Disagree' selector uses Llama-3.1-8B-Instruct as a judge of agreement between DeepSeek-R1 and Llama-3.3-70B, which does reuse a model of the same architecture as the student, but the resulting subset is then evaluated on held-out external benchmarks; the selection definition does not contain the evaluation outcome. The paper sources questions from NaturalReasoning, a prior dataset by overlapping authors, but this citation is used as an input pool and testbed rather than as a claimed result; the Empirical comparisons against OpenThoughts, LIMO, S1K, and DeepSeek-distilled checkpoints give independent grounding. The paper does not report a decontamination or overlap check between NaturalReasoning and the evaluation benchmarks; that is a real benchmark-containment risk that could inflate absolute accuracies if near-duplicates exist, but it is an empirical validity concern about the training pool, not a construction-level circularity in the derivation. Therefore, under the evidence rules requiring an exhibited reduction or fitted parameter renamed as a prediction, no circular step is established.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on several hand-picked thresholds and on annotations produced by the authors' models. The most important unexamined assumption is the absence of overlap between the web-sourced NaturalReasoning pool and the evaluation benchmarks.

free parameters (5)
  • Rmin and Rmax = 4 and 8
    Thresholds on the number of annotated reasoning strategies used to downsample examples in the Reasoning Strategies selection (Section 3.2). Chosen based on the distribution in Figure 7, not from an independent criterion.
  • Length sampling C and tau = C=5000, tau=2.5
    Controls how heavily short reasoning traces are downsampled in the Long subset (Section 3.2). Values are set by hand and no sensitivity sweep is reported in the main text.
  • System-2 word budget K = 3500 tokens
    Used for the Think instruction at inference time; set to the average System-2 response length in the training set (Section 3.3).
  • Random mixing probabilities p_System2 = 0.2, 0.4, 0.6
    Ratios of full System-2 examples mixed with System-1 examples in the random mixing experiments (Section 3.3).
  • Semantic clustering method and cluster count = Llama-3.1-8B embeddings with HDBSCAN (991 clusters)
    Selected in Appendix B as the best among the tested configurations for the semantic embedding diversity subset; the choice is data-dependent.
assumptions (5)
  • domain assumption LLM-annotated reasoning strategies and verbosity scores are reliable.
    The Reasoning Strategies and Verbosity selection methods in Section 3.2 depend on annotations from Llama-3.1-70B-Instruct; no human agreement or accuracy check is reported.
  • domain assumption NaturalReasoning is a diverse and representative source of general reasoning questions.
    The paper builds all training data by sampling from NaturalReasoning (Section 3), citing the authors' own prior work (Yuan et al. 2025) as justification.
  • domain assumption DeepSeek-R1 teacher traces are high-quality demonstrations.
    NaturalThoughts is generated by DeepSeek-R1 (Section 3) without independent verification of the reasoning steps beyond the final answer.
  • domain assumption Model disagreement is a valid proxy for question difficulty.
    The Models Disagree subset uses disagreement between DeepSeek-R1 and Llama-3.3-70B, judged by Llama-3.1-8B-Instruct, as a difficulty signal (Section 3.2).
  • domain assumption Evaluation data is not contained in the training pool.
    No decontamination analysis is reported; the validity of the benchmark comparisons in Section 5 depends on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks." pith.science (2026). https://pith.science/paper/7QYJMUUV

@misc{pith2026250701921,
  author       = {Pith},
  title        = {Pith review of: NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7QYJMUUV}},
  note         = {Machine review of arXiv:2507.01921}
}
read the original abstract

Recent work has shown that distilling reasoning traces from a larger teacher model via supervised finetuning outperforms reinforcement learning with the smaller student model alone (Guo et al. 2025). However, there has not been a systematic study of what kind of reasoning demonstrations from the teacher are most effective in improving the student model's reasoning capabilities. In this work we curate high-quality "NaturalThoughts" by selecting reasoning traces from a strong teacher model based on a large pool of questions from NaturalReasoning (Yuan et al. 2025). We first conduct a systematic analysis of factors that affect distilling reasoning capabilities, in terms of sample efficiency and scalability for general reasoning tasks. We observe that simply scaling up data size with random sampling is a strong baseline with steady performance gains. Further, we find that selecting difficult examples that require more diverse reasoning strategies is more sample-efficient to transfer the teacher model's reasoning skills. Evaluated on both Llama and Qwen models, training with NaturalThoughts outperforms existing reasoning datasets such as OpenThoughts, LIMO, etc. on general STEM reasoning benchmarks including GPQA-Diamond, MMLU-Pro and SuperGPQA.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Distilling an 8B reasoning teacher into a 0.6B student recovers most summary quality at ~50× speed, but teacher type—not scale alone—determines which capabilities transfer.

  2. A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Only gradient-based (LESS) representations make subset-to-query distance a reliable predictor of instruction-tuning performance; greedy round-robin helps most at small budgets, and random selection is surprisingly com...

Reference graph

Works this paper leans on

30 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    On-policy distillation of language models: Learning from self-generated mistakes, 2023

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes, 2023

    work page 2023

  3. [3]

    L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025. https://arxiv.org/abs/2503.04697

    arXiv 2025

  4. [4]

    fairseq2, 2023

    Can Balioglu, Martin Gleize, Artyom Kozhevnikov, Ilia Kulikov, Tuan Tran, and Julien Yao. fairseq2, 2023. http://github.com/facebookresearch/fairseq2

    work page 2023

  5. [5]

    Llama-nemotron: Efficient reasoning models, 2025

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Ziji...

    arXiv 2025

  6. [6]

    Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025. https://arxiv.org/abs/2412.21187

    arXiv 2025

  7. [7]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739, 2025

    arXiv 2025

  8. [8]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. https://github.com/huggingface/open-r1

    work page 2025

Show all 30 references
  1. [9]

    Simcse: Simple contrastive learning of sentence embeddings, 2022

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings, 2022. https://arxiv.org/abs/2104.08821

    2022 arXiv

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  3. [11]

    Openthoughts: Data recipes for reasoning models

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint arXiv:2506.04178, 2025

    2025 arXiv

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  6. [14]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  7. [15]

    Overthink: Slowdown attacks on reasoning llms

    Abhinav Kumar, Jaechul Roh, Ali Naseh, Marzena Karpinska, Mohit Iyyer, Amir Houmansadr, and Eugene Bagdasarian. Overthink: Slowdown attacks on reasoning llms. arXiv e-prints, pages arXiv--2502, 2025

    2025

  8. [16]

    Retro-search: Exploring untaken paths for deeper and efficient reasoning, 2025

    Ximing Lu, Seungju Han, David Acuna, Hyunwoo Kim, Jaehun Jung, Shrimai Prabhumoye, Niklas Muennighoff, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, and Yejin Choi. Retro-search: Exploring untaken paths for deeper and efficient reasoning, 2025. https://arxiv.org/abs/2504.04383

    2025 arXiv

  9. [17]

    Reasoning models can be effective without thinking, 2025

    Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking, 2025. https://arxiv.org/abs/2504.09858

    2025 arXiv

  10. [18]

    hdbscan: Hierarchical density based clustering

    Leland McInnes, John Healy, and Steve Astels. hdbscan: Hierarchical density based clustering. The Journal of Open Source Software, 2 0 (11): 0 205, 2017

    2017

  11. [19]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. https://arxiv.org/abs/2501.19393

    2025 arXiv

  12. [20]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA : A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. https://openreview.net/forum?id=Ti67584b98

    2024

  13. [21]

    Climbing the ladder of reasoning: What llms can-and still can't-solve after sft?, 2025

    Yiyou Sun, Georgia Zhou, Hao Wang, Dacheng Li, Nouha Dziri, and Dawn Song. Climbing the ladder of reasoning: What llms can-and still can't-solve after sft?, 2025. https://arxiv.org/abs/2504.11741

    2025

  14. [22]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020. https://arxiv.org/abs/2002.10957

    2020 arXiv

  15. [23]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024

    2024 arXiv

  16. [24]

    Thoughts are all over the place: On the underthinking of o1-like llms, 2025

    Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of o1-like llms, 2025. https://arxiv.org/abs/2501.18585

    2025 arXiv

  17. [25]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  18. [26]

    Limo: Less is more for reasoning, 2025

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025. https://arxiv.org/abs/2502.03387

    2025 arXiv

  19. [28]

    Distilling system 2 into system 1, 2024 b

    Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1, 2024 b . https://arxiv.org/abs/2407.06023

    2024 arXiv

  20. [29]

    Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions

    Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, et al. Naturalreasoning: Reasoning in the wild with 2.8 m challenging questions. arXiv preprint arXiv:2502.13124, 2025. Creative Commons Attributio...

    2025

  21. [30]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025. https://arxiv.org/abs/2504.13837

    2025 arXiv

  22. [31]

    Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning, 2024

    Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning, 2024. https://arxiv.org/abs/2402.04833

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.