REVIEW 6 cited by
Not All Correct Answers Are Equal: Why Your Distillation Source Matters
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Distillation has emerged as a practical and effective approach to enhance the reasoning capabilities of open-source language models. In this work, we conduct a large-scale empirical study on reasoning data distillation by collecting verified outputs from three state-of-the-art teacher models-AM-Thinking-v1, Qwen3-235B-A22B, and DeepSeek-R1-on a shared corpus of 1.89 million queries. We construct three parallel datasets and analyze their distributions, revealing that AM-Thinking-v1-distilled data exhibits greater token length diversity and lower perplexity. Student models trained on each dataset are evaluated on reasoning benchmarks including AIME2024, AIME2025, MATH500, and LiveCodeBench. The model distilled from AM-Thinking-v1 consistently achieves the best performance (e.g., 84.3 on AIME2024, 72.2 on AIME2025, 98.4 on MATH500, and 65.9 on LiveCodeBench) and demonstrates adaptive output behavior-producing longer responses for harder tasks and shorter ones for simpler tasks. These findings highlight the value of high-quality, verified reasoning traces. We release the AM-Thinking-v1 and Qwen3-235B-A22B distilled datasets to support future research on open and high-performing reasoning-oriented language models. The datasets are publicly available on Hugging Face\footnote{Datasets are available on Hugging Face: \href{https://huggingface.co/datasets/a-m-team/AM-Thinking-v1-Distilled}{AM-Thinking-v1-Distilled}, \href{https://huggingface.co/datasets/a-m-team/AM-Qwen3-Distilled}{AM-Qwen3-Distilled}.}.
Forward citations
Cited by 6 Pith papers
-
Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs
A multi-agent framework reconstructs the evolutionary graph of post-training LLM datasets, revealing domain patterns like vertical refinement in math data and systemic issues like redundancy and benchmark contaminatio...
-
MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling
MiroThinker shows that scaling agent-environment interactions via reinforcement learning lets a 72B open-source model reach up to 81.9% on GAIA and approach commercial performance on research benchmarks.
-
Gold-Guided Programmatic Distillation for Financial Reasoning over Hybrid Tables and Text
Execution-verified programmatic distillation with gold derivations and iterative recovery trains a 7B student that outperforms its 72B teacher on TAT-QA.
-
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Perplexity-based scoring systematically understates teacher–student gaps in distilled hybrid LLMs and can reverse design-choice rankings; generation-based evaluation is the more reliable protocol.
-
NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research
NebulaExp reports an empirical post-training pipeline on Qwen3-8B that raises instruct scores from 55.01 to 61.85 and reasoning scores from 73.88 to 75.17 via curated data, SFT, GRPO RL, and OPD/MOPD distillation.
-
A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models
A hybrid-granularity FP8 training recipe achieves near-lossless accuracy versus BF16 on reasoning benchmarks with up to 22% faster training and 14% lower memory.
Discussion (0). Sign in to comment.