REVIEW 4 major objections 3 minor 18 references
Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold
T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper claims that a deliberately constructed union of RL-trained teachers can be distilled into a single student that outperforms every individual teacher.
desk verdict The expand-then-compress idea is genuinely interesting, but the main distillation objective as written has zero expected gradient, so the compression results need a rewritten loss and stronger baselines before they can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a three-part loop. First, per-example coverage scores $s_i^{(r)}$ estimated from $J$ sampled responses per teacher define a coverage envelope $u_i^{(r)}=\max_{a\le r}s_i^{(a)}$, and examples with $u_i^{(r)}\le\tau_{\mathrm{drop}}$ form the residual training subset for the next teacher. Second, Reliability-Gated Teacher-Union OPD computes per-example reliable teacher sets $Q_i=\{r:s_i^{(r)}\ge\tau_{\mathrm{base}}\}$, softmax quality weights $\alpha_i^{(r)}$, and a sampled-token reverse-KL objective $L_{\mathrm{base}}$ that only queries teacher log-probabilities on tokens the student itself sampled. Third, Consensus-Residual Decomposition selects a winner teacher $r_i^\star$, builds a peer mixture from its reliable peers, and transfers, through $L_{\mathrm{res}}$, the winner's excess Top-$K$ token preferences when the winner beats its best peer by at least $\tau_{\mathrm{gap}}$; the total objective is $L_{\mathrm{total}}=L_{\mathrm{base}}+\lambda L_{\mathrm{res}}$.
What would settle it
Run the expansion stage with the same budget but different numbers of coverage samples, such as $J=4$, $J=16$, and $J=64$; if the residual subsets and final student scores change sharply with $J$, the coverage estimates are not stable enough to carry the mechanism. A positive control would be to train teachers on random subsets matched in size and difficulty to the residual subsets and check whether the residual subsets' union still beats the random union by the reported margin.
Extended reading notes
Core claim
The central claim is that stronger students can be obtained not by selecting the best teacher but by constructing a complementary teacher union and distilling the union rather than the average. The paper treats each RL-trained policy as a local probe of a multi-basin reasoning solution manifold, so different teachers can cover different solution modes even from the same initialization. Across math, code, and instruction following, the resulting single student exceeds every individual teacher on the primary aggregate metric of each domain, recovering part of the gap to a benchmark-wise teacher envelope that is not itself a deployable model. The paper further shows that coverage-aware residual allocation broadens the teacher union more than random data partitioning, and that the residual objective that preserves a winner teacher's excess token preferences over its reliable peers is what makes the compression succeed.
Load-bearing premise
The load-bearing premise is that per-example coverage scores estimated from $J=16$ sampled responses per teacher accurately separate examples the teacher union reliably solves from residual examples needing new solution modes; if those scores are noisy, the residual subsets and reliability gates become approximately random and the reported complementarity could be hard-example overfitting rather than manifold expansion.
Editorial extensions
If this is right
- If the central claim is right, the best single RL run should no longer be treated as the deployable artifact; the union of several runs is the object with transferable value.
- Residual GRPO turns teacher construction into a coverage problem, so later teachers are trained on shrinking subsets and can add new modes at lower marginal cost.
- Quality-weighted distillation with a residual objective outperforms uniform teacher mixing, so compression should preserve specialist preferences rather than only the consensus.
- The reported Qwen3-4B scaling result suggests the expand-then-compress benefit is not limited to one model size and may generalize to larger student policies.
Reading between the lines
- An implication left implicit is that the expand-then-compress cycle could be iterated: the distilled student itself could serve as the next common initialization, with a fresh residual round targeting modes still uncovered after compression.
- An extension worth testing is replacing the $J=16$ outcome-sample coverage estimates with a learned surrogate verifier; if noisy coverage is the bottleneck, cleaner scores should enlarge the gap between residual and random partitioning.
- A diagnostic the paper does not report is per-round difficulty or loss on residual subsets; if later residual rounds degenerate into hard-example overfitting, a difficulty cap on the residual subset would be the natural corrective.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an expand-then-compress framework for distilling a set of RL-trained reasoning teachers into a single student. In the expansion stage, Residual GRPO (RGRPO) trains successive teachers from a common initialization on examples not yet covered by the accumulated teacher union, where coverage is estimated from per-example success rates over 16 sampled responses. In the compression stage, Reliability-Gated Teacher-Union On-policy Distillation (TU-OPD) combines a base quality-weighted reverse-KL objective on student-sampled tokens with a Consensus-Residual Decomposition loss that transfers a winner teacher's excess token preferences over its reliable peers. Experiments on mathematical reasoning (AIME, HMMT, AMC23), code generation (LiveCodeBench v6), and instruction following (IFBench) with Qwen3-1.7B and Qwen3-4B report that the distilled student outperforms the strongest individual teacher on the primary aggregate metric of each domain, with relative gains of 2.0%, 8.3%, and 6.9% at the 1.7B scale. Ablations compare residual versus parallel teacher construction and uniform versus quality-weighted distillation objectives.
Significance. If the empirical claims hold, the expand-then-compress idea is valuable: it offers a concrete recipe for converting several specialized RL policies into a single deployable model that is stronger than any one teacher, with potentially broad applicability to reasoning, code, and instruction-following domains. The paper has notable strengths: the teacher-envelope analysis in Table 1 gives direct evidence of benchmark-level complementarity; the residual-versus-parallel comparison in Table 2 is a clean test of the expansion mechanism; the experiments span three domains and two model scales; and the writing is generally clear. The main weaknesses are technical and empirical: the base distillation objective in Eq. (11) appears to have zero expected gradient as written, the headline gains are small and unreplicated, and the comparisons lack single-best-teacher and compute-matched RL baselines. No code or data is provided, which limits reproducibility.
major comments (4)
- [§3.3, Eq. (11); §4.1] The base objective as written has zero expected gradient. For a fixed prefix h, the token y_t is sampled from π_θ(·|h), and the loss term log π_θ(y_t|h) − log π_r(y_t|h) has gradient ∇_θ log π_θ(y_t|h) with respect to the student parameters, since the teacher term is constant. The expectation of this gradient is ∑_y ∇_θ π_θ(y|h) = 0. Thus Eq. (11) is a zero-mean control variate, not a Monte Carlo estimator of the quality-weighted reverse KL; a correct REINFORCE-style estimator would require an additional (1 + log(π_θ/π_r)) ∇_θ log π_θ term or a reparameterized/surrogate objective. Because §4.1 states that the base objective is evaluated only on the token sampled by the student policy, this is not a mere notational ambiguity. Consequently, the compression gains, and especially the Table 3 ablations that remove L_res, cannot be explained by Eq. (11). If the implementation actually trains on teacher cross-entropy or a score-function surrogate, the paper must state this explicitly; otherwise the central compression claim is not supported by the described method.
- [§4.2, Table 1; §4.3, Table 3] The headline empirical claims rest on single-run, single-seed numbers with no error bars, multiple seeds, or significance tests. With eight sampled responses per problem, the 2.0% math-Mean gain (16.70 vs. 16.37) is within the range that could arise from sampling noise, and the ablation differences in Table 3 (e.g., 15.99 vs. 16.70, and 15.81 vs. 16.70) are likewise unreplicated. Please report multiple seeds or bootstrap confidence intervals for the main tables, and state explicitly whether the reported gains are consistent across seeds.
- [§4.2, §4.3] There is no single-best-teacher distillation baseline or compute-matched RL baseline. The title and central claim, 'beyond the best teacher,' require showing that the multi-teacher union distilled into a student outperforms a student distilled from the single best teacher with comparable compute and data, or a student trained directly with RL on the same training pool. Without such controls, the observed gains could be due to distillation from any teacher on the full data rather than to the expand-then-compress mechanism.
- [§3.2, Eqs. (5)–(7)] The residual construction relies on per-example success-rate estimates from J=16 samples per teacher, and the threshold τ_drop=0.5 is applied to these estimates. Binomial noise at J=16 is substantial: a teacher with a true per-example success rate of 0.6 has roughly 29% probability of being classified as residual, while one with a true rate of 0.4 has roughly 14% probability of being classified as covered. Please report the sensitivity of the residual subsets and downstream results to J (e.g., J=32 or 64) and to the threshold, or otherwise justify that the coverage estimates are accurate enough to support the 'residual region' interpretation.
minor comments (3)
- [§4.1, Eqs. (8)–(18)] Several hyperparameters used in the method are not specified, including τ_base in Eqs. (8)–(9), the peer temperature T_p in Eq. (16), and the margin γ in Eq. (18). Without these values the distillation setup is not fully reproducible.
- [§3.3, Eq. (11)] The notation for the expectation in Eq. (11) drops the per-example dependence of the teacher weights α_i^(r); the expression would be clearer as E_{x_i∼D_valid, y_i∼π_θ(·|x_i)} with explicit dependence on i retained throughout.
- [Table 1] The row labeled 'Avg. Residual-GRPO Teachers' averages benchmark-wise scores across teachers; this is not the same as the aggregate performance of an average teacher and could be misread. Consider labeling it explicitly as the teacher-average of per-benchmark scores.
Circularity Check
No circularity found: the expand-then-compress pipeline is self-contained against external benchmarks, and the reported student gains are not forced by construction from the teacher-coverage inputs.
full rationale
The paper’s derivation chain is not circular. In the expansion stage (Eqs. 4–7), Residual GRPO uses teacher-quality estimates s_i^(r) from Eq. 5 only to construct the next residual training subset D_{r+1}; this is training-data curation, not a fitted prediction of the reported benchmark results. In the compression stage (Eqs. 9–21), the reliability gates, quality weights, and residual objective are all defined in terms of teacher log-probabilities and student-sampled tokens; they specify a training loss, and none of the reported evaluation numbers (AIME, HMMT, AMC23, LiveCodeBench, IFBench) is an algebraic consequence of that loss. Hyperparameters such as tau_drop = 0.5, tau_base, tau_gap = 0.2, lambda = 0.05, and K = 8 are fixed by hand in Section 4.1, not fitted to the evaluation benchmarks. The central claim that the student outperforms the strongest individual teacher by 2.0%, 8.3%, and 6.9% is an external empirical comparison against held-out benchmarks. Citations to prior on-policy distillation work (Agarwal et al. 2024; Lu and Lab 2025; Li et al. 2026) provide background and are not load-bearing as self-citations or uniqueness theorems. The skeptical concern about Eq. 11 having zero expected gradient when optimized at the sampled token is an internal-consistency or correctness issue about whether the described objective trains as written, not a circularity in the sense of the output being equivalent to the input by construction. The scope limitation in the Conclusion (‘our current study focuses on verifiable tasks and a fixed teacher set’) is a statement of scope, not an admission of circularity.
Assumptions & free parameters
free parameters (8)
- tau_drop =
0.5
- tau_base
- tau_gap =
0.2
- lambda =
0.05
- K =
8
- teacher weighting temperature T =
0.25
- peer weighting temperature T_p
- J =
16
assumptions (4)
- domain assumption The outcome verifier R(x,y) is rule-based and reliably indicates correctness for every training and evaluation example.
- domain assumption A multi-basin reasoning solution manifold exists, and training on residual examples yields teachers that cover genuinely different valid solution modes.
- domain assumption Common initialization plus different training subsets makes teacher differences attributable to subset coverage rather than optimization randomness.
- standard math The sampled-token Monte Carlo estimator in Eq. (11) is a valid estimator of the quality-weighted reverse KL.
invented entities (1)
-
reasoning solution manifold
Cite this review
Pith. "Pith review of Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold." pith.science (2026). https://pith.science/paper/AETV3T3J
@misc{pith2026260727770,
author = {Pith},
title = {Pith review of: Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold},
year = {2026},
howpublished = {\url{https://pith.science/paper/AETV3T3J}},
note = {Machine review of arXiv:2607.27770}
}
read the original abstract
A single reinforcement-learning run can produce a strong reasoner yet an incomplete teacher: it often amplifies only a subset of the valid solution modes. We argue that reinforcement learning (RL)-trained policies should therefore be viewed as local probes of a multi-basin reasoning solution manifold, rather than as globally reliable supervisors. Based on this view, we propose an expand-then-compress framework that couples teacher construction with multi-teacher policy distillation. In the expansion stage, Residual Group Relative Policy Optimization (RGRPO) trains a sequence of teachers from a common initialization and redirects each later round toward examples not yet covered by the accumulated teacher union. In the compression stage, reliability-gated Teacher-Union On-policy Distillation (TU-OPD) lets the student learn from its own response prefixes. For each example, only reliable teachers contribute, and their sampled-token OPD losses are weighted by their per-example quality. We further introduce Consensus-Residual Decomposition, which preserves a winner teacher's excess token preferences over its reliable peers, preventing specialist behavior from being suppressed during teacher aggregation. Experiments on mathematical reasoning, code generation, and instruction following show that the resulting Qwen3-1.7B student consistently outperforms the strongest individual teacher across all three domains, yielding relative improvements of 2.0%, 8.3%, and 6.9%, respectively, while retaining single-model inference. These results establish a simple but powerful principle: stronger students can be obtained not by selecting a single better teacher, but by deliberately constructing and compressing a complementary teacher union.
Figures
Reference graph
Works this paper leans on
-
[1]
On-policy distillation of language models: Learning from self-generated mistakes
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InInternational Conference on Learning Representations, volume 2024, pages 21246–21263,
2024
-
[5]
Ruihan Jin, Pengpeng Shao, Zhengqi Wen, Jinyang Wu, Mingkuan Feng, Shuo Yang, Chu Yuan Zhang, and Jian- hua Tao. Exploring knowledge purification in multi-teacher knowledge distillation for llms.arXiv preprint arXiv:2602.01064, 2026a. Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. En...
-
[7]
Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,
-
[8]
Learning from committee: Reasoning distillation from a mixture of teachers with peer-review
Zhuochun Li, Yuelyu Ji, Rui Meng, and Daqing He. Learning from committee: Reasoning distillation from a mixture of teachers with peer-review. InFindings of the Association for Computational Linguistics: ACL 2025, pages 4190–4205,
work page 2025
-
[9]
https://thinkingmachines.ai/blog/on-policy-distillation
doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, et al. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training.arXiv preprint arXiv:2606.30406,
-
[10]
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand `es, and Tatsunori B Hashimoto. s1: Simple test-time scaling. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20286–20332,
work page 2025
-
[11]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,
-
[12]
Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256,
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256,
Show all 18 references
-
[13]
Self-supervised on-policy distillation for reasoning language models.arXiv preprint arXiv:2605.17497,
Zhiquan Tan and Yinrong Hong. Self-supervised on-policy distillation for reasoning language models.arXiv preprint arXiv:2605.17497,
-
[14]
Tip: Token importance in on-policy distillation.arXiv preprint arXiv:2604.14084,
Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation.arXiv preprint arXiv:2604.14084,
-
[15]
Qwen3 technical report.arXiv preprint arXiv:2505.09388,
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,
-
[16]
On-policy context distillation for language models
Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv preprint arXiv:2602.12275,
-
[17]
Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387,
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387,
-
[18]
Egad: Entropy-guided adaptive distillation for token-level knowledge transfer.arXiv preprint arXiv:2605.01732, 2026a
Hao Zhang, Zhibin Zhang, Guangxin Wu, Wanyi Ning, Jiafeng Guo, and Xueqi Cheng. Egad: Entropy-guided adaptive distillation for token-level knowledge transfer.arXiv preprint arXiv:2605.01732, 2026a. Xinsen Zhang, Zhenkai Ding, Tianjun Pan, Run Yang, Chun Kang, Xue Xiong, and Ji...
-
[2023]
Sead: Competence-aware on-policy distillation via entropy-guided supervision.arXiv preprint arXiv:2606.28562,
Chia-Hsuan Lee, Zelei Cheng, Yu Wang, Renkun Ni, Sambit Sahu, Shi-Xiong Zhang, and William Campbell. Sead: Competence-aware on-policy distillation via entropy-guided supervision.arXiv preprint arXiv:2606.28562,
-
[2024]
Openthoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178,
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178,
-
[2025]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,
-
[2026]
Skywork open reasoner 1 technical report.arXiv preprint arXiv:2505.22312, 2025a
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner 1 technical report.arXiv preprint arXiv:2505.22...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.