{"id":"7ac19379-23f2-4c1b-924d-1c08ef7fce70","arxiv_id":"2607.27770","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"An expand-then-compress pipeline that trains complementary teachers on residual examples and distills their union into one student yields a small model that beats each individual teacher on math, code, and instruction-following benchmarks.","lead":"This paper trains several AI reasoning models on different leftover parts of a problem set, then squeezes their combined skills into one smaller model. The result is a student model that outperforms each individual teacher on math, code, and instruction following, while running as a single model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 11's sampled-token reverse-KL objective has zero expected gradient when optimized as written, so the reported compression gains may not be attributable to the proposed TU-OPD base loss.","rationale":"The paper is proposing an expand-then-compress framework, and the central empirical claim is that the compressed student beats the strongest individual teacher. The expansion stage's J=16 coverage estimates are indeed a legitimate robustness concern, as the reader noted: noisy residual subsets could make the teacher complementarity partly an artifact. However, the compression stage is the mechanism that turns the teacher union into the reported student, and Eq. 11 as written cannot train the student if the sampled token is treated as a fixed target: the expected gradient of log pi_theta at a token sampled from pi_theta is zero. This is more fundamental than the coverage-noise issue because even perfect coverage scores would not rescue a distillation objective with no expected learning signal. The residual loss L_res still provides signal, so the full pipeline may work in practice, but the paper does not specify the surrogate that makes L_base effective, and the ablations in Table 3 are not interpretable without that specification. I therefore disagree with the reader's choice of weakest assumption: the load-bearing issue is the internal consistency of the proposed compression objective, not the accuracy of per-example coverage scores. The empirical tables and the scaling experiment are useful and the paper shows internal effort, but the central claim cannot be evaluated from the text as written, so the verdict should move from CONDITIONAL to UNVERDICTED until the optimization of L_base is clarified or corrected.","tokens_in":11039,"tokens_out":13427,"duration_ms":129778,"concrete_test":"Derive the single-step gradient of Eq. 11: for each prefix h, E_{y~pi_theta(.)}[grad_theta log pi_theta(y|h)] = 0, so the literal loss has zero expected gradient. Then run a minimal implementation with fixed teachers, L_res = 0, and only L_base on a small data subset; if the student distribution does not move toward the teacher on held-out prefixes, the described objective provides no learning signal. Additionally, compare against the correct REINFORCE surrogate E[(log pi_theta/pi_r - b) grad log pi_theta]; if only that surrogate reproduces the Table 3 quality-weighted improvement, the paper should be revised to report the actual optimization surrogate.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Section 3.3 defines L_base = E_{y~pi_theta} sum_t sum_{r in Q_i} alpha_i^r log[ pi_theta(y_{i,t}|h) / pi_r(y_{i,t}|h) ], and section 4.1 says the objective 'is evaluated only on the token sampled by the student policy.' For any fixed prefix h, E_{y_t ~ pi_theta(.|h)} grad_theta log pi_theta(y_t|h) = sum_y grad_theta pi_theta(y|h) = 0. Therefore, if L_base is backpropagated at the sampled token as the text implies, each token term contributes zero expected gradient to the student parameters; the loss is a zero-mean control variate rather than a training signal. Correctly optimizing the reverse KL requires a REINFORCE score-function term E[(log pi_theta/pi_r) grad log pi_theta] or a reparameterized/soft-target surrogate, neither of which appears in Eq. 11 or in the implementation details. Because L_res does have nonzero gradient, the full framework may still train, but the reliability-gated base distillation, and the Table 3 ablations separating uniform from quality-weighted OPD, cannot be explained by the objective as written. If the implementation actually uses teacher cross-entropy or a REINFORCE surrogate, the paper must state it; otherwise the central compression claim is not supported by the described method. This is an internal-consistency concern, not a comment on the provenance of the empirical tables.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an expand-then-compress framework for distilling a set of RL-trained reasoning teachers into a single student. In the expansion stage, Residual GRPO (RGRPO) trains successive teachers from a common initialization on examples not yet covered by the accumulated teacher union, where coverage is estimated from per-example success rates over 16 sampled responses. In the compression stage, Reliability-Gated Teacher-Union On-policy Distillation (TU-OPD) combines a base quality-weighted reverse-KL objective on student-sampled tokens with a Consensus-Residual Decomposition loss that transfers a winner teacher's excess token preferences over its reliable peers. Experiments on mathematical reasoning (AIME, HMMT, AMC23), code generation (LiveCodeBench v6), and instruction following (IFBench) with Qwen3-1.7B and Qwen3-4B report that the distilled student outperforms the strongest individual teacher on the primary aggregate metric of each domain, with relative gains of 2.0%, 8.3%, and 6.9% at the 1.7B scale. Ablations compare residual versus parallel teacher construction and uniform versus quality-weighted distillation objectives.","tokens_in":11357,"tokens_out":8926,"duration_ms":79585,"significance":"If the empirical claims hold, the expand-then-compress idea is valuable: it offers a concrete recipe for converting several specialized RL policies into a single deployable model that is stronger than any one teacher, with potentially broad applicability to reasoning, code, and instruction-following domains. The paper has notable strengths: the teacher-envelope analysis in Table 1 gives direct evidence of benchmark-level complementarity; the residual-versus-parallel comparison in Table 2 is a clean test of the expansion mechanism; the experiments span three domains and two model scales; and the writing is generally clear. The main weaknesses are technical and empirical: the base distillation objective in Eq. (11) appears to have zero expected gradient as written, the headline gains are small and unreplicated, and the comparisons lack single-best-teacher and compute-matched RL baselines. No code or data is provided, which limits reproducibility.","major_comments":[{"comment":"The base objective as written has zero expected gradient. For a fixed prefix h, the token y_t is sampled from π_θ(·|h), and the loss term log π_θ(y_t|h) − log π_r(y_t|h) has gradient ∇_θ log π_θ(y_t|h) with respect to the student parameters, since the teacher term is constant. The expectation of this gradient is ∑_y ∇_θ π_θ(y|h) = 0. Thus Eq. (11) is a zero-mean control variate, not a Monte Carlo estimator of the quality-weighted reverse KL; a correct REINFORCE-style estimator would require an additional (1 + log(π_θ/π_r)) ∇_θ log π_θ term or a reparameterized/surrogate objective. Because §4.1 states that the base objective is evaluated only on the token sampled by the student policy, this is not a mere notational ambiguity. Consequently, the compression gains, and especially the Table 3 ablations that remove L_res, cannot be explained by Eq. (11). If the implementation actually trains on teacher cross-entropy or a score-function surrogate, the paper must state this explicitly; otherwise the central compression claim is not supported by the described method.","section":"§3.3, Eq. (11); §4.1"},{"comment":"The headline empirical claims rest on single-run, single-seed numbers with no error bars, multiple seeds, or significance tests. With eight sampled responses per problem, the 2.0% math-Mean gain (16.70 vs. 16.37) is within the range that could arise from sampling noise, and the ablation differences in Table 3 (e.g., 15.99 vs. 16.70, and 15.81 vs. 16.70) are likewise unreplicated. Please report multiple seeds or bootstrap confidence intervals for the main tables, and state explicitly whether the reported gains are consistent across seeds.","section":"§4.2, Table 1; §4.3, Table 3"},{"comment":"There is no single-best-teacher distillation baseline or compute-matched RL baseline. The title and central claim, 'beyond the best teacher,' require showing that the multi-teacher union distilled into a student outperforms a student distilled from the single best teacher with comparable compute and data, or a student trained directly with RL on the same training pool. Without such controls, the observed gains could be due to distillation from any teacher on the full data rather than to the expand-then-compress mechanism.","section":"§4.2, §4.3"},{"comment":"The residual construction relies on per-example success-rate estimates from J=16 samples per teacher, and the threshold τ_drop=0.5 is applied to these estimates. Binomial noise at J=16 is substantial: a teacher with a true per-example success rate of 0.6 has roughly 29% probability of being classified as residual, while one with a true rate of 0.4 has roughly 14% probability of being classified as covered. Please report the sensitivity of the residual subsets and downstream results to J (e.g., J=32 or 64) and to the threshold, or otherwise justify that the coverage estimates are accurate enough to support the 'residual region' interpretation.","section":"§3.2, Eqs. (5)–(7)"}],"minor_comments":[{"comment":"Several hyperparameters used in the method are not specified, including τ_base in Eqs. (8)–(9), the peer temperature T_p in Eq. (16), and the margin γ in Eq. (18). Without these values the distillation setup is not fully reproducible.","section":"§4.1, Eqs. (8)–(18)"},{"comment":"The notation for the expectation in Eq. (11) drops the per-example dependence of the teacher weights α_i^(r); the expression would be clearer as E_{x_i∼D_valid, y_i∼π_θ(·|x_i)} with explicit dependence on i retained throughout.","section":"§3.3, Eq. (11)"},{"comment":"The row labeled 'Avg. Residual-GRPO Teachers' averages benchmark-wise scores across teachers; this is not the same as the aggregate performance of an average teacher and could be misread. Consider labeling it explicitly as the teacher-average of per-benchmark scores.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the mismatch between Eq. (11) and the reported implementation: if the base loss is truly evaluated only on the student-sampled token, it has zero expected gradient. I recommend asking the authors to provide the exact loss computation (e.g., a code snippet or pseudocode) and, if a REINFORCE or cross-entropy surrogate was used, to report it and re-run the relevant ablations accordingly. The missing baselines and lack of seed variance should also be addressed before the paper can be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper has a genuinely interesting expand-then-compress idea, and the expansion-stage evidence is decent. But the compression objective as written cannot do what the paper claims. Eq. (11) has zero expected gradient when evaluated on student-sampled tokens. For any prefix, E_{y_t ~ pi_theta} grad log pi_theta(y_t) = 0, so the sampled-token log-ratio contributes no expected gradient to the student. To optimize reverse KL you need a REINFORCE score-function term or a reparameterized surrogate, and neither appears in the text. That is not a minor typo; it means the reported distillation gains and the Table 3 ablations are not explained by the described method. The residual L_res does have gradient, so the framework may still train, but the base distillation component and the uniform-vs-quality-weighted comparison are unaccounted for.\n\nWhat is actually new and good: the coupling of Residual GRPO with distillation. Training teachers from a common initialization, measuring per-example coverage from sampled responses, and redirecting later rounds to uncovered residuals is a clean idea. Table 2's comparison against Parallel GRPO is a fair test and shows the residual allocation broadens the union, especially on AIME. Consensus-Residual Decomposition—preserving a winner teacher's top-K preferences over its reliable peers—is also a reasonable mechanism to prevent consensus collapse. The experiments span math, code, and instruction following, plus a Qwen3-4B scaling check, and the student does beat each individual teacher on the aggregate metric.\n\nSoft spots in proportion: the zero-gradient issue is the load-bearing one and needs to be fixed. Also missing: error bars, multiple seeds, code/data release, and a best-teacher distillation baseline. The tau_base threshold is never specified. The 2.0% math gain is small; the 8.3% and 6.9% gains are more convincing but still single-run. The expansion-side results are the strongest part of the paper.\n\nWho this is for: people working on multi-teacher distillation and LLM post-training. It deserves a serious referee—the idea is timely and the expansion experiments are worth engaging—but only with major revision. I would not cite the compression results in current form.","headline":"The expand-then-compress idea is genuinely interesting, but the main distillation objective as written has zero expected gradient, so the compression results need a rewritten loss and stronger baselines before they can be trusted.","tokens_in":11871,"tokens_out":4157,"would_cite":false,"duration_ms":39543,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a deliberately constructed union of RL-trained teachers can be distilled into a single student that outperforms every individual teacher.","keywords":["multi-teacher distillation","on-policy distillation","group relative policy optimization","residual training","reasoning solution manifold","policy compression","verifiable rewards","reasoning language models"],"falsifier":"Run the expansion stage with the same budget but different numbers of coverage samples, such as $J=4$, $J=16$, and $J=64$; if the residual subsets and final student scores change sharply with $J$, the coverage estimates are not stable enough to carry the mechanism. A positive control would be to train teachers on random subsets matched in size and difficulty to the residual subsets and check whether the residual subsets' union still beats the random union by the reported margin.","tokens_in":10837,"feed_emoji":"🧠","tokens_out":5788,"duration_ms":46502,"temperature":0.7,"pith_summary":"This paper argues that a single reinforcement-learning-trained reasoning model is a local specialist, not a complete teacher, and that deliberately training several complementary specialists before compressing them into one policy yields a student stronger than any individual teacher. It proposes an expand-then-compress pipeline: Residual GRPO trains successive teachers on examples not yet reliably solved by the existing teacher union, and Teacher-Union On-policy Distillation then consolidates the union into one student. The compression step uses reliability-gated teacher weighting and a Consensus-Residual Decomposition that preserves winner-specific token preferences that ordinary averaging would suppress. Across mathematical reasoning, code generation, and instruction following, the resulting Qwen3-1.7B student outperforms the strongest individual teacher by 2.0%, 8.3%, and 6.9%, respectively, while retaining single-model inference.","feed_headline":"A single student beats every one of its RL teachers","feed_subtitle":"Training a complementary teacher union, then distilling it, lifts math, code, and instruction scores by 2–8%.","key_machinery":"The load-bearing machinery is a three-part loop. First, per-example coverage scores $s_i^{(r)}$ estimated from $J$ sampled responses per teacher define a coverage envelope $u_i^{(r)}=\\max_{a\\le r}s_i^{(a)}$, and examples with $u_i^{(r)}\\le\\tau_{\\mathrm{drop}}$ form the residual training subset for the next teacher. Second, Reliability-Gated Teacher-Union OPD computes per-example reliable teacher sets $Q_i=\\{r:s_i^{(r)}\\ge\\tau_{\\mathrm{base}}\\}$, softmax quality weights $\\alpha_i^{(r)}$, and a sampled-token reverse-KL objective $L_{\\mathrm{base}}$ that only queries teacher log-probabilities on tokens the student itself sampled. Third, Consensus-Residual Decomposition selects a winner teacher $r_i^\\star$, builds a peer mixture from its reliable peers, and transfers, through $L_{\\mathrm{res}}$, the winner's excess Top-$K$ token preferences when the winner beats its best peer by at least $\\tau_{\\mathrm{gap}}$; the total objective is $L_{\\mathrm{total}}=L_{\\mathrm{base}}+\\lambda L_{\\mathrm{res}}$.","core_discovery":"The central claim is that stronger students can be obtained not by selecting the best teacher but by constructing a complementary teacher union and distilling the union rather than the average. The paper treats each RL-trained policy as a local probe of a multi-basin reasoning solution manifold, so different teachers can cover different solution modes even from the same initialization. Across math, code, and instruction following, the resulting single student exceeds every individual teacher on the primary aggregate metric of each domain, recovering part of the gap to a benchmark-wise teacher envelope that is not itself a deployable model. The paper further shows that coverage-aware residual allocation broadens the teacher union more than random data partitioning, and that the residual objective that preserves a winner teacher's excess token preferences over its reliable peers is what makes the compression succeed.","pith_inferences":["An implication left implicit is that the expand-then-compress cycle could be iterated: the distilled student itself could serve as the next common initialization, with a fresh residual round targeting modes still uncovered after compression.","An extension worth testing is replacing the $J=16$ outcome-sample coverage estimates with a learned surrogate verifier; if noisy coverage is the bottleneck, cleaner scores should enlarge the gap between residual and random partitioning.","A diagnostic the paper does not report is per-round difficulty or loss on residual subsets; if later residual rounds degenerate into hard-example overfitting, a difficulty cap on the residual subset would be the natural corrective."],"forward_implications":["If the central claim is right, the best single RL run should no longer be treated as the deployable artifact; the union of several runs is the object with transferable value.","Residual GRPO turns teacher construction into a coverage problem, so later teachers are trained on shrinking subsets and can add new modes at lower marginal cost.","Quality-weighted distillation with a residual objective outperforms uniform teacher mixing, so compression should preserve specialist preferences rather than only the consensus.","The reported Qwen3-4B scaling result suggests the expand-then-compress benefit is not limited to one model size and may generalize to larger student policies."],"supporting_citations":[{"why":"Introduces GRPO, the reinforcement-learning algorithm that produces each teacher.","marker":"Shao et al. 2024"},{"why":"Establishes that large-scale outcome RL can elicit long reasoning, the setting the paper argues is incomplete.","marker":"Guo et al. 2025"},{"why":"Introduces on-policy distillation, the mechanism the student objective builds on.","marker":"Agarwal et al. 2024"},{"why":"Provides the on-policy distillation formulation used for student-prefix supervision.","marker":"Lu and Lab 2025"},{"why":"Extends on-policy distillation to reasoning models, supporting the token-level transfer design.","marker":"Li et al. 2026"},{"why":"Prior multi-teacher reasoning distillation that aggregates an existing teacher set, the baseline the paper distinguishes itself from.","marker":"Tian et al. 2025"},{"why":"Committee-style multi-teacher distillation that the paper contrasts with reliability-gated aggregation.","marker":"Li et al. 2025"},{"why":"Supplies the Skywork-OR1-RL-Data training pool for the math and code experiments.","marker":"He et al. 2025a"},{"why":"Supplies the verifiable instruction-following data and IFBench evaluation.","marker":"Pyatkin et al. 2025"}],"fun_headline_variants":["Teacher union beats best single teacher in distillation","Complementary teachers, not the best, train top student","Expand teacher set, compress into one: student wins","A student stronger than any teacher: the union approach"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that per-example coverage scores estimated from $J=16$ sampled responses per teacher accurately separate examples the teacher union reliably solves from residual examples needing new solution modes; if those scores are noisy, the residual subsets and reliability gates become approximately random and the reported complementarity could be hard-example overfitting rather than manifold expansion.","fun_headline_variants_meta":{"raw":{"variants":["Teacher union beats best single teacher in distillation","Complementary teachers, not the best, train top student","Expand teacher set, compress into one: student wins","A student stronger than any teacher: the union approach"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1634,"prompt_tokens":990,"completion_tokens":644,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":582}},"tokens_in":606,"tokens_out":644,"duration_ms":6471,"temperature":1.0,"reasoning_tokens":582,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:23:22.451259+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the expansion stage with the same budget but different numbers of coverage samples, such as $J=4$, $J=16$, and $J=64$; if the residual subsets and final student scores change sharply with $J$, the coverage estimates are not stable enough to carry the mechanism. A positive control would be to train teachers on random subsets matched in size and difficulty to the residual subsets and check whether the residual subsets' union still beats the random union by the reported margin.","supporting_citations":[],"review_version":2}