{"id":"092b2cef-e62e-4125-bf38-ff65a2c9671f","arxiv_id":"2607.17247","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using teacher preferences to reweight RL gradients only on correct responses improves LLM post-training on math and knowledge benchmarks in both within- and cross-family settings.","lead":"This paper proposes Distilled RL, a post-training method that lets a stronger teacher model reweight the reinforcement-learning signal token by token instead of forcing the student to imitate the teacher. The method reports consistent gains over standard RL and on-policy distillation on math and knowledge benchmarks, with the biggest wins when teacher and student come from different model families.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Teacher-competence assumption is unvalidated: Distilled RL reweights positive-advantage responses, but never checks whether the teacher can actually solve the prompt, and the claimed advantage over RL may rest on this untested premise.","rationale":"The central claim is empirical: across the tested student models, Distilled RL outperforms RL, OPD, and OPD+RL on the reported benchmarks. The strongest support is the consistency of the gains and the ablations showing negative sample reset and geometric normalization matter. The weakest link is the teacher-competence assumption in Appendix A, exactly as the reader identified. With binary correctness rewards, A_i>0 corresponds to a correct student response, so the method applies teacher reweighting precisely on responses the student solved correctly; the assumption is that the teacher is also competent on those prompts. This is never checked. The teacher is never rolled out, so there is no direct evidence of teacher competence on the training distribution. As the student improves, it can locally outperform the fixed teacher, making the teacher's token preferences misleading on otherwise successful responses. The entropy case study is not a valid test of knowledge transfer because the 'teacher' is the student itself with temperature switched by the student's entropy — it is a feedback control loop, and the observed tracking does not demonstrate transfer from an independent teacher. The proposed concrete test — gating teacher weights on teacher-rollout correctness — would settle whether the assumption holds. The absence of error bars and the small deltas on some benchmarks (e.g., Qwen3-1.7B on HMMT25) further support a conditional verdict, but the teacher-competence check is the most decisive. Since the reader already assigned CONDITIONAL, this stress-test does not change the verdict; it reinforces the need for the condition.","tokens_in":13649,"tokens_out":10685,"duration_ms":106367,"concrete_test":"Sample a subset of DAPO-17k prompts; for each prompt, have the teacher (Qwen3-8B-GRPO) roll out G=8 responses and compute its own pass rate. Train Distilled RL in two variants: (a) the proposed method as-is, and (b) a gated variant that sets w_i,t=1 for all tokens from prompts where the teacher's own pass rate is 0 (or below a threshold), so teacher reweighting is only applied on prompts where the teacher demonstrates competence. Compare held-out pass@1 between (a) and (b). If (b) matches or exceeds (a), the unvalidated teacher-competence assumption is not helping or is actively harmful; if (a) is better, the reported gains depend on prompts where the teacher happens to be competent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Distilled RL's advantage over plain RL rests on the assumption stated in Appendix A: when the student's response has positive advantage (A_i>0, which with binary rewards means the response is correct), the teacher must be competent enough on that prompt for its token-level preferences to be useful guidance. The training loop never checks teacher competence: the teacher is only scored on student-generated tokens, never rolled out on the prompt. As training proceeds, the student may surpass the fixed teacher on a growing subset of problems, and then the teacher's weighting is actively misleading even on correct student responses. The paper provides no per-prompt teacher-correctness statistics, no diagnostic of how often the teacher's token preferences disagree with correct student reasoning, and no ablation that gates teacher weights on teacher competence. Without this, the reported pass@1 gains could be driven entirely by the subset of prompts where the teacher is competent, while the method silently degrades performance on the rest. The entropy case study in Section 4 does not resolve this because the 'teacher' is the student itself with temperature controlled by the student's current entropy — a feedback loop, not an independent competence test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Distilled RL, a post-training objective that injects token-level teacher preferences into a GRPO-style RL update. The method replaces unconditional KL-based imitation (OPD) with three components: reverse importance sampling with clipping of the teacher-to-student token ratio, a negative-sample reset that disables teacher reweighting on non-positive advantages, and sequence-level geometric normalization of the clipped ratios. The central empirical claim is that Distilled RL outperforms standard RL, OPD, and a simple OPD+RL combination on pass@1 and pass@k across within-family and cross-family distillation settings, with the largest gains in cross-family distillation. A controlled entropy-based case study is offered as evidence that the method transfers teacher-side distributional knowledge beyond the task reward.","tokens_in":13867,"tokens_out":3418,"duration_ms":36395,"significance":"If the empirical claims hold, Distilled RL would be a useful and simple recipe for combining teacher supervision with outcome-based RL in LLM post-training. The paper has clear strengths: the evaluation uses external benchmarks, the teacher (Qwen3-8B-GRPO) is external to the student family in the cross-family setting, the ablations in Table 4 support the two principal design components, and the authors provide a code link. The method is also conceptually appealing because it avoids unconditional imitation while retaining dense token-level feedback. However, the central empirical claim currently rests on point estimates without error bars or multiple seeds, and the key teacher-competence assumption is acknowledged but not validated. The entropy case study, as constructed, is a feedback-control demonstration rather than independent evidence of knowledge transfer. These issues are load-bearing for the paper's contributions and require additional empirical support or careful reframing.","major_comments":[{"comment":"The main empirical claim is that Distilled RL 'consistently outperforms' RL, OPD, and OPD+RL. However, the paper reports no error bars, no multiple seeds, and no statistical tests. Several reported improvements are small or negative at the individual-benchmark level; for example, Qwen3-1.7B shows -0.11 on HMMT25 and an average gain of only +1.16 over OPD. Since pass@1 is computed from 32 sampled responses, sampling noise is non-negligible. Without variance estimates or at least bootstrap confidence intervals, the central 'consistent improvement' claim is not yet established. Please add multiple runs (or seed-level results) and report variability, especially for the smaller gains.","section":"§5.1, Tables 2 and 3"},{"comment":"Appendix A explicitly states the implicit teacher-competence assumption: when a student response has positive advantage, the teacher's token-level preferences are assumed to be useful guidance. The training loop never checks this: the teacher is only scored on student-generated tokens and is never rolled out on the prompt. As the student improves, it may locally surpass the fixed teacher on some prompts, making the teacher's reweighting misleading even on correct student responses. This is not merely a theoretical concern; it directly affects the positive-advantage reweighting in Eq. (3). The paper provides no per-prompt teacher-correctness statistics, no diagnostic of teacher/student token agreement, and no ablation that gates teacher weights on teacher competence. Without such evidence, the reported gains could be concentrated on prompts where the teacher is competent while the method","section":"§3.2 / Appendix A"},{"comment":"The entropy case study is presented as evidence that Distilled RL can transfer 'previously unavailable knowledge' from the teacher. But the teacher is constructed from the student itself, with temperature chosen from the student's current entropy: low temperature when entropy is above 0.5 and high temperature when entropy is below 0.5. This is a feedback controller, not an independent teacher. The observed trajectory toward the entropy threshold is therefore expected from the sign of the control signal and does not demonstrate that Distilled RL acquires information beyond the reward. This claim is a listed contribution ('transfer previously unavailable knowledge'), so it needs either a genuinely independent teacher property or a reframing as a synthetic sanity check of the weighting mechanism rather than evidence of knowledge transfer.","section":"§4, Figure 4"}],"minor_comments":[{"comment":"Notation is inconsistent: the objective defines r_{i,t}(θ) but the clipped expression and surrounding text use r_{i,t}. The min expression is also visually dense; adding parentheses around the clipped term would improve readability.","section":"§3.2, Eq. (3)"},{"comment":"The layout of Table 4 is very hard to read: benchmark values and parenthetical deltas run together in a single row, especially for DSQW-1.5B. Please align columns by benchmark and separate the deltas clearly, ideally in a separate row or column.","section":"Table 4"},{"comment":"The entropy threshold of 0.5 is introduced without units or justification. State whether this is natural-log entropy, and explain how the threshold was chosen.","section":"§4"},{"comment":"The SuperGPQA subset is described as 'fixed' but no seed or construction procedure is given. Please specify how the 500 examples were selected.","section":"Appendix B"},{"comment":"Figure 2 and Figure 5 report training dynamics for a single run. Adding light smoothing and a second seed would improve interpretability, especially for the OPD+RL 'performance decline' claim.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine-learning venue and the core idea is reasonable, but the empirical support needs strengthening before publication. The lack of error bars and the unvalidated teacher-competence assumption are the main blockers. The entropy case study should be repositioned or replaced. The authors' Appendix A already acknowledges the teacher-competence limitation, which is good, but the manuscript does not yet address it experimentally."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a practical post-training recipe worth a serious look, not a field reorg. The specific combination — reverse teacher-to-student importance ratios, clipped and geometrically normalized, applied only to positive-advantage responses, with negative samples reset to plain RL — is genuinely new in this form. The cross-family setting (DeepSeek student with a Qwen teacher) is the most interesting test, and the gains there are the most compelling evidence that selective teacher weights do something beyond KL imitation.\n\nThe paper is well-written and honest. The opening observation that OPD saturates early while RL improves slowly is a nice framing. The ablations actually support the two main components: removing negative sample reset hurts a lot, removing geometric normalization hurts less but consistently. And Appendix A explicitly names the teacher-competence assumption that the method relies on — the teacher is only scored on student-generated tokens, never rolled out on its own, so the loop never checks whether the teacher can solve the prompt. That is more candid than most.\n\nThe soft spots are real but addressable. First, no error bars or multiple seeds. For a recipe that shows 1-3 point average gains over RL, a single run per setting leaves a lot of uncertainty. Second, the teacher-competence assumption is load-bearing and not validated. If the teacher is wrong on a prompt where the student is right, the token-level weights actively distort the RL update. The paper offers no per-prompt teacher-correctness statistics or ablation gating on teacher competence. This is the main thing a referee should push on. Third, the entropy case study is a feedback loop: a temperature chosen from the student's current entropy drives the student toward that entropy. It is not independent evidence of knowledge transfer, and the paper does not really lean on it, so it is a minor issue.\n\nOn the citation pattern: the paper cites the relevant KD+RL work (KDRL is there) and presents itself as an extension, which is fair. The code link is unverified, but at least it is prominent.\n\nBottom line: this deserves a serious referee, not a desk reject. I'd recommend engagement — ideally with a request for multiple seeds, a teacher-competence diagnostic, and a check of the entropy story if the authors want to keep it.","headline":"A practical and honest post-training recipe that reweights RL gradients with teacher token preferences; the cross-family gains are the main result, but the single-seed evidence and unvalidated teacher-competence assumption keep me from fully buying the averages.","tokens_in":14425,"tokens_out":2372,"would_cite":true,"duration_ms":23755,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Distilled RL folds teacher token preferences into the RL gradient, so a teacher can guide a student step-by-step without unconditional imitation.","keywords":["reinforcement learning","knowledge distillation","on-policy distillation","LLM post-training","credit assignment","teacher reweighting","reasoning","token-level guidance"],"falsifier":"Train Distilled RL on a set of prompts where the teacher provably fails on many prompts the student solves, and compare against plain RL: if teacher weighting on correct student responses hurts or offers no gain, the teacher-competence assumption is violated. A more direct check is to record, on correct student responses, whether the tokens the teacher prefers lead to later failure in the same rollout; if such reversed preferences are frequent, the token-level guidance is misleading.","tokens_in":13484,"feed_emoji":"🎓","tokens_out":3713,"duration_ms":39699,"temperature":0.7,"pith_summary":"The paper tries to establish that LLM post-training improves when teacher supervision enters the reinforcement learning gradient itself, rather than as a separate KL-based imitation loss. The proposed method, Distilled RL, reweights the RL advantage at the token level using a clipped, normalized teacher-to-student probability ratio, and it resets those weights on negative-advantage responses. The authors argue this gives the student fine-grained credit assignment while preserving the reward-driven optimization of RL, and they report that it outperforms both standard RL and on-policy distillation, especially in cross-family teacher-student settings. A controlled entropy-transfer experiment is offered as evidence that the student acquires information beyond the outcome reward.","feed_headline":"Distilled RL beats RL and distillation on reasoning benchmarks","feed_subtitle":"Mixing teacher preferences into the RL gradient fixes cross-family training and improves pass@k.","key_machinery":"The carrying object is the reverse importance ratio rho_{i,t} = pi_teacher(oi_t | q, o_{i,<t}) / pi_theta_old(oi_t | q, o_{i,<t}), evaluated on tokens the student itself generated. The ratio is clipped to a bounded interval, divided by the per-sequence geometric mean of clipped ratios, and multiplied into the clipped policy-gradient objective; for negative-advantage responses the weight is reset to 1. This converts the teacher's relative token preferences into a redistribution of the sequence-level RL advantage, so the teacher guides which tokens to reinforce without globally amplifying or suppressing the response.","core_discovery":"The central claim is that teacher supervision should be coupled into the RL surrogate objective through a reverse importance ratio rather than added as an unconditional imitation term. The objective is min(r_{i,t} w_{i,t} A_i, clip(r_{i,t}) w_{i,t} A_i), where r is the ordinary student-policy ratio, A_i is the group-normalized sequence advantage, and w_{i,t} is a teacher-derived weight: the clipped teacher-to-old-student token probability ratio, normalized per sequence so its geometric mean is one, and set to one whenever A_i is non-positive. On successful responses, tokens the teacher prefers are amplified and tokens it disprefers are suppressed; on failed responses, the update reduces to t","pith_inferences":["Beyond the paper: the same teacher-weighting mechanism could be gated by an explicit per-prompt estimate of teacher competence, such as occasional teacher rollouts or a verifier check, potentially extending gains to cases where the teacher is locally weaker than the student.","Beyond the paper: the sequence-level geometric normalization is a generic device that could also be applied to token-level process rewards or advantage estimation, not only to teacher ratios.","Beyond the paper: the entropy-transfer result suggests Distilled RL could be used to steer distributional properties such as output entropy, response length, or style, by constructing teachers with prescribed properties rather than relying solely on correctness rewards."],"forward_implications":["If the central claim is correct, teacher guidance no longer needs to be framed as distribution matching, so the teacher can be used selectively without dragging the student into a KL-induced local optimum.","Cross-family distillation becomes viable: the largest reported gains appear precisely where the teacher and student differ in architecture or reasoning distribution, the regime where KL-based on-policy distillation tends to degrade.","The student can acquire knowledge beyond the outcome reward signal, as the entropy-control case study demonstrates by transferring a prescribed distributional property that RL alone does not provide.","Pass@k improves as well as pass@1, implying the method strengthens the whole response distribution, not just the single most probable answer.","Because the update reduces to standard RL on negative-advantage trajectories, the method preserves the reward-driven penalty structure and avoids counterproductive imitation of bad responses."],"fun_headline_variants":["Teacher-aware RL objective outperforms RL and distillation","Distilled RL uses teacher preference to fix credit assignment","Cross-family LLM post-training improved by teacher-weighted RL","Reverse importance sampling couples teacher supervision into RL updates"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes that whenever a student response is correct, the teacher is competent enough on that prompt that its per-token preferences are useful guidance; Appendix A notes that the teacher is never rolled out, so the training procedure never checks this assumption.","fun_headline_variants_meta":{"raw":{"variants":["Teacher-aware RL objective outperforms RL and distillation","Distilled RL uses teacher preference to fix credit assignment","Cross-family LLM post-training improved by teacher-weighted RL","Reverse importance sampling couples teacher supervision into RL updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1224,"prompt_tokens":751,"completion_tokens":473,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":411}},"tokens_in":495,"tokens_out":473,"duration_ms":4772,"temperature":1.0,"reasoning_tokens":411,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T18:35:13.833803+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Distilled RL on a set of prompts where the teacher provably fails on many prompts the student solves, and compare against plain RL: if teacher weighting on correct student responses hurts or offers no gain, the teacher-competence assumption is violated. A more direct check is to record, on correct student responses, whether the tokens the teacher prefers lead to later failure in the same rollout; if such reversed preferences are frequent, the token-level guidance is misleading.","supporting_citations":[],"review_version":1}