{"id":"29a5533a-de10-43a0-9d69-c743d1985ead","arxiv_id":"2607.04037","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Reward-gated on-policy distillation keeps teacher reverse-KL only on trajectories where verifier reward and teacher–student likelihood gap are directionally aligned, improving reasoning and coding scores.","lead":"RG-OPD gates on-policy teacher distillation with verifier rewards so a student only copies teacher logits when reward and likelihood gap agree. The filter yields stronger small reasoning models than unconditional reverse-KL or TSD-KD.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified","rationale":"The paper's central claim is the empirical superiority of reward-gated reverse-KL on-policy distillation over unconditional reverse-KL and TSD-KD on the stated benchmarks and model pair. That claim is supported by complete multi-seed tables, training dynamics, ablations on top-k, and released code. The reader's weakest assumption correctly flags that Eq. 2 is a heuristic whose reliability is not independently proven; however, for an empirical method paper the assumption is not load-bearing in the sense that would force a verdict change—if the proxy were poor, the reported gains would simply fail to appear. The concrete random-gate control would cleanly separate “any filtering helps” from “directional agreement helps,” but its absence does not invalidate the existing comparisons. Scope limits (one student–teacher pair, UltraInteract subset, mixed IFEval) are already reflected in the reader's moderate novelty score and do not create an internal flaw. Therefore the ACCEPT / HIGH verdict stands without adjustment.","tokens_in":10152,"tokens_out":485,"duration_ms":5381,"concrete_test":"Re-run the 1K and 8K Table 1 protocol with an ablated gate that keeps trajectories at random with the same kept-token fraction as RG-OPD (or with δ-swept / reverse-gate variants); if the random-gate student matches or exceeds RG-OPD averages, the directional-agreement rule is not doing the claimed work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (that directional agreement of the trajectory-level teacher–student log-likelihood gap with GRPO advantage is a reliable proxy for teacher usefulness) is real but not load-bearing against the central claim as stated. The claim is empirical and scoped: on the reported suite, RG-OPD students outperform reverse-KL and TSD-KD (Table 1). The gate (Eq. 2) is a simple, falsifiable design choice whose justification is the observed gains, multi-seed tables, dynamics (Figure 2), and public code—not a derivation that the gap is always informative. Mixed IFEval and likelihood-ranking results are already disclosed and do not overturn the generation-based averages. No internal inconsistency, circularity, or evaluation artifact undermines the reported deltas.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes RG-OPD, a reward-gated form of on-policy distillation for reasoning post-training. A student samples trajectories, a verifier supplies trajectory-level rewards/advantages (GRPO-style), and a fixed teacher supplies token-level logits on those states. Distillation (top-k reverse KL) is applied only when the sign of the teacher–student trajectory log-likelihood gap agrees with the reward: keep reward-positive trajectories only if the teacher is more likely than the student, and reward-negative ones only if the teacher is less likely (Eqs. 1–3, optional margin δ). Experiments distill Qwen2.5-1.5B from Qwen2.5-14B on UltraInteract and evaluate generation-based math/coding/instruction benchmarks at 1K and 8K max length (Table 1), with training dynamics (Figure 2), top-k ablation (Table 2), and likelihood-ranking results (Table 3). Reported averages favor RG-OPD over reverse-KL and TSD-KD on generation tasks, with code released.","tokens_in":10382,"tokens_out":1055,"duration_ms":8447,"significance":"If the empirical gains hold under broader settings, the work offers a simple, practical bridge between sparse verifier rewards and dense teacher logits without rewriting the RL objective. The gate is easy to implement, the failure modes of unconditional OPD are clearly stated, multi-seed tables and dynamics support the mechanism, and public code aids reproducibility. The contribution is incremental rather than foundational—an empirical filter on top of reverse-KL OPD—but it is well-scoped for reasoning distillation and useful to practitioners combining RLVR with teacher supervision.","major_comments":[{"comment":"The central design choice (Eq. 2) treats directional agreement of the trajectory-level teacher–student log-likelihood gap with the GRPO advantage as a proxy that teacher logits are informative rather than misleading. Table 1 shows consistent generation gains, but the manuscript does not independently validate this proxy (e.g., fraction of kept vs. dropped trajectories that are actually correct under the verifier, or ablations that reverse/randomize the gate). Without that, it remains unclear whether the gate is selecting useful teacher signal or merely a correlated subset of trajectories; a short diagnostic would make the mechanism claim load-bearing rather than post-hoc.","section":null},{"comment":"Comparisons to joint RL–KD / reward-aware distillation methods cited in §2 (KDRL, RL-aware KD, Self-Distilled RLVR) are only conceptual. Table 1 baselines are reverse-KL and TSD-KD; there is no head-to-head or controlled ablation against a simple RLVR + reverse-KL regularizer under the same student, teacher, data, and generation budgets. That leaves open whether gating is necessary relative to applying distillation as a broad auxiliary signal on the same setup.","section":null}],"minor_comments":[{"comment":"Table 1 caption and body: TSD-KD∗ numbers are borrowed and excluded from Avg. Rank; state more clearly in the main text whether the primary TSD-KD row is a full re-run under identical seeds/generation settings, to avoid apples-to-oranges reading of the 4.9-point claim.","section":null},{"comment":"IFEval is weaker for RG-OPD than TSD-KD at both lengths; the student-centric explanation is plausible but brief—add a short note on whether instruction-following is systematically traded off by reverse-KL gating.","section":null},{"comment":"Figure 2: kept-token fraction falls from ~0.9 to ~0.7; report absolute numbers of kept trajectories or tokens per step so selectivity is interpretable, not only the fraction.","section":null},{"comment":"Eq. (4) and Table 2: top-k with tail correction is well described; state the default K used for the main Table 1 runs explicitly in §4 (Appendix says K=50).","section":null},{"comment":"Notation: Ai is called both reward and GRPO advantage; fix one term in §3 and the figure caption for consistency.","section":null},{"comment":"Table 3: likelihood-ranking underperforms generation extraction; the caveat is good—consider moving the generation-extraction numbers into the main table or appendix for the same four tasks so readers do not over-weight the ranking protocol.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The reader’s accept-with-high-confidence take is reasonable on the reported suite; I am slightly more conservative (minor_revision) only because the gate’s proxy is not diagnostically validated and joint RL–KD baselines are missing. Neither issue looks unfixable. Fit for a solid ML venue is good; novelty is incremental but cleanly executed with code."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing is that this is a simple, well-motivated filter rather than another joint RL–KD objective. Student rolls out, verifier gives a trajectory advantage, teacher and student log-likelihoods are compared on the same tokens, and reverse-KL is applied only when the two signals point the same way (Eq. 2). That is the whole method.\n\nWhat is new is the directional agreement gate itself. Prior OPD (GKD) and TSD-KD apply teacher supervision more broadly; concurrent KDRL / RL-aware / self-distillation work either mixes signals into the RL objective or stays self-distillation. RG-OPD keeps external-teacher on-policy distillation and just drops trajectories where the teacher looks uninformative or actively wrong relative to the reward. The paper states the two failure modes cleanly (teacher endorses a failure; teacher under-rates a success) and the gate is the direct fix.\n\nEmpirically it works on the suite they chose. Table 1 (3 seeds, 1K and 8K) shows consistent gains on math and coding over reverse-KL and TSD-KD; long-context average is +8.2 over the untuned 1.5B student. Figure 2 (kept-token fraction falling, length rising, gated loss falling) and the top-k ablation support the mechanism. Code is public. Math is elementary; no circularity.\n\nSoft spots are real but scoped. Single student–teacher pair (Qwen2.5 1.5B/14B) and UltraInteract subset; IFEval is weaker than TSD-KD (they own this); likelihood-ranking tables are mixed and less natural for instruct models. The load-bearing assumption—that LT − LS co-signed with advantage is a good proxy for “teacher is useful here”—is only validated by the final numbers, not by an independent diagnostic. That is a design choice, not a derivation error, and the claim is empirical as stated.\n\nThis is for people already running RLVR or on-policy KD pipelines who want a cheap reliability filter. It deserves a serious referee; I would engage with it and expect to cite the gate if I am in that lane.","headline":"Clean, usable gate for on-policy distillation: keep reverse-KL only when verifier reward and teacher–student likelihood gap agree; real gains over reverse-KL and TSD-KD on the reported suite.","tokens_in":10984,"tokens_out":553,"would_cite":true,"duration_ms":4938,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Teacher logits help a student reasoner only when they agree with verifier rewards on the same trajectories.","keywords":["on-policy distillation","knowledge distillation","verifiable rewards","reinforcement learning","reasoning","language models","reward gating","reverse KL"],"falsifier":"Hold the keep-rate fixed and replace the reward–likelihood gate with random trajectory dropout; if the random filter matches or beats RG-OPD on the same student–teacher pair and benchmarks, directional agreement is not doing the claimed work.","tokens_in":11100,"feed_emoji":"🧠","tokens_out":853,"duration_ms":22956,"temperature":0.7,"pith_summary":"On-policy distillation trains a small student by sampling its own answers and matching a strong teacher’s token probabilities on those states. The paper argues this is not always safe: a teacher can favor plausible wrong solutions or undervalue correct student paths that differ from its own. RG-OPD keeps the dense reverse-KL update only on trajectories where the verifier reward and the teacher–student likelihood gap point the same way—teacher more confident than the student on successes, less confident on failures. Sparse correctness thus filters dense teacher signals instead of replacing them. On math and coding benchmarks, the gated students beat unconditional reverse-KL and a recent token-selective baseline, with larger gains when longer generations are allowed.","feed_headline":"Gate teacher logits by reward agreement, lift student scores","feed_subtitle":"Dense teacher signals help only when likelihood and correctness point the same way on each trajectory.","key_machinery":"The reward–teacher likelihood gate gi: a binary filter that admits a sampled trajectory only if the verifier advantage Ai is directionally aligned with the teacher–student log-likelihood gap (teacher more likely than student on reward-positive rollouts, less likely on reward-negative ones). Reverse-KL is then computed only over the kept trajectories.","core_discovery":"Unconditional on-policy distillation can reinforce bad modes the teacher endorses or erase correct student behaviors the teacher would not produce. RG-OPD addresses this with a trajectory-level reward–teacher gate: distill reverse KL only when a positive advantage co-occurs with higher teacher likelihood than the student’s, or a non-positive advantage co-occurs with lower teacher likelihood. The method preserves token-level supervision while dropping misleading teacher signals, and the resulting students outperform reverse-KL and TSD-KD on reasoning and coding benchmarks.","pith_inferences":["The same directional gate could be applied when the “teacher” is a privileged or older copy of the student rather than an external model.","Noisy verifiers may require a soft or multi-sample gate; a hard binary rule can drop useful updates or keep harmful ones.","Trajectory-level gating leaves finer token-level disagreements unused; a token-level reliability score could extend the idea.","The design points to a general pattern for RL with dense auxiliaries: keep the dense signal only when it agrees with the sparse primary reward."],"forward_implications":["On-policy distillation for reasoning should condition teacher updates on verifier agreement rather than apply them to every rollout.","Sparse outcome rewards can act as a reliability filter for dense teacher logits without discarding token-level supervision.","Students can keep correct behaviors the teacher would not produce while still absorbing guidance where the teacher is informative.","Gains can grow with longer generation budgets as students imitate more verbose teacher reasoning on the trajectories that pass the gate.","Joint reward-and-distillation training can use a selective gate instead of always mixing the two signals."],"fun_headline_variants":["Gate teacher logits with rewards to fix on-policy distillation flaws","Reward agreement filters bad teacher signals for stronger students","RG-OPD trusts teacher logits only on verified reward trajectories","Verifier-gated reverse KL beats vanilla OPD and TSD-KD baselines","Drop misleading teacher modes via reward-teacher co-occurrence gate"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The method assumes that when a teacher’s total likelihood of a trajectory lines up with whether that answer was correct, the teacher’s token distribution is worth imitating on that trajectory.","fun_headline_variants_meta":{"raw":{"variants":["Gate teacher logits with rewards to fix on-policy distillation flaws","Reward agreement filters bad teacher signals for stronger students","RG-OPD trusts teacher logits only on verified reward trajectories","Verifier-gated reverse KL beats vanilla OPD and TSD-KD baselines","Drop misleading teacher modes via reward-teacher co-occurrence gate"]},"model":"grok-4.5","effort":"low","cost_usd":0.005576,"raw_usage":{"total_tokens":1470,"prompt_tokens":811,"num_sources_used":0,"completion_tokens":88,"cost_in_usd_ticks":55760000,"prompt_tokens_details":{"text_tokens":811,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":571,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":811,"tokens_out":88,"duration_ms":6154,"temperature":1.0,"reasoning_tokens":571,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T22:07:01.027350+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Hold the keep-rate fixed and replace the reward–likelihood gate with random trajectory dropout; if the random filter matches or beats RG-OPD on the same student–teacher pair and benchmarks, directional agreement is not doing the claimed work.","supporting_citations":[],"review_version":1}