{"id":"073f2ef0-7496-4554-ac0e-06b3653b46d9","arxiv_id":"2607.04751","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Interpolating teacher and student token probabilities bounds distillation rewards and, with PPO-style reuse, yields large AIME gains over standard on-policy distillation.","lead":"TOP-D stabilizes on-policy distillation of language models by interpolating teacher and student probabilities into a proximal teacher, then reusing data with trust-region updates. It reports large gains on math reasoning benchmarks with no extra model calls.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The headline 25.84-point gain over OPD is confounded by TOP-D's off-policy reuse; the pure proximal-teacher contribution is not isolated against a matched OPD baseline.","rationale":"The Reader correctly flags Assumption 4.1 and short single-seed runs, but the more load-bearing empirical soft spot is the mismatched baseline that directly underwrites the 25.84-point number in the strongest claim. Theory (bounded variance, contraction, monotonic lower bound) can stand even if M is only locally finite; the paper's positioning as a \"promising alternative to the well-established OPD paradigm\" rests on the empirical delta. Because Table 5 and Alg. 1 give TOP-D free off-policy reuse that the OPD column is denied, that delta is not yet isolated. The proposed control is cheap (one extra 8B run with α=1 under the TOP-D schedule) and decisive. I therefore keep the Reader's CONDITIONAL verdict and only partially agree on the weakest assumption: Assump. 4.1 is real but secondary to the baseline confound for the claim as stated.","tokens_in":19276,"tokens_out":713,"duration_ms":32943,"concrete_test":"Re-train the Qwen3-8B student under the exact TOP-D hyper-parameters of Table 5 (global batch 512, mini-batch 32, 16 mini-batches, E=1, same optimizer/LR) but with α=1.0 (i.e., ordinary OPD reward + identical off-policy reuse). Report AIME24 avg@32. If the gap to TOP-D (α=0.1/0.2) shrinks below ~5 points, the headline 25.84-point claim is largely an off-policy-reuse effect rather than a proximal-teacher effect.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim packages three things: (i) the proximal reward ˜r_k = log(α ρ_k + 1-α) that bounds variance (Thm. 4.2), (ii) internal trust-region / off-policy reuse (Alg. 1, E>1, mini-batch 32), and (iii) a 25.84-point AIME24 avg@32 lift over \"standard OPD\". Table 5 shows the confound: OPD is run with mini-batch size = global batch (strictly on-policy, 1 mini-batch) while TOP-D uses 16 mini-batches and off-policy epochs. Figure 5's \"TOP-D w/o off-policy\" curve already collapses sample efficiency, and the α=1.0 curve (unbounded reward + reuse) is also weak, but the paper never reports the natural control: standard OPD reward with the identical PPO-style reuse schedule. Without that cell, the massive absolute gain cannot be attributed primarily to the proximal teacher that the theory advertises; it may largely be ordinary off-policy RLVR-style data reuse applied to a dense reward. Theorems 4.2/4.4/4.9 remain formally interesting, but the empirical pillar of the strongest claim is not cleanly supported.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes Trust Region Policy Distillation (TOP-D) to stabilize On-Policy Distillation (OPD). It replaces the unbounded token reward r_k = log ρ_k with a proximal-teacher reward ˜r_k = log(α ρ_k + 1 − α) obtained by interpolating teacher and student probabilities, and couples this with PPO-style internal trust-region iterations that reuse on-policy rollouts. Theoretically it proves a uniform second-moment bound on the resulting policy-gradient estimator (Thm. 4.2), a contraction of the student toward the teacher up to per-step optimization error (Thm. 4.4), and a performance-difference lower bound that yields monotonic improvement of reverse KL under internal updates (Thm. 4.9). Empirically, on DAPO-Math-17k with Qwen3-1.7B/8B students and stronger Qwen teachers, TOP-D reports large avg@32 gains over GRPO, DAPO and standard OPD (e.g., +25.84 points on AIME24 for the 8B student), with ablations on α and off-policy reuse and zero extra model FLOPs beyond the algebraic reward transform.","tokens_in":19561,"tokens_out":1453,"duration_ms":19163,"significance":"If the claims hold, TOP-D is a practically useful, zero-overhead drop-in for a widely used post-training recipe: a simple, closed-form reward reparameterization that simultaneously bounds gradient variance and licenses safe data reuse. The theory package (variance control + global contraction + monotonic improvement) is more complete than typical OPD engineering notes, and the reported AIME lifts are large enough to matter for mathematical reasoning. Strengths that should be credited include the fully algebraic proximal reward (no extra teacher forward pass), the detailed Appendix proofs of Thms. 4.2/4.4/4.9, and the explicit ablations of α and off-policy reuse in Figure 5. The main open question is how cleanly the headline gains can be attributed to the proximal teacher versus ordinary off-policy reuse of a dense reward.","major_comments":[{"comment":"The headline 25.84-point AIME24 gain over “standard OPD” (Table 2, abstract, §5.2) confounds the proximal teacher with off-policy reuse. Table 5 shows OPD is run with mini-batch size = global batch (strictly one pass, no reuse) while TOP-D uses 16 mini-batches and off-policy epochs. Figure 5’s α=1.0 curve (unbounded reward + reuse) and “w/o off-policy” curve show both ingredients matter, but the natural control—standard OPD reward with the identical PPO-style reuse schedule—is never reported. Without that cell the central empirical claim that the proximal teacher is what “transforms” OPD cannot be cleanly attributed; the gain may largely be dense-reward off-policy RLVR. Please add this matched baseline (or re-run OPD under TOP-D’s mini-batch schedule) and revise the attribution language accordingly.","section":null},{"comment":"Theorem 4.2’s uniform variance bound rests entirely on Assumption 4.1 (∥∇_θ log π_θ(y_k | x, y_<k)∥ ≤ M for every prompt, prefix and token). The bound scales as M²|V|; if the score function is unbounded under modern LLM parameterizations or long contexts, the claimed “strict” control fails. The assumption is stated without Lipschitz analysis, gradient-norm measurements, or even a qualitative argument for transformer students. Either supply empirical score-norm statistics on the trained models or weaken the claim from “inherently controls gradient variance” to a conditional bound under Assumption 4.1, and discuss when the assumption is expected to hold.","section":null},{"comment":"The closed-loop argument in §4.3 (internal iterations drive ∥ϵ_k∥_1 ≤ δ via Pinsker, thereby closing the ϵ_∞/α gap of Thm. 4.4) is stated for exact optimization of the surrogate M_π(˜π). Algorithm 1 instead optimizes a clipped, token-level-normalized PPO objective (Eqs. 7–8) whose relationship to M_π is not established. The length-normalized future return in Eq. (8) also differs from the undiscounted cumulative reward used in the MDP of §2.2. Please either prove that the practical surrogate still guarantees the same monotonicity / error reduction, or clearly separate the theoretical guarantee (exact trust-region steps) from the implemented algorithm.","section":null}],"minor_comments":[{"comment":"Figure 3 caption and surrounding text claim ˜r_k is “strictly bounded from below”; the bound log(1−α) is correct, but the figure’s vertical axis is truncated and the positive-reward regime is hard to read. Consider a log-scale or inset for ρ_k ≫ 1.","section":null},{"comment":"Notation: ρ_k is introduced in Eq. (3) as π*/π_θ, yet Algorithm 1 line 6 evaluates the ratio under π_θold. A short remark that the proximal reward is always computed w.r.t. the behavior policy would avoid confusion.","section":null},{"comment":"Table 1 lists theoretical guarantees as “strong” for TOP-D and “weak” for OPD/RLVR; given that the guarantees are conditional on Assumptions 4.1 and 4.6 and on exact internal optimization, a more measured wording (“stronger / conditional”) would be fairer.","section":null},{"comment":"Limitations section correctly notes the 8B scale and short training horizon; it would help to also flag that all teachers are from the same Qwen3 family, so cross-family capacity-gap behavior remains untested.","section":null},{"comment":"Typos / polish: “aproximal teacher” (abstract), “T oken-level” in Fig. 3, and occasional missing spaces before citations. A light copy-edit pass is warranted.","section":null}],"recommendation":"major_revision","confidential_remarks":"The experimental confound (OPD vs TOP-D mini-batch schedule) is the single most important fix; if the authors add the matched OPD+reuse baseline and the proximal teacher still wins by a large margin, the paper becomes a clear accept. The theory is standard but carefully executed; I would not reject on Assumption 4.1 alone if it is properly scoped. Scope fits a solid ML conference / journal track on LLM post-training."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful core is simple: replace the OPD token reward log(ρ) with log(αρ + 1 − α). That single algebraic change lower-bounds the signal, yields a uniform second-moment bound (Thm 4.2), and turns the proximal teacher into a contraction whose asymptotic gap is ϵ∞/α (Thm 4.4). They then close the loop with a standard performance-difference lower bound that justifies multi-epoch clipped updates (Thm 4.9). The proofs in the appendix are detailed and standard once the finite-horizon MDP and the interpolated reward are fixed; nothing looks circular.\n\nEmpirically the method works. On Qwen3-8B and 1.7B students the AIME lifts over both OPD and strong RLVR (GRPO/DAPO) are large, ablations show α = 1 and pure on-policy both hurt, and α itself is robust in {0.1, 0.2, 0.3}. Zero extra model compute is genuine—the proximal teacher never has to be materialized.\n\nThe soft spot is attribution, not existence of the gains. Table 5 shows OPD is run strictly on-policy (mini-batch = global batch) while TOP-D gets 16 mini-batches and off-policy epochs. Figure 5’s “w/o off-policy” curve already tanks sample efficiency, and the α = 1 curve (unbounded reward + reuse) is weak, but the natural control—standard OPD reward with the identical PPO-style reuse schedule—is missing. So the 25.84-point AIME24 number packages the proximal reward and ordinary data reuse; we cannot yet say how much each contributes. Assumption 4.1 (uniformly bounded score-function norm) is also load-bearing for the variance theorem and is never checked. Runs are short, single-seed, and stop at 8B; no code.\n\nStill, the idea is clean, the math is honest, and the practical recipe is immediately usable. Anyone doing dense-reward post-training or reverse-KL distillation should read it. I would send it to referees; the missing control and the bounded-score assumption are fixable revision items, not desk-reject reasons.","headline":"Clean one-line reward fix plus solid theory; the 25-point AIME claim is real but confounded by unmatched off-policy reuse, so treat the headline number carefully.","tokens_in":20209,"tokens_out":572,"would_cite":true,"duration_ms":6667,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"TOP-D stabilizes on-policy distillation by building a proximal teacher that bounds gradient variance and yields large gains on math reasoning with no extra compute.","keywords":["policy distillation","on-policy distillation","trust region","proximal teacher","gradient variance","large language models","mathematical reasoning"],"falsifier":"Train an identical student–teacher pair with α = 1 (recovering ordinary on-policy distillation) versus α ≈ 0.1–0.2 under the same optimizer and data; if the measured gradient norms remain unbounded or the AIME accuracy gap disappears, the claimed stabilizing effect of the proximal teacher is refuted.","tokens_in":20141,"feed_emoji":"📐","tokens_out":691,"duration_ms":5655,"temperature":0.7,"pith_summary":"On-policy distillation of large language models is attractive because it gives dense token-level rewards and avoids the forgetting of pure supervised fine-tuning, yet it is fragile: when a strong teacher assigns near-zero probability to a student token the logarithmic reward explodes and gradients become unusable. Trust Region Policy Distillation (TOP-D) replaces that unbounded signal with a simple algebraic rewrite that mixes the teacher and student probabilities, producing a reward that is strictly lower-bounded. The same construction is proved to keep the second moment of every token gradient finite, to contract the student toward the teacher up to a controllable optimization residual, and, when combined with clipped trust-region updates, to guarantee monotonic improvement of a lower bound on expected return. Empirically the method turns unstable training curves into smooth ones and lifts average accuracy on AIME-style math contests by more than twenty points over ordinary on-policy distillation, all without any extra model evaluations or memory cost.","feed_headline":"Proximal teacher stabilizes on-policy distillation","feed_subtitle":"Bounded rewards cut variance, guarantee improvement, and lift AIME scores 25 points with zero extra compute","key_machinery":"The proximal teacher ˜π* = α π* + (1 − α) π_θ and the resulting bounded reward ˜r_k = log(α ρ_k + 1 − α). It is the object that simultaneously caps gradient second moments (Theorem 4.2), drives the global contraction (Theorem 4.4), and underpins the monotonic-improvement lower bound used by the internal trust-region steps (Theorem 4.9).","core_discovery":"By interpolating teacher and student probabilities into a proximal teacher, the token reward becomes ˜r = log(α ρ + 1 − α). This single algebraic change both prevents variance explosion and, together with internal trust-region iterations, supplies a closed theoretical loop of bounded variance, global contraction, and monotonic policy improvement, converting the previously unstable on-policy distillation paradigm into a reliable training method that needs no additional compute.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Proximal teacher bounds variance in on-policy distillation","Trust-region iterations stabilize policy distillation fully","Interpolated teacher yields monotonic improvement for free","TOP-D turns unstable OPD into reliable zero-compute method","Dynamic proximal teacher locks global convergence guarantees"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The analysis assumes that the gradient of the log-probability of every token under the student is bounded by a single fixed constant for all prompts and prefixes; if that uniform bound fails for modern language models, the variance-control guarantee no longer holds.","fun_headline_variants_meta":{"raw":{"variants":["Proximal teacher bounds variance in on-policy distillation","Trust-region iterations stabilize policy distillation fully","Interpolated teacher yields monotonic improvement for free","TOP-D turns unstable OPD into reliable zero-compute method","Dynamic proximal teacher locks global convergence guarantees"]},"model":"grok-4.5","effort":"low","cost_usd":0.003862,"raw_usage":{"total_tokens":1159,"prompt_tokens":677,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":38620000,"prompt_tokens_details":{"text_tokens":677,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":408,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":677,"tokens_out":74,"duration_ms":3335,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T14:05:28.314385+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train an identical student–teacher pair with α = 1 (recovering ordinary on-policy distillation) versus α ≈ 0.1–0.2 under the same optimizer and data; if the measured gradient norms remain unbounded or the AIME accuracy gap disappears, the claimed stabilizing effect of the proximal teacher is refuted.","supporting_citations":[],"review_version":1}