{"id":"d201fd23-5704-4f15-a4d7-761c2f3b4020","arxiv_id":"2608.11698","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"REOPD replaces the single global extrapolation coefficient in on-policy distillation with per-token compatibility weights and an adaptive batch budget, matching or modestly beating fixed-coefficient baselines on math and code.","lead":"This paper introduces REOPD, a training method that lets an LLM student decide per token how strongly to push beyond its teacher instead of using one fixed strength everywhere. It reports small gains over fixed-coefficient distillation baselines on math and code benchmarks while removing per-domain tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are within likely sampling noise: no repeated seeds or uncertainty are reported, and pooled-completion statistics overstate independence; the central 'comparable or better' claim needs variance-aware evidence.","rationale":"The reader's stated weakest assumption is that the compatibility proxy q_t reliably gates extrapolation safety, and their rationale notes the q_t-is-not-correctness limitation. I agree that is a genuine limitation, but it is explicitly acknowledged in Sec. 5 and is not the load-bearing point: even a purely heuristic compatibility gate could in principle deliver the claimed empirical performance. The load-bearing concern is statistical: the central claim is that REOPD matches or exceeds strong baselines, and the only evidence for that claim is single pooled numbers with no seeds, no error bars, and cluster-correlated completions. The text itself says 'across repeated experiments' without reporting the repetitions, while Sec. 4.1 exposes only seed 42 for ablations. The Section 4.3 oracle comparison is a further single-run comparison. These evidentiary gaps directly undermine the strongest_claim as stated, which is an empirical claim. My recommended verdict is unchanged: CONDITIONAL remains appropriate, now conditioned specifically on repeated-seed results with cluster-robust uncertainty and released artifacts. I do not see a reason to move to ACCEPT or REJECT based on the manuscript text. The algorithm is well specified, the derivation of Eq. (7) is internally consistent (delta_hat is nonnegative by e^x >= x+1, so q_t is in (0,1]), and the ablation isolating token compatibility is informative. The remaining question is whether the reported margins survive proper statistical treatment.","tokens_in":10996,"tokens_out":6552,"duration_ms":73923,"concrete_test":"Re-run all four main configurations (single-teacher math, single-teacher code, multi-teacher math, multi-teacher code) with at least 10 independent training seeds under the exact Sec. 4.1 protocol, recording per-problem accuracies. Compute cluster-bootstrap 95% confidence intervals over problems for the REOPD-minus-ExOPD(lambda=1.25) and REOPD-minus-OPD differences. If any claimed 'outperforms' interval includes zero, or if the single-teacher code margin versus the best fixed lambda reverses sign, the central claim should be downgraded to 'not statistically distinguishable' until artifacts are released.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: REOPD reaches accuracy comparable to or better than OPD/ExOPD without a verifier or coefficient sweep. The evidence underdetermines this claim. Table 1 reports single pooled numbers from a step-50 checkpoint with no seeds, confidence intervals, or variance; the text claims 'across repeated experiments' (Secs. 4.1 and 5) but only seed 42 is mentioned, and only for ablations. Pooling 3,840 math completions from 120 problems (or 2,868 code completions from 717 tasks) treats responses to the same problem as independent; the effective sample is the problem, so the reported 0.19 to 1.38 point differences can easily be within cluster noise. Section 4.3's comparison to the best fixed lambda chosen on the same evaluation sets is an oracle baseline; while that is conservative for REOPD in one sense, the single-run comparison still gives no estimate of the variability of the 0.19 to 0.42 point margins. Consequently the abstract's 'outperforms' and 'matching' statements are not statistically supported, and the shared-controller claim (one controller for math and code without sweeps) rests on margins of +0.03 and +0.42 points. This is not an internal inconsistency in the algorithm; the method is clearly specified and the ablation supports token gating as active. But the headline claim's load-bearing weight is carried by differences that the reported statistics cannot resolve. The acknowledged q_t-is-not-correctness limitation is real but secondary: even if q_t is only a compatibility proxy, the paper's empirical claim could still hold, so the decisive issue is the missing uncertainty quantification and artifact release.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes REOPD, a reliability-adaptive extension of ExOPD for on-policy distillation. REOPD replaces the global extrapolation coefficient with a token-level compatibility weight q_t and a micro-batch-level adaptive budget γ_b, yielding an effective token-wise coefficient λ_{b,t}=1+γ_b q_t that preserves the standard teacher-alignment term while gating only the beyond-teacher residual. The method requires no verifier, reward model, value model, or extra rollout beyond standard OPD. The paper evaluates REOPD on single-teacher mathematics, single-teacher code, and mixed-domain multi-teacher distillation, reporting that REOPD outperforms OPD and the fixed λ=1.25 ExOPD baseline on mathematics and on both multi-teacher domains, while performing comparably on single-teacher code. A component ablation in the mathematics setting shows that removing the token-level gate lowers accuracy by 4.27 points, identifying token compatibility as the main active component.","tokens_in":11242,"tokens_out":6545,"duration_ms":66943,"significance":"If the empirical claims hold, the contribution is useful: the decomposition of teacher alignment from extrapolation control is conceptually clean, the special cases (γ_b=0 recovers OPD, q_t=1 and fixed γ_b recovers ExOPD) are well stated, and the ablation cleanly attributes the main effect to the token gate rather than to the batch controller. The method is not circular: q_t and γ_b are stop-gradient functions of already-computed log-probabilities, and B0 is calibrated on alignment RMS, not on evaluation accuracy. The main weakness is that the headline comparisons rest on small differences (0.19 to 1.38 percentage points) reported without repeated seeds, confidence intervals, or cluster-aware uncertainty. Because the evaluation pools many completions per problem, the effective sample size is much smaller than the reported number of completions, so the central 'outperforms/matching' claims are currently underdetermined. The paper acknowledges that q_t is a compatibility measure rather than a correctness estimator; that limitation is real but secondary relative to the missing uncertainty quantification.","major_comments":[{"comment":"The central empirical claim is not yet supported by the reported statistics. All main results come from a single step-50 checkpoint, no repeated seeds are reported for the main table, and the only seed mentioned (42) is used for ablations. The text says 'across repeated experiments' (Secs. 4.1 and 5), but no repeated experiments are shown. Pooling 3,840 mathematics completions from 120 problems, or 2,868 code completions from 717 tasks, treats responses to the same problem as independent, while the effective sampling unit is the problem. The reported differences of 0.19 to 1.38 percentage points can therefore easily be within cluster noise. Please provide variance-aware evidence: multiple seeds with means and standard deviations, or cluster bootstrap confidence intervals at the problem level, and state explicitly whether the REOPD advantages over OPD and ExOPD are statistically significant. Without this, the abstract's 'outperforms' and 'matching' statements are not supported.","section":"Secs. 4.1, 4.2, Table 1"},{"comment":"The comparison against the best fixed coefficient chosen from the same evaluation sweep is an oracle baseline, and the paper is transparent that this is not a post-hoc favorable baseline for REOPD. However, the four margins reported here are +0.19, −0.15, +0.03, and +0.42 percentage points, which fall within the uncertainty range described in the previous comment. The claim that REOPD 'reaches comparable or better performance than the task-specific best fixed coefficient' needs error bars or confidence intervals for both the REOPD runs and the swept ExOPD curves. Without such intervals, the reader cannot distinguish a genuine adaptation advantage from sampling noise.","section":"Sec. 4.3"}],"minor_comments":[{"comment":"The phrase 'low-variance k 3 discrepancy proxy' is unclear: the symbol 'k' or 'k3' is never defined, and the equation as written only introduces x and δ. Please either define the term or remove it.","section":"Sec. 3.3, Eq. (6)"},{"comment":"The auto-calibration of B0 is described only verbally as a 'scaled moving average of the alignment RMS, κ RMS_b(a)' during the first K0 controller calls. Please provide the exact formula and initialization details, since B0 directly affects the magnitude of the extrapolation budget.","section":"Sec. 4.1"},{"comment":"The terminology is inconsistent: the abstract and Sec. 4.2 refer to 'G-OPD' as the comparison method, while Table 1 labels the corresponding rows 'OPD'. Please clarify whether G-OPD denotes the general framework and 'OPD' denotes the λ=1 instantiation, and use the terminology consistently throughout.","section":"Table 1 and Sec. 4.2"},{"comment":"Panel (b) reports 'mean compatibility q_t and effective coefficient λ_{b,i,t}'; because the effective coefficient is token-wise and varies within a batch, please clarify over which tokens the mean is taken and whether the plotted quantity is the micro-batch average of λ or the average of γ_b times the average q_t.","section":"Figure 3"},{"comment":"The abstract and title use 'reliability' to describe the gating mechanism, but Sec. 3.3 correctly states that q_t measures local compatibility, not correctness. Since the paper explicitly acknowledges that the gate cannot detect cases where both student and teacher are wrong, please qualify the abstract's phrase 'reliable teacher-reference directions' to avoid overstating the semantic content of the gate.","section":"Sec. 3.3 and Conclusion"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"REOPD is a clean, well-specified method paper. The token-level compatibility gate multiplied by a synchronized, EMA-smoothed micro-batch budget inside the extrapolation coefficient is a genuine extension of ExOPD; I don't see it in the cited TIP or adaptive-distillation work. The separation between preserving teacher alignment and gating only the beyond-teacher residual is well motivated, and the whole controller is stop-gradient, using only log-probabilities already computed in the pipeline. That is a useful subfield-level contribution.\n\nThe ablation is the strongest part. Removing the token gate drops pooled accuracy by 4.27 points on math; removing the explicit bound drops it by only 0.50. That cleanly identifies token compatibility as the active component. The limitations section is also honest: q_t is a compatibility proxy, not a correctness estimator, and the paper says it cannot catch cases where both student and teacher are wrong. Good.\n\nThe soft spot is the evidence. Table 1 reports single pooled numbers from a step-50 checkpoint, with no seeds, no confidence intervals, no variance. Pooling 3,840 math completions from 120 problems treats responses to the same problem as independent, and the effective sample is the problem, so margins of 0.19 to 1.38 points are plausibly within cluster noise. The text claims “across repeated experiments” but only seed 42 appears, and only for ablations. Section 4.3 compares REOPD to the best fixed lambda selected on the same evaluation sets, which is an oracle baseline; that is conservative for REOPD in one sense, but it still gives no estimate of variability. The shared-controller claim rests on +0.03 and +0.42 point margins. No code, checkpoints, or data are released. The controller also retains tau, gamma_max, B0 calibration, and EMA/warm-up hyperparameters; saying no lambda sweep doesn't mean no tuning, though the paper does acknowledge this in the limitations.\n\nThe method is plausible, the math is consistent, and the citation pattern looks appropriate. The central claim just needs variance-aware evidence. I'd send this to a serious referee, with the clear request for repeated seeds, clustered standard errors, and released artifacts. Worth engaging with as a method proposal; not yet proven as a performance claim.","headline":"Clean, well-specified method proposal with a strong ablation, but the headline empirical claims are not yet supported by single-run, no-error-bar numbers.","tokens_in":11918,"tokens_out":2505,"would_cite":false,"duration_ms":25075,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single shared adaptive controller for on-policy distillation matches or beats per-domain tuned extrapolation coefficients across math, code, and multi-teacher settings.","keywords":["on-policy distillation","reward extrapolation","token-level compatibility","adaptive budget","large language models","teacher alignment","multi-teacher distillation","reliability-adaptive control"],"falsifier":"Take a training set where teacher outputs are systematically corrupted at the token level (for example, math solutions with deliberately inserted arithmetic slips), keep the student initialization and prompts identical, and compare REOPD against OPD and a verifier-gated baseline. If the tokens receiving the largest amplified residual are the corrupted tokens and accuracy drops relative to OPD, then the compatibility gate is not a sufficient safety signal.","tokens_in":10659,"feed_emoji":"🎯","tokens_out":8743,"duration_ms":77470,"temperature":0.7,"pith_summary":"The paper tackles a specific instability in on-policy distillation: when a student is trained to push beyond its teacher by scaling the teacher–reference log-ratio with a single global constant $\\lambda$, a few extreme tokens can dominate the update, and the preferred $\\lambda$ changes across domains. REOPD separates teacher alignment from extrapolation, using a token-wise coefficient $\\lambda_{b,t}=1+\\gamma_b q_t$ that keeps the standard alignment term intact and gates only the beyond-teacher residual through a compatibility weight $q_t$ and a shared micro-batch budget $\\gamma_b$. The paper claims this makes per-domain coefficient sweeps unnecessary: a single adaptive controller outperforms G-OPD on single-teacher mathematics and on both multi-teacher domains, matches it on single-teacher code, and requires no verifier, reward model, value model, or extra rollout. A reader should care because the method, if right, replaces a brittle global hyperparameter with an online control signal built entirely from log-probabilities that OPD already computes.","feed_headline":"Shared controller beats per-domain tuning in LLM distillation","feed_subtitle":"Matches or beats per-domain tuned baselines on math and code, with no verifier or extra rollout.","key_machinery":"The load-bearing object is the token-wise effective coefficient $\\lambda_{b,t}=1+\\gamma_b q_t$, inserted into the distillation cost as $C_{b,t}=a_{b,t}-\\gamma_b q_t r_{b,t}$. Here $a_t$ is the student–teacher alignment cost, $r_t$ is the teacher–reference log-ratio that defines the implicit reward, $q_t$ is a compatibility weight built from a stop-gradient student–teacher discrepancy proxy $\\hat{\\delta}_t$ (with $q_t=\\exp(-\\hat{\\delta}_t/\\tau)$, so agreement keeps $q_t$ near $1$ and disagreement attenuates the residual), and $\\gamma_b$ is a shared micro-batch budget computed from the compatibility-weighted residual proportion $\\rho_b$ and the reliable residual RMS $s_b$ after EMA smoothing, clipped to $[0,\\gamma_{\\max}]$. This object does the work of separating teacher alignment from extrapolation: it reduces to standard OPD at $\\gamma=0$, to fixed-$\\lambda$ ExOPD when $q=1$ and $\\gamma=\\lambda-1$, and it guarantees $1\\le \\lambda_{b,t}\\le 1+\\gamma_{\\max}$ so no single token can receive unbounded extrapolation.","core_discovery":"The central claim is that reward extrapolation in on-policy distillation fails not because extrapolation is dangerous, but because a single global coefficient applies the same gain to every token. REOPD establishes that the token-wise coefficient $\\lambda_{b,t}=1+\\gamma_b q_t$ resolves this by decomposing ExOPD's uniform residual multiplier into two adaptive parts: a token gate $q_t$ that preserves the residual when the student and teacher agree locally and suppresses it when they disagree, and a shared, bounded budget $\\gamma_b$ that reflects how much compatible residual the current batch can support. The paper states that this construction preserves teacher alignment at all times, bounds the effective coefficient to $[1, 1+\\gamma_{\\max}]$, and reuses only the student, teacher, and reference log-probabilities already needed by G-OPD. Empirically, REOPD reports better or matching accuracy relative to G-OPD across single-teacher math, single-teacher code, and shared multi-teacher distillation, and its ablation shows that removing the token gate costs 4.27 points while removing the budget bound costs 0.50 points, identifying token-level filtering as the principal component.","pith_inferences":["Editorial inference: since $q_t$ measures compatibility rather than correctness, the method's safety depends on the teacher being mostly right; injecting a sparse, cheap correctness signal, such as verifier labels only where $q_t$ is high, could suppress confident joint errors without giving up the adaptive budget.","Editorial inference: the logged dynamics show $\\gamma_b$ saturating at its upper bound late in training, suggesting the controller's main work happens early; a simpler schedule that starts small and anneals to $\\gamma_{\\max}$ may reproduce much of REOPD's benefit in single-teacher settings.","Editorial inference: the discrepancy proxy is computed per token on the sampled response, so a token with low $q_t$ because of an earlier student mistake is merely down-weighted, not corrected; combining REOPD with rollout truncation or rollback could produce alignment gains the current method leaves on the table.","Editorial inference: the controller is tested only at one model scale and teacher family, so the strongest testable extension is running REOPD on larger student models and heterogeneous teacher families to see whether the shared budget remains stable."],"forward_implications":["Fixed-coefficient ExOPD's per-domain $\\lambda$ sweep can be replaced by online adaptation using only quantities already computed in OPD, eliminating a costly full-training tuning loop.","Token-level compatibility, not the batch budget, is the main source of improvement, so residual filtering per token is the intervention that stabilizes reward extrapolation.","A single shared adaptive controller works across routed domain teachers, so multi-teacher distillation does not need separate per-teacher coefficients.","The added safety comes at no extra inference cost: no verifier, reward model, value model, or rollout beyond standard OPD is required.","Because the effective coefficient is bounded by $1+\\gamma_{\\max}$, extreme teacher–reference log-ratio peaks cannot dominate the policy update beyond the explicit cap."],"supporting_citations":[{"why":"Defines the dense KL-regularized RL formulation of OPD and the ExOPD global-coefficient baseline that REOPD extends.","marker":"[4]"},{"why":"Introduces on-policy distillation of language models, the training setting whose tokens REOPD re-weights.","marker":"[1]"},{"why":"Provides the phenomenology of student–teacher compatibility in OPD that motivates the token-level compatibility gate.","marker":"[2]"},{"why":"Supplies the PPO-style policy surrogate used for the actor update in both the baselines and REOPD.","marker":"[15]"},{"why":"Supplies the filtered DeepMath-103K level-6 mathematics training examples used in the single- and multi-teacher runs.","marker":"[17]"},{"why":"Supplies the Eurus code training split used for the code-domain experiments.","marker":"[18]"},{"why":"Provides the EvalPlus evaluation harness used for HumanEval+ and MBPP+ scores in the code results.","marker":"[22]"},{"why":"Provides the LiveCodeBench v6 test6 evaluation split used for the code accuracy numbers.","marker":"[23]"}],"fun_headline_variants":["Token-level reliability adapts reward extrapolation in distillation","No more global lambda: token-wise coefficient boosts LLM distillation","Reliability-adaptive reward beats global coefficient in distillation","Per-token gains replace global lambda in on-policy distillation","Token-aware coefficient matches per-domain tuning without extra rollout"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that local student–teacher agreement marks the teacher's residual signal as safe to amplify, so if the student and teacher both agree on tokens that are nevertheless wrong, REOPD extrapolates confidently in wrong directions.","fun_headline_variants_meta":{"raw":{"variants":["Token-level reliability adapts reward extrapolation in distillation","No more global lambda: token-wise coefficient boosts LLM distillation","Reliability-adaptive reward beats global coefficient in distillation","Per-token gains replace global lambda in on-policy distillation","Token-aware coefficient matches per-domain tuning without extra rollout"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000643,"raw_usage":{"total_tokens":2980,"prompt_tokens":993,"completion_tokens":1987,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1921}},"tokens_in":609,"tokens_out":1987,"duration_ms":14320,"temperature":1.0,"reasoning_tokens":1921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:30:46.786711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a training set where teacher outputs are systematically corrupted at the token level (for example, math solutions with deliberately inserted arithmetic slips), keep the student initialization and prompts identical, and compare REOPD against OPD and a verifier-gated baseline. If the tokens receiving the largest amplified residual are the corrupted tokens and accuracy drops relative to OPD, then the compatibility gate is not a sufficient safety signal.","supporting_citations":[{"cited_title":"On-policy distillation of language mod- els: Learning from self-generated mistakes","cited_arxiv_id":null,"evidence_quote":"Introduces on-policy distillation of language models, the training setting whose tokens REOPD re-weights."}],"review_version":1}