{"id":"ffeb8f90-d7af-4846-88c0-e4358810553d","arxiv_id":"2605.13230","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"TGPO improves on-policy reasoning distillation in LLMs by using teacher-guided token generation conditioned on student contexts together with trajectory rewards, outperforming RKL-based methods under large policy divergence.","lead":"The paper proposes Teacher-Guided Policy Optimization (TGPO) as an on-policy distillation technique for LLM reasoning that adds direct teacher token-level guidance on student contexts plus RLVR rewards to handle large teacher-student divergence. A smart generalist might read it because better distillation methods could make post-training of reasoning models more reliable when initial policies differ substantially.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Conditioning the teacher on student-generated prefixes under large divergence may not reliably yield higher-quality tokens, as the teacher's conditional predictions could degrade without explicit validation.","rationale":"The reader's weakest assumption directly identifies the untested conditioning step; the full-text experiments (as described in the abstract) do not appear to contain the isolating measurement above, so the concern remains load-bearing and moves the verdict from UNVERDICTED to CONDITIONAL pending that check.","tokens_in":1712,"tokens_out":348,"duration_ms":14673,"concrete_test":"On a held-out set of student-generated prefixes from the high-divergence regime (e.g., after 2–3 RLVR steps), compute the average verifiable reward of the top-1 token continuation under teacher guidance versus under the student's policy and versus RKL teacher targets; if the guided continuations do not improve reward by >5% relative to baselines, the token-level steering benefit is not demonstrated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The core claim is that TGPO's token-level teacher guidance on student prefixes (plus RLVR) steers exploration better than RKL when trajectories fall outside the teacher distribution. This requires that the teacher's p(· | student_prefix) produces continuations with higher expected reward/correctness than the student's own sampling or RKL targets. No section isolates this: experiments compare end-to-end performance but do not report token-level quality metrics, divergence between teacher guidance and optimal continuations, or ablations that disable the conditioning while keeping RLVR. If the conditioned teacher simply reproduces the same uninformative signal on OOD prefixes, the claimed advantage collapses to standard RLVR.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that existing reverse KL (RKL)-based on-policy distillation (OPD) methods for reasoning LLMs fail under large teacher-student policy divergence because RL-driven exploration produces trajectories outside the teacher distribution, yielding uninformative negative feedback. It proposes Teacher-Guided Policy Optimization (TGPO), which augments RLVR-style trajectory rewards with direct token-level teacher guidance conditioned on student-generated prefixes to steer exploration toward improved continuations, and reports that TGPO outperforms RKL-based OPD baselines while remaining robust across teacher models on reasoning benchmarks.","tokens_in":1858,"tokens_out":454,"duration_ms":23632,"significance":"If the core mechanism is validated, TGPO would address a practical limitation in combining on-policy RL with distillation for LLMs, enabling more effective reasoning post-training when student policies diverge substantially. The approach is a straightforward algorithmic combination rather than a parameter-free derivation or machine-checked result, so its significance hinges on whether the reported gains are attributable to the proposed conditioning rather than RLVR alone.","major_comments":[{"comment":"The central claim that teacher token predictions conditioned on student prefixes produce higher-quality continuations (and thus better steering than RKL) under large divergence is load-bearing, yet the manuscript provides no token-level quality metrics, no measurement of divergence between the conditioned teacher distribution and optimal continuations, and no ablation that disables the conditioning while retaining RLVR-style rewards.","section":"Abstract (mechanism description) and Experiments"},{"comment":"End-to-end benchmark outperformance is reported, but without isolating the contribution of the student-prefix conditioning (e.g., via a controlled comparison to pure RLVR or to RKL on the same trajectories), it is impossible to determine whether the claimed advantage over RKL-based OPD collapses to standard RLVR when trajectories fall outside the teacher distribution.","section":"Experiments"}],"minor_comments":[{"comment":"The abstract refers to 'RLVR-style trajectory level rewards' without defining the precise reward formulation or how it interacts with the token-level guidance.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address the major comments below and commit to revisions that strengthen the isolation of the proposed mechanism.","responses":[{"response":"We agree that token-level metrics and an explicit ablation would strengthen the mechanistic claims. While our primary results focus on end-to-end reasoning performance (the relevant downstream metric), we will add in revision: (i) an ablation of TGPO versus RLVR without the teacher conditioning, and (ii) token-level statistics such as average teacher log-probability on tokens generated under student prefixes. Direct measurement of divergence to 'optimal' continuations is not feasible without an oracle, but the added ablation will isolate the conditioning effect.","revision_made":"yes","referee_comment":"[Abstract (mechanism description) and Experiments] The central claim that teacher token predictions conditioned on student prefixes produce higher-quality continuations (and thus better steering than RKL) under large divergence is load-bearing, yet the manuscript provides no token-level quality metrics, no measurement of divergence between the conditioned teacher distribution and optimal continuations, and no ablation that disables the conditioning while retaining RLVR-style rewards."},{"response":"Our reported comparisons are against RKL-based OPD baselines under identical large-divergence settings and trajectories. We acknowledge that a direct head-to-head with pure RLVR (no teacher guidance) is missing and will include this controlled ablation in the revision to quantify the incremental benefit of the student-prefix conditioning.","revision_made":"yes","referee_comment":"[Experiments] End-to-end benchmark outperformance is reported, but without isolating the contribution of the student-prefix conditioning (e.g., via a controlled comparison to pure RLVR or to RKL on the same trajectories), it is impossible to determine whether the claimed advantage over RKL-based OPD collapses to standard RLVR when trajectories fall outside the teacher distribution."}],"tokens_in":1356,"tokens_out":411,"duration_ms":20980,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"TGPO targets the case where RL exploration pushes student trajectories outside the teacher's support, making standard RKL supervision uninformative. The fix is to let the teacher directly condition its next-token predictions on the student's own prefix and combine that with trajectory-level verifiable rewards.\n\nThe new piece is the explicit token-level guidance step rather than trajectory-only RKL. The abstract says this steers exploration better and holds up across different teachers on reasoning benchmarks. That matches a real pain point in current on-policy distillation setups.\n\nThe description stays high-level. No loss formulation, no description of how the teacher is prompted or masked on the student prefix, and no ablations that turn the conditioning off while keeping RLVR. The stress-test worry about teacher quality on OOD prefixes is not answered by end-to-end benchmark wins alone; without token-level quality checks or divergence measurements, it is possible the added signal is neutral or noisy.\n\nThe work is aimed at groups already running RLVR-style reasoning post-training. Readers who need a concrete way to keep teacher signal alive under large divergence will find the idea worth testing. The central limitation is practical rather than conceptual, and the paper engages honestly with the stated problem.\n\nSend it to referees; the targeted issue is common enough that a clear mechanism plus controlled experiments would be useful even if the gains turn out modest.","headline":"TGPO adds token-level teacher conditioning on student prefixes to RLVR, but the abstract gives no equations, ablations, or token metrics to show the conditioning actually helps under divergence.","tokens_in":2364,"tokens_out":354,"would_cite":false,"duration_ms":22223,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"TGPO uses teacher token guidance on student prefixes plus trajectory rewards to enable effective on-policy reasoning distillation under large policy divergence.","keywords":["on-policy distillation","reasoning distillation","policy optimization","large language models","reinforcement learning","teacher guidance","policy divergence","token-level supervision"],"falsifier":"Run TGPO and an RKL baseline on the same reasoning task with deliberately large divergence; if the TGPO continuations show no quality or benchmark improvement over RKL, the mechanism claim does not hold.","tokens_in":2619,"feed_emoji":"🤖","tokens_out":544,"duration_ms":24015,"temperature":0.7,"pith_summary":"Existing reverse KL supervision in on-policy distillation breaks down when student exploration produces trajectories far from the teacher distribution, yielding uninformative negative signals. TGPO counters this by letting the teacher directly predict each token conditioned on the student's own generated context, then combines those signals with trajectory-level verifiable rewards to steer sampling toward better sequences. The result is distillation that stays informative even when policies diverge substantially. Experiments show consistent gains over RKL baselines and stability across teacher choices on reasoning benchmarks.","feed_headline":"Teacher token guidance rescues distillation at large divergence","feed_subtitle":"Conditioning teacher predictions on student prefixes plus trajectory rewards steers exploration on reasoning tasks.","key_machinery":"Teacher token-level guidance conditioned on student-generated prefixes, paired with trajectory-level RLVR rewards.","core_discovery":"TGPO remains effective under large teacher-student policy divergence by using the teacher to directly guide token level generation conditioning on student-generated contexts; together with RLVR-style trajectory level rewards, TGPO steers exploration toward improved continuations.","pith_inferences":["The same conditioning trick could be tried in other distillation settings where student sampling routinely leaves the teacher support.","It may reduce the need to keep teacher and student policies artificially close during training.","One could measure the fraction of tokens where teacher guidance actually changes the student's choice and correlate that with final gains."],"forward_implications":["Outperforms RKL-based OPD methods on reasoning benchmarks.","Maintains performance robustness when different teacher models are used.","Avoids uninformative feedback from trajectories outside the teacher distribution.","Combines token-level teacher signals with trajectory rewards to direct exploration."],"fun_headline_variants":["TGPO conditions teacher on student prefixes for large divergence","Teacher token guidance on student contexts with trajectory rewards","TGPO steers exploration using teacher on student-generated contexts","On-policy distillation aided by teacher guidance at large divergence"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Conditioning the teacher's next-token predictions on prefixes sampled from the student will produce higher-quality continuations without adding new biases or shifts.","fun_headline_variants_meta":{"raw":{"variants":["TGPO conditions teacher on student prefixes for large divergence","Teacher token guidance on student contexts with trajectory rewards","TGPO steers exploration using teacher on student-generated contexts","On-policy distillation aided by teacher guidance at large divergence"]},"model":"grok-4.3","cost_usd":0.008918,"raw_usage":{"total_tokens":3885,"prompt_tokens":582,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":89178000,"prompt_tokens_details":{"text_tokens":582,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3243,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":582,"tokens_out":60,"duration_ms":12306,"temperature":1.0,"reasoning_tokens":3243,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T21:38:22.468381+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run TGPO and an RKL baseline on the same reasoning task with deliberately large divergence; if the TGPO continuations show no quality or benchmark improvement over RKL, the mechanism claim does not hold.","supporting_citations":[],"review_version":2}