{"id":"5f801be7-a015-416e-9d79-83dd8d7957bb","arxiv_id":"2605.29089","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"OISD improves mathematical reasoning in language models by using the final layer as an internal teacher to align logits and attention patterns in selected intermediate layers via signed advantage-weighted Jensen-Shannon divergence during GRPO optimization.","lead":"The paper introduces OISD, a framework that distills predictive signals from a language model's final layer to its intermediate layers during on-policy RL training with GRPO. A smart generalist might read it to understand a potential way to boost reasoning performance in AI models without extra external data or teachers.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Whether signed advantage-weighted JS alignment preserves GRPO's on-policy property when added to the objective","rationale":"Reader's weakest assumption targets signal transferability; my concern targets the orthogonal but equally load-bearing claim of unchanged on-policy GRPO dynamics under the added alignment. They are related but distinct; full text would be needed to confirm whether the weighting is implemented to avoid bias.","tokens_in":1720,"tokens_out":312,"duration_ms":27029,"concrete_test":"In the methods section, locate the combined loss (likely Eq. for GRPO + OISD); verify whether the JS term receives stop-gradient on the teacher or advantage weights; if gradients flow, re-derive the policy gradient to confirm advantages remain outcome-only; if not, the consistency claim holds.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that OISD distills signals while 'preserving policy consistency under a unified acting policy.' The mechanism is signed advantage-weighted Jensen-Shannon alignment applied during GRPO rollouts and updates, with the final layer as detached teacher. If the alignment loss is jointly optimized with the GRPO objective and the advantage weighting is not fully detached, the effective gradient for the acting policy could incorporate representation-matching terms, altering the relative advantages computed from outcome rewards alone. This would violate the on-policy assumption that advantages reflect only the current policy's outcome distribution. The abstract provides no equation showing loss combination or stop-gradient placement, leaving this interaction as the least secure link.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces the OISD framework for on-policy internal self-distillation during RL post-training of language models with GRPO. The final layer serves as a detached teacher that aligns selected intermediate layers via logit alignment (transferring reasoning behaviors) and attention alignment (enforcing consistent patterns) using signed advantage-weighted Jensen-Shannon divergence, without external privileged information. The central claim is that this distills informative intermediate representations while preserving policy consistency under a unified acting policy, yielding substantial and consistent improvements over strong reasoning RL baselines on four mathematical reasoning tasks.","tokens_in":1857,"tokens_out":494,"duration_ms":27250,"significance":"If the on-policy property is preserved and the improvements hold under proper controls, the approach would address a gap in outcome-only RL by leveraging internal predictive signals for better reasoning representations. The use of a unified policy and detached teacher is a clean design choice that could generalize beyond the reported tasks.","major_comments":[{"comment":"Abstract: the claim of 'substantial and consistent improvements' and 'preserving policy consistency under a unified acting policy' is load-bearing, yet the abstract supplies no quantitative results, baseline names, effect sizes, statistical tests, or ablation numbers. Without these, the empirical support for the central claim cannot be assessed.","section":"Abstract"},{"comment":"OISD + GRPO description (alignment mechanism paragraph): the signed advantage-weighted Jensen-Shannon alignment is jointly optimized with the GRPO objective, but no equation shows the combined loss, the coefficient on the alignment term, or stop-gradient placement on the advantage weights and final-layer teacher. If the advantage weighting is not fully detached, the gradient for the acting policy incorporates representation-matching terms, which would alter the on-policy assumption that advantages derive solely from outcome rewards.","section":"OISD framework description"}],"minor_comments":[{"comment":"The abstract mentions 'four mathematical reasoning tasks' but does not name them; adding the task names would improve clarity.","section":"Abstract"},{"comment":"The code release link is welcome, but the manuscript should include a short reproducibility checklist (hyperparameters for the alignment loss, layer selection criteria, and exact JS weighting) to support the promised release.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript to strengthen the presentation and clarify technical details.","responses":[{"response":"We agree that the abstract would benefit from quantitative support for the central claims. In the revised version we will incorporate specific results, including average performance gains over the GRPO baseline across the four mathematical reasoning tasks, the primary baseline names, and a brief reference to the evaluation protocol.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the claim of 'substantial and consistent improvements' and 'preserving policy consistency under a unified acting policy' is load-bearing, yet the abstract supplies no quantitative results, baseline names, effect sizes, statistical tests, or ablation numbers. Without these, the empirical support for the central claim cannot be assessed."},{"response":"We acknowledge that the manuscript describes the alignment mechanisms but omits an explicit combined-loss equation and the precise stop-gradient placements. The intended design detaches both the final-layer teacher and the advantage weights (computed exclusively from outcome rewards) so that representation-matching gradients do not affect the policy update. We will add the equation L_total = L_GRPO + λ L_OISD, report the coefficient λ used in experiments, and explicitly document the stop-gradient operations to confirm that the on-policy property is preserved.","revision_made":"yes","referee_comment":"[OISD framework description] OISD + GRPO description (alignment mechanism paragraph): the signed advantage-weighted Jensen-Shannon alignment is jointly optimized with the GRPO objective, but no equation shows the combined loss, the coefficient on the alignment term, or stop-gradient placement on the advantage weights and final-layer teacher. If the advantage weighting is not fully detached, the gradient for the acting policy incorporates representation-matching terms, which would alter the on-policy assumption that advantages derive solely from outcome rewards."}],"tokens_in":1387,"tokens_out":415,"duration_ms":30079,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this paper proposes on-policy internal self-distillation inside GRPO training: the final layer serves as a detached teacher for selected intermediate layers, aligning logits (to transfer reasoning behavior) and attention patterns (to enforce consistent focus) via signed advantage-weighted Jensen-Shannon divergence, all without external teachers. It claims this yields substantial gains on four math reasoning tasks.\n\nWhat is new is the combination of logit-plus-attention alignment under a unified acting policy during rollout and optimization. The framing correctly notes that standard outcome-reward RL overlooks predictive signals in intermediate layers, and the two mechanisms target different aspects of that signal.\n\nThe paper does a reasonable job laying out why internal signals matter and why keeping the teacher on-policy could avoid distribution shift. The idea of using the model's own final layer as teacher is straightforward and avoids privileged information.\n\nThe soft spots are more central. The abstract supplies zero numbers, no baseline details, no statistical tests, and no equations showing how the alignment loss combines with the GRPO objective or where stop-gradients sit. Without those, the central claim cannot be evaluated. The stress-test concern lands: if the signed advantage weighting is not fully detached, the alignment term could alter the effective advantages that GRPO relies on, undermining the on-policy guarantee. The abstract gives no indication this interaction was checked.\n\nThis work is aimed at groups already running GRPO-style RL on reasoning models who want to experiment with internal distillation. A reader already familiar with the GRPO paper and basic distillation losses could extract the high-level recipe, but would still need the full methods to implement or judge it.\n\nI would send it to peer review so the experiments and loss derivations can be examined, though the current abstract is too thin to assess impact on its own.","headline":"OISD adds a specific on-policy internal distillation step to GRPO but the abstract shows no results and leaves the policy-consistency claim unverified.","tokens_in":2360,"tokens_out":437,"would_cite":false,"duration_ms":28546,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Language models improve reasoning by distilling final-layer signals into intermediate layers during on-policy RL training.","keywords":["on-policy self-distillation","internal representations","language model reasoning","reinforcement learning","GRPO","logit alignment","attention alignment","mathematical reasoning"],"falsifier":"Applying OISD during GRPO training and observing no improvement or outright degradation on the four mathematical reasoning benchmarks compared with GRPO alone would falsify the central claim.","tokens_in":2625,"feed_emoji":"","tokens_out":652,"duration_ms":22440,"temperature":0.7,"pith_summary":"The paper introduces on-policy internal self-distillation (OISD) to move predictive signals from the final layer of a language model into selected intermediate layers. It does this inside the same rollout used for Group Relative Policy Optimization by treating the final layer as a detached internal teacher. Alignment occurs through two channels: logit alignment transfers high-level reasoning behaviors and attention alignment transfers consistent focus patterns, both using signed advantage-weighted Jensen-Shannon loss to keep the policy unchanged. Readers would care because standard RL post-training only rewards final answers with sparse signals and therefore leaves potentially useful intermediate representations under-exploited.","feed_headline":"Final layer teaches intermediate layers during RL to boost reasoning","feed_subtitle":"OISD aligns logits and attention on-policy with the final layer, delivering gains on math tasks without external teachers or data.","key_machinery":"Signed advantage-weighted Jensen-Shannon alignment that distills logits and attention from the final layer to intermediate layers under a unified on-policy acting policy.","core_discovery":"OISD uses the final layer as both the acting policy and a detached internal teacher during rollout and GRPO optimization. Selected intermediate layers are guided to match the final layer through logit alignment, which copies reasoning behaviors, and attention alignment, which copies focus patterns. The alignment employs signed advantage-weighted Jensen-Shannon divergence so that distillation occurs while preserving policy consistency under a single unified acting policy. Experiments show this produces substantial and consistent gains over strong reasoning RL baselines on four mathematical reasoning tasks.","pith_inferences":["The same internal-teacher pattern could be tested on non-mathematical domains where intermediate layers already encode useful task signals.","OISD might be combined with other forms of auxiliary supervision inside the same RL loop to compound gains.","If the final layer's signals prove transferable, future post-training pipelines could routinely add lightweight internal alignment heads rather than only optimizing final outputs."],"forward_implications":["The approach transfers high-level reasoning behaviors without any external privileged information or separate teacher models.","It enforces consistent attention patterns across layers while the model continues to act under one policy.","The method yields measurable gains on four separate mathematical reasoning tasks over strong RL baselines.","Distillation happens on-policy during the same rollouts used for policy optimization."],"fun_headline_variants":["OISD self-distills final layer signals to intermediate layers on-policy","Final layer aligns logits and attention in intermediate layers via OISD","On-policy internal self-distillation guides mid layers in OISD framework","GRPO optimization with internal teacher from final layer in OISD"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The final layer's on-policy representations contain transferable predictive signals about reasoning that are worth distilling to intermediate layers without causing policy degradation.","fun_headline_variants_meta":{"raw":{"variants":["OISD self-distills final layer signals to intermediate layers on-policy","Final layer aligns logits and attention in intermediate layers via OISD","On-policy internal self-distillation guides mid layers in OISD framework","GRPO optimization with internal teacher from final layer in OISD"]},"model":"grok-4.3","cost_usd":0.008644,"raw_usage":{"total_tokens":3905,"prompt_tokens":680,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":86437000,"prompt_tokens_details":{"text_tokens":680,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3154,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":680,"tokens_out":71,"duration_ms":25548,"temperature":1.0,"reasoning_tokens":3154,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T13:41:00.279257+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Applying OISD during GRPO training and observing no improvement or outright degradation on the four mathematical reasoning benchmarks compared with GRPO alone would falsify the central claim.","supporting_citations":[],"review_version":1}