{"id":"a1e04a70-171e-47ba-b71d-4e645c0b08a4","arxiv_id":"2608.06802","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Simple-OPD warms up a student on the teacher's chain-of-thought traces with a LoRA adapter before on-policy distillation, improving in-domain reasoning while preserving out-of-domain performance.","lead":"This paper studies how to initialize a small AI model before distilling knowledge from a larger teacher model, a stage called warm-up. It finds that warming up on the teacher's own reasoning traces, even incorrect ones, with a lightweight LoRA adapter works best, and proposes a recipe called Simple-OPD.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The training-recipe claim that LoRA warm-up beats full-parameter SFT is confounded by unequal learning rates (5e-5 vs 5e-6) and no LR sweep, so the central Simple-OPD design choice rests on an uncontrolled comparison.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the LoRA-versus-full-SFT comparison is not controlled because the two methods use different learning rates (5e-5 vs 5e-6) and no LR sweep is reported. I agree that this is the most important issue because the paper's proposed method, Simple-OPD, is defined by its choice of LoRA warm-up, and the central empirical claim is that this choice yields a better ID-OOD balance than full-parameter SFT. If the comparison were confounded by learning rate or training strength, the method's design rationale would be substantially weakened, even if the empirical Simple-OPD results themselves remain plausible. The reader's other concerns (missing error bars/seeds, inconsistent OOD benchmark count, overgeneralization of the incorrect-rollout finding) are real but secondary: they affect the strength of the evidence, whereas the LR confound affects the internal validity of the key training-recipe comparison. A focused LR sweep for full-parameter SFT would settle whether the observed OOD degradation is inherent to full-parameter updating or an artifact of the chosen hyperparameter. The paper's central results are still sufficient to warrant a conditional acceptance rather than rejection, since the Simple-OPD recipe is simple, reproducible in principle, and shows consistent improvements across several settings; the unresolved confound should be addressed before the training-recipe claim is taken as established.","tokens_in":17020,"tokens_out":4714,"duration_ms":48805,"concrete_test":"Run the Section 4.1 comparison again with a full-parameter SFT learning-rate sweep (e.g., 1e-6, 5e-6, 1e-5, 5e-5) and also a cosine-decay schedule, holding warm-up data, steps (40/100/150/175), batch size, and OPD setup fixed. For each configuration, record ID (AMC23/MATH-500/AIME24/AIME25) and OOD (IFEval/GPQA/HumanEval/MMLU-Pro subsets) immediately after warm-up and after 300 OPD steps. If any full-FT configuration lies on or above the LoRA ID-OOD Pareto frontier at the final OPD checkpoint, the claim that LoRA is preferable to full-parameter warm-up is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recipe claim (Sec. 4.1, 'LoRA warm-up balances adaptation and generalization') is established by comparing full-parameter SFT warm-up against LoRA warm-up at learning rates 5e-6 and 5e-5, respectively (Appendix A, Table 3), with no learning-rate sweep or schedule matching. These two methods differ not only in parameterization but in effective update size, optimization geometry, and training strength. The paper's own Figure 5 shows full SFT achieves high ID quickly but loses OOD; however, this could be a property of the chosen LR/step count rather than of full-parameter updating per se. For example, a lower full-FT LR or a shorter full-FT warm-up might preserve OOD while still giving ID gains, and a higher LR might make the OOD collapse worse; without a sweep along the full-FT axis, the claimed Pareto dominance of LoRA is not identified. The Simple-OPD design choice (LoRA rather than SFT) rests directly on this comparison, so this is the load-bearing assumption. The correctness-rollout and CoT-source findings are secondary and would not by themselves invalidate Simple-OPD, but the LoRA-vs-SFT result is the one that selects the method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the warm-up stage preceding on-policy distillation (OPD). Through ablations on the Qwen3-1.7B student / Qwen3-8B teacher setup, it reports three data-side findings (CoT supervision is essential, CoT from the OPD teacher is better than from a stronger external model, and incorrect teacher rollouts are nearly as useful as correct ones) and a training-side finding (LoRA warm-up gives a better ID/OOD balance than full-parameter SFT, with a near-saturation duration preferred). It then proposes Simple-OPD, which warms up the student on teacher-generated CoT with LoRA before the normal OPD stage, and evaluates this recipe across three OPD objectives, thinking and non-thinking models, and a same-size teacher-student consolidation setup.","tokens_in":17263,"tokens_out":3960,"duration_ms":41298,"significance":"If the central claims hold, Simple-OPD is a practical, plug-and-play initialization recipe that could improve OPD training in a variety of settings without modifying the distillation objective. The paper's strengths are its systematic ablation of the warm-up data and training choices, its coverage of multiple OPD objectives and model configurations, and its detailed appendix with benchmark-level results and example rollouts. The code release and the transparent reporting of hyperparameters are also valuable. However, the headline training-recipe claim rests on a comparison that is not fully controlled, and the robustness claims would be stronger with multiple seeds; these issues affect the confidence one can place in the specific design recommendation.","major_comments":[{"comment":"The central claim that LoRA warm-up balances ID adaptation and OOD generalization better than full-parameter SFT is based on a comparison in which the two methods use different learning rates (full FT uses 5e-6, LoRA uses 5e-5) and different parameterizations. Since full-parameter fine-tuning and LoRA have very different numbers of trainable parameters and optimization geometry, the observed differences in ID/OOD trajectories could be due to the learning rate or effective update magnitude rather than to the parameterization itself. Please add a learning-rate sweep for both methods (or otherwise match the effective update scale, e.g., by equivalent step sizes or update norms) and report the resulting ID/OOD trajectories. Without this, the design choice of LoRA over full SFT is not identified from the presented experiments.","section":"§4.1, Appendix A, Table 3"},{"comment":"Section 4 states that the OOD result is the average over six benchmarks, listing IFEval, GPQA-Diamond, HumanEval, and the Chemistry, Physics, and History subsets of MMLU-Pro. However, Tables 1 and 2 report only five OOD benchmarks (IFEval, GPQA-Diamond, Physics, Chemistry, History) and do not show HumanEval in the average. This is an inconsistency in the reported evaluation protocol. Please clarify whether HumanEval was evaluated and included in the OOD averages or correct the stated count, since the claim of \"preserving overall OOD generalization\" depends on which benchmarks are actually in the aggregate.","section":"§4 (OOD evaluation) and Tables 1 and 2"},{"comment":"All reported results appear to come from single runs, with no error bars, confidence intervals, or multiple seeds. Differences as small as 0.95 points (PowerOPD in Table 1) and 1.35 points (vanilla OPD in Table 1) are used to claim consistent improvements, but without variance estimates these differences may not be significant. Please provide at least three seeds for the core comparisons, or explicitly state that results are single-run and temper the language accordingly.","section":"§5.1, §5.2, §5.3 (Tables 1, 2, 6) and §4.1 (Figure 5)"},{"comment":"The conclusion that \"incorrect teacher rollouts can provide comparable benefits\" is supported by paired correct/wrong rollouts that are almost identical in reasoning structure, differing only in a local arithmetic error and its propagation. This is visible in Table 7. As presented, the finding supports the narrower claim that process-structurally similar incorrect traces still transfer the teacher's thinking pattern; it does not show that incorrectness per se is irrelevant. Please clarify how the paired incorrect rollouts were selected and whether the result holds for arbitrary incorrect teacher rollouts, or restrict the claim accordingly.","section":"§3.3 and Table 7"}],"minor_comments":[{"comment":"The \"near-saturation\" warm-up duration is identified from the training curves without a formal criterion. Please state an operational rule (e.g., a loss plateau or a threshold on improvement) or at least acknowledge that the choice is heuristic.","section":"§4.2"},{"comment":"The multi-part bar figures contain a large amount of textual annotation that is difficult to parse at small print. Please consider separating the SFT and OPD trajectories into distinct panels or using a table for the numerical values.","section":"Figures 2, 3, and 4"},{"comment":"The text says that \"a stronger SFT checkpoint does not necessarily provide a better initialization,\" but this claim is inferred from comparing different training configurations rather than from a controlled experiment that holds all else fixed. Please rephrase to reflect the correlational nature of the observation.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the LoRA-versus-full-SFT comparison is valid and is the main reason for my recommendation. The paper's central recipe claim depends on comparing two methods at different learning rates without a sweep, so the 'LoRA is better' conclusion is not yet identified. The OOD benchmark count inconsistency is also worth flagging to the authors before acceptance, as it affects the reported aggregate results. The data-side finding about incorrect rollouts is interesting but needs a clearer statement of its scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's the short version: Simple-OPD does something genuinely useful — it turns warm-up into a principled recipe and shows that teacher-compatible CoT, not answer correctness, is the active ingredient. The cross-setting gains on in-domain reasoning are credible, and the paper is honest about per-benchmark OOD movement. But the headline recommendation that LoRA beats full-parameter SFT rests on a comparison that is not fully controlled, and the lack of seeds makes the 1–2 point gains hard to weigh.\n\nThe new empirical content is real. The CoT-source ablation (OPD teacher vs GPT-5.5) is the strongest result; the message that compatibility with the downstream teacher matters more than raw teacher strength is non-obvious and useful. The incorrect-rollout finding is interesting but narrower than the abstract implies — the paired examples are nearly identical trajectories with a single arithmetic slip, so claiming 'incorrect rollouts work as well' needs a caveat. The LoRA-vs-SFT section is where the support weakens. Full SFT uses 5e-6 LR and LoRA uses 5e-5, and there is no LR sweep for either. It is entirely possible that full SFT at a lower LR or a different schedule preserves OOD while still giving ID gains. The Pareto claim about LoRA is therefore not identified. This is the load-bearing design choice in Simple-OPD, so it needs fixing before publication.\n\nAlso worth flagging: no error bars or multiple seeds; OOD benchmark count says six including HumanEval but tables show five (HumanEval missing); exact Simple-OPD hyperparameters (rank and steps) used in Tables 1 and 2 are not given in the main text. These are easy to fix.\n\nOn balance, the paper deserves peer review. The recipe is simple, reproducible (code on GitHub), and the ID gains are consistent across three OPD variants and two model families. A careful referee should ask for the LR sweep and repeated runs, and the authors should clarify the benchmark list. I would cite this for the CoT-source finding even with the current caveats. For a reading group, it's a good case study in how an uncontrolled hyperparameter can threaten a training-recipe claim.\n\nRecommendation: send it to review, with a request for a full learning-rate sweep on the full-parameter comparison and at least a couple of seeds on the main tables.","headline":"A credible warm-up recipe whose LoRA-over-SFT claim is confounded by an unequal learning-rate comparison; worth serious review but needs a sweep and seeds.","tokens_in":17801,"tokens_out":2767,"would_cite":true,"duration_ms":26476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the warm-up stage before on-policy distillation works by transferring the teacher's reasoning pattern—not correct answers—and that a LoRA-warmed, teacher-compatible chain-of-thought initialization consistently…","keywords":["on-policy distillation","warm-up","chain-of-thought","low-rank adaptation","LoRA","in-domain adaptation","out-of-distribution generalization","reasoning distillation"],"falsifier":"Run the same teacher-CoT warm-up with full-parameter SFT while sweeping learning rates across at least an order of magnitude, including $5\\times10^{-5}$, and trace ID and OOD curves through OPD. If any full-SFT checkpoint reaches the same ID-OOD Pareto frontier as the LoRA checkpoints, the paper's central training-recipe claim is falsified; if no full-SFT setting does, the low-rank constraint is doing the work.","tokens_in":16817,"feed_emoji":"🧠","tokens_out":7793,"duration_ms":64755,"temperature":0.7,"pith_summary":"On-policy distillation trains a student on its own rollouts under token-level supervision from a teacher, and this paper argues that the warm-up preceding it does one specific job: it imprints the teacher's reasoning pattern into the student before distillation begins. The paper establishes three facts about that warm-up: it needs chain-of-thought supervision generated by the same teacher that will supervise OPD, not by a stronger external model; the correctness of those teacher rollouts barely matters; and a low-rank LoRA adapter trained to near saturation balances in-domain adaptation against out-of-distribution generalization better than full-parameter fine-tuning. Together these facts yield Simple-OPD, a plug-and-play warm-up recipe that improves in-domain reasoning while preserving overall generalization across OPD objectives, thinking and non-thinking models, and same-size teacher-student consolidation. A sympathetic reader should care because warm-up is currently treated as an unexamined heuristic, and this paper turns it into a principled initialization with a cheap implementation.","feed_headline":"Warm-up's real job: copy the teacher's thinking, not the answer","feed_subtitle":"LoRA on teacher-generated chain-of-thought improves on-policy distillation without hurting out-of-domain skills.","key_machinery":"The load-bearing object is the teacher-compatible chain-of-thought rollout used as warm-up data together with the low-rank adapter that constrains how the student absorbs it. LoRA freezes the pretrained weights and learns only $A$ and $B$ low-rank matrices ($W = W_0 + \\frac{\\alpha}{r} BA$), so the student acquires the teacher's reasoning distribution without large changes to its pretrained capabilities. During warm-up, the student is trained on teacher-generated CoT with this low-rank constraint; during OPD, the student's rollouts are scored token-wise by the teacher's reverse KL. The mechanism's job is to move the student's rollout distribution into the teacher's generation space before distillation starts, which is why teacher-compatible reasoning matters more than answer correctness.","core_discovery":"The central discovery is that the warm-up stage transfers a teacher-compatible thinking pattern rather than correct answers, and that the training recipe should be constrained accordingly. Using rollouts that include chain-of-thought from the OPD teacher consistently beats using CoT from a much stronger external model, and paired correct and wrong teacher rollouts produce nearly identical SFT and OPD trajectories, with final scores within one point of each other. Full-parameter SFT warm-up accelerates in-domain convergence but permanently lowers out-of-domain performance, whereas LoRA warm-up reaches comparable in-domain performance while keeping OOD scores substantially higher, with a smaller rank (16) doing best. The recommended point is a near-saturation LoRA warm-up: enough steps to learn the teacher-aligned pattern, not so many that in-domain overfitting erodes generalization. The resulting Simple-OPD recipe—teacher CoT, LoRA, near-saturation duration, then standard OPD—improves ID performance consistently while leaving average OOD performance essentially unchanged.","pith_inferences":["An inference the authors leave implicit: if warm-up is really distributional imprinting, the same recipe should transfer to other token-level on-policy imitation tasks beyond math reasoning, such as code or tool use, wherever the teacher's reasoning trace is available.","The correctness-insensitivity result suggests that answer-accuracy filtering in distillation data curation may be less valuable than style- and compatibility-filtering; a testable extension is whether systematically biased wrong rollouts (for example, all failing at the same reasoning step) eventually break the student, unlike the locally diverse errors studied here.","The authors' implicit claim is that warm-up acts as an initialization in function space, not just parameter space; this predicts that the overlap ratio between student and teacher top-k token sets should jump immediately after warm-up and stay high, which the reported overlap curve already hints at and could be measured precisely.","Because the main experiments use a math-reasoning curriculum, an open question is whether near-saturation LoRA warm-up remains optimal when the teacher's CoT is much longer or multi-step; the answer may depend on a rank-to-duration tradeoff not explored in the paper."],"forward_implications":["Warm-up data for OPD should be sampled from the OPD teacher's own rollouts with CoT; buying CoT from a stronger external model is not just wasteful but harmful.","Distillation pipelines can skip filtering warm-up rollouts for correctness, since wrong teacher rollouts with intact reasoning structure give nearly the same boost.","A low-rank LoRA warm-up (rank 16 to 32, trained to near saturation) is a cheap and Pareto-better alternative to full-parameter SFT warm-up.","Simple-OPD improves in-domain reasoning under vanilla OPD, G-OPD, and PowerOPD, and in thinking and non-thinking settings, so it can be layered onto existing OPD objectives without changing them."],"supporting_citations":[{"why":"Supplies the LoRA parameterization that constitutes the warm-up training mechanism.","marker":"Hu et al., 2022"},{"why":"Introduces MiniLLM, the reverse-KL on-policy distillation objective used as the vanilla OPD baseline.","marker":"Gu et al., 2024"},{"why":"Establishes the on-policy distillation framework of token-level teacher supervision on student rollouts that the paper warms up for.","marker":"Agarwal et al., 2024"},{"why":"Prior work introducing off-policy cold start for OPD; the paper's point of departure for studying warm-up.","marker":"Li et al., 2026"},{"why":"The teacher-student model family's technical report describes the two-stage strong-to-weak distillation pipeline being analyzed.","marker":"Yang et al., 2025"},{"why":"Provides the DAPO-Math-17K dataset and teacher training setup used for warm-up and OPD.","marker":"Yu et al., 2026"},{"why":"A forward-KL warm-up alternative that the paper contrasts with its teacher-compatible CoT recipe.","marker":"Xu et al., 2026c"},{"why":"Supplies the specific on-policy self-distillation teacher used in the main warm-up experiments.","marker":"Yang et al., 2026b"}],"fun_headline_variants":["LoRA warm-up preserves OOD skills in on-policy distillation","Teacher thinking pattern, not answer correctness, drives warm-up","Simple-OPD: LoRA + teacher CoT for robust on-policy distillation","Even wrong teacher rollouts match correct ones in warm-up","Near-saturation LoRA warm-up beats full-parameter SFT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training-recipe claim assumes that the comparison between LoRA and full-parameter SFT is fair, but the paper's own hyperparameter table reports different learning rates (LoRA at $5\\times10^{-5}$, full SFT at $5\\times10^{-6}$), so the conclusion that the low-rank parameterization—not the optimization budget—causes the better ID/OOD balance could fail if a differently tuned full SFT matched it.","fun_headline_variants_meta":{"raw":{"variants":["LoRA warm-up preserves OOD skills in on-policy distillation","Teacher thinking pattern, not answer correctness, drives warm-up","Simple-OPD: LoRA + teacher CoT for robust on-policy distillation","Even wrong teacher rollouts match correct ones in warm-up","Near-saturation LoRA warm-up beats full-parameter SFT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000379,"raw_usage":{"total_tokens":2005,"prompt_tokens":925,"completion_tokens":1080,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":989}},"tokens_in":541,"tokens_out":1080,"duration_ms":9042,"temperature":1.0,"reasoning_tokens":989,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:33.711472+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same teacher-CoT warm-up with full-parameter SFT while sweeping learning rates across at least an order of magnitude, including $5\\times10^{-5}$, and trace ID and OOD curves through OPD. If any full-SFT checkpoint reaches the same ID-OOD Pareto frontier as the LoRA checkpoints, the paper's central training-recipe claim is falsified; if no full-SFT setting does, the low-rank constraint is doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the on-policy distillation framework of token-level teacher supervision on student rollouts that the paper warms up for."}],"review_version":1}