{"id":"7e4c3fbe-c96d-4b8c-9d2b-1198f498db9a","arxiv_id":"2602.01058","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Reweighting SFT tokens by target-to-behavior likelihood ratios before RL improves post-RL accuracy, while raw SFT strength does not predict RL success.","lead":"This paper shows that a better supervised fine-tuning checkpoint does not reliably become a better model after reinforcement learning, and proposes a loss-reweighting method that prepares SFT checkpoints for the RL stage. Practitioners building reasoning models could use it to design SFT data and objectives with downstream RL in mind.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"“PEAR consistently improves” is not yet established: single-seed RL and flat/negative cells in Table 2 leave the central comparative claim without statistical support.","rationale":"The reader's weakest_assumption centers on the requirement of a known behavior policy and support conditions. That is a real applicability limitation, but in this paper's controlled setup the behavior policy is deliberately known (responses are sampled from Qwen3-8B and verified), so it does not directly attack the central empirical claim in the tested regime. The more load-bearing weakness is statistical: the paper's headline comparative claim—'PEAR consistently improves post-RL performance'—is contradicted by zero/negative cells in its own Table 2 and by the unexplained duplicate PEAR rows in Table 6. Since no error bars, multiple seeds, or significance tests are reported, the observed gains and rank reversals could arise from RL stochasticity or evaluation noise rather than from the SFT objective. The controlled, contamination-free experimental design and the breadth of models/benchmarks are strengths, and the conceptual point that offline gains need not transfer to post-RL performance is well motivated. But the central quantitative claim requires multi-seed evidence before it can be accepted. Thus the appropriate verdict remains CONDITIONAL, matching the reader's conclusion, with the condition being additional statistical validation of the headline comparisons.","tokens_in":22714,"tokens_out":9575,"duration_ms":108770,"concrete_test":"Run 5+ independent GRPO seeds for SFT and PEAR on Qwen2.5-1.5B-Math and Qwen3-8B-Base using the paper's exact hyperparameters; report mean±std for AMC23/AIME25/MATH500 pass@8 and pass@64, and compute paired confidence intervals. Also evaluate an intermediate RL budget (e.g., half and double the original steps) to check whether any PEAR advantage is only a speed-up that disappears at convergence. If the PEAR−SFT interval contains 0 on these cells, the 'consistently improves' claim fails; if intervals are positive and exclude 0, this concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PEAR's central claim is empirical: after identical GRPO, PEAR-initialized checkpoints consistently beat SFT, and stronger SFT checkpoints can underperform weaker ones. The supporting evidence is weakened by (i) no multiple seeds for RL, (ii) no confidence intervals or significance tests, and (iii) the paper's own tables containing zero/negative PEAR deltas. Table 2 reports AMC23 pass@8 80→78 (−2) for Qwen2.5-1.5B-Math, AIME25 pass@8 35→35 (0) and MATH500 pass@8 93→93 (0) for Qwen3-8B-Base. Table 6 has two PEAR rows for Qwen3-1.7B-Base with opposite offline-to-online patterns (32.5→22.5 vs 20→45), without explaining the difference (variant? seed? hyperparameter?). Since pass@8 is computed from a small number of samples and RL is stochastic, these deltas are within plausible noise. The rank-reversal phenomenon, central to the paper's motivation, is demonstrated on single checkpoints; a single stochastic RL run per initialization cannot distinguish a true initialization effect from run-to-run variance. This is load-bearing because the abstract's 'consistently improves' is a universal comparative claim, and the method's practical value rests on it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the two-stage SFT-to-RL post-training pipeline and argues that optimizing SFT loss in isolation is a poor proxy for post-RL performance. It first presents a controlled comparison on synthetic logic puzzles showing rank reversals between offline and online performance across several SFT objectives. It then proposes PEAR, an importance-sampling reweighting of the SFT loss that uses likelihood ratios between the target policy and a known behavior policy. Three variants are given: sequence-level, token-level (suffix-based), and block-level weighting, plus an optional negative-example term. The authors evaluate PEAR as an RL initialization across Qwen2.5/3 and DeepSeek-distilled models on logic games and math benchmarks, reporting Pass@1/Pass@8 gains after GRPO, and include analyses of gradient alignment and parameter drift.","tokens_in":23112,"tokens_out":4703,"duration_ms":51475,"significance":"If the central empirical claim holds, the paper makes a useful practical contribution: it provides a simple, plug-in modification to standard SFT objectives that can better prepare a checkpoint for subsequent RL, and it documents a nontrivial phenomenon (offline gains do not necessarily transfer to post-RL performance). The controlled setup is a strength: the same RL recipe is used, the offline data is decontaminated, and the behavior policy is known, so the importance weights are well-defined. The method is also clearly described and can be implemented with modest overhead. However, the headline claim of 'consistently improves' is currently under-supported by the reported statistics, and the theoretical framing as importance sampling is not fully justified when the target policy is the model being optimized.","major_comments":[{"comment":"The paper's central claim, repeated in the abstract and conclusion, is that PEAR 'consistently improves' post-RL performance over canonical SFT. Table 2 contains zero and negative deltas: Qwen3-8B-Base AIME25 pass@8 is 35% for both SFT+GRPO and PEAR+GRPO (0), and Qwen2.5-1.5B-Math AMC23 pass@8 is 80% vs 78% (-2). Since pass@8 is computed from a limited number of samples and no confidence intervals or multiple RL seeds are reported, these cells are within plausible noise. This is load-bearing because the abstract's universal claim rests on the aggregate pattern. Please report multiple independent RL runs (at least 3 seeds) and standard errors or bootstrap intervals, and temper the claim to match the statistical evidence.","section":"Section 4.3 / Table 2"},{"comment":"Table 6 lists two rows labeled 'PEAR' for Qwen3-1.7B-Base with opposite offline-to-online patterns: one goes from 32.5% offline Pass@1 to 22.5% online Pass@1, while the other goes from 20.0% to 45.0%. The table does not state which PEAR variant, hyperparameters, or data configuration each row corresponds to. This ambiguity makes the rank-reversal evidence impossible to interpret and raises the concern that these are two runs of the same configuration, which would directly demonstrate run-to-run variance. Please identify the two rows precisely; if they are repeats, report variance information and adjust the conclusions accordingly.","section":"Table 6"},{"comment":"The paper motivates PEAR with standard OPE identities, but in the actual algorithm the importance weights are computed from πθ, the same model being optimized. The objective L_PEAR uses sg[Ĝ_t], so the weights are frozen only for the gradient step; the method is a self-bootstrapping procedure rather than minimization of a fixed population objective. Appendix B's unbiasedness claim 'E_{yt:T ∼ πβ}[ρ_{t:T} φ] = E_{yt:T ∼ πθ}[φ]' holds only pointwise for a fixed θ, not across the training trajectory. No convergence, consistency, or fixed-point argument is supplied. This does not invalidate PEAR as a heuristic, but the paper should either provide a theoretical justification or explicitly reframe the method as heuristic and rely on controlled experiments for its validity.","section":"Section 3.2 / Appendix B"},{"comment":"The method requires that the data-generating policy πβ is known and that per-token probabilities πβ(y_t | x, y<t) are available for every token in the offline dataset. The experiments satisfy this because responses are sampled from a single known generator (Qwen3-8B). However, many real SFT datasets are compiled from unknown or mixed sources, in which case the likelihood ratios cannot be computed. The paper should discuss how PEAR would be applied when πβ must be estimated or when support conditions (Appendix B, πβ > 0 wherever πθ > 0) are violated. At minimum, the scope limitation should be stated in the introduction or method section.","section":"Section 3.1"}],"minor_comments":[{"comment":"The sentence 'PEARB=1 stands for token-level weighting (§3.3)' appears to point to the wrong section: §3.3 describes sequence-level weighting, while token-level weighting is §3.4.","section":"Section 4.3"},{"comment":"Duplicate word: 'the former outperform the latter by by 40% absolute accuracy' should read 'by 40%'.","section":"Introduction"},{"comment":"Typo: 'Perforamnce' should be 'Performance'.","section":"Figure 6 caption"},{"comment":"Typo: 'modle' should be 'model'.","section":"Appendix A.4"},{"comment":"Typo: 'can take minutes to hPEAR to finish' should be 'can take minutes to complete' or similar; 'hPEAR' appears to be an artifact.","section":"Appendix A"},{"comment":"The text refers to 'IS-SFT' while the main body names the method PEAR; the naming should be consistent.","section":"Appendix A.5"},{"comment":"The sentence 'As shown in Table 4' appears after 'Figure 4a'; the reference should be to Table 4 only, and the surrounding text should be cleaned up.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to the core idea and the controlled experimental design, but the missing replication statistics and the unexplained duplicate PEAR rows in Table 6 are the main blockers. The theoretical gap around stop-gradient importance weights is also significant but could be addressed by reframing the method as a heuristic. With seed-level variance reporting and clarified tables, the paper could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper has a genuinely useful idea and an empirical claim that is not yet backed up. The idea: offline SFT objectives should be judged by how well they initialize online RL, not by offline accuracy. The authors show rank reversals on controlled logic puzzles—stronger SFT checkpoints can end up worse after identical GRPO. That is a real phenomenon worth taking seriously, even if it builds on Kang et al. 2025. The method, PEAR, reweights the SFT loss by suffix importance ratios between the target policy and the behavior policy, with token/block/sequence variants. It is simple, low-overhead, and clearly different from prior reweighting-SFT methods that target stabilization rather than downstream RL readiness. They also test it on math and with KD objectives, so the scope is decent.\n\nWhat I like: the controlled setup—same RL recipe, decontaminated splits, only the SFT objective varied. The negative-example variant is a nice addition, and the parameter-drift analysis is suggestive even if not conclusive.\n\nWhere it is soft: the central comparative claim, \"PEAR consistently improves,\" is contradicted by the authors' own Table 2. There are zero deltas on AIME25 and MATH500 for Qwen3-8B, and a −2 on AMC23 for Qwen2.5-1.5B. Pass@8 is noisy, and there are no confidence intervals or multiple RL seeds. The evidence supports \"often helps, sometimes neutral or slightly worse,\" not \"consistently improves.\" That is a proportionate correction, not a dismissal. The weights also depend on πθ, the model being optimized; the stop-gradient is a reasonable heuristic, but there is no unbiasedness or convergence argument. And the support condition (πβ positive wherever πθ is) will be violated in practice. These are addressable issues, but they should be stated as limitations. No code or data is released, which hurts reproducibility.\n\nThe stress-test note is basically right. I would add that the rank-reversal observation is not entirely new, but the authors do credit Kang et al. and push it further. The circularity concern is real but minor; this is an empirical method and treating the weights as fixed is defensible. The reader's score of 6 significance seems about right if the method holds up, but the soundness score of 5 is generous given the statistical gaps.\n\nBottom line: this paper deserves a serious referee, not because the claim is proven, but because the question is important and the method is plausible and cheap. A reviewer should ask for multiple RL seeds, error bars, a more careful wording of the abstract, and ideally code. I would send it to peer review with the expectation of major revisions.","headline":"A useful OPE-style reweighting for SFT-to-RL initialization, anchored on a real rank-reversal observation, but the 'consistently improves' claim overreaches what the evidence shows.","tokens_in":23570,"tokens_out":2144,"would_cite":true,"duration_ms":25009,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Supervised fine-tuning should be judged by how well it prepares a model for subsequent reinforcement learning, not by its offline accuracy; the paper shows that a suffix-likelihood reweighting of the SFT loss produces checkpoints that subse","keywords":["supervised fine-tuning","reinforcement learning initialization","off-policy evaluation","importance weighting","likelihood ratios","reasoning large language models","post-training","GRPO"],"falsifier":"Run identical RL starting from two checkpoints on the same base model and data, one trained with PEAR and one with standard SFT, over several seeds and a held-out verifiable benchmark; the central claim would be refuted if PEAR-initialized models do not consistently beat SFT-initialized models, or if rank reversals between offline and online performance disappear when the data-generating policy is known exactly. A more targeted test is to contaminate the offline buffer with a mixture of two known behavior policies: PEAR's weights assume a single known π_β, so its advantage should degrade measu","tokens_in":22623,"feed_emoji":"⚖️","tokens_out":6989,"duration_ms":67002,"temperature":0.7,"pith_summary":"The paper challenges the common assumption that a better offline supervised fine-tuning (SFT) checkpoint is automatically a better starting point for the online reinforcement-learning (RL) stage that follows it. It reports that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform models initialized from weaker ones, so offline accuracy is an unreliable proxy for post-RL performance. The cause it identifies is a distribution mismatch: SFT learns from logged data generated by one policy, while online RL learns from the model's own rollouts. To correct this, the paper proposes PEAR, a reweighting scheme that scales each token's SFT loss by the likelihood ratio of the remaining continuation under the trained policy relative to the data-generating policy. Across verifiable reasoning games and math benchmarks, PEAR-initialized models consistently outperform canonical SFT initializations after RL, with pass@8 gains up to 14.6 percent on AIME-2025.","feed_headline":"Offline training wins can vanish after RL — unless loss is reweighted","feed_subtitle":"Suffix-likelihood reweighting makes offline checkpoints better RL starting points, up to +14.6% pass@8 on AIME-2025.","key_machinery":"The key object is the PEAR weighting scheme, L_PEAR(θ) = E_{(x,y)~D}[ ∑_t sg[Ĝ_t] ℓ_θ(x, y_<t, y_t)], where the weight Ĝ_t is a numerically stabilized, discounted suffix importance ratio: roughly γ^{T−t} ∏_{j=t+1}^T π_θ(y_j | x, y_<j) / π_β(y_j | x, y_<j), with π_β the known behavior policy that generated the offline token and π_θ the policy being trained. This ratio measures how plausible the logged continuation is under the target policy relative to how it was logged; low-ratio suffixes are down-weighted and high-ratio suffixes up-weighted. The paper gives three variants — sequence-level, token-level, and block-level — that differ in where the product is aggregated, and a negative-example","core_discovery":"The paper's central claim is that the offline stage of LLM post-training should be optimized with the downstream RL stage in mind, and that this is achievable through off-policy-evaluation-style importance weighting. Concretely, after identical RL training, checkpoints produced by PEAR — which reweights the per-token SFT loss by a suffix likelihood ratio between the trained policy and the behavior policy that generated the offline data — outperform canonical SFT checkpoints, even when PEAR's offline scores are not the best. The paper also establishes that stronger offline performance does not entail stronger post-RL performance: across a wide spectrum of SFT-style objectives, rank reversals","pith_inferences":["If the distribution-mismatch explanation is correct, any offline data-heavy stage — including data curation or synthetic-data filtering — could be evaluated by rollout compatibility with the final RL policy, not by offline metrics; that suggests a new family of offline diagnostics based on suffix likelihood ratios.","The method's dependence on a known behavior policy marks a clear boundary: on real web-scale data whose generative source is unknown, PEAR's weights cannot be computed directly; a testable extension is to estimate π_β with a frozen reference model and measure how PEAR's advantage degrades as that estimator becomes inaccurate.","The fixed discount factor and clipping ranges are free hyperparameters; an extension would be to make them adaptive per sequence or per block, which the paper does not explore.","The negative-example variant hints that offline objectives can productively repel as well as attract, provided the repulsion is weighted in a policy-consistent way — a direction that could be pushed further in future post-training recipes."],"forward_implications":["A model with a higher offline SFT score can end up worse after identical RL training, so offline accuracy alone is not a reliable selection criterion for RL initializations.","Reweighting the SFT loss by suffix likelihood ratios gives consistent post-RL gains over canonical SFT across model sizes and math benchmarks, up to 14.6 percent pass@8 on AIME-2025.","Token- and block-level suffix weighting outperform single-step importance weighting, because what matters is whether the entire remaining continuation is plausible under the target policy, not just the next token.","PEAR transfers to a shifted online RL task distribution, indicating the improvement is not limited to the offline training domain.","PEAR can be combined with KL-based knowledge distillation and with negative examples, making it a plug-in reweighting of the offline objective rather than a new architecture or data pipeline."],"fun_headline_variants":["Reweight SFT loss for RL: PEAR lifts pass@8 by 14.6%","Offline SFT strength flips after RL unless loss is reweighted","Stronger SFT offline can underperform after RL — PEAR fixes","SFT should prepare for RL: suffix-likelihood reweighting works","PEAR: SFT reweighted for RL outperforms standard SFT post-RL"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"PEAR assumes the data-generating behavior policy is known and that its per-token probabilities are available for every offline token, and that this policy has strictly positive probability wherever the trained policy does; if the offline data comes from unknown sources, or training pushes the model onto tokens the logging policy could not have produced, the importance weights cannot be computed and the method loses its foundation.","fun_headline_variants_meta":{"raw":{"variants":["Reweight SFT loss for RL: PEAR lifts pass@8 by 14.6%","Offline SFT strength flips after RL unless loss is reweighted","Stronger SFT offline can underperform after RL — PEAR fixes","SFT should prepare for RL: suffix-likelihood reweighting works","PEAR: SFT reweighted for RL outperforms standard SFT post-RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000386,"raw_usage":{"total_tokens":1912,"prompt_tokens":817,"completion_tokens":1095,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":997}},"tokens_in":561,"tokens_out":1095,"duration_ms":11969,"temperature":1.0,"reasoning_tokens":997,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:46:58.778828+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run identical RL starting from two checkpoints on the same base model and data, one trained with PEAR and one with standard SFT, over several seeds and a held-out verifiable benchmark; the central claim would be refuted if PEAR-initialized models do not consistently beat SFT-initialized models, or if rank reversals between offline and online performance disappear when the data-generating policy is known exactly. A more targeted test is to contaminate the offline buffer with a mixture of two known behavior policies: PEAR's weights assume a single known π_β, so its advantage should degrade measu","supporting_citations":[],"review_version":1}