Pith. sign in

REVIEW 4 major objections 7 minor 5 cited by

Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

T0 review · 4 major / 7 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Supervised fine-tuning should be judged by how well it prepares a model for subsequent reinforcement learning, not by its offline accuracy; the paper shows that a suffix-likelihood reweighting of the SFT loss produces checkpoints that subse

desk verdict A useful OPE-style reweighting for SFT-to-RL initialization, anchored on a real rank-reversal observation, but the 'consistently improves' claim overreaches what the evidence shows. read the letter →

arxiv 2602.01058 v2 pith:CRXVDAUC submitted 2026-02-01 cs.LG cs.AIcs.CL

Dylan Zhang , Yufeng Xu , Haojin Wang , Qingzhi Chen , Hao Peng This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords supervisedfine-tuningreinforcementlearninginitializationoff-policyevaluationimportanceweightinglikelihoodratiosreasoninglargelanguagemodelspost-trainingGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper challenges the common assumption that a better offline supervised fine-tuning (SFT) checkpoint is automatically a better starting point for the online reinforcement-learning (RL) stage that follows it. It reports that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform models initialized from weaker ones, so offline accuracy is an unreliable proxy for post-RL performance. The cause it identifies is a distribution mismatch: SFT learns from logged data generated by one policy, while online RL learns from the model's own rollouts. To correct this, the paper proposes PEAR, a reweighting scheme that scales each token's SFT loss by the likelihood ratio of the remaining continuation under the trained policy relative to the data-generating policy. Across verifiable reasoning games and math benchmarks, PEAR-initialized models consistently outperform canonical SFT initializations after RL, with pass@8 gains up to 14.6 percent on AIME-2025.

What carries the argument

The key object is the PEAR weighting scheme, L_PEAR(θ) = E_{(x,y)~D}[ ∑_t sg[Ĝ_t] ℓ_θ(x, y_<t, y_t)], where the weight Ĝ_t is a numerically stabilized, discounted suffix importance ratio: roughly γ^{T−t} ∏_{j=t+1}^T π_θ(y_j | x, y_<j) / π_β(y_j | x, y_<j), with π_β the known behavior policy that generated the offline token and π_θ the policy being trained. This ratio measures how plausible the logged continuation is under the target policy relative to how it was logged; low-ratio suffixes are down-weighted and high-ratio suffixes up-weighted. The paper gives three variants — sequence-level, token-level, and block-level — that differ in where the product is aggregated, and a negative-example

What would settle it

Run identical RL starting from two checkpoints on the same base model and data, one trained with PEAR and one with standard SFT, over several seeds and a held-out verifiable benchmark; the central claim would be refuted if PEAR-initialized models do not consistently beat SFT-initialized models, or if rank reversals between offline and online performance disappear when the data-generating policy is known exactly. A more targeted test is to contaminate the offline buffer with a mixture of two known behavior policies: PEAR's weights assume a single known π_β, so its advantage should degrade measu

Watch

Extended reading notes

Core claim

The paper's central claim is that the offline stage of LLM post-training should be optimized with the downstream RL stage in mind, and that this is achievable through off-policy-evaluation-style importance weighting. Concretely, after identical RL training, checkpoints produced by PEAR — which reweights the per-token SFT loss by a suffix likelihood ratio between the trained policy and the behavior policy that generated the offline data — outperform canonical SFT checkpoints, even when PEAR's offline scores are not the best. The paper also establishes that stronger offline performance does not entail stronger post-RL performance: across a wide spectrum of SFT-style objectives, rank reversals

Load-bearing premise

PEAR assumes the data-generating behavior policy is known and that its per-token probabilities are available for every offline token, and that this policy has strictly positive probability wherever the trained policy does; if the offline data comes from unknown sources, or training pushes the model onto tokens the logging policy could not have produced, the importance weights cannot be computed and the method loses its foundation.

Editorial extensions

If this is right

  • A model with a higher offline SFT score can end up worse after identical RL training, so offline accuracy alone is not a reliable selection criterion for RL initializations.
  • Reweighting the SFT loss by suffix likelihood ratios gives consistent post-RL gains over canonical SFT across model sizes and math benchmarks, up to 14.6 percent pass@8 on AIME-2025.
  • Token- and block-level suffix weighting outperform single-step importance weighting, because what matters is whether the entire remaining continuation is plausible under the target policy, not just the next token.
  • PEAR transfers to a shifted online RL task distribution, indicating the improvement is not limited to the offline training domain.
  • PEAR can be combined with KL-based knowledge distillation and with negative examples, making it a plug-in reweighting of the offline objective rather than a new architecture or data pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the distribution-mismatch explanation is correct, any offline data-heavy stage — including data curation or synthetic-data filtering — could be evaluated by rollout compatibility with the final RL policy, not by offline metrics; that suggests a new family of offline diagnostics based on suffix likelihood ratios.
  • The method's dependence on a known behavior policy marks a clear boundary: on real web-scale data whose generative source is unknown, PEAR's weights cannot be computed directly; a testable extension is to estimate π_β with a frozen reference model and measure how PEAR's advantage degrades as that estimator becomes inaccurate.
  • The fixed discount factor and clipping ranges are free hyperparameters; an extension would be to make them adaptive per sequence or per block, which the paper does not explore.
  • The negative-example variant hints that offline objectives can productively repel as well as attract, provided the repulsion is weighted in a policy-consistent way — a direction that could be pushed further in future post-training recipes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper studies the two-stage SFT-to-RL post-training pipeline and argues that optimizing SFT loss in isolation is a poor proxy for post-RL performance. It first presents a controlled comparison on synthetic logic puzzles showing rank reversals between offline and online performance across several SFT objectives. It then proposes PEAR, an importance-sampling reweighting of the SFT loss that uses likelihood ratios between the target policy and a known behavior policy. Three variants are given: sequence-level, token-level (suffix-based), and block-level weighting, plus an optional negative-example term. The authors evaluate PEAR as an RL initialization across Qwen2.5/3 and DeepSeek-distilled models on logic games and math benchmarks, reporting Pass@1/Pass@8 gains after GRPO, and include analyses of gradient alignment and parameter drift.

Significance. If the central empirical claim holds, the paper makes a useful practical contribution: it provides a simple, plug-in modification to standard SFT objectives that can better prepare a checkpoint for subsequent RL, and it documents a nontrivial phenomenon (offline gains do not necessarily transfer to post-RL performance). The controlled setup is a strength: the same RL recipe is used, the offline data is decontaminated, and the behavior policy is known, so the importance weights are well-defined. The method is also clearly described and can be implemented with modest overhead. However, the headline claim of 'consistently improves' is currently under-supported by the reported statistics, and the theoretical framing as importance sampling is not fully justified when the target policy is the model being optimized.

major comments (4)
  1. [Section 4.3 / Table 2] The paper's central claim, repeated in the abstract and conclusion, is that PEAR 'consistently improves' post-RL performance over canonical SFT. Table 2 contains zero and negative deltas: Qwen3-8B-Base AIME25 pass@8 is 35% for both SFT+GRPO and PEAR+GRPO (0), and Qwen2.5-1.5B-Math AMC23 pass@8 is 80% vs 78% (-2). Since pass@8 is computed from a limited number of samples and no confidence intervals or multiple RL seeds are reported, these cells are within plausible noise. This is load-bearing because the abstract's universal claim rests on the aggregate pattern. Please report multiple independent RL runs (at least 3 seeds) and standard errors or bootstrap intervals, and temper the claim to match the statistical evidence.
  2. [Table 6] Table 6 lists two rows labeled 'PEAR' for Qwen3-1.7B-Base with opposite offline-to-online patterns: one goes from 32.5% offline Pass@1 to 22.5% online Pass@1, while the other goes from 20.0% to 45.0%. The table does not state which PEAR variant, hyperparameters, or data configuration each row corresponds to. This ambiguity makes the rank-reversal evidence impossible to interpret and raises the concern that these are two runs of the same configuration, which would directly demonstrate run-to-run variance. Please identify the two rows precisely; if they are repeats, report variance information and adjust the conclusions accordingly.
  3. [Section 3.2 / Appendix B] The paper motivates PEAR with standard OPE identities, but in the actual algorithm the importance weights are computed from πθ, the same model being optimized. The objective L_PEAR uses sg[Ĝ_t], so the weights are frozen only for the gradient step; the method is a self-bootstrapping procedure rather than minimization of a fixed population objective. Appendix B's unbiasedness claim 'E_{yt:T ∼ πβ}[ρ_{t:T} φ] = E_{yt:T ∼ πθ}[φ]' holds only pointwise for a fixed θ, not across the training trajectory. No convergence, consistency, or fixed-point argument is supplied. This does not invalidate PEAR as a heuristic, but the paper should either provide a theoretical justification or explicitly reframe the method as heuristic and rely on controlled experiments for its validity.
  4. [Section 3.1] The method requires that the data-generating policy πβ is known and that per-token probabilities πβ(y_t | x, y<t) are available for every token in the offline dataset. The experiments satisfy this because responses are sampled from a single known generator (Qwen3-8B). However, many real SFT datasets are compiled from unknown or mixed sources, in which case the likelihood ratios cannot be computed. The paper should discuss how PEAR would be applied when πβ must be estimated or when support conditions (Appendix B, πβ > 0 wherever πθ > 0) are violated. At minimum, the scope limitation should be stated in the introduction or method section.
minor comments (7)
  1. [Section 4.3] The sentence 'PEARB=1 stands for token-level weighting (§3.3)' appears to point to the wrong section: §3.3 describes sequence-level weighting, while token-level weighting is §3.4.
  2. [Introduction] Duplicate word: 'the former outperform the latter by by 40% absolute accuracy' should read 'by 40%'.
  3. [Figure 6 caption] Typo: 'Perforamnce' should be 'Performance'.
  4. [Appendix A.4] Typo: 'modle' should be 'model'.
  5. [Appendix A] Typo: 'can take minutes to hPEAR to finish' should be 'can take minutes to complete' or similar; 'hPEAR' appears to be an artifact.
  6. [Appendix A.5] The text refers to 'IS-SFT' while the main body names the method PEAR; the naming should be consistent.
  7. [Section 4.4] The sentence 'As shown in Table 4' appears after 'Figure 4a'; the reference should be to Table 4 only, and the surrounding text should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PEAR's weighting is a standard OPE-style reweighting, and the headline claim is an empirical benchmark comparison, not a fitted prediction.

full rationale

The only formal derivation in the paper is the suffix change-of-measure identity in Appendix B, which is the standard Radon-Nikodym reweighting E_{y~pi_theta}[phi] = E_{y~pi_beta}[rho phi] with the support assumption stated explicitly. PEAR's objective L_PEAR(theta) = E[sum_t sg[G_hat_t] ell_theta(...)] is not derived from a fitted parameter and does not reduce to any benchmark outcome by construction; post-RL gains are measured on held-out SynLogic/Enigmata puzzles and MATH-500/AIME/AMC benchmarks after identical GRPO. The weights G_hat_t depend on the current pi_theta with stop-gradient, so this is a self-referential training heuristic, but the paper does not claim a theorem that this heuristic must outperform, and the lack of a convergence/unbiasedness proof is a rigor concern rather than circularity. The only author-overlapping citations I found (Mukherjee et al. 2025b for the sparsity metric; Qian et al. 2025 as an ancillary RLVR-environment reference) are not load-bearing for the central claim. The zero/negative PEAR deltas in Tables 2 and 6 and the two unexplained PEAR rows in Table 6 are evidentiary weaknesses, not circular steps. Accordingly, there is no load-bearing circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central method's validity rests on knowing pi_beta, on support conditions, on treating theta-dependent weights as constants, and on the empirical assumption that reweighting by current-model likelihood ratios transfers to GRPO. These are domain assumptions rather than standard math.

free parameters (5)
  • discount gamma = 0.999
    Controls variance of suffix importance weights; chosen by hand in Section 4.2, no sensitivity analysis reported.
  • per-decision log-ratio clip lower/upper = [-0.08, 0.3]
    Clipping of log Delta_t; affects which tokens are up/down-weighted; set in Section 4.2.
  • final weight clip lower/upper = [-10, 5]
    Clip on log G_t; chosen empirically; location Section 4.2.
  • negative-example weight lambda = not reported
    Repulsive term in Section 3.6; lambda is never specified in the text.
  • block size B = 1 (default), 4 variant
    Granularity/stability tradeoff; B=1 is the default form, B=4 is tested in Figure 6.
assumptions (4)
  • domain assumption Offline data y is generated by a known behavior policy pi_beta with accessible per-token probabilities.
    Section 3.1 defines D produced by a known pi_beta; PEAR weights require pi_beta(y_t | x, y_<t). In most public SFT datasets this is unavailable.
  • domain assumption Support condition pi_beta(...) > 0 whenever pi_theta(...) > 0.
    Appendix B explicitly assumes this for the suffix change-of-measure; it can fail when training pushes pi_theta off the logged data's support.
  • ad hoc to paper Stop-gradient importance weights yield a valid training objective.
    L_PEAR in Section 3.2 treats G_t as fixed coefficients even though they are functions of theta; no convergence or unbiasedness argument is provided.
  • ad hoc to paper Reweighting by current/evolving pi_theta improves downstream GRPO performance.
    This is the central empirical hypothesis; no theoretical proof is given, and it is evaluated only on selected tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning." pith.science (2026). https://pith.science/paper/CRXVDAUC

@misc{pith2026260201058,
  author       = {Pith},
  title        = {Pith review of: Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CRXVDAUC}},
  note         = {Machine review of arXiv:2602.01058}
}
read the original abstract

Post-training of reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone. We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We attribute this to a mismatch typical in current SFT-RL pipelines: the distribution that generates the offline SFT data can differ substantially from the policy optimized during online RL, which learns from its own rollouts. We propose PEAR (Policy Evaluation-inspired Algorithm for Offline Learning Loss Re-weighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected. We conduct controlled experiments on verifiable reasoning games and mathematical reasoning tasks on Qwen 2.5 and 3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass at 8 gains up to a 14.6 percent on AIME2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.

Figures

Figures reproduced from arXiv: 2602.01058 by the authors.

Figure 1
Figure 1. Offline v.s. Online pass@1 on SynLogic Games on a total of 19 Models. It exhibits significant ranking changes indicating offline performance will not entail online performance. In addtion, our proposed approach remains the most effective initialization for online RL. et al., 2000; Thomas & Brunskill, 2016; Jiang & Li, 2016; Levine et al., 2020), we address this by reweighting of￾fline data using importance weights, … view at source ↗
Figure 2
Figure 2. A sketch of our weighing intuition. Red numbers are probabilities under target policy, Blue numbers are probabilities un￾der behavior policy. After token A, the behavior (data-generating) policy often continues with A→B→C (e.g., 0.9 × 0.8), but this continuation is highly unlikely for the policy we ultimately want to optimize. As a result, the offline data over-represents A→B→C, which can push the model to associate… view at source ↗
Figure 3
Figure 3. An illustration of how PEAR’s 3 variant works. APPROACH SUMMARY Step 1. Compute token log-likelihood ratios on tokens from offline dataset. Step 2. Aggregate into weights using either one of 3 variants and stabilize it. Step 3. Weigh the loss for each token. See [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Results on SynLogic dataset. We demonstrate that PEAR consistently improves post-RL performance.The bars reflect Pass@1 and dots mark Pass@8. In Figure 4a, Single Step IS is a baseline that corrects each token only based on the probability ratio of the token itself. Se…
Figure 5
Figure 5. Figure 5: PEAR-to-base KL divergence across weight levels.y-axis is the weight (clipped). Token distribution is more heavily driven on important tokens that drives success probability. distribution. Concretely, we initialize RL from the PEAR checkpoint and run online training on…
Figure 6
Figure 6. Figure 6: Perforamnce of different variants of PEAR. The bars reflect Pass@1 and dots mark Pass@8. while avoiding token-/suffix-level signed ratio products that can be particularly unstable on long horizons. We sub￾sampled 50K positive data and included 50K negative data from th…
Figure 7
Figure 7. Figure 7: Mean principal angle between offline and online GRPO gradients. Recent analyses further connect RL’s reduced catastrophic forgetting to its on-policy sam￾pling bias,(Shenfeld et al., 2025; Chen et al., 2025a; Jin et al., 2025a). Beyond behavior-level metrics, recent an…
Figure 8
Figure 8. Figure 8: Parameter drift of different approaches. (a) is the NSS score between offline and base model. (b) is the NSS score between online and offline model. probability-based objectives beyond NLL (Li et al., 2025a), entropy-regularized distribution matching (Diao et al., 2026…
Figure 9
Figure 9. Figure 9: The comparison between different metrics versus SynLogic online pass@1. (a) offline model forward KL divergence against the base model. (b) offline model update sparsity against the base model. (c) average spectrum drift of different linear modules in the base and offl…
Figure 10
Figure 10. Figure 10: Offline update strength measured by KL to base model and sparsity of parameter updates [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Visualization of offline vs online performance. (a): Qwen3-1.7B-Base-Base offline pass@1 versus online pass@1. (b): Qwen3- 4B-Base offline pass@1 versus online pass@1. (c): Qwen3-1.7B-Base offline pass@128 versus online pass@1. (d): Qwen3-1.7B-Base offline majority vo…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Supervised fine-tuning collapses LLM action diversity in board-game play beyond what the accuracy–diversity tradeoff requires; augmenting SFT data with all optimal actions per state partially prevents this.

  2. RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    RASFT is an adaptive SFT method that strengthens or relaxes expert imitation per problem based on on-policy rollout solvability and adds clipped reference-policy ratio to limit drift, reporting better results than sta...

  3. SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    SFT depth increases pre-RL pass@1 but can cause entropy collapse that inverts GRPO outcomes on Qwen models via reduced group advantage variance.

  4. PriFT: Prior-Support Guided Supervised Fine-Tuning

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    PriFT uses token reweighting signals from a frozen pretrained model to stabilize SFT and achieve better results than standard SFT baselines on reasoning tasks.

  5. When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Excessive SFT reduces LLM plasticity for RL; Rejuvenation restores it via base-anchored fusion and targeted neuron resets, yielding better RL performance and OOD generalization.

Reference graph

Works this paper leans on

3 extracted references · 3 linked inside Pith · cited by 5 Pith papers

  1. [1378]

    translates

    URL https://aclanthology.org/2025. emnlp-main.1378/. Zhao, R., Meterez, A., Kakade, S., Pehlevan, C., Jelassi, S., and Malach, E. Echo chamber: Rl post-training amplifies behaviors learned in pretraining, 2025. URL https: //arxiv.org/abs/2504.07912. Zhao, Y ., Boney, R., Ilin, A., Kannala, J., and Pajarinen, J. Adaptive behavior cloning regularization for...

    arXiv 2025

  2. [2018]

    Thomas, P

    ISBN 0262039249. Thomas, P. S. and Brunskill, E. Data-efficient off-policy policy evaluation for reinforcement learning, 2016. URL https://arxiv.org/abs/1604.00923. Uehara, M., Shi, C., and Kallus, N. A review of off-policy evaluation in reinforcement learning, 2022. URLhttps: //arxiv.org/abs/2212.06355. Wang, H., Hao, S., Dong, H., Zhang, S., Bao, Y ., Y...

    arXiv 2016

  3. [2020]

    Sanyal, S., Prairie, H., Das, R., Kavis, A., and Sanghavi, S

    URL https://proceedings.mlr.press/ v108/rowland20b.html. Sanyal, S., Prairie, H., Das, R., Kavis, A., and Sanghavi, S. Upweighting easy samples in fine-tuning mitigates forgetting, 2025. URL https://arxiv.org/abs/ 2502.02797. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. Deepseekmath: Pushing...

    arXiv 2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.