REVIEW 4 major objections 6 minor 2 references
RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs
T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Singular-vector rotations, not singular values, drive SFT forgetting and RL recovery
desk verdict Real empirical pattern, but the central causal claim is confounded because restoring base U,V with near-identical Sigma just moves weights back toward base. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the singular value decomposition of the transformer's weight matrices, applied to Q, K, V projections, together with principal angles between the singular subspaces of the base and fine-tuned models. A named minor theorem, 'Rotation Matters,' explains the empirical signature: in a two-layer linear sub-chain, updates of the form ΔW1 = +ηW1A and ΔW2 = −ηAW2 with skew-symmetric A leave the composite map unchanged to first order, rotate singular vectors, and cost only O(η²) in Frobenius norm, whereas direct singular-value changes cost O(η). The diagnostic work of this machinery is to distinguish 'which directions the model points' from 'how much it amplifies along those dir
What would settle it
Scale the top 20% singular values of a post-SFT model up and down by 10–20% while keeping every singular-vector direction fixed. If OOD accuracy moves by the same margin as the 70–80% recovery achieved by restoring directions, magnitude changes matter and the paper's claim collapses. Alternatively, restore base directions for the middle 20% of ranks and middle layers: if OOD recovers equally well, the localization claim (extremal components and shallow layers) is wrong.
Extended reading notes
Core claim
The paper's central claim is that the OOD generalization lost during SFT and recovered during RL is controlled by rotations of the singular vectors of the Q, K, V weight matrices, while the singular values themselves remain stable. Using principal angles between the base model's and fine-tuned model's singular subspaces, the authors observe that SFT and RL produce nearly identical rotation profiles: leading and trailing directions are rotated most, and the bulk spectrum stays fixed. A causal intervention supports the claim: replacing the U and V matrices of post-SFT weights with the base model's U and V, while keeping SFT's singular values, restores a large fraction of OOD performance. The r
Load-bearing premise
The causal conclusion rests on the assumption that swapping only the U and V singular-vector matrices of post-SFT weights for the base model's U and V, while keeping singular values fixed, isolates the effect of direction drift rather than introducing uncontrolled side effects of the surgery.
Editorial extensions
If this is right
- RL's OOD gains after SFT are mostly a restoration of directions SFT rotated away from the base model, so RL is not a source of brand-new reasoning ability.
- Practitioners can recover 70–80% of lost OOD performance by swapping in the base model's top-20% singular-vector directions, or the first 25% of layers' directions, without running PPO.
- The choice of SFT checkpoint sets the ceiling: stronger/intermediate SFT checkpoints can be rescued by RL, while overfitted checkpoints resist both RL and UV-direction restoration.
- Spectral angle curves, rather than in-distribution loss, provide an early signal of when SFT starts trading away OOD generalization.
Reading between the lines
- If direction drift is the causal channel, then SFT early stopping could be automated by monitoring the angle between base and current singular subspaces at the spectrum's head and tail, before any OOD evaluation.
- The KL penalty in PPO may be doing the heavy lifting by pulling directions back toward the base model; this predicts that orthogonal regularization or Procrustes-style direction alignment would reproduce part of RL's OOD benefit, which the paper does not test.
- The distinction between directional alignment and stored knowledge resonates with continual-learning reports of 'spurious forgetting'; extending this lens suggests OOD recovery should be formulated as geometric alignment repair rather than knowledge replay.
- Because the empirical base is a single card-game OOD variant, the direction-over-magnitude claim should be checked on math and code reasoning tasks under PPO; if it transfers, principal-angle drift could become a general health metric for post-training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies supervised fine-tuning (SFT) followed by reinforcement-learning fine-tuning (RL/PPO) on an out-of-distribution variant of the 24-point card game. It reports that SFT first improves then degrades OOD accuracy, that RL restores much of the lost OOD performance, and that spectral analysis of Q/K/V weight matrices shows singular values remain nearly unchanged while singular-vector directions rotate. The authors conclude that OOD forgetting during SFT and recovery during RL are driven by rotations of singular-vector directions, not by singular-value magnitude, and that restoring top-20% singular directions or first-25% layers recovers 70--80% of OOD performance. An appendix attempts a proof that rotation-type updates dominate singular-value changes. The empirical loss/accuracy curves are interesting, but the causal interpretation is not established by the experiments as presented.
Significance. If the direction-vs-magnitude claim were properly established, the paper would offer an inexpensive 'restoration' recipe (low-rank UV merging, shallow-layer resets) and reframe RL as an anti-forgetting mechanism rather than a source of new capabilities. The controlled OOD task, continuous checkpoint tracking, and explicit intervention design are strengths. However, the central causal experiment is confounded by the near-constancy of singular values, and the theoretical appendix assumes the conclusion. With additional controls and statistical rigor, this could be a useful contribution; in its current form, the headline mechanistic claim is untested.
major comments (4)
- [§4.3, Figs. 5–7] The direction-restoration intervention does not isolate direction from magnitude. §4.2 reports Δσ ≤ 0.005 and Figure 3 shows the SVD spectra before/after are nearly indistinguishable. Hence W' = U_base Σ_sft V_base^T is approximately U_base Σ_base V_base^T = W_base in Frobenius norm. Restoring 'directions' is therefore almost equivalent to restoring the whole weight matrix; any operation that moves W_sft toward W_base (e.g., linear interpolation) would likely recover OOD accuracy. The same confound affects Figure 7: imposing SFT directions on the RL model gives U_sft Σ_rl V_sft^T ≈ W_sft, so the OOD drop is essentially a return to the SFT checkpoint. The paper needs controls that change directions by a comparable Frobenius amount using non-base orthogonal matrices, plus a linear-mixing baseline, before claiming that directions matter more than magnitudes.
- [Appendix A, Eq. (5)] The proof assumes the very behavior it needs to establish: that a gradient step takes the orthogonal-gauge form ΔW1 = +ηW1A and ΔW2 = −ηAW2 for skew-symmetric A. No derivation from the loss in Eq. (3) or from the actual SFT/PPO objectives is provided, and the connection to Q/K/V layers is only asserted. The conclusion that 'the optimiser naturally drifts along this orthogonal-gauge manifold' is therefore unsupported. At most, the appendix shows that if updates were of that form, the forward function would be preserved and the Frobenius change would be O(η²); it cannot explain why gradient descent would prefer such updates.
- [§3.2, §6] The central quantitative claims rest on single runs with no error bars. Several reported differences are small relative to plausible seed-to-seed variation (e.g., Qwen RL gain 17.09%→19.66%; Figure 7 drop 19.8%→19.0%). Moreover, Inter_SFT is defined as the checkpoint with the highest OOD performance among those evaluated (§3.2), so the subsequent SFT→RL 'recovery' percentages are optimistic by construction. The 'top 20%' and 'first 25%' thresholds are also read post hoc from Figures 5–6. Multiple seeds and pre-specified thresholds are needed for the paper's quantitative claims.
- [§6, Conclusion] The paper explicitly states: 'To substantiate conclusions (2) and (3), we are currently undertaking additional experiments.' Conclusions (2) and (3) are the paper's mechanistic and scope claims, so this is an admission that the main results are not yet fully supported. TODO notes in §3.2 and §4.1 also indicate missing analysis (e.g., the singular-value-recovery experiment). This is more than a presentation issue; it affects the completeness of the contribution.
minor comments (6)
- [§3.1] 'Qwen-2.5-8B' appears in Models and Setup; the rest of the paper (and the abstract) uses Qwen-2.5-7B. Please fix the inconsistency.
- [§2.4 and throughout] 'Principle angle' should be 'principal angle'. Also, §2.4 says the singular values of M = U_base^T U_tgt lie in [−1,1]; since these are singular values of an orthonormal product, they are nonnegative and belong to [0,1].
- [§3.2, §4.1, Appendix A] Unfinished author notes remain in the text, e.g., '[hangzhan: TODO, ...]' and 'TODO connection between QKV and sub-chain'. These should be removed or resolved before submission.
- [Appendix A] The heading 'A Prove' should be 'A Proof'. The theorem statement refers to a transformer with Q, K, V, but the proof is for a two-layer linear chain; the mapping between the two is not made precise.
- [Figure 4] Figure 4 caption says 'MLP weight matrix' while the text describes it as 'layer-0 upproj'. Please clarify which matrix is plotted and whether the same trend holds for Q/K/V matrices.
- [Appendix B] Experimental details are incomplete: no reward-shaping or KL coefficient for PPO, no rollout count, no evaluation sample size, and no compute budget beyond '4 H100 GPUs'. These details are necessary for reproducibility.
Circularity Check
Mechanistic claim partially circular: SVD surgery recreates base/SFT checkpoints and Appendix A assumes rotation to prove rotation.
-
self definitional
[Appendix A, Theorem 1 and 'Why the optimiser prefers rotation']
"Suppose A is skew-symmetric and W1 and W2 are bounded with finite norm, learning rate η ≪ 1, and after one gradient descent step, suppose we have the following update ∆W1 = +ηW1A, ∆W2 = −ηAW2, then, ∥∆W ∥2F = O(η2)"
The update (4) is exactly the rotation-only behavior the proof is meant to establish. The theorem only computes consequences of assuming this gauge update; it never derives it from the gradient of L. The subsequent subsection then cites the assumed update to assert that the optimiser 'naturally drifts along this orthogonal-gauge manifold,' so the first-principles proof assumes its own conclusion.
-
renaming known result
[Section 4.3, 'Causal Validation of RL's Rotational Recovery' (Fig. 7)]
"By taking the high-performing RL-tuned model and forcing its feature space to adopt the geometric orientation of the poorly-generalizing SFT model, we effectively reverse the benefits of RL."
This intervention replaces the RL model's U,V with SFT U,V while keeping RL's Σ. Since §4.2 reports Δσ between 0 and 0.005 for both SFT and RL, Σ_rl≈Σ_sft, so W'=U_sft Σ_rl V_sft^T ≈ U_sft Σ_sft V_sft^T = W_sft. The operation approximately reconstitutes the SFT checkpoint, and the observed OOD drop is just the already-known SFT-vs-RL gap, renamed as a direction effect. The mirror experiment restoring base U,V on SFT gives W''≈W_base for the same reason. Without controls using random orthogonal matrices or linear mixing toward base, the 'directions matter more than values' conclusion is a renamed restatement of the base/SFT/RL comparison.
1 more flagged steps
-
other
[Section 3.2, TODO note after Fig. 2]
"[hangzhan: TODO, For the singular value curves that override each other, we may need to emphasize this phenomenon that indicates the value doesn’t change that much after fine-tuning. In addition, we should explain that we recovered the singular value and kept the singular vectors, the OOD and Ind capabilities didn’t change after this recovery, this means the singular values have nothing to do with the performance of a model after fine-tuning.]"
The planned inference is circular: because the paper reports Δσ ≤ 0.005, 'recovering' the singular values (replacing SFT Σ with base Σ) is a near-identity operation. Observing no change in OOD/ID is guaranteed by the near-equality of the inputs, not evidence that singular values are causally irrelevant.
full rationale
The raw empirical measurements—OOD/ID training curves, singular-value stability, and principal-angle spectra—are self-contained and not circular: they are direct observations, and the OOD metric is not defined in terms of the SVD quantities. The paper's mechanistic conclusion, however, rests on two circular moves. First, Appendix A's Theorem 1 assumes a skew-symmetric gauge update (Eq. 4) and then uses that assumption to conclude the optimizer 'prefers rotation'; this is a tautology, not a derivation from the loss gradient. Second, the surgical experiments in §4.3 do not isolate directions from magnitudes: since §4.2 reports Δσ ≤ 0.005, replacing U,V while keeping Σ is approximately replacing the entire weight matrix. Imposing SFT directions on RL yields W' ≈ W_sft, and restoring base directions on SFT yields W'' ≈ W_base. The observed OOD drop/recovery is therefore the already-known SFT/RL/base performance gap, renamed as a direction effect. The planned Σ-replacement null result (TODO note) is likewise a near-no-op. These issues make the 'directions matter more than magnitudes' claim partially circular/confounded, though the underlying data could support it with proper controls (e.g., random orthogonal rotations or linear interpolation toward base). No load-bearing self-citation is present; the circularity is internal to the argument.
Assumptions & free parameters
free parameters (3)
- Inter_SFT checkpoint position =
20% of SFT steps, selected for highest OOD accuracy
- top-k singular direction rank threshold =
20% of singular values (e.g., k=512 for Qwen)
- top-k layers threshold =
first 25% of layers
assumptions (4)
- standard math SVD and principal-angle computations on raw weight matrices are numerically meaningful and capture functionally relevant geometric change.
- domain assumption The GeneralPoints OOD variant (face cards as 11/12/13) measures general pretraining-acquired reasoning ability.
- domain assumption Findings from two models on one card game extrapolate to LLM post-training generally.
- ad hoc to paper The gradient update takes the rotation form ΔW1=+ηW1A, ΔW2=-ηAW2.
Cite this review
Pith. "Pith review of RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs." pith.science (2026). https://pith.science/paper/FGDSKN47
@misc{pith2026250816546,
author = {Pith},
title = {Pith review of: RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/FGDSKN47}},
note = {Machine review of arXiv:2508.16546}
}
read the original abstract
Training large language models (LLMs) from scratch is increasingly impractical, making post-training methods such as supervised fine-tuning (SFT) and reinforcement-learning fine-tuning (RL-FT, e.g., PPO) central to modern practice. Using an out-of-distribution (OOD) variant of the 24-point card game and new spectrum-based diagnostics, we revisit how these two stages reshape model representation and OOD performance. Our key findings are- (1) RL-FT can restore much of the OOD performance loss from SFT (e.g., Llama-11B 8.97% to 15.38%, Qwen-7B 17.09% to 19.66%). But when SFT induces severe overfitting and a clear distribution shift, RL-FT cannot fully recover OOD performance. (2) Direction shifts of singular vectors matter more than singular value magnitudes. These shifts concentrate on directions linked to the largest and smallest singular values, leaving the bulk spectrum intact. (3) Low-rank and shallow recovery is effective: restoring singular vector directions for the top 20% of values or first 25% of layers recovers 70-80% of OOD performance. (4) Stronger SFT checkpoints enable better recovery by RL, while overfitted ones resist restoration. These results reconcile prior reports of RL superior OOD performance: RL primarily counteracts SFT-induced directional drift rather than finding new solutions. Our spectrum-aware analysis highlights inexpensive recovery knobs low-rank UV merging and shallow-layer resets that practitioners can use before costly RL fine-tuning.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[2024]
ISSN 2994-970X. doi: 10.1145/3643763. URL http://dx.doi.org/10.1145/3643763. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– ...
arXiv 2022
-
[2025]
URL https://arxiv.org/abs/2501.12948. Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506.19767, 2025. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav ...
arXiv 2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.