{"id":"7f8f683b-d578-46be-bb42-56ee4110ba2a","arxiv_id":"2607.22039","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.","lead":"This paper finds that language models fine-tuned with reinforcement learning (RL) lose much less performance when their weights are merged with other task-specialized models than models fine-tuned with standard supervised learning (SFT), across five tasks and several base models. It then proposes three mechanisms—on-policy data, adaptive update sizes, and learning from both correct and incorrect samples—to explain why RL-trained models merge better.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RL-vs-SFT merging comparison is confounded by the RL arm's shared multi-task SFT warm-up; apparent mergeability may stem from common initialization, not the RL objective.","rationale":"The reader's weakest_assumption identifies the conflation of training objective with SFT warm-up, lower LR, KL regularization, and budget. I agree with the general point but think the strongest single issue is the shared SFT warm-up: it changes the initialization of every RL specialist, so the task vectors being merged are not comparable to SFT task vectors. This threatens the central claim about the RL objective, not just the mechanism. The proposed test—SFT initialized from the same warm-up—directly isolates the objective. If SFT-from-warm-up merges as well as RL, then the headline 'RL mitigates task conflicts' is misleading, though the practical recipe may remain useful. The existing evidence is single-run and lacks artifacts, so the concern cannot be resolved from the paper. I recommend keeping the reader's CONDITIONAL verdict: the empirical observation is plausible but the attribution is unverified. No change to verdict.","tokens_in":19732,"tokens_out":4945,"duration_ms":51126,"concrete_test":"Run a matched SFT-from-warm-up condition: take the same multi-task SFT warm-up checkpoint used to initialize GRPO (F.1), continue SFT on each task separately (same data, same 3 epochs, LR 5e-6 or a matched 1e-6), then apply TIEs merging to the resulting specialists and compute relative performance drops. If the SFT-from-warm-up drops are comparable to the RL arm (~7% average in Table 1) and much smaller than SFT-from-base (~19%), the mergeability advantage is attributable to the shared warm-up/initialization rather than to the RL objective, and the mechanism claims (2) and (3) lose support. Additionally, report RFT hyperparameters (LR, epochs, KL) to confirm whether Table 2's norm reduction is due to data provenance or regularization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison (Section 3.2, Table 1) is not between SFT and RL from a common starting point. Appendix F.1 states: 'Before applying reinforcement learning, we first fine-tune the base model on the math, code, and instruction-following datasets to equip it with basic instruction-following capabilities.' Thus each RL specialist is initialized from the same multi-task SFT checkpoint, whereas each SFT specialist is trained from the pretrained base. The task vectors τ_i = θ_i − θ_0 in Eq. (1) therefore contain a large shared component (θ_warm − θ_base) for all RL models. Merging RL models then amounts to merging small, possibly orthogonal deltas around a common center, which can yield small drops regardless of the RL objective. Table 2 supports the concern: RFT, an SFT-loss model on model-generated data, already reduces update norms by ~2–3× (6.50→2.36 for math), so data provenance alone matters; RL (0.78) is a further reduction but is trained with LR 1e-6 vs SFT 5e-6 and KL regularization (F.2), and no budget matching is reported. The paper's factor (2) and (3) claims—vanishing advantages and negative-sample joint optimization—cannot be separated from this initialization confound. The practical headline ('RL pipelines yield mergeable specialists') may survive, but the paper's central claim that the RL objective mitigates task conflicts is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper asks whether the post-training paradigm (SFT vs RL) changes how well individually fine-tuned LLMs can be merged. Across five verifiable tasks, three base models, three RL algorithms, and four merging strategies, it reports that RL-trained specialists retain substantially more performance after merging than SFT-trained specialists (e.g., Table 1: TIEs average drop 7.1% vs 19%). It then proposes three mechanisms: (1) RL's on-policy data reduce update norms; (2) RL's advantage scaling makes updates vanish at convergence; (3) joint optimization of positive and negative samples produces less conflicting parameter updates. The mechanistic claims are supported by update-norm measurements, conflict-norm curves, a landscape visualization, and an RL-Pos ablation.","tokens_in":20021,"tokens_out":7163,"duration_ms":74072,"significance":"If the headline result were established under controlled conditions, this would be an important and practically useful finding: it would identify the training paradigm as a first-order determinant of model mergeability and point to RL post-training as a way to build mergeable specialist suites. The paper's empirical breadth is a genuine strength: 5 tasks, 3 base models, 3 RL algorithms, 4 merging methods, with the headline effect measured externally on merged-model performance rather than derived from the theory. The central empirical claim is therefore not circular. However, a load-bearing confound in the main comparison (RL warm-start, lower LR, KL regularization) means the paper's causal attribution to the RL objective is not currently identified. The paper is transparent enough to expose this (F.1, F.2), which is to its credit, but the missing controls are decisive for the mechanism story.","major_comments":[{"comment":"The headline comparison is confounded by initialization and regularization. F.1 states that before RL the authors 'first fine-tune the base model on the math, code, and instruction-following datasets,' so for at least these three tasks each RL specialist starts from the same multi-task SFT checkpoint, whereas each SFT specialist starts from the pretrained base. The task vectors τ_i = θ_i − θ_0 in Eq. (1) therefore share the large common component θ_warm − θ_0 for all RL models; merging them is equivalent to merging small deltas around a common center, which can yield small drops regardless of the RL objective. The RL arm also uses a 5× lower learning rate (1e-6 vs 5e-6) and KL regularization (1e-3) (F.2), with no matched compute budget. This is not a local issue: it affects the central comparison in Table 1 and the 'RL objective mitigates task conflicts' conclusion. I would need a contro","section":"§3.2 / Table 1, with F.1–F.2"},{"comment":"The 'vanishing advantage' mechanism is largely definitional. Theorem 2 assumes E(r_n) → r* (Appendix B.1, Eq. 13) and then concludes E|A_n| → 0; that is a restatement of convergence, not a demonstrated property of RL training on these models. The update-magnitude formula Δθ_RL = Σ η A_s G_s in §4.2 omits the KL-regularization gradient (coefficient 1e-3) and the PPO/GRPO clipping or normalization in Eq. (4), both of which are present in the actual training and do not vanish with the advantage. Figure 5/6 plot norms over 120 steps but there is no evidence that the RL runs have converged or that the advantage has become small relative to the KL term. To support factor (2), the authors should measure the actual per-step update norms and conflict norms from the optimizer state (including KL and clipping), and show that the advantage term itself is the dominant cause of the reduced updates, no","section":"§4.2, Theorems 1–2, Eq. (8), Appendix B"},{"comment":"All reported numbers are single runs; there are no seeds, error bars, or significance tests. Given the empirical claim is a comparative statement about training paradigms, at least the headline comparisons (e.g., Table 1 TIEs row, Figure 3) need multiple seeds and a measure of dispersion. Some gaps are narrow (e.g., Puzzle under averaging: SFT −65% vs RL −56%), and without variance the 'substantially' claim is not fully quantified. This is fixable but should be addressed before the empirical section can be relied upon.","section":"§3.2, Tables 1, 4, 5; Figures 2, 3"},{"comment":"The RL-Pos ablation is a reasonable way to isolate negative samples, but it does not support the paper's subspace claim: no measurement of the 'unbiased task-specific parameter subspace' is provided. Figure 7 shows only merged-performance drops; H1/H2 are tested, but the proposed mechanism (negative examples steer updates to an unbiased subspace) is not directly verified. Also, since RL-Pos is trained with the same warm-start and KL regularization, the comparison with full RL is clean for the negative-sample factor but still inherits the warm-start confound relative to SFT. The section should either present direct evidence about the parameter subspace or weaken the claim.","section":"§4.3 / Figure 7"}],"minor_comments":[{"comment":"The caption reads 'Comparison of performance changes between SFT and RFT in model merging,' but the surrounding text refers to RL-trained models. This appears to be a typo; please correct.","section":"Figure 1 caption"},{"comment":"The unmerged SFT average is listed as 61.5, but the five per-task values (61.9, 60.5, 63.9, 86.2, 52.8) average to about 65.1. Please check the arithmetic or clarify how the average is computed.","section":"Table 1"},{"comment":"The warm-start sentence names only math, code, and instruction-following datasets, while the main-text Table 1 also includes puzzle and ranking. Please clarify whether the RL models for those tasks are warm-started, and if so, on what data; if not, the design is inconsistent across the five tasks.","section":"Appendix F.1"},{"comment":"The notation L(·) is used for a 'performance function' but is conventionally a loss. Please define whether higher L is better and whether it is accuracy or loss.","section":"§3.3, Eq. (6)"},{"comment":"The '≪' relation is used without a formal scale. Since this is a heuristic statement, consider stating it as an empirical finding or proving a bound with explicit constants.","section":"§4.2, Eq. (11)"},{"comment":"The text says the pairwise merging setup focuses on two models at a time, but Figure 1 shows results for up to five merged models. Please clarify how the two settings relate.","section":"§3.1 / Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practically important question, and the empirical breadth is a strong point. The main risk is the warm-start confound identified in the stress-test note: it lands squarely in the manuscript (F.1) and affects the central causal claim, not merely a peripheral analysis. I do not recommend rejection because the confound is identifiable and fixable within the manuscript's scope (warm-start controls, budget matching, and uncertainty quantification would address it). I would additionally ask the editor to ensure the authors do not merely soften the causal language without providing the missing controls, since the paper's stated contributions depend on the mechanism analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, the paper asks a question nobody else has really asked: does the post-training paradigm (SFT vs RL) change how well task-specialized LLMs merge? That is worth a serious look. Second, the answer as stated—\"RL inherently mitigates task conflicts\"—is not actually established. The RL arm is warm-started from the same multi-task SFT checkpoint, trained at a five-times lower learning rate, with KL regularization, and no budget matching. So the comparison conflates objective with initialization and regularization.\n\nThe paper's strength is the empirical sweep: five tasks, three base models, three RL algorithms, four merging methods. The pattern is consistent, and Table 1's TIEs numbers (19% average drop for SFT vs 7.1% for GRPO on Llama-3.1-8B) are striking. The RL-Pos ablation is a good idea, and the update-norm/conflict-norm analysis is a reasonable way to look inside the models.\n\nBut the soft spots are load-bearing. The warm-start confound is not minor; Table 2 shows that RFT—SFT on model-generated data, no RL at all—cuts update norms by 2–3x. So small deltas may trace to data provenance or regularization, not the RL objective. The mechanism section overreaches. Theorem 2 makes \"advantages vanish\" definitional under convergence, and Eq. 11 compares cumulative conflicts of n RL steps against n SFT steps without accounting for gradient magnitudes or convergence rates; vanishing advantages don't bound the sum, and SFT gradients also decay as the model fits. No error bars, seeds, or released checkpoints.\n\nWho is this for? Practitioners who merge specialists and want a hint that RL post-training produces more mergeable models; researchers studying merging or post-training. It deserves a serious referee because the question is important and the data are broad, but I'd want major revisions before accepting the mechanism claims. If you're asked to review, push for a matched-budget comparison with a common-start point and at least a few seeds. The core pattern might survive, but the current manuscript's central claim goes beyond what it shows.","headline":"A broad and genuinely new empirical sweep on post-training paradigm and model merging, but the headline causal claim about RL is undercut by the SFT warm-start and unmatched hyperparameters; worthwhile to read, not yet convincing on the mechanism.","tokens_in":20564,"tokens_out":2488,"would_cite":false,"duration_ms":28531,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This study claims RL-trained LLMs can be merged with a 7% average loss, versus 19% for SFT-trained models.","keywords":["model merging","task conflicts","reinforcement learning","supervised fine-tuning","task vectors","parameter interference","advantage decay","LLM post-training"],"falsifier":"Train SFT under RL's exact hyperparameters (same warm-up init, learning rate 1e-6, KL/weight-decay regularization) and merge it: if the post-merge drop falls to RL-like levels, the mechanism story is falsified though the practical headline survives. Alternatively, compare conflict norms per unit update: if RL and SFT have equal conflict at matched total update magnitude, the 'task-orthogonal subspace' claim fails.","tokens_in":19558,"feed_emoji":"🤖","tokens_out":7355,"duration_ms":71286,"temperature":0.7,"pith_summary":"This paper tries to establish that the post-training paradigm of a specialist language model—supervised fine-tuning (SFT) versus reinforcement learning (RL)—determines how much its parameters conflict with another specialist's when the two are merged into one model. On five verifiable tasks and with four merging methods, RL-trained models preserve performance far better after merging: the average relative drop is about 7% for RL and 19–22% for SFT, with some SFT tasks collapsing by 65%. If true, task conflicts are not inherent to merging; they are shaped by how the models were trained, and RL becomes a practical route to building generalist LLMs by fusing specialists without retraining. The paper proposes three mechanisms—smaller gradient updates from on-policy data, an advantage term that decays as the model converges ('enough is as good as a feast'), and joint optimization over positive and negative examples—and supports them with update-norm measurements, a conflict-norm statistic, and a convergence bound on the advantage.","feed_headline":"RL-trained LLMs lose 7% when merged; SFT models lose 19%","feed_subtitle":"If true, task conflicts are a training-paradigm problem: reinforcement learning makes specialist models safe to fuse into a generalist.","key_machinery":"The central mechanism is advantage-scaled gradient descent combined with on-policy sampling: RL multiplies each per-sample gradient by an advantage A = r - baseline, so the effective update is η·A·∇log π; SFT applies η·∇log π with fixed intensity. The paper proves (Theorems 1–2) that E|A| ≤ sqrt(Var(r)) and that E|A|→0 as training converges, meaning RL updates shrink automatically. The accompanying conflict norm, ||C(Δθ_t1,Δθ_t2)|| = L2 norm of element-wise products of two task-update vectors restricted to negative entries, quantifies how much two tasks push parameters in opposite directions; the paper uses this machinery to claim RL reduces conflict both because |A| decays and because posit","core_discovery":"The central claim is that RL-trained LLMs are substantially more suitable for merging than SFT-trained ones, largely preserving individual performance after parameter fusion. In the paper's own numbers, SFT models lose on average 19% relative performance under TIES merging while GRPO-trained models lose 7.1%; instruction-following tasks, which collapse by 25–29% under SFT, lose essentially nothing under RL. The authors attribute the advantage to three mechanisms: (1) on-policy data yield smaller gradient-update norms, so RL training is less likely to overwrite knowledge from other tasks; (2) the RL objective is adaptive—advantage scales updates and provably converges to zero as reward approa","pith_inferences":["Editorial inference: the mechanism attribution is confounded—RL runs are initialized by SFT warm-up, use a 5× smaller learning rate, and add KL regularization, and the RFT ablation already cuts update norms 2–3× without any RL objective or negative samples; so the practical headline may survive even if the three-factor explanation is wrong.","Editorial inference: a cheap falsifying experiment—SFT with matched learning rate, same init, and a KL/weight-decay regularizer—could tell whether 'small updates' or 'RL objective' is the true cause; if matched-budget SFT becomes mergeable, the paper's mechanism story reduces to a regularization story.","Editorial inference: the conflict norm measures element-wise sign disagreement; measuring cosine similarity between task vectors or linear mode connectivity along merge paths would test whether RL truly yields task-orthogonal subspaces or simply smaller updates with the same alignment pattern."],"forward_implications":["If the central claim holds, practitioners can merge RL-trained task specialists with far less performance loss than SFT specialists, across merging methods—even simple parameter averaging.","The benefit generalizes across RL algorithms (PPO, GRPO, REINFORCE++) and base model scales (3B to 24B), so the effect is attributable to RL post-training broadly, not to a specific recipe.","Multi-model merging scales gracefully on RL models: per-task drops stay gradual as more models are fused, whereas SFT models degrade steeply, supporting RL as a basis for generalist-model construction.","Because RL models show fewer conflicting parameter signs (especially among high-magnitude parameters), merging methods that keep only large task-vector entries (TIES, DARE) should become more reliable on RL-trained models."],"fun_headline_variants":["Merging RL-trained LLMs: only 7% loss vs SFT's 19%","RL models fuse better: 7% dip vs SFT's 19% on merge","Reinforcement learning cuts LLM merge conflict loss to 7%","Why RL LLMs merge better: on-policy data, adaptive goals","Enough is as good as a feast: RL models merge better"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the SFT-versus-RL comparison isolates the training objective—but the RL arm is initialized by SFT warm-up on the same task data, trained with a five-times smaller learning rate, and regularized toward the reference model, so the smaller, more mergeable updates may come from those confounds rather than from RL itself.","fun_headline_variants_meta":{"raw":{"variants":["Merging RL-trained LLMs: only 7% loss vs SFT's 19%","RL models fuse better: 7% dip vs SFT's 19% on merge","Reinforcement learning cuts LLM merge conflict loss to 7%","Why RL LLMs merge better: on-policy data, adaptive goals","Enough is as good as a feast: RL models merge better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2684,"prompt_tokens":803,"completion_tokens":1881,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1776}},"tokens_in":547,"tokens_out":1881,"duration_ms":12483,"temperature":1.0,"reasoning_tokens":1776,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T05:59:37.679154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SFT under RL's exact hyperparameters (same warm-up init, learning rate 1e-6, KL/weight-decay regularization) and merge it: if the post-merge drop falls to RL-like levels, the mechanism story is falsified though the practical headline survives. Alternatively, compare conflict norms per unit update: if RL and SFT have equal conflict at matched total update magnitude, the 'task-orthogonal subspace' claim fails.","supporting_citations":[],"review_version":1}