{"id":"86cc5370-1e14-452e-afd4-87b35a3b033b","arxiv_id":"2507.07562","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Long-CoT SFT and RL trade off strengths in reasoning VLMs, and five combination strategies fail to produce additive gains.","lead":"The paper finds that long CoT supervised fine-tuning helps a vision-language model on hard questions but hurts easy ones, while reinforcement learning gives steady, broad gains. Combining the two through five common strategies produces trade-offs instead of synergy, which the authors call a 'synergy dilemma' for multimodal reasoning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 6's central negative result is not statistically supported: the closest combination (progressive, 54.9) is within 0.2 of RL (55.1), no error bars or significance tests are reported, and best-checkpoint-on-eval selection can bias the comparison.","rationale":"The paper's intended contribution is an empirical negative result: after controlling for data and base model, five ways of combining long-CoT SFT and RL fail to beat RL alone, implying a synergy dilemma. For that negative result to be load-bearing, the comparison itself must be trustworthy. I do not see a fatal internal contradiction, and the study has real strengths: multiple benchmarks, difficulty-level analyses, ablations (KL, easy questions), and a public code repo. But the headline comparison is reported as point estimates with no error bars, despite four runs, and the protocol selects best checkpoints on the evaluation benchmarks. The closest combination is within 0.2 average points of RL, and per-benchmark gaps are the size of one binomial standard error. That makes 'fundamental dilemma' a claim about a difference that the table cannot resolve. The reader's weakest assumption was external validity: one base model, one distilled dataset, one RL algorithm. I agree that is a limitation, but the more immediate issue is internal: even for the exact configurations tested, we cannot tell whether the null result is real or an artifact of noise and checkpoint selection. A single concrete test—release per-run results and compute paired CIs, plus a held-out checkpoint-selection check—would settle this. If the CI for RL minus progressive includes zero, the honest conclusion is 'no statistically significant combination benefit observed,' not 'synergy is impossible.' If it excludes zero in favor of progressive, the central claim fails. For these reasons I would keep a conditional verdict rather than reject: the paper is a useful data point and the code is available, but the central claim must be reworded and supported with uncertainty estimates. This is why I mark agreement with the reader as partial: we flag related but different weaknesses.","tokens_in":14132,"tokens_out":8336,"duration_ms":100114,"concrete_test":"Ask the authors to release per-run, per-benchmark scores for the 4 runs behind Table 6 (or rerun with 8 seeds) and compute paired bootstrap 95% confidence intervals for RL minus each combination on the five-benchmark average and on the difficulty-5 subset. In the same release, run a checkpoint-selection sensitivity check: split each eval benchmark in half, select each method's best checkpoint on one half, and score on the other half. If any 95% CI excludes zero in favor of a combination, or if the best-on-test advantage disappears under held-out checkpoint selection, the 'synergy dilemma' claim is not supported. At minimum, the paper should report CIs and replace 'fundamental' with 'not observed under the tested configurations.'","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The claim 'we uncover a fundamental synergy dilemma' (Sec. 3) rests on Table 6, where no combination exceeds standalone RL's 55.1 average. But the table entries are point estimates: the paper says results are 'averaged over 4 runs' (Sec. 2.1 evaluation settings) yet gives no variance, confidence intervals, or significance tests. The strongest contender, Progressive SFT & RL, averages 54.9, only 0.2 below RL; per-benchmark deltas are MathVision +0.8, MathVerse -1.1, MathVista -0.2, MMMU +0.4, MMStar -0.6. On typical benchmark sizes (hundreds to ~1k questions), these differences are within one standard error of a binomial proportion, so 'all fail to produce additive benefits' may be a null result within sampling noise. Additionally, the evaluation protocol selects 'best checkpoint results' on the eval benchmarks themselves. Because SFT, RL, and each combination have different training curves and checkpoint counts, best-on-test checkpoint selection can create or erase point-estimate gaps and makes the comparisons non-comparable. Thus the load-bearing condition—that the dilemma is a real property of SFT/RL combinations rather than a measurement artifact—is not yet established. The single-base-model/single-dataset generality concern is real, but even the internal evidence is insufficient as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript studies post-training of Qwen2.5-VL-7B with long-CoT SFT and GRPO-based RL. It reports that SFT improves the most difficult questions but hurts easy ones and produces verbose outputs, whereas RL produces concise, broad gains. It then evaluates five combination strategies—two-stage, interleaved, progressive training, data mixing, and model merging—and concludes on the basis of Table 6 that none surpasses standalone RL, terming this a 'synergy dilemma.' The difficulty-level analyses in Figures 1 and 6 are used to support the complementary-but-incompatible characterization of SFT and RL.","tokens_in":14418,"tokens_out":5196,"duration_ms":57929,"significance":"If the negative result were robust, the paper would be a useful counterpoint to the prevailing assumption that SFT and RL automatically complement each other in reasoning VLMs. The study is systematic in its coverage of combination strategies and benchmarks, and it reports training/evaluation details and a code repository link. However, the central quantitative claim currently rests on point estimates without measures of uncertainty, and the evaluation protocol's best-checkpoint selection compounds the issue. The headline 'synergy dilemma' is therefore not yet supported at the strength claimed.","major_comments":[{"comment":"Table 6 is the load-bearing evidence for the 'synergy dilemma,' but no variance or significance testing is reported despite the statement in Sec. 2.1 that results are averaged over 4 runs. The closest combination, Progressive SFT & RL, averages 54.9 versus RL's 55.1, and per-benchmark deltas such as MathVerse -1.1 and MMStar -0.6 are within one standard error of a binomial proportion on benchmarks of only hundreds to about a thousand items. The conclusion that all five combination strategies 'fail to produce additive benefits' is statistically indistinguishable from a null result under sampling noise. Please report standard deviations or confidence intervals and run paired or bootstrap significance tests across the four runs, and soften the 'fundamental' framing until these analyses support the point estimates.","section":"Sec. 2.1, Table 6"},{"comment":"The evaluation protocol states 'For fine-tuned models, we report their best checkpoint results.' Because each method has a different training curve and a different number of saved checkpoints, selecting the best checkpoint on the evaluation benchmarks can differentially inflate each method's score and makes the comparisons in Table 6 non-comparable. This is especially problematic given the small margins (e.g., 54.9 versus 55.1). Please either evaluate a fixed checkpoint per training budget, report full training curves or the sensitivity of rankings to checkpoint selection, or perform selection on a held-out split and then report the resulting test numbers.","section":"Sec. 2.1, Evaluation settings"},{"comment":"The difficulty-level accuracy gains in Fig. 6 (and the corresponding gains in Fig. 1) are shown without error bars, despite the four-run protocol. The claim that 'SFT degrades easier questions and is strongest on the hardest questions' depends on these per-level deltas, which can be small and are based on small per-level item counts. Please provide per-level intervals or raw per-level counts; without them, the qualitative difficulty-dependent claims are not quantitatively supported.","section":"Sec. 2.3, Fig. 6"},{"comment":"The paper presents the 'synergy dilemma' as a general property of reasoning VLMs ('fundamental,' 'often behave more like a trade-off'), but the study uses one base model, one training set (Eureka-Distill, distilled from the authors' own SFT model), one RL algorithm with one KL value, and method-specific choices that are known to be consequential—for example, interleaved SFT applied only to pass-rate-zero questions and progressive prefix weight 0.2 in Sec. 3.1. These choices may be representative, but the current evidence cannot rule out that the dilemma is an artifact of this particular recipe. Please add supporting variation (e.g., a second base model or data source, or a sensitivity analysis over the method-specific hyperparameters) or explicitly restrict the claim to 'under the configurations tested here.'","section":"Sec. 3.1, Sec. 3.3"}],"minor_comments":[{"comment":"There are numerous typos and misspellings, for example 'contorling' in Sec. 2.2, 'Abalation' in the Table 2 caption, 'Acuracy' in the Table 6 caption, 'varing' in Sec. 2.1, and 'thourgh' in Sec. 3.1; these should be corrected.","section":"Throughout"},{"comment":"The data section refers to 's1.1-R1' while the training settings refer to 's1.1'; please clarify whether these are the same dataset and use one consistent name.","section":"Sec. 2.1, Data"},{"comment":"The token-level KL divergence used to identify 'reasoning pivotal tokens' is not defined in the text; please specify how tokens are aligned and what reference distribution is used.","section":"Sec. 2.3, Fig. 7"},{"comment":"The difficulty levels are defined using 16 independent runs of the baseline model, while the evaluation protocol in Sec. 2.1 says results are averaged over 4 runs; please clarify the relation between the two settings.","section":"Sec. 2.3, Difficulty levels"},{"comment":"The statement 'We have also attempted reducing the number of preliminary SFT epochs to 1' is not accompanied by any result or table; please add a sentence summarizing the outcome or remove the claim.","section":"Sec. 3.1, Two-stage SFT & RL"},{"comment":"The reference list contains duplicates: Chen et al. 2025a and 2025b are the same arXiv paper, and Yang et al. 2025a and 2025b are also the same; please merge or distinguish them correctly.","section":"References"},{"comment":"The SFT count of 249,091 for the word 'wait' appears implausibly large for a single benchmark unless counting across many generated responses; please clarify the counting procedure and verify the reported numbers.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical study whose main weakness is statistical support for its central negative result; the requested revisions are feasible within the manuscript's scope. I do not have concerns about attribution, but the GitHub repository link should be verified as a genuine reproducibility artifact before publication, since the text does not describe what code or data are released."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nIf you do post-training for reasoning VLMs, this paper is worth a look, but read Table 6 with suspicion. The authors compare five ways of combining long-CoT SFT and RL on Qwen2.5-VL-7B and find that none beats pure RL on average accuracy. The difficulty-level analysis is the most valuable part: SFT clearly helps hard questions while hurting easy ones, RL gives steady broad gains, and the response-length and reasoning-word frequencies show why. That is a solid, reproducible empirical contribution, and they release code and data.\n\nThe soft spot is the headline claim. The five combination strategies land at 51.4, 53.6, 54.9, 54.0, and 54.3 average accuracy, versus RL's 55.1. The closest is 0.2 points behind. The paper says results are averaged over 4 runs but reports no variance, confidence intervals, or significance tests. On benchmark sizes of a few hundred to a thousand questions, that gap is inside sampling noise. They also select the best checkpoint on the eval benchmarks, which can bias comparisons when training curves differ. So \"we uncover a fundamental synergy dilemma\" is too strong; the defensible version is \"under this recipe, naive combinations fail to add.\" That is still a useful data point, but it is a hypothesis, not a law.\n\nThe generality is also limited: one base model, one training set distilled from their own SFT model, one RL algorithm with fixed KL, and particular design choices for each combination. Fine as a case study, but the framing should say so.\n\nWho gets value? Practitioners choosing a post-training recipe, and researchers wanting a clean comparison of two-stage, interleaved, progressive, data mixing, and model merging in one setting. The difficulty-level decomposition itself is worth citing. I would send it to review, but ask the authors for per-run numbers, error bars, and significance tests, and to soften the generalization. If those come back, the paper becomes much more useful.\n\nMy vote: engage, but treat the dilemma as provisional.","headline":"Useful difficulty-level analysis of SFT vs RL for reasoning VLMs, but the 'synergy dilemma' headline is not supported by the reported statistics.","tokens_in":15039,"tokens_out":3607,"would_cite":true,"duration_ms":36340,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Long-CoT SFT and RL do not combine synergistically in vision-language reasoning models: all five tested fusion methods underperform standalone RL.","keywords":["vision-language models","long chain-of-thought","supervised fine-tuning","reinforcement learning","multimodal reasoning","synergy dilemma","difficulty stratification","GRPO"],"falsifier":"Replace Eureka-Distill with long-CoT traces distilled from the target model itself, keep every other setting identical, and rerun the five combination strategies on the same five benchmarks; if any variant averages above 55.1 percent across the five, the dilemma is a property of the tested recipe, not of reasoning VLMs generally.","tokens_in":13912,"feed_emoji":"🧠","tokens_out":9427,"duration_ms":86726,"temperature":0.7,"pith_summary":"This paper asks whether long chain-of-thought supervised fine-tuning (SFT) and reinforcement learning (RL), two post-training techniques that reportedly work well together in language-only models, can be combined to improve reasoning in vision-language models. Analyzing benchmark questions by difficulty, it finds that SFT teaches deep, structured reasoning that helps the hardest questions but hurts easy ones by making the model verbose, while RL gives broad, consistent gains with concise responses but helps the hardest questions less. The paper then tests five ways to combine the two—two-stage, interleaved, and progressive training, data mixing, and model merging—and reports that none beats standalone RL on average accuracy across five multimodal benchmarks. This \"synergy dilemma\" is the paper's central claim: SFT and RL behave like a trade-off, not a complement, under the recipes tested.","feed_headline":"No SFT+RL recipe beats pure RL on reasoning VLMs","feed_subtitle":"SFT helps hardest questions; RL helps everywhere. Fusing them loses both advantages.","key_machinery":"The load-bearing mechanism is difficulty-stratified analysis: every benchmark question is placed into one of five difficulty levels using the baseline model's pass rate over 16 runs, so gains can be separated by question type rather than averaged away. On top of that, the paper uses token-level KL divergence before and after fine-tuning to expose \"reasoning pivotal tokens\"—words like \"first\", \"then\", \"check\", and \"maybe\"—that SFT injects at sentence beginnings, and it tracks response lengths and RL training dynamics under the GRPO objective with a KL regularization term. These tools let the authors attribute SFT's gains to hard questions and verbosity, RL's gains to concise generalization, and the failure of hybrids to unresolved trade-offs.","core_discovery":"On the paper's own terms, the central discovery is that long-CoT SFT and RL play distinct, difficulty-dependent roles in reasoning VLMs, and naive ways of stacking them do not combine those roles. SFT raises accuracy on the hardest questions (levels 4-5) by injecting structured, step-by-step reasoning traces, but makes responses roughly an order of magnitude longer than baseline and lowers accuracy on easy questions. RL produces concise responses with stable token distributions and improves accuracy across all five difficulty levels, yet its gains on the hardest questions are smaller than SFT's. Five integration strategies—two-stage, interleaved, and progressive training, data mixing, and model merging—all land between SFT and RL or below RL on average accuracy (51.4, 53.6, 54.9, 54.0, and 54.3 percent versus RL's 55.1), showing trade-offs in accuracy, reasoning style, and response length.","pith_inferences":["A testable extension: apply SFT loss only to difficulty levels 4-5 and RL loss elsewhere, instead of the paper's zero-pass-rate threshold; the paper's own difficulty curves suggest this could recover parts of both gains.","The authors' data-mixing recipe collects RL-generated correct responses and then runs SFT on them; the resulting 10x response-length increase hints that SFT imitates format and verbosity from the distilled traces, which a format-reward ablation could confirm.","If the dilemma is a general property, inference-time routing—send hard questions to the SFT model and easy ones to the RL model—would sidestep training incompatibility entirely and could beat either model alone, a consequence the paper does not test.","Language-model synergy reported elsewhere may depend on SFT data being self-distilled or model-compatible; reproducing this study with self-distilled traces would clarify whether the dilemma is multimodal or a data-mismatch artifact."],"forward_implications":["For average accuracy on current multimodal reasoning benchmarks, standalone RL with KL regularization is the strongest recipe tested; stacking SFT first, interleaving it, mixing data, or merging models does not add value.","SFT has a measurable strength on the hardest questions, so any future combination should target hard questions rather than apply SFT uniformly.","Retaining easy questions in RL training, even though their advantages normalize to zero, preserves accuracy on easy problems through the KL term.","High-quality reasoning traces matter more than volume or length: 1k textual traces outperformed a 34k multimodal set, and longer traces of lower quality hurt performance."],"supporting_citations":[{"why":"Supplies the Qwen2.5-VL-7B base model and Instruct checkpoint whose behavior every fine-tuned variant is measured against.","marker":"Bai et al. 2025"},{"why":"Supplies the s1 and s1.1-R1 textual long-CoT datasets, including the 1k-question set that transfers to multimodal reasoning.","marker":"Muennighoff et al. 2025"},{"why":"Provides the MM-Eureka query set from which Eureka-Distill, the shared SFT and RL training set, is distilled.","marker":"Meng et al. 2025"},{"why":"Defines the GRPO algorithm and the KL-regularized objective used in every RL run.","marker":"Shao et al. 2024"},{"why":"Provides the RL training system whose configuration (rollouts, batch size, learning rate) the paper uses.","marker":"Sheng et al. 2024"},{"why":"Provides the training system and hyperparameters used for all SFT runs.","marker":"Zheng et al. 2024"},{"why":"Provides the evaluation toolkit that scores the five multimodal reasoning benchmarks.","marker":"Duan et al. 2024"},{"why":"Supplies MathVision, the hardest benchmark where SFT shows its clearest advantage over RL.","marker":"Wang et al. 2024"},{"why":"Documents the prior conflicting evidence on whether SFT helps or hurts reasoning VLMs, motivating the systematic study.","marker":"Chen et al. 2025a"},{"why":"Provides the model-merging toolkit used for the linear, TIES, and SLERP merging experiments.","marker":"Goddard et al. 2024"}],"fun_headline_variants":["RL alone beats every SFT+RL combo on reasoning VLMs","Why SFT+RL stacks can't outdo RL on reasoning VLMs","Synergy dilemma: fusing SFT and RL loses to RL alone","For reasoning VLMs, pure RL outperforms all SFT+RL mixes","SFT+RL trade-offs: RL alone still wins on reasoning VLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The generalization of the dilemma rests on one base model, one distilled long-CoT training set, one RL algorithm with one KL setting, and the authors' specific choices for each combination; if those choices are not representative, the trade-off could be an artifact of the recipe rather than a property of reasoning VLMs.","fun_headline_variants_meta":{"raw":{"variants":["RL alone beats every SFT+RL combo on reasoning VLMs","Why SFT+RL stacks can't outdo RL on reasoning VLMs","Synergy dilemma: fusing SFT and RL loses to RL alone","For reasoning VLMs, pure RL outperforms all SFT+RL mixes","SFT+RL trade-offs: RL alone still wins on reasoning VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3270,"prompt_tokens":942,"completion_tokens":2328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":2230}},"tokens_in":558,"tokens_out":2328,"duration_ms":19432,"temperature":1.0,"reasoning_tokens":2230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:37:41.582766+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace Eureka-Distill with long-CoT traces distilled from the target model itself, keep every other setting identical, and rerun the five combination strategies on the same five benchmarks; if any variant averages above 55.1 percent across the five, the dilemma is a property of the tested recipe, not of reasoning VLMs generally.","supporting_citations":[{"cited_title":"Arcee’s mergekit: A toolkit for merging large language models","cited_arxiv_id":null,"evidence_quote":"Provides the model-merging toolkit used for the linear, TIES, and SLERP merging experiments."}],"review_version":1}