{"id":"16cf5236-9057-496d-9a2b-e6d92e530dda","arxiv_id":"2506.13284","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 7B reasoning model trained with carefully balanced SFT and RL beats prior small models on math and code benchmarks, with the paper documenting scaling and temperature heuristics.","lead":"This paper studies how supervised fine-tuning and reinforcement learning combine to make small language models better at math and coding, and releases a 7B model trained with the resulting recipe. It offers practical rules for scaling training data and setting sampling temperature, so teams building reasoning models can avoid trial and error.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim on AIME25 and LiveCodeBench v6 depends on decontamination that is described only for SFT data; the inherited RL data in §3.2.2 is not shown to be filtered against these newer benchmarks, so leakage could inflate the headline numbers.","rationale":"I agree with the reader's weakest assumption: the central claim is carried by the AIME25 and LiveCodeBench v6 results, and the paper's decontamination account is incomplete for the RL data. Section 3.1.1 describes a 9-gram overlap filter for SFT prompt collection, but Section 3.2.2 simply says the RL data comes from AceReason-Nemotron-1.0, without stating that those data were filtered against the newer benchmarks. Because AIME25 and LiveCodeBench v6 postdate the earlier benchmark suite and overlap with the training period, this is a concrete leakage channel. Moreover, the 9-gram filter is an exact n-gram test and can be bypassed by paraphrasing, renamed variables, or reordered code, so even a properly applied filter of this type does not fully decontaminate. The model and data releases are valuable because they make the check feasible; without running the check, the SOTA claim is conditional. I also noticed a secondary reporting inconsistency: Section 5 states the final model scores 63.2% on AIME25 and 52.8% on LiveCodeBench v5, while Table 1 lists 64.8 and 57.2; this appears to be a stale stage-3 number and is a real copy-paste/consistency flaw, but it is less load-bearing than the contamination channel because the SOTA claim is keyed to AIME25 and LCBv6 in §4.3. The reader's CONDITIONAL verdict is appropriate and does not need adjustment based on this pass.","tokens_in":22691,"tokens_out":8736,"duration_ms":91868,"concrete_test":"Download the released SFT and RL datasets from the HuggingFace repository and run two checks: (1) exact and near-duplicate detection against AIME25 and LiveCodeBench v6 prompts, using the stated 9-gram filter plus a stricter normalized-edit-distance or embedding threshold; (2) a paraphrase robustness probe on the released checkpoint, re-evaluating AIME25 and LCBv6 after surface edits such as changed variable names, numbers, and wording. If any training sample matches a test problem, or if accuracy on paraphrased problems drops far below the reported numbers while the original prompts remain high, the SOTA claim is contaminated. If neither occurs, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 makes the SOTA claim specifically on AIME25 and LiveCodeBench v6, calling them lower-contamination benchmarks. The paper's only decontamination statement is the 9-gram overlap filter in §3.1.1, described for SFT prompt collection. Section 3.2.2 says the RL data is inherited from AceReason-Nemotron-1.0, but does not restate any decontamination step against AIME25 (released February 2025) or LiveCodeBench v6 (covering 2025/02/01-2025/05/01). Since RL training occurred after these releases, this is a live leakage channel. A 9-gram exact-overlap filter is also too weak to exclude memorization via paraphrased prompts or renamed code. If AIME25 or LCBv6 problems leaked into RL training, the reported gains in §4.5.1 and §4.5.6, and the SOTA headline, would be attributable to answer memorization rather than to the SFT-RL synergy the paper claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates the synergy between supervised fine-tuning (SFT) and reinforcement learning (RL) for building a 7B math/code reasoning model. It curates SFT data with two scaling axes (number of prompts, number of responses per prompt), fits a regression to claim prompt-count scaling matters more, and then applies a stage-wise math-then-code RL recipe inherited from AceReason-Nemotron-1.0. Through ablations, the paper reports that RL from stronger SFT models yields better final performance, that a sampling temperature keeping 'temperature-adjusted entropy' near 0.3 is beneficial, that overlong filtering helps only at short length budgets, and that math-only RL transfers to code. The final model, AceReason-Nemotron-1.1-7B, is claimed to be state-of-the-art among Qwen2.5-7B-based reasoning models on AIME25 and LiveCodeBench v6. The model and data are released.","tokens_in":22936,"tokens_out":8125,"duration_ms":79572,"significance":"If the central claims hold, the paper provides a practical and largely reproducible post-training recipe, with useful ablations on SFT scaling, RL temperature, overlong filtering, and cross-domain transfer. The release of the model and data is a concrete contribution, and the systematic comparison across multiple SFT initializations is valuable. However, the current evidence for the two headline insights—prompt-count scaling dominance and the temperature-adjusted-entropy rule—is statistically thin, and the SOTA claim relies on decontamination assurances that are not documented for the RL data. These issues limit the confidence with which the conclusions can be accepted as stated.","major_comments":[{"comment":"The RL data decontamination status is not established. Section 3.2.2 states only that the RL data are inherited from AceReason-Nemotron-1.0, while the 9-gram overlap filter described in §3.1.1 is explicitly for SFT prompt collection. The SOTA claim in §4.3 and the RL-gain analyses in §4.5.1 and §4.5.6 are made specifically on AIME25 (released February 2025) and LiveCodeBench v6 (covering 2025/02/01–2025/05/01), which the paper itself calls lower-contamination benchmarks. The paper must state whether the inherited RL data were decontaminated against these benchmarks, and if so, how. Without this, the reported gains over the SFT model could be inflated by memorization rather than by SFT-RL synergy.","section":"§3.2.2, §4.3"},{"comment":"The regression analysis uses seven data points (v1–v7) and three fitted parameters (a, b, c) and reports R²=0.989, but provides no standard errors, confidence intervals, cross-validation, or residual diagnostics. The conclusion that scaling the number of prompts has a larger impact than scaling responses per prompt is a central contribution in the abstract and §1, yet it rests on a comparison of two point estimates (a=4.831 vs. b=2.635) with no measure of uncertainty. The authors should add leave-one-out or bootstrap intervals and check sensitivity to the endpoints; otherwise the claim is underdetermined.","section":"§4.4.2"},{"comment":"The 'temperature-adjusted entropy' is never defined. The text and figure refer to it as the key quantity behind the rule that the training temperature should keep it around 0.3, but no equation is given, so the rule is not reproducible. In addition, the evidence comes from three temperature settings (0.6, 0.85, 1.0) on what appears to be a single SFT model. The paper should provide the definition, state which SFT initialization was used, and either show evidence that the rule transfers to other SFT models or explicitly restrict the claim to a rule of thumb for the tested setting.","section":"§4.5.2, Figure 8"}],"minor_comments":[{"comment":"The conclusion states the final model scores 63.2% on AIME25 and 52.8% on LiveCodeBench V5, but Table 1 reports 64.8% and 57.2% for the same model on those benchmarks. Please correct the conclusion or clarify which model variant these numbers refer to.","section":"§5 (Conclusion)"},{"comment":"'we modify the rope_theta parameter from 10,000 to 1,000,000 enable support for a context length of 128K' is missing the word 'to' before 'enable'.","section":"§3.1.2"},{"comment":"Typo: 'effecive RL training' should be 'effective RL training'.","section":"§1 (Contribution 1)"},{"comment":"The training pipeline figure appears to be a screenshot from an external editor with a page header; it should be redrawn for legibility and to match the journal style.","section":"Figure 2"},{"comment":"The temperature study is described as applying to 'a given model', but the manuscript does not specify which SFT checkpoint (v5, v7, or DeepSeek-R1-Distill-Qwen-7B) was used for Figure 8. Please state the model and, ideally, show that the trend holds for at least one other initialization.","section":"§4.5.2"}],"recommendation":"major_revision","confidential_remarks":"The most important issue is the RL-data decontamination gap: the SOTA claim is conditional on AIME25 and LCBv6 being leak-free for the RL data, and the paper needs to either document the filtering or substantially soften the claim. The regression and entropy-rule concerns are secondary but should be addressed with uncertainty quantification or reframing. The self-citation pattern is reasonable given the direct lineage from AceReason-1.0, though the paper should be careful to distinguish inherited components from new contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a careful read. This is a solid empirical paper about combining SFT and RL for a 7B reasoning model, and the team has released the model and data, which counts for a lot. The headline claim, that the resulting model is SOTA among Qwen2.5-7B-based models on math and code, is plausible but rests on one gap that needs clearing before I'd trust the exact numbers.\n\nWhat's actually new: a quantitative comparison of scaling prompts vs. responses per prompt (with regression coefficients), a stage-dependent overlong-filtering analysis, and a temperature-adjusted entropy rule of thumb around 0.3. None of this is deep theory, but it is useful, reproducible empirical work. The RL experiments starting from different SFT models are also genuinely informative: the gap-narrowing between weaker and stronger SFT initializations is shown with curves, not just asserted. The observation that math-only RL improves code performance, even from a strong SFT start, is a real and non-obvious finding.\n\nWhere I'm cautious. The regression uses 7 data points and 3 parameters, so R^2 = 0.989 overstates the reliability of the conclusion that prompts matter more than responses. That conclusion is a reasonable tendency, not a law, and the paper mostly treats it that way. The entropy rule comes from a few runs on one base model. Fine as a rule of thumb; it should not be oversold. The overlong-filtering study is more systematic than prior work and the stage-dependent conclusion is believable.\n\nThe bigger issue is decontamination, and the stress-test note has this right. The SFT section describes 9-gram overlap filtering against test benchmarks, but the RL data is inherited from AceReason-Nemotron-1.0 and the paper does not state whether it was filtered against AIME25 and LiveCodeBench v6, both released before the RL training happened. The SOTA claims rest specifically on those two benchmarks. I am not accusing the authors of leakage; I am saying the paper leaves a load-bearing step undescribed. A 9-gram exact-overlap filter would also be too weak to catch paraphrased contamination. This needs to be clarified or fixed before the numbers are taken as established.\n\nOne minor thing: the baseline table mixes official and self-computed numbers, flagged with a dagger. Common practice, but it can create apples-to-oranges comparisons.\n\nBottom line: the paper deserves peer review. The recipe is useful, the release is valuable, and most claims are backed by ablations. The decontamination gap on RL data and the fragile regression should be addressed, but neither kills the paper. I'd cite it for the post-training recipe; I'd hold off on quoting the exact SOTA numbers until the RL data decontamination is confirmed.","headline":"Solid empirical post-training recipe from a credible group; model and data are out, but the headline SOTA numbers hinge on a decontamination step that is only described for SFT data, not for the inherited RL data.","tokens_in":23456,"tokens_out":2137,"would_cite":true,"duration_ms":26363,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7B model leads Qwen2.5-7B peers on math and code reasoning, the paper reports.","keywords":["supervised fine-tuning","reinforcement learning","chain-of-thought reasoning","math reasoning","code generation","GRPO","SFT scaling","model post-training"],"falsifier":"Run an independent contamination audit of the released SFT and RL data against AIME25 and LiveCodeBench v6 using paraphrased or lightly edited test problems that share no 9-gram with the originals; if the training data still contains recognizable versions of held-out problems, the reported RL gains on those benchmarks would shrink or disappear when retrained on a clean corpus.","tokens_in":22497,"feed_emoji":"🧮","tokens_out":6407,"duration_ms":67358,"temperature":0.7,"pith_summary":"The paper argues that supervised fine-tuning and reinforcement learning work as a pair, not as independent stages, and that a 7B model can reach the top of its size class if SFT data is scaled correctly before RL begins. It claims two SFT scaling axes help: adding unique prompts improves accuracy more than adding extra responses per prompt (fitted coefficients $a=4.831$ versus $b=2.635$ on a log scale), and training through five to six epochs, past the point of conventional overfitting, continues to improve long-chain reasoning. For RL, the paper claims a stronger SFT initialization yields a stronger final model, though the gap between initializations narrows substantially during training. The central operating rule is to set the sampling temperature so that the temperature-adjusted entropy stays near 0.3, which balances exploration and exploitation, and to apply stage-wise RL with growing response-length budgets. The resulting model, AceReason-Nemotron-1.1-7B, reaches scores the paper reports as the highest among Qwen2.5-7B-based reasoning models on AIME25 and LiveCodeBench v6.","feed_headline":"A 7B model leads Qwen2.5-7B peers on math and code","feed_subtitle":"The paper's recipe: scale SFT prompts, then stage-wise RL with entropy held near 0.3.","key_machinery":"The load-bearing mechanism is a two-part recipe rather than a single formula. First, SFT data scaling: 247K math prompts plus 136K code prompts, with response-length filtering to rebalance difficulty, and multi-epoch training to the point of mild overfitting. Second, a stage-wise RL curriculum using GRPO with token-level policy-gradient advantages, starting with a short 8K warm-up that compresses reasoning paths (average response length drops from roughly 5K to 4K tokens), then progressively longer budgets of 16K, 24K, and 32K on harder prompts, with overlong filtering used in early stages and dropped in the final 32K stage. The named control rule is to keep the temperature-adjusted entropy around 0.3 during RL training: at a sampling temperature of 0.85 the entropy starts near 0.26 and climbs to about 0.38, which the paper associates with the best exploration-exploitation balance and the highest benchmark scores.","core_discovery":"Starting from Qwen2.5-Math-7B, the authors build an SFT corpus of 383K prompts, filter it for contamination with a 9-gram overlap test, and generate responses with DeepSeek-R1. Scaling the dataset from 36K to 2.2M samples, with prompt count scaled more aggressively than responses per prompt, raises AIME24 from 41.0 to 63.0 across dataset versions v1 through v7. They then apply the stage-wise RL recipe from their earlier AceReason work: GRPO with token-level advantages, strictly on-policy rollouts, no KL term, and response-length budgets growing from 8K to 32K, interleaving math-only and code-only stages. The finding is that RL adds 10.6 points on AIME24 and 8.3 points on LiveCodeBench v6 over the already-strong SFT checkpoint, while math-only RL alone lifts code scores by several points. The final model reaches 72.6% on AIME24 (avg@64), 64.8% on AIME25, and 52.1% on LiveCodeBench v6, which the paper reports as the highest among Qwen2.5-7B-based reasoning models.","pith_inferences":["The entropy-around-0.3 rule is stated as a rule of thumb for one model family; if it generalizes, it gives other teams a cheap diagnostic for setting RL temperature without full sweeps.","The narrowing of SFT-to-SFT gaps under RL hints that, at fixed compute, RL can substitute for some SFT data diversity, though the paper's AIME25 results also suggest the substitution has a ceiling when starting models differ greatly.","The math-to-code transfer is evidence that RL on verifiable math may train a shared reasoning skill rather than only memorizing problem templates; a testable implication is that RL on other verifiable domains should similarly improve code scores.","If later audits find train/test leakage beyond the 9-gram filter, the numerical rankings on AIME25 and LiveCodeBench v6 would need to be re-read as upper bounds; this is the main threat to the state-of-the-art claim."],"forward_implications":["Scaling unique prompts is the higher-leverage SFT move: the fitted regression gives a prompt-count coefficient of 4.831 versus 2.635 for responses per prompt, so collecting more diverse questions should be preferred when feasible.","A stronger SFT start pays off after RL: the final model from SFT v7 beats the one from SFT v5, but the initial 6.6-point AIME24 gap closes to 1.6 points, implying RL can compensate for weaker SFT diversity.","The 8K warm-up RL stage is worth keeping even though it temporarily lowers accuracy: skipping it costs the Stage-2 result on AIME25 (51.8 versus 56.7).","Overlong filtering is length-budget dependent: it helps at 8K and 16K, is nearly neutral at 24K, and hurts at 32K, so the decision should be made per stage rather than globally.","Math-only RL transfers to code: math stages alone raise LiveCodeBench performance, and the final model solves a long tail of hard problems the SFT model misses within 128 attempts."],"supporting_citations":[{"why":"Supplies the stage-wise math/code RL recipe, the on-policy GRPO practice, and the AceReason-Nemotron-1.0 baseline that this work extends.","marker":"Chen et al. (2025)"},{"why":"Used to generate SFT responses and as the reference long-CoT reasoning source; also the origin of the DeepSeek-R1-Distill-Qwen-7B baseline compared against.","marker":"Guo et al. (2025)"},{"why":"Source of the GRPO algorithm and its token-level reward normalization used in the RL objective.","marker":"Shao et al. (2024)"},{"why":"DAPO: the prior result on overlong penalties that this paper systematically re-tests across all RL stages.","marker":"Yu et al. (2025)"},{"why":"Supplies the 9-gram overlap decontamination filter applied to SFT data against the test benchmarks.","marker":"Muennighoff et al. (2025)"},{"why":"Provides the Qwen2.5-Math-7B base model that all training starts from; the paper modifies its rope_theta to support long context.","marker":"Yang et al. (2024)"},{"why":"Skywork-OR1 is a math/RL baseline and a prior source of findings on overlong filtering that this work extends and partly contradicts.","marker":"He et al. (2025)"},{"why":"OpenMathReasoning is one of the three math SFT data sources whose scaling drives the measured gains.","marker":"Moshkov et al. (2025)"}],"fun_headline_variants":["7B model tops Qwen2.5-7B on math and code after SFT+RL","SFT scaling and RL: how a 7B model set new math-code records","Entropy 0.3 unlocks RL gains on scaled SFT for 7B reasoning","Scale prompts, then RL: 7B model leads Qwen2.5-7B reasoning","SFT+RL synergy lifts 7B to SOTA among Qwen2.5-7B peers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison is only as clean as the 9-gram decontamination filter: the paper assumes that removing samples with any 9-gram overlap with test benchmarks fully removes train/test leakage, and that lower-contamination benchmarks like AIME25 and LiveCodeBench v6 are not leaking through other channels.","fun_headline_variants_meta":{"raw":{"variants":["7B model tops Qwen2.5-7B on math and code after SFT+RL","SFT scaling and RL: how a 7B model set new math-code records","Entropy 0.3 unlocks RL gains on scaled SFT for 7B reasoning","Scale prompts, then RL: 7B model leads Qwen2.5-7B reasoning","SFT+RL synergy lifts 7B to SOTA among Qwen2.5-7B peers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000342,"raw_usage":{"total_tokens":1967,"prompt_tokens":1116,"completion_tokens":851,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":732,"completion_tokens_details":{"reasoning_tokens":727}},"tokens_in":732,"tokens_out":851,"duration_ms":7872,"temperature":1.0,"reasoning_tokens":727,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:36:00.623808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an independent contamination audit of the released SFT and RL data against AIME25 and LiveCodeBench v6 using paraphrased or lightly edited test problems that share no 9-gram with the originals; if the training data still contains recognizable versions of held-out problems, the reported RL gains on those benchmarks would shrink or disappear when retrained on a clean corpus.","supporting_citations":[],"review_version":1}