{"id":"9ea0f29b-949f-4dfc-981a-b88f88207baf","arxiv_id":"2501.14002","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Problem-solving data in continued pretraining improves LLM math reasoning more than general math corpora, and tutorship amplification is the most effective synthesis method.","lead":"This paper shows that feeding language models worked math problems during continued pre-training, rather than only math textbook text, improves their math reasoning far more. The authors also find that a 'teacher corrects student mistakes' method of synthesizing training data works best, and they release a competitive 8B-parameter math model, MathGPT-8B, trained on far fewer tokens than larger rivals.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing control for Result 1 is Appendix A.4's decontamination step, but the paper reports removals only for OpenWebMath and never for Lila/NuminaMath/synthetic data; 100-byte MinHash filtering does not rule out benchmark leakage in the problem-solving tokens that produce the Figure…","rationale":"The reader identified the MinHash decontamination threshold as the weakest assumption; I agree that it is the most load-bearing control. The paper has real strengths: Llama2 is chosen because it predates OpenWebMath, the evaluation set is expanded to newer GAOKAO and ZHONGKAO items, and the model is released. However, A.1 lists Lila as part of the problem-solving training data, and Lila is a benchmark bundle that aggregates existing math problems, so contamination of GSM8K and MATH is not merely hypothetical. Appendix A.4's parenthetical admission that removing 2594 OpenWebMath documents had a significant effect on the conclusions demonstrates that the decontamination step is not a minor detail; it directly controls whether Result 1 reflects genuine capability or memorized answers. Reporting only the OpenWebMath removal count leaves the most suspicious training corpora unexamined. The absence of per-corpus contamination counts, the paraphrase blind spot of MinHash, and the propagation risk from synthetic data together make the central comparison unverifiable as currently documented. Other concerns, such as checkpoint selection, max-of-zero/few-shot scoring, and lack of error bars, are secondary once Result 1 is secured. A concrete contamination audit and retraining with stricter filtering would settle the question: clean results would support the claim, while a collapsed gap would invalidate it. Because this specific check has not been run, the reader's CONDITIONAL verdict remains appropriate.","tokens_in":17994,"tokens_out":4327,"duration_ms":43872,"concrete_test":"Run a decontamination audit and a targeted retraining experiment. First, apply the same MinHash 100-byte procedure separately to the general corpus, math corpus, problem-solving data, and each synthetic expansion, and report how many documents from each corpus are removed against each of GSM8K, MATH, GAOKAO, and ZHONGKAO. Second, run a stronger paraphrase/n-gram containment probe, e.g., tokenized 8-gram containment with normalized digits, names, and variables, to estimate near-duplicate contamination not caught by the 100-byte threshold. Third, retrain the Section 3 comparison (at minimum Base1, Test2, and Test4) after dropping every flagged document from the problem-solving and synthetic data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Result 1, supported by Figure 1 and Table 7) is that replacing general math-corpus tokens with problem-solving tokens during continued pre-training improves mathematical reasoning. The load-bearing control is Appendix A.4's decontamination procedure. The paper explicitly notes that removing 2594 OpenWebMath documents \"had a significant impact on the conclusions during our initial experiments,\" yet it reports no per-corpus contamination counts for the problem-solving data. This matters because the training mix in A.1 includes Lila and NuminaMath; Lila is a collection of existing benchmark problems, and GSM8K and MATH are precisely the kinds of sources such collections aggregate. If a non-trivial fraction of the 7.2B problem-solving tokens, or of the 14M synthetic samples generated from those seeds, are near-duplicates of evaluation questions, then the higher accuracy of Test1-4 over Base1 could reflect memorization rather than the hypothesized benefit of problem-solving data. The 100-byte MinHash threshold removes exact and near-duplicate spans, but it is not designed to catch paraphrase-level contamination or modified versions of benchmark questions; synthetic expansion can propagate a contaminated seed into many paraphrased training samples. The paper's own parenthetical admission makes this a live risk, not a hypothetical, and the absence of reported decontamination counts for the corpora most likely to contain benchmark items leaves the central comparison unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies alternatives to standard continued pre-training (CPT) for mathematical reasoning in LLMs. It asks three research questions: (RQ1) whether problem-solving data (questions plus solutions) is more effective than general mathematical corpora during CPT; (RQ2) which data synthesis methods are most efficient; and (RQ3) how capabilities acquired from the same problem-solving data differ between CPT and SFT. Using Llama2 as the base model, the authors report that problem-solving data yields larger accuracy gains than an equal amount of math corpus tokens, that tutorship amplification is the best synthesis method among four tested, and that CPT outperforms SFT at similar data volumes. They then apply these findings to train MathGPT-8B, claiming it is comparable to Qwen2.5-Math-7B while using fewer math tokens. The main results are supported by Figure 1 and Table 7 (RQ1), Table 1 (RQ2), and Tables 2, 3, and 8 (RQ3).","tokens_in":18313,"tokens_out":5120,"duration_ms":43985,"significance":"If the central claim is sound, the paper offers a compute-efficient recipe for building math-specialized base models and contributes a released model (MathGPT-8B) with a detailed data recipe. The study covers an unusually large experimental space, including data mixture ratios, four synthesis methods, and stage comparisons, with transparent appendices. The choice of Llama2 to reduce contamination from OpenWebMath is thoughtful, and the addition of GAOKAO/ZHONGKAO broadens evaluation beyond GSM8K and MATH. However, the strength of the evidence is limited by incomplete contamination reporting and by single-run evaluations without variance estimates; these issues directly affect the load-bearing claims.","major_comments":[{"comment":"The decontamination procedure is the load-bearing control for Result 1, but the paper reports removal counts only for OpenWebMath (2594 documents). The problem-solving data include Lila and NuminaMath, which aggregate existing benchmark questions, and 14M seeds are used to generate synthetic data. If a nontrivial fraction of these tokens overlaps with GSM8K or MATH evaluation questions, the advantage of Test1-4 over Base1 in Figure 1 and Table 7 could reflect memorization rather than improved reasoning. The 100-byte MinHash threshold is not sufficient to detect paraphrase-level overlap, and the paper's own statement that removing the OpenWebMath documents 'had a significant impact on the conclusions during our initial experiments' demonstrates that the conclusions are sensitive to contamination. The authors should report per-corpus contamination counts for the problem-solving and synthetic data, and re-run the key comparisons with stricter decontamination or otherwise demonstrate that removal of all overlapping documents does not change the conclusions.","section":"Appendix A.4, Section 3"},{"comment":"All conclusions are based on single training runs with no error bars or multiple seeds. In Table 7, the average-accuracy differences between Test2 (39.88) and Test4 (40.80) are small and not consistent across datasets: Test4 is lower than Test2 on GSM8K (40.11 vs. 48.29) but higher on MATH and GAOKAO. The claim that 'the green line consistently surpasses the other two' in Figure 1 is not supported without variance estimates. Additionally, the checkpoint selection rule (lowest validation loss) is applied per run, which can introduce selection bias when comparing training configurations. The authors should provide at least two or three seeds for the main comparisons, or report statistical significance of the average-accuracy differences.","section":"Section 3, Figure 1, Table 7"},{"comment":"The evaluation protocol uses the maximum of zero-shot and few-shot accuracy for each dataset. This is an unconventional metric that inflates reported scores relative to standard single-setting evaluations. If the baselines in Table 4 were not evaluated with the same max rule, the comparison with published numbers is uncalibrated; if they were, the individual zero-shot and few-shot scores should be reported to allow assessment. This is particularly important for the headline claim that MathGPT-8B is comparable to Qwen2.5-Math-7B (70.62 vs. 69.13 average), since a 1.5-point difference may depend on which setting is selected per dataset.","section":"Appendix A.5, Table 4"}],"minor_comments":[{"comment":"The sentence 'we evaluate our model using the Gao et al. (2024) on the MMLU benchmarks' is missing a noun such as 'framework'; please revise.","section":"Section 6"},{"comment":"There are several typos and grammatical issues: 'leading to leading to' in Section 5.1, 'markgroup' in Section 5.1, and subject-verb disagreements such as 'the improvements ... is'. A careful proofread is needed.","section":"Throughout"},{"comment":"The token counts for synthetic data vary widely (e.g., Tutor-Amp 13.90B vs. Query-Exp 4.78B), but the token-to-question ratio is not discussed; since the comparison aims to identify the most efficient synthesis method, it would be helpful to control for token budget or to report compute per method.","section":"Table 1"},{"comment":"The GAOKAO and ZHONGKAO evaluation sets are translated and reviewed; the paper does not state whether the translation process could introduce artifacts, or whether the answer comparison model was validated on these sets.","section":"Appendix A.3"},{"comment":"The t-SNE analysis uses 'up to 1,000 problems' and a cosine similarity calculation, but the t-SNE parameters and the exact sample sizes are not reported, making the visual claim that GSM8K is distributionally farthest difficult to verify.","section":"Appendix B, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical contribution with a large experimental matrix and a released model. The main risk is data contamination; the authors should be asked to provide per-corpus decontamination counts and, if necessary, re-run the central experiments with stricter filtering. The single-run analysis also weakens the conclusions. If these issues are resolved, the paper would be a useful reference for training math-specialized models. The unconventional max(zero-shot, few-shot) evaluation protocol should also be clarified in relation to the baseline numbers in Table 4."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core finding—that problem-solving data beats general math corpora during continued pretraining—is plausible and likely right, but the paper does not fully verify it because the decontamination reporting is incomplete. The authors admit in Appendix A.4 that removing 2,594 OpenWebMath documents “had a significant impact on the conclusions,” yet they report no contamination counts for Lila, NuminaMath, or the synthetic data. Those are exactly the corpora most likely to contain benchmark-style problems, and 100-byte MinHash filtering does not catch paraphrase-level leakage. So the stress-test concern is real: the advantage of problem-solving data in Figure 1 could be partly memorization. That said, the effect is large and consistent across four datasets, including GAOKAO and ZHONGKAO which post-date the base model, so I would not dismiss the result. The paper needs per-corpus contamination counts and ideally repeated runs with public training data before the recipe is relied on.\n\nWhat is genuinely new: the head-to-head of problem-solving data versus general math corpora under controlled token budgets, the comparison of four synthesis methods specifically for CPT, and the difficulty-based analysis of CPT versus SFT. The tutorship amplification result is interesting and the released MathGPT-8B gives the community something concrete to build on. Those are real contributions.\n\nThe soft spots are mostly about measurement. The main comparisons are single runs with no error bars; the evaluation protocol takes the max of zero-shot and few-shot accuracy, which flatters every model; and checkpoint selection uses the lowest validation loss or best of ten checkpoints, which invites selection bias. The final MathGPT-8B comparison is uncontrolled—different base model, different token budget, different data mix—so the “comparable to Qwen2.5-Math-7B with 1/10 the tokens” claim is softer than the paper suggests. None of these alone falsifies the central argument, but together they mean the headline numbers should be treated as provisional.\n\nWho should read this: anyone working on math-specific LLMs or data curation for continued pretraining. It deserves serious refereeing—the questions are important and the experimental design, aside from the decontamination gap, is thoughtful. I would ask for a revision that reports contamination counts per corpus, adds variance information or repeated runs, and tones down the final-model claim until it is matched on the same base and compute. Worth citing, worth discussing, but verify the contamination story before building on it.","headline":"Useful, credible empirical comparison of problem-solving vs corpus data in math CPT, but the decontamination evidence for the load-bearing Result 1 is thinner than it should be.","tokens_in":18853,"tokens_out":1791,"would_cite":true,"duration_ms":17525,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that during continued pre-training, problem-solving data—questions with reasoning steps—improves mathematical reasoning more than general mathematical text, and that a tutorship-style synthesis method yields the…","keywords":["mathematical reasoning","continued pre-training","problem-solving data","synthetic data","tutorship amplification","supervised fine-tuning","data efficiency","MathGPT-8B"],"falsifier":"Re-run the Test2 and Test4 recipes on a base model that predates the evaluation sets, but evaluate only on math questions released after training data collection, with the same decontamination filter; if the problem-solving-data advantage over general math corpora shrinks or disappears on those future questions, memorization of benchmark text, not reasoning, would explain the result.","tokens_in":17798,"feed_emoji":"🧮","tokens_out":4030,"duration_ms":33419,"temperature":0.7,"pith_summary":"The paper asks whether, during continued pre-training of a large language model on mathematics, it is better to feed it general mathematical text or to feed it solved problems. It claims that problem-solving data—questions together with their reasoning steps—improves mathematical reasoning more than the same number of tokens of mathematical corpus, and that a higher proportion of such data is better. It further claims that synthetic problems generated from existing ones are beneficial, with a tutorship amplification method that has a teacher model correct a student model's mistakes working best. Finally, it claims that continued pre-training develops more mathematical capability from the same problem-solving data than supervised fine-tuning, largely because harder problems are learned better during pre-training. If these claims hold, training pipelines can get stronger math models with far fewer tokens, as illustrated by MathGPT-8B.","feed_headline":"Solved problems beat general math text in LLM pretraining","feed_subtitle":"A pre-training study finds question-solution pairs outperform math corpora; tutoring-style synthetic data helps most.","key_machinery":"The mechanism is the data type itself: problem-solving data, defined as math questions paired with full solution steps, given during continued pre-training. The paper treats this as teaching knowledge application rather than knowledge accumulation. The named synthesis method that carries the strongest result is tutorship amplification, where a teacher model checks a student model's solution, points out erroneous steps, and continues solving from the error, giving the model a self-correction loop.","core_discovery":"The central discovery is a training-stage asymmetry: the same problem-solving data yields more mathematical reasoning ability when used in continued pre-training (CPT) than in supervised fine-tuning (SFT). According to the paper, CPT teaches the model to apply knowledge it already has, while SFT mainly teaches instruction following and learns simpler in-distribution problems. The paper attributes SFT's weaker gain partly to its poorer learning of harder, multi-step problems, and concludes that more challenging problem-solving data should be prepared for the CPT phase.","pith_inferences":["The paper's evidence suggests the advantage of CPT over SFT might grow with problem difficulty, so a direct test would train on only the hardest subsets and measure transfer to unseen competition problems.","The tutorship amplification result implies that error-correcting trajectories, not just correct solutions, may be the most token-efficient way to teach self-correction during pre-training; this could be tested by ablating the correction step.","If the decontamination assumption holds, the same recipe should transfer to other reasoning domains such as code or science, where problem-solution pairs are abundant."],"forward_implications":["Pipelines that currently spend pre-training tokens on filtered math web text should reallocate a larger share to solved problems.","A small amount of SFT data suffices for instruction-following; extra SFT tokens add only modest reasoning gains compared to CPT.","Synthetic data from query expansion, response diversification, and tutorship amplification can extend limited problem sets, with tutorship amplification giving the largest lift.","Models trained this way can reach the math accuracy of much larger models on GSM8K, MATH, GAOKAO, and ZHONGKAO while using a fraction of the math tokens."],"supporting_citations":[{"why":"Supplies OpenWebMath, the main general mathematical corpus used as the base-line and comparison condition for problem-solving data.","marker":"Paster et al., 2023"},{"why":"Provides the LLEMMA continued pre-training approach and the AlgebraicStack data, establishing the earlier paradigm the paper tests against.","marker":"Azerbayev et al., 2023"},{"why":"Supplies NuminaMath as a major source of the problem-solving data used in the controlled experiments.","marker":"Li et al., 2024"},{"why":"Provides the MinHash deduplication framework used for the decontamination of training data against evaluation sets.","marker":"Lee et al., 2022"},{"why":"Introduces the reshaping and query-expansion ideas that the paper's query expansion synthesis method builds on.","marker":"Yu et al., 2023"},{"why":"Motivates retrospective enhancement and the idea that models can be taught to detect and correct their own mistakes.","marker":"Ye et al., 2024"},{"why":"Provides DeepSeekMath-Base-7B, a math-specific baseline that MathGPT-8B aims to outperform.","marker":"Shao et al., 2024"},{"why":"Supplies Qwen2-Math-7B and Qwen2-Math-72B, baselines used to position MathGPT-8B's efficiency and capability.","marker":"Yang et al., 2024a"},{"why":"Supplies Qwen2.5-Math-7B, the 1-trillion-token model that MathGPT-8B matches with roughly one-tenth the math tokens.","marker":"Yang et al., 2024b"},{"why":"Provides Llama2 as the base model for the controlled CPT-versus-SFT experiments, chosen because it predates OpenWebMath and the evaluation sets.","marker":"Touvron et al., 2023"}],"fun_headline_variants":["Solved problems beat math text for LLM pretraining","Tutoring-style synthetic data best for math LLM pretraining","CPT with solved problems beats SFT for math reasoning","Why CPT beats SFT for math LLM problem-solving data","Problem-solving data for pretraining, not just fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that removing every training document sharing more than 100 bytes with an evaluation question is enough to rule out benchmark memorization, so the measured gains come from genuine reasoning rather than from having seen the test questions.","fun_headline_variants_meta":{"raw":{"variants":["Solved problems beat math text for LLM pretraining","Tutoring-style synthetic data best for math LLM pretraining","CPT with solved problems beats SFT for math reasoning","Why CPT beats SFT for math LLM problem-solving data","Problem-solving data for pretraining, not just fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000944,"raw_usage":{"total_tokens":4037,"prompt_tokens":956,"completion_tokens":3081,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":2998}},"tokens_in":572,"tokens_out":3081,"duration_ms":19867,"temperature":1.0,"reasoning_tokens":2998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:46:18.441873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Test2 and Test4 recipes on a base model that predates the evaluation sets, but evaluate only on math questions released after training data collection, with the same decontamination filter; if the problem-solving-data advantage over general math corpora shrinks or disappears on those future questions, memorization of benchmark text, not reasoning, would explain the result.","supporting_citations":[{"cited_title":"Deduplicating training data makes language models better","cited_arxiv_id":null,"evidence_quote":"Provides the MinHash deduplication framework used for the decontamination of training data against evaluation sets."}],"review_version":1}