Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that during continued pre-training, problem-solving data—questions with reasoning steps—improves mathematical reasoning more than general mathematical text, and that a tutorship-style synthesis method yields the…

desk verdict Useful, credible empirical comparison of problem-solving vs corpus data in math CPT, but the decontamination evidence for the load-bearing Result 1 is thinner than it should be. read the letter →

arxiv 2501.14002 v3 pith:H7YCQHAU submitted 2025-01-23 cs.CL cs.AI

Zui Chen , Tianqiao Liu , Mi Tian , Qing Tong , Weiqi Luo , Zitao Liu This is my paper · ORCID
classification cs.CLcs.AI
keywords mathematicalreasoningcontinuedpre-trainingproblem-solvingdatasynthetictutorshipamplificationsupervisedfine-tuningefficiencyMathGPT-8B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether, during continued pre-training of a large language model on mathematics, it is better to feed it general mathematical text or to feed it solved problems. It claims that problem-solving data—questions together with their reasoning steps—improves mathematical reasoning more than the same number of tokens of mathematical corpus, and that a higher proportion of such data is better. It further claims that synthetic problems generated from existing ones are beneficial, with a tutorship amplification method that has a teacher model correct a student model's mistakes working best. Finally, it claims that continued pre-training develops more mathematical capability from the same problem-solving data than supervised fine-tuning, largely because harder problems are learned better during pre-training. If these claims hold, training pipelines can get stronger math models with far fewer tokens, as illustrated by MathGPT-8B.

What carries the argument

The mechanism is the data type itself: problem-solving data, defined as math questions paired with full solution steps, given during continued pre-training. The paper treats this as teaching knowledge application rather than knowledge accumulation. The named synthesis method that carries the strongest result is tutorship amplification, where a teacher model checks a student model's solution, points out erroneous steps, and continues solving from the error, giving the model a self-correction loop.

What would settle it

Re-run the Test2 and Test4 recipes on a base model that predates the evaluation sets, but evaluate only on math questions released after training data collection, with the same decontamination filter; if the problem-solving-data advantage over general math corpora shrinks or disappears on those future questions, memorization of benchmark text, not reasoning, would explain the result.

Watch

Extended reading notes

Core claim

The central discovery is a training-stage asymmetry: the same problem-solving data yields more mathematical reasoning ability when used in continued pre-training (CPT) than in supervised fine-tuning (SFT). According to the paper, CPT teaches the model to apply knowledge it already has, while SFT mainly teaches instruction following and learns simpler in-distribution problems. The paper attributes SFT's weaker gain partly to its poorer learning of harder, multi-step problems, and concludes that more challenging problem-solving data should be prepared for the CPT phase.

Load-bearing premise

The load-bearing premise is that removing every training document sharing more than 100 bytes with an evaluation question is enough to rule out benchmark memorization, so the measured gains come from genuine reasoning rather than from having seen the test questions.

Editorial extensions

If this is right

  • Pipelines that currently spend pre-training tokens on filtered math web text should reallocate a larger share to solved problems.
  • A small amount of SFT data suffices for instruction-following; extra SFT tokens add only modest reasoning gains compared to CPT.
  • Synthetic data from query expansion, response diversification, and tutorship amplification can extend limited problem sets, with tutorship amplification giving the largest lift.
  • Models trained this way can reach the math accuracy of much larger models on GSM8K, MATH, GAOKAO, and ZHONGKAO while using a fraction of the math tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's evidence suggests the advantage of CPT over SFT might grow with problem difficulty, so a direct test would train on only the hardest subsets and measure transfer to unseen competition problems.
  • The tutorship amplification result implies that error-correcting trajectories, not just correct solutions, may be the most token-efficient way to teach self-correction during pre-training; this could be tested by ablating the correction step.
  • If the decontamination assumption holds, the same recipe should transfer to other reasoning domains such as code or science, where problem-solution pairs are abundant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies alternatives to standard continued pre-training (CPT) for mathematical reasoning in LLMs. It asks three research questions: (RQ1) whether problem-solving data (questions plus solutions) is more effective than general mathematical corpora during CPT; (RQ2) which data synthesis methods are most efficient; and (RQ3) how capabilities acquired from the same problem-solving data differ between CPT and SFT. Using Llama2 as the base model, the authors report that problem-solving data yields larger accuracy gains than an equal amount of math corpus tokens, that tutorship amplification is the best synthesis method among four tested, and that CPT outperforms SFT at similar data volumes. They then apply these findings to train MathGPT-8B, claiming it is comparable to Qwen2.5-Math-7B while using fewer math tokens. The main results are supported by Figure 1 and Table 7 (RQ1), Table 1 (RQ2), and Tables 2, 3, and 8 (RQ3).

Significance. If the central claim is sound, the paper offers a compute-efficient recipe for building math-specialized base models and contributes a released model (MathGPT-8B) with a detailed data recipe. The study covers an unusually large experimental space, including data mixture ratios, four synthesis methods, and stage comparisons, with transparent appendices. The choice of Llama2 to reduce contamination from OpenWebMath is thoughtful, and the addition of GAOKAO/ZHONGKAO broadens evaluation beyond GSM8K and MATH. However, the strength of the evidence is limited by incomplete contamination reporting and by single-run evaluations without variance estimates; these issues directly affect the load-bearing claims.

major comments (3)
  1. [Appendix A.4, Section 3] The decontamination procedure is the load-bearing control for Result 1, but the paper reports removal counts only for OpenWebMath (2594 documents). The problem-solving data include Lila and NuminaMath, which aggregate existing benchmark questions, and 14M seeds are used to generate synthetic data. If a nontrivial fraction of these tokens overlaps with GSM8K or MATH evaluation questions, the advantage of Test1-4 over Base1 in Figure 1 and Table 7 could reflect memorization rather than improved reasoning. The 100-byte MinHash threshold is not sufficient to detect paraphrase-level overlap, and the paper's own statement that removing the OpenWebMath documents 'had a significant impact on the conclusions during our initial experiments' demonstrates that the conclusions are sensitive to contamination. The authors should report per-corpus contamination counts for the problem-solving and synthetic data, and re-run the key comparisons with stricter decontamination or otherwise demonstrate that removal of all overlapping documents does not change the conclusions.
  2. [Section 3, Figure 1, Table 7] All conclusions are based on single training runs with no error bars or multiple seeds. In Table 7, the average-accuracy differences between Test2 (39.88) and Test4 (40.80) are small and not consistent across datasets: Test4 is lower than Test2 on GSM8K (40.11 vs. 48.29) but higher on MATH and GAOKAO. The claim that 'the green line consistently surpasses the other two' in Figure 1 is not supported without variance estimates. Additionally, the checkpoint selection rule (lowest validation loss) is applied per run, which can introduce selection bias when comparing training configurations. The authors should provide at least two or three seeds for the main comparisons, or report statistical significance of the average-accuracy differences.
  3. [Appendix A.5, Table 4] The evaluation protocol uses the maximum of zero-shot and few-shot accuracy for each dataset. This is an unconventional metric that inflates reported scores relative to standard single-setting evaluations. If the baselines in Table 4 were not evaluated with the same max rule, the comparison with published numbers is uncalibrated; if they were, the individual zero-shot and few-shot scores should be reported to allow assessment. This is particularly important for the headline claim that MathGPT-8B is comparable to Qwen2.5-Math-7B (70.62 vs. 69.13 average), since a 1.5-point difference may depend on which setting is selected per dataset.
minor comments (5)
  1. [Section 6] The sentence 'we evaluate our model using the Gao et al. (2024) on the MMLU benchmarks' is missing a noun such as 'framework'; please revise.
  2. [Throughout] There are several typos and grammatical issues: 'leading to leading to' in Section 5.1, 'markgroup' in Section 5.1, and subject-verb disagreements such as 'the improvements ... is'. A careful proofread is needed.
  3. [Table 1] The token counts for synthetic data vary widely (e.g., Tutor-Amp 13.90B vs. Query-Exp 4.78B), but the token-to-question ratio is not discussed; since the comparison aims to identify the most efficient synthesis method, it would be helpful to control for token budget or to report compute per method.
  4. [Appendix A.3] The GAOKAO and ZHONGKAO evaluation sets are translated and reviewed; the paper does not state whether the translation process could introduce artifacts, or whether the answer comparison model was validated on these sets.
  5. [Appendix B, Figure 4] The t-SNE analysis uses 'up to 1,000 problems' and a cosine similarity calculation, but the t-SNE parameters and the exact sample sizes are not reported, making the visual claim that GSM8K is distributionally farthest difficult to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are direct experimental comparisons, not derivations; the single self-citation is peripheral and the contamination caveat is a validity risk, not a circularity.

full rationale

The paper's load-bearing claims are empirical measurements, not derivations. Result 1 (problem-solving data improves CPT) is supported by controlled comparisons in Section 3 and Appendix D where the total math-token budget is held constant and only the mix of math corpus versus problem-solving data changes (Base1 vs. Test1-4, Table 7). There is no fitted parameter that is later renamed as a prediction, no uniqueness theorem used to force a choice, and no ansatz smuggled in via self-citation. The only self-citation is one supporting reference in Section 4 for the general effectiveness of response-data synthesis: 'The effectiveness of response data synthesis is established through various implementations (Yuan et al., 2023; Yu et al., 2023; Chen et al., 2024).' That citation is not load-bearing because the paper's own controlled experiments in Table 1 carry the conclusion. Appendix A.4's admission that removing 2594 OpenWebMath documents 'had a significant impact on the conclusions during our initial experiments' is a contamination-risk caveat, not circularity: it concerns whether training data leaked evaluation data, not whether a conclusion is equivalent to an input by construction. Similarly, reporting the higher of zero-shot and few-shot accuracy is an evaluation-reporting choice, not a fitted-input-turned-prediction. The MathGPT-8B section builds a recipe from the paper's own experimental findings and then evaluates it, which is a demonstration, not a circular derivation.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central empirical findings rest on several hand-chosen data ratios, difficulty thresholds, and evaluation protocols, plus the domain assumptions that step count measures difficulty and that 1% SFT isolates instruction following. The decontamination threshold is the most consequential because the authors state it changed the conclusions.

free parameters (6)
  • math data mixture ratio (math corpus to problem-solving data) = 3:7 for best configuration and MathGPT-8B
    Chosen after comparing 5:5, 3:7, 7:3 in Section 3; the 3:7 ratio (more problem-solving data) was selected as Test2/MathGPT configuration. This choice is an input to Results 1 and 6, not predicted by theory.
  • reasoning-step difficulty thresholds = easy: 1-3, medium: 4-7, hard: 8+ steps
    Used in Section 5.3 to categorize problem-solving data; these cutoffs are chosen by hand and materially define Result 5's 'hard' category.
  • hard-problem seed filter for MathGPT = problems with more than five reasoning steps
    Used in Section 6 to select seed data for synthesis; the 5-step threshold is a hand-chosen heuristic that contributes to MathGPT-8B's reported performance.
  • SFT data-volume splits = 1%, 10%, 20%, 50%, 100% of problem-solving data
    Used in Section 5.1 to measure instruction-following vs reasoning gains; the 1% split is assumed to isolate instruction following, an assumption that shapes Result 3.
  • decontamination byte thresholds = 2048 bytes for dedup, 100 bytes for contamination removal
    Chosen for MinHash filtering in Appendix A.4; the 100-byte threshold plus removal of 2594 OpenWebMath documents was adjusted after observing 'significant impact on the conclusions'.
  • checkpoint selection scheme = lowest validation loss for CPT; 'optimal' checkpoint for SFT
    CPT uses lowest validation loss; SFT selects 'the optimal result from 10 checkpoints' with an unspecified criterion, both of which affect all reported numbers.
assumptions (6)
  • domain assumption Problem-solving data teaches the model to apply memorized math knowledge, whereas general math corpora only increase memorized knowledge.
    Stated in the Introduction; Results 1-5 depend on this distinction being the causal mechanism.
  • domain assumption Number of reasoning steps in a solution is a valid proxy for problem difficulty.
    Section 5.3 classifies easy/medium/hard entirely by step counts; if step count tracks verbosity rather than difficulty, Result 5 is weakened.
  • domain assumption Taking the higher of zero-shot and few-shot accuracy per dataset gives a fair, robust estimate for comparing models.
    Section 2 Evaluation Metrics; this protocol maximizes scores and may favor models with format quirks over true reasoning.
  • domain assumption SFT with 1% of the problem-solving data imparts instruction-following ability without materially improving reasoning.
    Section 5.1 uses this to decompose the SFT delta; if 1% SFT also improves reasoning, the comparison of CPT vs SFT deltas is misattributed.
  • domain assumption The MinHash decontamination at 100 bytes removes evaluation-set contamination without removing legitimate training signal.
    Appendix A.4; the authors note the removed OpenWebMath documents had a significant impact on conclusions, so the validity of the decontamination is load-bearing.
  • domain assumption Llama2 predating OpenWebMath ensures the base model has not already memorized the math corpus or evaluation sets.
    Appendix A.2; this justifies the experimental setup, but does not protect against contamination from the problem-solving data itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages." pith.science (2026). https://pith.science/paper/H7YCQHAU

@misc{pith2026250114002,
  author       = {Pith},
  title        = {Pith review of: Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H7YCQHAU}},
  note         = {Machine review of arXiv:2501.14002}
}
read the original abstract

Mathematical reasoning remains a challenging area for large language models (LLMs), prompting the development of math-specific LLMs such as LLEMMA, DeepSeekMath, and Qwen2-Math, among others. These models typically follow a two-stage training paradigm: pre-training with math-related corpora and post-training with problem datasets for supervised fine-tuning (SFT). Despite these efforts, the improvements in mathematical reasoning achieved through continued pre-training (CPT) are often less significant compared to those obtained via SFT. This study addresses this discrepancy by exploring alternative strategies during the pre-training phase, focusing on the use of problem-solving data over general mathematical corpora. We investigate three primary research questions: (1) Can problem-solving data enhance the model's mathematical reasoning capabilities more effectively than general mathematical corpora during CPT? (2) Are synthetic data from the same source equally effective, and which synthesis methods are most efficient? (3) How do the capabilities developed from the same problem-solving data differ between the CPT and SFT stages, and what factors contribute to these differences? Our findings indicate that problem-solving data significantly enhances the model's mathematical capabilities compared to general mathematical corpora. We also identify effective data synthesis methods, demonstrating that the tutorship amplification synthesis method achieves the best performance. Furthermore, while SFT facilitates instruction-following abilities, it underperforms compared to CPT with the same data, which can be partially attributed to its poor learning capacity for more challenging problem-solving data. These insights provide valuable guidance for optimizing the mathematical reasoning capabilities of LLMs, culminating in our development of a powerful mathematical base model called MathGPT-8B.

Figures

Figures reproduced from arXiv: 2501.14002 by the authors.

Figure 1
Figure 1. The average accuracy of the four groups varies with the number of steps. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of the acquisition of learning capabilities between the CPT and SFT stages [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Ability dimensions of four evaluation sets [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (a) Data distribution of problems of the four evaluation sets. (b) Dataset similarity based [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: The average accuracy of the five groups varies with the number of steps. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: The relationship between average accuracy and SFT data quantity. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text

    cs.AI 2026-05 conditional novelty 6.0 of 10

    A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.

  2. GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper releases GSM-Plus-BN, a human-verified Bengali translation of the GSM-Plus perturbed math benchmark, and reports accuracy baselines for six open LLMs under standard and CoT prompting.

Reference graph

Works this paper leans on

43 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    Physics of language models: Part 3.1, knowledge storage and extraction

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. In Forty-first International Conference on Machine Learning, 2023

    work page 2023

  2. [2]

    Llemma: An open language model for mathematics

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2023

    work page 2023

  3. [3]

    Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2022

    work page 2022

  4. [4]

    An empirical study of data ability boundary in llms' math reasoning

    Zui Chen, Yezeng Chen, Jiaqi Han, Zhijie Huang, Ji Qi, and Yi Zhou. An empirical study of data ability boundary in llms' math reasoning. arXiv preprint arXiv:2403.00799, 2024

    arXiv 2024

  5. [5]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [6]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024

    arXiv 2024

  7. [7]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  8. [8]

    Kwaiyiimath: Technical report

    Jiayi Fu, Lei Lin, Xiaoyang Gao, Pengli Liu, Zhengzong Chen, Zhirui Yang, Shengnan Zhang, Xue Zheng, Yan Li, Yuliang Liu, et al. Kwaiyiimath: Technical report. arXiv preprint arXiv:2310.07488, 2023

    arXiv 2023

Show all 43 references
  1. [9]

    A framework for few-shot language model evaluation, 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  2. [10]

    Tora: A tool-integrated reasoning agent for mathematical problem solving

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, 2023

    2023

  3. [11]

    Infimm-webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning

    Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, et al. Infimm-webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and AI at Ne...

    2024

  4. [12]

    Reasoning with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 8154--8173, 2023

    2023

  5. [13]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In The Eighth International Conference on Learning Representations, 2020

    2020

  6. [14]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021

    2021

  7. [15]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  8. [16]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 0 22199--22213, 2022

    2022

  9. [17]

    Step-dpo: Step-wise preference optimization for long-chain reasoning of llms

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629, 2024

    2024 arXiv

  10. [18]

    Deduplicating training data makes language models better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp.\ 84...

    2022

  11. [19]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face reposito...

    2024

  12. [20]

    Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023

    Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, LI Jia, Jenny Chim, Qian Liu, et al. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023

    2023

  13. [21]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2022

    2022

  14. [22]

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023

    2023 arXiv

  15. [23]

    Lila: A unified benchmark for mathematical reasoning

    Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, et al. Lila: A unified benchmark for mathematical reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Nat...

    2022

  16. [24]

    Orca-math: Unlocking the potential of slms in grade school math

    Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830, 2024

    2024 arXiv

  17. [25]

    Finding GPT-4 's mistakes with GPT-4 , 2024

    OpenAI. Finding GPT-4 's mistakes with GPT-4 , 2024. URL https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/

    2024

  18. [26]

    Openwebmath: An open dataset of high-quality mathematical web text

    Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2023

    2023

  19. [27]

    Fabio Petroni, Tim Rockt \"a schel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint C...

    2019

  20. [28]

    Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp.\ 5418--5426, 2020

    2020

  21. [29]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  22. [30]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  23. [31]

    Redpajama: an open dataset for training large language models

    Maurice Weber, Daniel Y Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. In The Thirty-eight Conference on Neural Information ...

    2023

  24. [32]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  25. [33]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024 a

    2024 arXiv

  26. [34]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024 b

    2024 arXiv

  27. [35]

    Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems. arXiv preprint arXiv:2408.16293, 2024

    2024 arXiv

  28. [36]

    Mumath-code: Combining tool-use large language models with multi-perspective data augmentation for mathematical reasoning

    Shuo Yin, Weihao You, Zhilong Ji, Guoqiang Zhong, and Jinfeng Bai. Mumath-code: Combining tool-use large language models with multi-perspective data augmentation for mathematical reasoning. arXiv preprint arXiv:2405.07551, 2024

    2024 arXiv

  29. [37]

    Internlm-math: Open math large language models toward verifiable reasoning

    Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024

    2024 arXiv

  30. [38]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  31. [39]

    Scaling relationship on learning mathematical reasoning with large language models

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023

    2023 arXiv

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  33. [41]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  34. [42]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  35. [43]

    -Tv ," (k Ņ-l]ӦIf ![44- dfrg JNν B ! B.u= B ! B F ! B !܃ B ! B !` B ! B= I

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1999

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.