Pith. sign in

REVIEW 5 major objections 4 minor 41 references

Large-Scale Diverse Synthesis for Mid-Training

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a 100B-token synthetic QA dataset, called BoostQA, can be built without human annotation and that mid-training on a 40B-token slice of it lifts Llama-3 8B by 12.74% on average across MMLU and CMMLU while also…

desk verdict A plausible synthetic-data recipe whose headline gain is not verifiable from the submitted text; needs a contamination check before I would believe the 12.74%. read the letter →

arxiv 2508.01326 v1 pith:XQZPFSBV submitted 2025-08-02 cs.CL

Xuemiao Zhang , Chengying Tu , Can Ren , Rongxiang Weng , Hongfei Yan , Jingang Wang , Xunliang Cai This is my paper
classification cs.CL
keywords largelanguagemodelssyntheticdataquestionansweringmid-trainingdiversitySTEMdifficultyannotationbenchmarkimprovement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the bottleneck in improving large language models is not just model size or compute, but the scarcity of high-quality, knowledge-intensive training data, and that a large, diverse synthetic question-answering corpus can fill that gap during the mid-training phase between pretraining and post-training. It introduces BoostQA, a 100B-token dataset built by curating seed data from heterogeneous sources, using DeepSeek-R1 for STEM-focused multi-grade synthesis and high-difficulty synthesis, and refining answers with DeepSeek-V3. The authors report that mid-training Llama-3 8B on 40B tokens of BoostQA improves average MMLU and CMMLU performance by 12.74% and achieves state-of-the-art average performance across 12 benchmarks. The result matters because it suggests a scalable, annotation-light route to injecting domain knowledge into general-purpose models.

What carries the argument

The central object is BoostQA, a 100B-token question-answering dataset produced by a three-stage synthesis framework. The framework curates seed data from heterogeneous sources, then uses DeepSeek-R1 to generate questions and answers across school-grade levels within STEM disciplines and to synthesize high-difficulty examples that counteract difficulty degradation, and finally refines answers using DeepSeek-V3. A paired discipline-and-difficulty annotation system lets the authors probe specific model weaknesses, particularly in STEM disciplines and hard items, and the dataset is applied in mid-training, a stage positioned between pretraining and post-training to optimize domain-specific knowledge acquisition.

What would settle it

Run an n-gram overlap analysis between BoostQA training samples and MMLU/CMMLU test items, or train on a version of BoostQA with answers deliberately corrupted while keeping the same question format; if either intervention reproduces the reported gains, the knowledge-transfer claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a deliberately diversified synthetic question-answering corpus, rather than more raw text, can drive broad and large benchmark gains when used in mid-training. On the paper's reported numbers, Llama-3 8B mid-trained on 40B tokens of BoostQA improves by 12.74% on average over MMLU and CMMLU and achieves the best average performance among compared systems across 12 benchmarks. The improvements are attributed to three mechanisms: heterogeneous seed curation for diversity, DeepSeek-R1-based synthesis that targets STEM disciplines and high difficulty to counteract difficulty degradation, and DeepSeek-V3 answer refinement for output quality. The paper also reports that performance continues to improve as model size, data volume, and initial FLOPs scale, indicating the data pipeline is not tied to one model size.

Load-bearing premise

The reported gains rest on the assumption that the QA pairs generated by DeepSeek-R1 and DeepSeek-V3 are factually reliable and sufficiently diverse to transfer to MMLU and CMMLU without overlapping those benchmarks.

Editorial extensions

If this is right

  • Synthetic QA data can be produced at 100B-token scale without new human annotations, making mid-training data a controllable resource for model developers.
  • Mid-training on 40B tokens of BoostQA is enough to produce double-digit relative gains on knowledge benchmarks, so data diversity may matter more than raw corpus size at this stage.
  • STEM-focused and high-difficulty synthesis can be used to target known model weaknesses, giving practitioners a way to steer capability improvements toward specific disciplines.
  • The reported gains scale with model size, data volume, and initial FLOPs, suggesting the BoostQA pipeline remains useful as training budgets grow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gains are real, part of the improvement may come from aligning the model with the question format and reasoning style of the benchmarks rather than from acquiring genuinely new world knowledge, so evaluation on held-out factual QA would help separate these effects.
  • The same pipeline could plausibly be applied to other base models and languages, and the difficulty annotations could be used for curriculum ordering or adaptive data selection during training.
  • The 12.74% figure should be treated as an upper bound until an independent contamination analysis is available, because no such analysis is reported in the abstract.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes BoostQA, a 100B-token synthetic question-answering dataset built by a three-part pipeline: heterogeneous seed curation, DeepSeek-R1-based STEM-focused and high-difficulty synthesis, and DeepSeek-V3 answer refinement. The authors then apply BoostQA to mid-training Llama-3 8B on a 40B-token subset and report an average improvement of 12.74% on MMLU and CMMLU, along with state-of-the-art average performance across 12 benchmarks. However, the submitted full text is almost entirely corrupted mojibake; only the abstract and a few fragments are readable. As a result, essentially all technical claims — including the experimental setup, baselines, per-benchmark numbers, and ablations — cannot be inspected or independently verified from the manuscript as received.

Significance. If the reported results hold, the paper would make a substantive contribution to data-centric LLM research: it proposes a large-scale, diverse QA synthesis pipeline, demonstrates its use in mid-training, and reports sizable gains over strong baselines. The claimed scaling behavior (with model size, data volume, and initial FLOPs) is also potentially valuable. However, significance is conditional on the verifiability of the empirical claims, and the current manuscript does not permit verification. No machine-checked proofs or reproducible code are evident from the readable material, which further lowers confidence.

major comments (5)
  1. [Full text] The full text of the manuscript is corrupted (mojibake) and unreadable: all sections, tables, equations, and experimental details are inaccessible. This is a load-bearing problem because the central claim of a 12.74% average improvement on MMLU and CMMLU cannot be verified, nor can the per-benchmark results, baselines, training configurations, or ablation studies be inspected. The abstract alone is insufficient for a rigorous review; the paper must be resubmitted with a readable full text before the scientific content can be evaluated.
  2. [Abstract] The abstract reports no contamination analysis or deduplication against evaluation benchmarks, even though the synthetic data are generated by DeepSeek-R1 and DeepSeek-V3, which were pretrained on web-scale corpora that likely include MMLU, CMMLU, and other benchmark items. If BoostQA contains near-duplicates of evaluation questions, the reported gains could be inflated by memorization rather than attributable to the synthesis method. A contamination analysis (e.g., n-gram or embedding overlap between training and evaluation data, reported per benchmark) is required to support the causal claim.
  3. [Abstract] The abstract lacks a matched control: the only comparison is between mid-training with BoostQA and the base Llama-3 8B model. Without mid-training on a comparable 40B-token general corpus (or on an alternative QA dataset of similar size and diversity), the reported improvement cannot be attributed to BoostQA specifically, as opposed to generic continued pretraining or the simple addition of QA data. The abstract should describe such a control, or the full text must be readable to confirm that one exists.
  4. [Abstract] The abstract claims 'SOTA average performance across 12 benchmarks' but names only MMLU and CMMLU; the other ten benchmarks, their baseline scores, and the per-benchmark breakdown are not provided in the abstract. The phrase 'average improvement of 12.74%' also lacks a definition of the average (over which benchmarks, with what weighting) and any measure of variance or number of seeds. This information, presumably in the unreadable full text, must be presented explicitly in a revised version.
  5. [Abstract] The abstract states that the pipeline 'probes model deficiencies in STEM disciplines and high-difficulty data' using a 'discipline and difficulty annotation system,' but does not describe how these annotations are obtained, how deficiencies are measured, or how the annotations influence the synthesis. This is a central methodological component, and without a readable description (which may exist in the corrupted full text) the novelty of the method cannot be assessed.
minor comments (4)
  1. [Abstract] The phrase 'traditional corpora provide limited information' is vague; it should be clarified whether the limitation is in factual content, diversity, or both.
  2. [Abstract] The term 'mid-training' is used without definition or citation; the paper should situate this stage relative to pretraining and post-training, with references.
  3. [Abstract] The sentence 'leveraging our designed discipline and difficulty annotation system' has awkward phrasing; consider 'using our discipline and difficulty annotation system' for clarity.
  4. [Abstract] The reported '12.74%' improvement should be accompanied by the exact evaluation protocol, such as few-shot settings and answer extraction methods, even in an abstract of a full paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation identified: the abstract's synthesis-to-mid-training pipeline is an empirical causal claim, and no step in the stated pipeline reduces to its own inputs.

full rationale

The inspectable portion of the paper (the abstract plus heavily corrupted body text) does not contain a derivation in which a predicted quantity is defined by the fitted inputs, nor does it invoke a load-bearing self-citation. The stated pipeline is: curate seeds from heterogeneous sources, synthesize QA data with DeepSeek-R1 and DeepSeek-V3, mid-train Llama-3 on a 40B-token subset, then evaluate on MMLU, CMMLU, and ten additional benchmarks. Each stage is conceptually distinct from the next, and the reported 12.74% improvement is not used to construct the seed data, the synthesis prompts, or the mid-training loss. The absence of a reported contamination or deduplication analysis is a serious external-validity concern—if benchmark items entered the 40B-token training mixture, the gain would not support the synthesis method—but that is a benchmark-overlap threat, not circularity. The manuscript itself provides no quoted statement or equation showing that the training data were derived from the evaluation benchmarks, and the reader's uncertainty about contamination cannot be converted into a demonstrated circular step under the rule that circularity must be exhibited by specific reduction. Therefore the honest finding is no significant circularity, with the caveat that the corrupted full text prevents a complete audit of the experimental details.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

Full text is unreadable, so only abstract-level assumptions can be listed. No theoretical entities are introduced by the abstract.

free parameters (2)
  • mid-training token budget = 40B tokens
    The abstract states Llama-3 8B is mid-trained on a 40B-token subset; the choice of this budget is an experimental setting, not derived from theory.
  • total BoostQA size = 100B tokens
    The synthetic corpus size is a design choice and may influence results; no justification is given in the abstract.
assumptions (3)
  • domain assumption DeepSeek-R1 and DeepSeek-V3 generate factually correct and sufficiently diverse synthetic QA data for training.
    The entire pipeline relies on the quality of synthetic outputs from these models, as stated in the abstract.
  • domain assumption The evaluation benchmarks (MMLU, CMMLU, etc.) are not contaminated by BoostQA training data.
    No contamination analysis is visible in the abstract; if the synthetic data contains benchmark questions, the gains may be inflated.
  • domain assumption Mid-training on 40B tokens of BoostQA is directly comparable to the baseline training setups.
    The claimed 12.74% improvement depends on fair comparisons with previous mid-training or continued pretraining methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large-Scale Diverse Synthesis for Mid-Training." pith.science (2026). https://pith.science/paper/XQZPFSBV

@misc{pith2026250801326,
  author       = {Pith},
  title        = {Pith review of: Large-Scale Diverse Synthesis for Mid-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQZPFSBV}},
  note         = {Machine review of arXiv:2508.01326}
}
abstract

The scarcity of high-quality, knowledge-intensive training data hinders the development of large language models (LLMs), as traditional corpora provide limited information. Previous studies have synthesized and integrated corpora-dependent question-answering (QA) data to improve model performance but face challenges in QA data scalability and knowledge diversity, particularly in cross-domain contexts. Furthermore, leveraging our designed discipline and difficulty annotation system, we probe model deficiencies in STEM disciplines and high-difficulty data. To overcome these limitations, we propose a novel diversified pipeline to synthesize BoostQA, a 100B-token large-scale QA dataset. Our synthesis framework: (1) curates seed data from heterogeneous sources; (2) utilizes DeepSeek-R1 to implement STEM-focused multi-grade synthesis to boost data diversity and high-difficulty synthesis to mitigate difficulty degradation; (3) refines answers via DeepSeek-V3 to improve output quality. We utilize BoostQA in mid-training, a mid-stage between pre-training and post-training, to optimize domain-specific knowledge acquisition and enhance data quality. Our method enables Llama-3 8B, mid-trained on a 40B-token dataset, to achieve an average improvement of $\mathbf{12.74\%}$ on MMLU and CMMLU and establish SOTA average performance across 12 benchmarks. BoostQA also demonstrates robust scalability, with performance consistently improving as model size, data volume, and initial FLOPs scale.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 23 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

    work page

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    N.; Prabhumoye, S.; Kamalu, J.; Satheesh, S.; Nyberg, E.; Patwary, M.; Shoeybi, M.; and Catanzaro, B

    Akter, S. N.; Prabhumoye, S.; Kamalu, J.; Satheesh, S.; Nyberg, E.; Patwary, M.; Shoeybi, M.; and Catanzaro, B. 2025. MIND : Math Informed syNthetic Dialogues for Pretraining LLM s. In The Thirteenth International Conference on Learning Representations

    work page 2025

  4. [4]

    Amini, A.; Gabriel, S.; Lin, S.; Koncel-Kedziorski, R.; Choi, Y.; and Hajishirzi, H. 2019. M ath QA : Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short P...

    work page 2019

  5. [5]

    Chang, Y.; Cui, L.; Dong, L.; Huang, S.; Huang, Y.; Huang, Y.; Li, S.; Lv, T.; Ma, S.; Sun, Q.; Wang, W.; Wei, F.; Xin, Y.; Yang, M.; Yin, Q.; and Zhang, X. 2024. RedStone: Curating General, Code, Math, and QA Data for Large Language Models. CoRR, abs/2412.03398

    arXiv 2024

  6. [6]

    Chen, J.; Chen, Z.; Wang, J.; Zhou, K.; Zhu, Y.; Jiang, J.; Min, Y.; Zhao, X.; Dou, Z.; Mao, J.; Lin, Y.; Song, R.; Xu, J.; Chen, X.; Yan, R.; Wei, Z.; Hu, D.; Huang, W.; and Wen, J.-R. 2025. Towards Effective and Efficient Continual Pre-training of Large Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lingu...

    work page 2025

  7. [7]

    Cheng, D.; Gu, Y.; Huang, S.; Bi, J.; Huang, M.; and Wei, F. 2024. Instruction Pre-Training: Language Models are Supervised Multitask Learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2529--2550

    work page 2024

  8. [8]

    Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. CoRR, abs/1803.05457

    arXiv 2018

Show all 41 references
  1. [9]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. CoRR, abs/2110.14168

    2021 arXiv

  2. [10]

    DeepSeek-AI; Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; Zhang, X.; Yu, X.; Wu, Y.; Wu, Z. F.; Gou, Z.; Shao, Z.; Li, Z.; Gao, Z.; Liu, A.; Xue, B.; Wang, B.; Wu, B.; Feng, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; D...

    2025 arXiv

  3. [11]

    L.; Liang, J.; Guo, J.; Ni, J.; Li, J.; Wang, J.; Chen, J

    DeepSeek-AI; Liu, A.; Feng, B.; Xue, B.; Wang, B.; Wu, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; Dai, D.; Guo, D.; Yang, D.; Chen, D.; Ji, D.; Li, E.; Lin, F.; Dai, F.; Luo, F.; Hao, G.; Chen, G.; Li, G.; Zhang, H.; Bao, H.; Xu, H.; Wang, H.; Zhang, H.; Ding, H.; Xi...

    2024 arXiv

  4. [12]

    Du, Z.; Zeng, A.; Dong, Y.; and Tang, J. 2024. Understanding Emergent Abilities of Language Models from the Loss Perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  5. [13]

    Dua, D.; Wang, Y.; Dasigi, P.; Stanovsky, G.; Singh, S.; and Gardner, M. 2019. DROP : A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Lingu...

    2019

  6. [14]

    C.; Nikolaidis, C.; Allonsius, D.; Song, D.; Pintz, D

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; Goyal, A.; Hartshorn, A.; Yang, A.; Mitra, A.; Sravankumar, A.; Korenev, A.; Hinsvark, A.; Rao, A.; Zhang, A.; Rodriguez, A.; Gregerson, A.; Spataru, A.; Roziè...

    2024 arXiv

  7. [15]

    Gao, L.; Biderman, S.; Black, S.; Golding, L.; Hoppe, T.; Foster, C.; Phang, J.; He, H.; Thite, A.; Nabeshima, N.; Presser, S.; and Leahy, C. 2021 a . The Pile: An 800GB Dataset of Diverse Text for Language Modeling. CoRR, abs/2101.00027

    2021 arXiv

  8. [16]

    Gao, L.; Tow, J.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; McDonell, K.; Muennighoff, N.; et al. 2021 b . A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10: 8--9

    2021

  9. [17]

    Ge, T.; Chan, X.; Wang, X.; Yu, D.; Mi, H.; and Yu, D. 2025. Scaling Synthetic Data Creation with 1,000,000,000 Personas. arXiv:2406.20094

    2025 arXiv

  10. [18]

    Gekhman, Z.; Yona, G.; Aharoni, R.; Eyal, M.; Feder, A.; Reichart, R.; and Herzig, J. 2024. Does Fine-Tuning LLM s on New Knowledge Encourage Hallucinations? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 7765--7784

    2024

  11. [19]

    Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021 a . Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations

    2021

  12. [20]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021 b . Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

    2021

  13. [21]

    Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; jiayi lei; Fu, Y.; Sun, M.; and He, J. 2023. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. In Thirty-seventh Conference on Neural Information Process...

    2023

  14. [22]

    Jiang, J.; Li, J.; Zhao, X.; Song, Y.; Zhang, T.; and Wen, J.-R. 2025. Mix- CPT : A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment. In The Thirteenth International Conference on Learning Representations

    2025

  15. [23]

    Li, H.; Zhang, Y.; Koto, F.; Yang, Y.; Zhao, H.; Gong, Y.; Duan, N.; and Baldwin, T. 2024. CMMLU : Measuring massive multitask language understanding in C hinese. In Findings of the Association for Computational Linguistics: ACL 2024, 11260--11285

    2024

  16. [24]

    Li, P.; Li, W.; He, Z.; Wang, X.; Cao, Y.; Zhou, J.; and Xu, W. 2016. Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering. arXiv:1607.06275

    2016 arXiv

  17. [25]

    Maini, P.; Seto, S.; Bai, R.; Grangier, D.; Zhang, Y.; and Jaitly, N. 2024. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14044--14072

    2024

  18. [26]

    Narayanan, D.; Shoeybi, M.; Casper, J.; LeGresley, P.; Patwary, M.; Korthikanti, V.; Vainbrand, D.; Kashinkunti, P.; Bernauer, J.; Catanzaro, B.; Phanishayee, A.; and Zaharia, M. 2021. Efficient large-scale language model training on GPU clusters using megatron-LM. In Proceedi...

    2021

  19. [27]

    W.; Liu, J.; Malik, S.; Merrill, W.; Miranda, L

    OLMo, T.; Walsh, P.; Soldaini, L.; Groeneveld, D.; Lo, K.; Arora, S.; Bhagia, A.; Gu, Y.; Huang, S.; Jordan, M.; Lambert, N.; Schwenk, D.; Tafjord, O.; Anderson, T.; Atkinson, D.; Brahman, F.; Clark, C.; Dasigi, P.; Dziri, N.; Guerquin, M.; Ivison, H.; Koh, P. W.; Liu, J.; Mal...

    2025 arXiv

  20. [28]

    Parmar, J.; Satheesh, S.; Patwary, M.; Shoeybi, M.; and Catanzaro, B. 2024. Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models. arXiv:2407.07263

    2024 arXiv

  21. [29]

    B.; Lozhkov, A.; Mitchell, M.; Raffel, C.; Werra, L

    Penedo, G.; Kydl \' c ek, H.; allal, L. B.; Lozhkov, A.; Mitchell, M.; Raffel, C.; Werra, L. V.; and Wolf, T. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and B...

    2024

  22. [30]

    Qwen; :; Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao, K.; Yang, K.; Yu, L.; Li, M.; Xue, M.; Zhang, P.; Zhu, Q.; Men, R.; Lin,...

    2025 arXiv

  23. [31]

    L.; Bhagavatula, C.; and Choi, Y

    Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2021. WinoGrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9): 99–106

    2021

  24. [32]

    K.; Wu, Y.; and Guo, D

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300

    2024 arXiv

  25. [33]

    Su, D.; Kong, K.; Lin, Y.; Jennings, J.; Norick, B.; Kliegl, M.; Patwary, M.; Shoeybi, M.; and Catanzaro, B. 2025. Nemotron- CC : Transforming C ommon C rawl into a Refined Long-Horizon Pretraining Dataset. In Che, W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds., Proce...

    2025

  26. [34]

    W.; Chowdhery, A.; Le, Q.; Chi, E.; Zhou, D.; and Wei, J

    Suzgun, M.; Scales, N.; Sch \"a rli, N.; Gehrmann, S.; Tay, Y.; Chung, H. W.; Chowdhery, A.; Le, Q.; Chi, E.; Zhou, D.; and Wei, J. 2023. Challenging BIG -Bench Tasks and Whether Chain-of-Thought Can Solve Them. In Findings of the Association for Computational Linguistics: ACL...

    2023

  27. [35]

    Wang, Y.; Ma, X.; Zhang, G.; Ni, Y.; Chandra, A.; Guo, S.; Ren, W.; Arulraj, A.; He, X.; Jiang, Z.; Li, T.; Ku, M.; Wang, K.; Zhuang, A.; Fan, R.; Yue, X.; and Chen, W. 2024. MMLU -Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. In The Thirty-ei...

    2024

  28. [36]

    Wang, Z.; Zhou, F.; Li, X.; and Liu, P. 2025. OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling. arXiv:2506.20512

    2025 arXiv

  29. [37]

    Wettig, A.; Gupta, A.; Malik, S.; and Chen, D. 2024. QuRating: Selecting High-Quality Data for Training Language Models. In International Conference on Machine Learning, 52915--52971

    2024

  30. [38]

    Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. H ella S wag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4791--4800

    2019

  31. [39]

    Zhang, G.; Qu, S.; Liu, J.; Zhang, C.; Lin, C.; Yu, C. L.; Pan, D.; Cheng, E.; Liu, J.; Lin, Q.; Yuan, R.; Zheng, T.; Pang, W.; Du, X.; Liang, Y.; Ma, Y.; Li, Y.; Ma, Z.; Lin, B.; Benetos, E.; Yang, H.; Zhou, J.; Ma, K.; Liu, M.; Niu, M.; Wang, N.; Que, Q.; Liu, R.; Liu, S.; G...

    2024 arXiv

  32. [40]

    Zhou, F.; Wang, Z.; Ranjan, N.; Cheng, Z.; Tang, L.; He, G.; Liu, Z.; and Xing, E. P. 2025. MegaMath: Pushing the Limits of Open Math Corpora. arXiv:2504.02807

    2025 arXiv

  33. [41]

    Zhou, K.; Zhang, B.; jiapeng wang; Chen, Z.; Zhao, X.; Sha, J.; Sheng, Z.; Wang, S.; and Wen, J.-R. 2024. JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.