Pith. sign in

REVIEW 5 major objections 4 minor 47 references

LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that graph-linked knowledge points turn a small QA seed set into a 50B-token corpus that lifts MMLU and CMMLU by 11.51% when used for continual pre-training.

desk verdict The abstract promises a practically important 11.51% MMLU/CMMLU gain from 50B synthetic QA tokens, but with the full text unreadable and no contamination controls, the number remains an unverified assertion. read the letter →

arxiv 2508.01317 v2 pith:YWS35UJN submitted 2025-08-02 cs.CL

Xuemiao Zhang , Can Ren , Chengying Tu , Rongxiang Weng , Hongfei Yan , Jingang Wang , Xunliang Cai This is my paper
classification cs.CL
keywords knowledgepointgraphsyntheticdataquestionansweringcontinualpre-trainingMMLUCMMLUsynthesislargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes LinkSyn, a pipeline that extracts knowledge points from question-answer seed pairs, links those points into a graph, and walks the graph to feed several logically connected seeds to a synthesis model that writes new questions and answers. The result is LinkQA, a 50-billion-token, multi-disciplinary QA dataset. The paper claims that continually pre-training Llama-3 8B on LinkQA improves average MMLU and CMMLU accuracy by 11.51%, setting a new state of the art. If that holds, synthetic data generated from a small seed set can substitute for much larger human-written corpora in knowledge-focused continual pre-training.

What carries the argument

The central object is the knowledge point (KP) graph: nodes are atomic knowledge points extracted from seed QA data, and edges connect knowledge points that are strongly associated or logically related. Graph walks over this KP graph produce sequences of multiple linked seeds, and a knowledge distribution value function adjusts the path sampling probability to balance coverage of rare knowledge points against popularity. These multi-seed paths are then used to prompt DeepSeek-R1 for diffusion-based synthesis, with a difficulty-adjustment component for high-difficulty QA within a chosen discipline.

What would settle it

Continually pre-train Llama-3 8B on a 50B-token control corpus generated from the same seeds sampled uniformly at random instead of via graph walks; if the control matches the 11.51% average MMLU and CMMLU gain, the graph linking is not what drives the result. Alternatively, have annotators audit a random sample of LinkQA for factual errors and exact or near-duplicate questions; high error or duplication rates would indicate the gains come from style or benchmark overlap rather than new knowledge.

Watch

Extended reading notes

Core claim

The central claim is that the organizing structure of synthetic training data matters as much as its volume. LinkSyn represents each seed QA item as a set of knowledge points, builds a knowledge-point graph whose edges encode strong associations, and uses a knowledge distribution value function to bias graph walks toward a balance between covering rare knowledge points and repeating popular ones. Each graph walk yields a path of multiple seeds with dense logical associations, which is given to DeepSeek-R1 to synthesize a new question-answer pair; a separate module raises difficulty by flexibly adjusting question hardness within a discipline. The authors report that the resulting 50B-token LinkQA corpus, used for continual pre-training of Llama-3 8B, yields an average improvement of 11.51% on MMLU and CMMLU, establishing state-of-the-art results, and that LinkQA consistently enhances performance across model sizes and initial FLOPs scales.

Load-bearing premise

The whole method rests on the assumption that the knowledge points extracted from seed QA and the edges drawn between them are accurate enough that walking the graph leads DeepSeek-R1 to write useful, correct, non-repetitive questions rather than plausible-looking ones.

Editorial extensions

If this is right

  • A small seed corpus of QA pairs can be expanded into a 50B-token training corpus, so data-hungry continual pre-training no longer requires collecting human-written questions at that scale.
  • Because the value function exposes controls over discipline and difficulty, practitioners can steer synthetic data toward weak subjects or harder questions without regenerating the whole corpus.
  • The reported gains on MMLU and CMMLU imply that graph-linked multi-seed prompts transfer to general knowledge benchmarks, not just to the distribution of the original seeds.
  • If the result holds across model sizes and FLOPs budgets, the method is a recipe for upgrading a base model's knowledge with synthetic QA at any scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 11.51% figure is measured on MMLU and CMMLU, which are themselves question-answering benchmarks; a harder test would be whether LinkQA improves open-ended generation or multi-step reasoning tasks that do not resemble the training format.
  • Because DeepSeek-R1 is the generator, LinkQA inherits both the knowledge and the blind spots of that model; rerunning the same graph walks with a different generator and checking whether the benchmark gain persists would separate the graph's contribution from the generator's.
  • A direct ablation would synthesize a matched 50B-token control corpus from the same seeds sampled independently rather than along graph walks; if the control matches the 11.51% gain, the knowledge-point linking is not the active ingredient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The manuscript proposes LinkSyn, a knowledge-point (KP) graph-based framework that extracts KPs from seed QA data, builds a KP graph, and samples multi-seed paths via graph walks; DeepSeek-R1 then synthesizes new QA items conditioned on the path. The authors report constructing LinkQA, a 50B-token multilingual QA dataset, and claim that continual pretraining of Llama-3 8B on LinkQA improves MMLU and CMMLU by an average of 11.51%, establishing state-of-the-art results. The submitted full text is almost entirely corrupted by an encoding error, so beyond the abstract, the method, the experiments, and the results cannot be verified from the manuscript.

Significance. If the claimed 11.51% improvement is real and attributable to the KP-graph curriculum, LinkSyn would be a practically significant contribution to synthetic data generation for continual pretraining, with useful controls over discipline and difficulty. However, the paper as submitted provides no verifiable experimental protocol: no baselines, ablations, variance, contamination checks, or dataset-quality audits, and the full text is unreadable. The central claim is therefore currently a single unvalidated number.

major comments (5)
  1. [Full text (all sections)] The full text is encoded as mojibake (e.g., '������� ������������ ������� ��'), making every section, equation, figure, and table unreadable. Because the central 11.51% claim, the LinkSyn algorithm, and the experimental controls are presented only in this corrupted text, the manuscript cannot be assessed in its current form. The authors must resubmit a properly rendered version; no technical evaluation is possible otherwise.
  2. [Abstract] The headline result, 'average improvement of 11.51% on MMLU and CMMLU,' is reported without any experimental protocol: the abstract gives no baseline data mixture, training token budget for the baseline versus LinkQA, number of runs, standard deviation, evaluation harness, or comparison to prior SOTA. Without a matched baseline trained on the same compute and on an alternative data source, the improvement cannot be attributed to LinkSyn. Please report these details and the per-benchmark breakdown.
  3. [Abstract / Method (unreadable)] The paper does not address the possibility that LinkQA overlaps with MMLU/CMMLU test items or near-paraphrases, nor does it evaluate the factual correctness of DeepSeek-R1-generated questions. If the synthesized data contains evaluation items, the reported gain could reflect memorization rather than knowledge-curriculum value. Please report (a) n-gram or embedding overlap between LinkQA and the evaluation benchmarks, (b) a sample-based correctness and diversity audit of generated QA, and (c) a control in which the question-answer mapping is shuffled or the question order is reversed while keeping the same tokens. The shuffled-answer control is the minimal experiment needed to separate content-driven gains from distributional artifacts.
  4. [Method (KP extraction and graph)] The claimed mechanism depends on accurate knowledge-point extraction and on graph edges that encode genuine logical associations. No evaluation of KP extraction accuracy, edge precision, or graph-walk path quality is reported. Without these measurements, or an ablation replacing the graph with random seed sampling, the paper has not shown that the KP graph, rather than the mere volume of synthetic data, causes the improvement.
  5. [Method (value function and difficulty)] The knowledge distribution value function and difficulty adjustment introduce several free parameters (coverage-popularity tradeoff, path length, difficulty weights), yet no sensitivity analysis or ablation isolating each of the three claimed components is visible in the readable portions. Since the abstract explicitly lists these components as contributions, the paper should show the marginal effect of each on the final benchmark scores.
minor comments (4)
  1. [Abstract] The phrase 'establishing new SOTA results' is unsupported without citing the best prior numbers for the same base model and compute setting; please name the prior SOTA and the margin of improvement.
  2. [Abstract] The phrase 'average improvement' is ambiguous; please specify whether it is a relative or absolute percentage-point improvement and over which benchmarks the average is taken.
  3. [Dataset description] The dataset is described as 50B tokens, but no release link, format, license, or duplicate-analysis is provided; if the dataset will be released, state the URL and license.
  4. [Experimental tables] Several table-like structures in the full text are corrupted (e.g., the rows labeled 'Humanities', 'STEM', and similar), so the per-discipline results and the claim of 'consistent enhancement across model size and initial FLOPs scales' cannot be checked.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity: the benchmark gain is an external empirical measurement, and the sampled distribution is defined over internal KP coverage/popularity, not fitted to MMLU/CMMLU.

full rationale

The paper's claimed derivation chain is: extract knowledge points from QA seeds; build a KP graph; sample graph walks under a knowledge-distribution value function that balances coverage and popularity; synthesize new QA with DeepSeek-R1; continual pre-train Llama-3 on the resulting LinkQA; and measure MMLU/CMMLU. The value function adjusts path sampling using coverage and popularity, which are defined over the extracted KP graph and seed QA, not over the evaluation benchmarks. The provided text does not report optimizing the value function or difficulty controls against MMLU/CMMLU scores, and no equation or passage equates the synthesized data distribution to the evaluation distribution. The absence of seed-source disclosure and contamination analysis is a missing-evidence concern, not a circular reduction: no quoted passage shows that the benchmark items or their near-paraphrases are inputs to LinkSyn. No load-bearing self-citation chain is identifiable in the readable text. Therefore no circular step meets the required standard of exhibiting a specific reduction by construction.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central empirical claim depends on unreported design choices (value function weights, walk lengths, difficulty controls, decoding settings) and on unvalidated assumptions about KP extraction accuracy, generated-question correctness, and benchmark integrity.

free parameters (4)
  • Coverage-popularity tradeoff weight in knowledge distribution value function
    Guides graph-walk sampling; its value affects KP coverage vs popularity and hence dataset composition; not reported in abstract.
  • Graph walk path length
    Determines how many linked KPs are merged per generated QA; not reported in abstract.
  • Difficulty adjustment parameters
    Used for high-difficulty QA enhancement per discipline; thresholds or strengths not reported in abstract.
  • DeepSeek-R1 decoding parameters
    Generation temperature and sampling settings shape diversity and quality; not reported in abstract.
assumptions (3)
  • domain assumption Knowledge points extracted from seed QA and their graph edges represent a valid knowledge structure for synthesis.
    The entire graph-walk synthesis depends on KP extraction quality and edge correctness; no validation is described in the abstract.
  • domain assumption DeepSeek-R1-generated QA from multi-seed paths is factually correct, diverse, and free of harmful repetition at 50B scale.
    Synthesis quality is the direct input to continual pretraining; the abstract offers no human or automated quality audit.
  • domain assumption MMLU and CMMLU gains reflect general knowledge improvement rather than benchmark overlap or token-count effects.
    The abstract reports only average improvement and SOTA, with no contamination analysis or controlled token-budget comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points." pith.science (2026). https://pith.science/paper/YWS35UJN

@misc{pith2026250801317,
  author       = {Pith},
  title        = {Pith review of: LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YWS35UJN}},
  note         = {Machine review of arXiv:2508.01317}
}
abstract

The advancement of large language models (LLMs) struggles with the scarcity of high-quality, diverse training data. To address this limitation, we propose LinkSyn, a novel knowledge point (KP) graph-based synthesis framework that enables flexible control over discipline and difficulty distributions while balancing KP coverage and popularity. LinkSyn extracts KPs from question-answering (QA) seed data and constructs a KP graph to synthesize diverse QA data from multiple seeds strongly linked by KPs and sampled from graph walks. Specifically, LinkSyn incorporates (1) a knowledge distribution value function to guide the adjustment of path sampling probability and balance KP coverage and popularity during graph walks; (2) diffusion-based synthesis via DeepSeek-R1 by leveraging multiple seeds with dense logical associations along each path; and (3) high-difficulty QA enhancement within given disciplines by flexible difficulty adjustments. By executing LinkSyn, we synthesize LinkQA, a diverse multi-disciplinary QA dataset with 50B tokens. Extensive experiments on Llama-3 8B demonstrate that continual pre-training with LinkQA yields an average improvement of $\mathbf{11.51\%}$ on MMLU and CMMLU, establishing new SOTA results. LinkQA consistently enhances performance across model size and initial FLOPs scales.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 11 canonical work pages

  1. [1]

    N.; Prabhumoye, S.; Kamalu, J.; Satheesh, S.; Nyberg, E.; Patwary, M.; Shoeybi, M.; and Catanzaro, B

    Akter, S. N.; Prabhumoye, S.; Kamalu, J.; Satheesh, S.; Nyberg, E.; Patwary, M.; Shoeybi, M.; and Catanzaro, B. 2025. MIND : Math Informed syNthetic Dialogues for Pretraining LLM s. In The Thirteenth International Conference on Learning Representations

    2025

  2. [2]

    Chen, J.; Chen, Z.; Wang, J.; Zhou, K.; Zhu, Y.; Jiang, J.; Min, Y.; Zhao, X.; Dou, Z.; Mao, J.; Lin, Y.; Song, R.; Xu, J.; Chen, X.; Yan, R.; Wei, Z.; Hu, D.; Huang, W.; and Wen, J.-R. 2025. Towards Effective and Efficient Continual Pre-training of Large Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lingu...

    2025

  3. [3]

    Cheng, D.; Gu, Y.; Huang, S.; Bi, J.; Huang, M.; and Wei, F. 2024. Instruction Pre-Training: Language Models are Supervised Multitask Learners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2529--2550

    2024

  4. [4]

    Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. CoRR, abs/1803.05457

    arXiv 2018

  5. [5]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168

    arXiv 2021

  6. [6]

    DeepSeek-AI; Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; Zhang, X.; Yu, X.; Wu, Y.; Wu, Z. F.; Gou, Z.; Shao, Z.; Li, Z.; Gao, Z.; Liu, A.; Xue, B.; Wang, B.; Wu, B.; Feng, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; Dai, D.; Chen, D.; Ji, D.; Li, E.; Lin, F.; Dai, F.; Luo, F.; Hao, G.; Chen, G.; ...

    arXiv 2025

  7. [7]

    DeepSeek-AI; Liu, A.; Feng, B.; Xue, B.; Wang, B.; Wu, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; Dai, D.; Guo, D.; Yang, D.; Chen, D.; Ji, D.; Li, E.; Lin, F.; Dai, F.; Luo, F.; Hao, G.; Chen, G.; Li, G.; Zhang, H.; Bao, H.; Xu, H.; Wang, H.; Zhang, H.; Ding, H.; Xin, H.; Gao, H.; Li, H.; Qu, H.; Cai, J. L.; Liang, J.; Guo, J.; Ni, J.; Li, J.; ...

    arXiv 2024

  8. [8]

    Du, Z.; Zeng, A.; Dong, Y.; and Tang, J. 2024. Understanding Emergent Abilities of Language Models from the Loss Perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

Show all 47 references
  1. [9]

    Dua, D.; Wang, Y.; Dasigi, P.; Stanovsky, G.; Singh, S.; and Gardner, M. 2019. DROP : A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Lingu...

    2019

  2. [10]

    Duan, F.; Zhang, X.; Wang, S.; Que, H.; Liu, Y.; Rong, W.; and Cai, X. 2025. Enhancing llms via high-knowledge data selection. In Proceedings of the AAAI Conference on Artificial Intelligence, 23832--23840

    2025

  3. [11]

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; Goyal, A.; Hartshorn, A.; Yang, A.; Mitra, A.; Sravankumar, A.; Korenev, A.; Hinsvark, A.; Rao, A.; Zhang, A.; Rodriguez, A.; Gregerson, A.; Spataru, A.; Roziè...

    2024 arXiv

  4. [12]

    Gunasekar, S.; Zhang, Y.; Aneja, J.; Mendes, C. C. T.; Giorno, A. D.; Gopi, S.; Javaheripi, M.; Kauffmann, P.; de Rosa, G.; Saarikivi, O.; Salim, A.; Shah, S.; Behl, H. S.; Wang, X.; Bubeck, S.; Eldan, R.; Kalai, A. T.; Lee, Y. T.; and Li, Y. 2023. Textbooks Are All You Need. ...

    2023 arXiv

  5. [13]

    Hao, F.; Gong, Y.; Yu, W.; and Loia, V. 2022. Knowledge points navigation based on three-way concept lattice for autonomous learning. Pattern Recognition Letters, 163: 96--103

    2022

  6. [14]

    Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021 a . Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations

    2021

  7. [15]

    Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021 b . Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations

    2021

  8. [16]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021 c . Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

    2021

  9. [17]

    W.; Chen, Z.; and Zhang, X

    Huang, B.; Wen, Y.; Zhao, Y.; Hu, Y.; Liu, Y.; Jia, F.; Mao, W.; Wang, T.; Zhang, C.; Chen, C. W.; Chen, Z.; and Zhang, X. 2024. SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Control. arXiv:2403.19438

    2024 arXiv

  10. [18]

    Huang, L.; Yu, W.; Ma, W.; Zhong, W.; Feng, Z.; Wang, H.; Chen, Q.; Peng, W.; Feng, X.; Qin, B.; and Liu, T. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems, 1–55

    2025

  11. [19]

    Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; jiayi lei; Fu, Y.; Sun, M.; and He, J. 2023. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. In Thirty-seventh Conference on Neural Information Process...

    2023

  12. [20]

    Jiang, J.; Li, J.; Zhao, X.; Song, Y.; Zhang, T.; and Wen, J.-R. 2025 a . Mix- CPT : A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment. In The Thirteenth International Conference on Learning Representations

    2025

  13. [21]

    Jiang, X.; Ma, S.; Xu, C.; Yang, C.; Zhang, L.; and Guo, J. 2025 b . Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models. arXiv:2505.00979

    2025 arXiv

  14. [22]

    Kandpal, N.; Deng, H.; Roberts, A.; Wallace, E.; and Raffel, C. 2023. Large Language Models Struggle to Learn Long-Tail Knowledge. arXiv:2211.08411

    2023 arXiv

  15. [23]

    Li, H.; Zhang, Y.; Koto, F.; Yang, Y.; Zhao, H.; Gong, Y.; Duan, N.; and Baldwin, T. 2024. CMMLU : Measuring massive multitask language understanding in C hinese. In Findings of the Association for Computational Linguistics: ACL 2024, 11260--11285

    2024

  16. [24]

    D.; Gunasekar, S.; and Lee, Y

    Li, Y.; Bubeck, S.; Eldan, R.; Giorno, A. D.; Gunasekar, S.; and Lee, Y. T. 2023. Textbooks Are All You Need II: phi-1.5 technical report. arXiv:2309.05463

    2023 arXiv

  17. [25]

    Maini, P.; Seto, S.; Bai, R.; Grangier, D.; Zhang, Y.; and Jaitly, N. 2024. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14044--14072

    2024

  18. [26]

    M.; Barak, B.; Scao, T

    Muennighoff, N.; Rush, A. M.; Barak, B.; Scao, T. L.; Piktus, A.; Tazi, N.; Pyysalo, S.; Wolf, T.; and Raffel, C. 2025. Scaling Data-Constrained Language Models. arXiv:2305.16264

    2025 arXiv

  19. [27]

    Nadăș, M.; Dioșan, L.; and Tomescu, A. 2025. Synthetic Data Generation Using Large Language Models: Advances in Text and Code. IEEE Access, 1–1

    2025

  20. [28]

    Narayanan, D.; Shoeybi, M.; Casper, J.; LeGresley, P.; Patwary, M.; Korthikanti, V.; Vainbrand, D.; Kashinkunti, P.; Bernauer, J.; Catanzaro, B.; Phanishayee, A.; and Zaharia, M. 2021. Efficient large-scale language model training on GPU clusters using megatron-LM. In Proceedi...

    2021

  21. [29]

    H.; Constant, N.; Ma, J.; Hall, K

    Ni, J.; Ábrego, G. H.; Constant, N.; Ma, J.; Hall, K. B.; Cer, D.; and Yang, Y. 2021. Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models. arXiv preprint arXiv:2108.08877

    2021 arXiv

  22. [30]

    B.; Lozhkov, A.; Mitchell, M.; Raffel, C.; Werra, L

    Penedo, G.; Kydl \' c ek, H.; allal, L. B.; Lozhkov, A.; Mitchell, M.; Raffel, C.; Werra, L. V.; and Wolf, T. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and B...

    2024

  23. [31]

    H.; Fung, Y

    Qin, Z.; Dong, Q.; Zhang, X.; Dong, L.; Huang, X.; Yang, Z.; Khademi, M.; Zhang, D.; Awadalla, H. H.; Fung, Y. R.; Chen, W.; Cheng, M.; and Wei, F. 2025. Scaling Laws of Synthetic Data for Language Models. arXiv:2503.19551

    2025

  24. [32]

    Qwen; :; Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao, K.; Yang, K.; Yu, L.; Li, M.; Xue, M.; Zhang, P.; Zhu, Q.; Men, R.; Lin,...

    2025 arXiv

  25. [33]

    L.; Bhagavatula, C.; and Choi, Y

    Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9): 99--106

    2021

  26. [34]

    K.; Wu, Y.; and Guo, D

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300

    2024 arXiv

  27. [35]

    Su, D.; Kong, K.; Lin, Y.; Jennings, J.; Norick, B.; Kliegl, M.; Patwary, M.; Shoeybi, M.; and Catanzaro, B. 2025. Nemotron- CC : Transforming C ommon C rawl into a Refined Long-Horizon Pretraining Dataset. In Che, W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds., Proce...

    2025

  28. [36]

    W.; Chowdhery, A.; Le, Q.; Chi, E.; Zhou, D.; and Wei, J

    Suzgun, M.; Scales, N.; Sch \"a rli, N.; Gehrmann, S.; Tay, Y.; Chung, H. W.; Chowdhery, A.; Le, Q.; Chi, E.; Zhou, D.; and Wei, J. 2023. Challenging BIG -Bench Tasks and Whether Chain-of-Thought Can Solve Them. In Findings of the Association for Computational Linguistics: ACL...

    2023

  29. [37]

    Tong, Y.; Zhang, X.; Wang, R.; Wu, R.; and He, J. 2024. DART -Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  30. [38]

    Villalobos, P.; Ho, A.; Sevilla, J.; Besiroglu, T.; Heim, L.; and Hobbhahn, M. 2024. Will we run out of data? Limits of LLM scaling based on human-generated data. arXiv:2211.04325

    2024 arXiv

  31. [39]

    Wang, Y.; Ma, X.; Zhang, G.; Ni, Y.; Chandra, A.; Guo, S.; Ren, W.; Arulraj, A.; He, X.; Jiang, Z.; Li, T.; Ku, M.; Wang, K.; Zhuang, A.; Fan, R.; Yue, X.; and Chen, W. 2024. MMLU -Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. In The Thirty-ei...

    2024

  32. [40]

    Wang, Z.; Zhou, F.; Li, X.; and Liu, P. 2025. OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling. arXiv:2506.20512

    2025 arXiv

  33. [41]

    Wettig, A.; Gupta, A.; Malik, S.; and Chen, D. 2024. QuRating: Selecting High-Quality Data for Training Language Models. In International Conference on Machine Learning, 52915--52971

    2024

  34. [42]

    Yang, Z.; Band, N.; Li, S.; Candes, E.; and Hashimoto, T. 2025. Synthetic continued pretraining. In The Thirteenth International Conference on Learning Representations

    2025

  35. [43]

    Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. H ella S wag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4791--4800

    2019

  36. [44]

    Zhou, F.; Wang, Z.; Ranjan, N.; Cheng, Z.; Tang, L.; He, G.; Liu, Z.; and Xing, E. P. 2025. MegaMath: Pushing the Limits of Open Math Corpora. arXiv:2504.02807

    2025 arXiv

  37. [45]

    Zhou, K.; Zhang, B.; jiapeng wang; Chen, Z.; Zhao, X.; Sha, J.; Sheng, Z.; Wang, S.; and Wen, J.-R. 2024. JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  38. [46]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.