Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Synthetic pretraining data quality is driven by prompt design, not generator scale.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 16:14 UTC pith:LTY3KIMI

load-bearing objection First controlled three-axis ablation of synthetic pretraining data with real findings and an overreaching abstract; deserves refereeing after fixing aggregation, hedging, and ideally a scale check. the 3 major comments →

arxiv 2604.13977 v2 pith:LTY3KIMI submitted 2026-04-15 cs.CL cs.AIcs.LG

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Joel Niklaus , Atsuki Yamaguchi , Michal \v{S}tef\'anik , Guilherme Penedo , Hynek Kydl\'i\v{c}ek , Elie Bakouch , Lewis Tunstall , Edward Emanuel Beeching
show 4 more authors
Thibaud Frere Colin Raffel Leandro von Werra Thomas Wolf
This is my paper
classification cs.CL cs.AIcs.LG
keywords synthetic pretraining datarephrasingprompt designdata mixinggenerator scaleFinePhraseweb text
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that when rephrasing web text into synthetic pretraining data, the rephrasing prompt is the dominant factor in downstream performance, and that generators beyond 1B parameters add no benefit. It identifies four structured pedagogical formats (math, FAQ, table, tutorial) that consistently beat curated web text and prior synthetic methods. Mixing synthetic data with original web tokens is necessary, and the mix-in dataset matters more than the quality of the source text. These findings are consolidated into FinePhrase, a 486-billion-token dataset that outperforms existing synthetic baselines at up to 30x lower generation cost.

Core claim

The paper's central claim is that rephrasing web text into structured pedagogical formats (math problems, FAQs, tables, tutorials) produces higher-quality pretraining data than both curated web corpora and prior synthetic rephrasing strategies. A controlled study across 1 trillion generated tokens shows that generator scale saturates at roughly 1B parameters: a 1.7B model outperforms a 27B model for most prompts. It also shows that mixing synthetic tokens with original web text is essential, and that a robust mix-in corpus can up-cycle low-quality source text into competitive training tokens. Applying these findings, the authors build FinePhrase, a 486B-token open dataset that beats all exis

What carries the argument

The central object is a set of four 'pedagogical structured' rephrasing prompts (math, faq, table, tutorial) that convert flat web documents into discrete, logical, queryable units. The experimental framework trains a 1.2B-parameter model from scratch on 21B tokens for each configuration and evaluates on 12 English benchmarks. The mechanism that carries the argument is the systematic ablation of prompt design, generator model, and source/mix-in data, which isolates the effect of each design choice.

Load-bearing premise

The ranking of prompts, generators, and mix-ins was measured on a single 1.2B-parameter model trained on 21B tokens and evaluated on 12 English benchmarks; if that ranking changes at larger model scales or on different benchmark distributions, the prescriptive findings and the FinePhrase recipe may not transfer.

What would settle it

Train a larger learner (e.g., 7B parameters) on the same data configurations and check whether the relative ordering of prompts and generator sizes persists; if a 27B generator's data surpasses a 1B generator's data at that scale, the claimed saturation point is scale-dependent and the main conclusion fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Researchers can prioritize prompt engineering over generator scale when synthesizing pretraining data, cutting compute costs substantially.
  • Structured pedagogical formats (math, FAQ, table, tutorial) are a practical recipe for converting noisy web text into high-value training tokens.
  • Synthetic data should always be mixed with original web data; pure synthetic training degrades NLU capabilities and risks model collapse.
  • The choice of mix-in dataset is a critical lever, capable of making low-quality source text competitive.
  • FinePhrase provides an off-the-shelf 486B-token synthetic dataset that matches or exceeds prior synthetic baselines at a fraction of the cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The result suggests that relatively small, instruction-tuned generators, combined with carefully designed prompts, could democratize high-quality synthetic data production for teams without massive compute budgets.
  • The 'template collapse' phenomenon implies that output diversity metrics could serve as a cheap filter or reward signal when selecting generators or sampling strategies.
  • Because the mix-in dataset matters more than source quality, data-scarce domains might be improved by pairing rephrased low-resource text with a strong general-purpose mix-in corpus.
  • The saturation at 1B parameters hints at a potential trade-off: extremely large generators may over-regularize output, and future work could test whether diversity-encouraging decoding further narrows the gap between small and large generators.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper presents a systematic empirical study of synthetic pretraining data generation from web text. The authors vary rephrasing prompts (eight existing and four new 'pedagogical' formats), generator models (six families, 135M–27B parameters), and source/mix-in corpora, training a 1.2B-parameter Qwen-2 model on 21B tokens for each configuration and evaluating on 12 English benchmarks. They report that structured pedagogical prompts (math, FAQ, table, tutorial) consistently beat curated web baselines and prior synthetic methods, that generator scale beyond 1B gives no consistent benefit, and that mix-in selection matters more than source quality. They release FinePhrase, a 486B-token dataset generated with SmolLM2 1.7B, and report a 30× generation cost reduction over prior pipelines. The paper's core contribution is an empirical comparison matrix rather than a new method, with the dataset and code release as practical outcomes.

Significance. The controlled comparison is the paper's main value: it is broader than prior isolated studies, and the release of the dataset, prompts, and pipeline is a substantial community contribution. If the reported rankings survive correction of the aggregate-score inconsistencies, the conclusion that prompt design dominates generator scale would be practically useful. The cost analysis is a strong feature, as is the reproducibility statement with public code, benchmark results, and generation framework. The central claims are conditional on a single small-scale evaluation proxy, which limits their generalizability but does not undermine the value of the dataset itself.

major comments (3)
  1. [§3 Evaluation; §4.1 Tables 4 and 12] The headline macro-averages are not reproducible from the appendix. For the math prompt, Table 12 gives a 12-task mean of 14.64 and a six-category mean of 15.20, but Table 4 reports 15.31; analogous offsets appear for faq (13.91/14.35 vs 14.45), table (14.26/14.74 vs 14.83), and tutorial (13.77/14.16 vs 14.30). Since these aggregates drive the central comparisons and the FinePhrase claims, recompute all aggregates, state the exact averaging scheme, and verify every table entry.
  2. [§3; §4.2; Abstract; Limitations] All conclusions rest on 1.2B-target/21B-token training runs. The paper's own Limitations concede results may vary at larger scales, yet the Abstract and §6 make unqualified claims (generator saturation at 1B, FinePhrase outperforms all baselines). Prior synthetic-data scaling work (Qin et al. 2025; Kang et al. 2025) shows interactions between data composition and target scale/compute, so the saturation point observed here could be an artifact of the proxy's capacity. Please add at least one scale-transfer point (e.g., train a 3–7B model on a few configurations) or restrict the conclusions to the tested scale.
  3. [§6, Figure 5] Figure 5 reports FinePhrase variants outperforming baselines, but it does not state how many tokens of the 486B dataset were used for the 21B-token training runs. If the bars use a small sample of FinePhrase, say so and specify the sample; if the full dataset was used, explain how it fits into a 21B-token budget. Without this, the claim that FinePhrase outperforms all baselines is under-specified.
minor comments (4)
  1. [Appendix B.2] Typo: 'acomplish' should be 'accomplish'.
  2. [Table 9] The SYNTH row is unclear: 'fine-tuned 80B 4K 20M' mixes units. Specify generator size, token volume, and GPU-hours explicitly, consistent with the other rows.
  3. [Figure 4] Many of the point labels overlap, making the Pareto-frontier argument hard to read. Consider using a table or separating the labels.
  4. [§5.2] The template-collapse analysis is based on 1,000 outputs from two models and one prompt. State whether the 68% completion and repetition statistics are representative and report variability or confidence intervals.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims rest on controlled experiments evaluated against external benchmarks, not on a self-referential derivation.

full rationale

The paper's central assertions—that structured pedagogical prompts outperform web baselines and prior synthetic datasets, that generator scale beyond 1B provides no additional benefit, and that mix-in selection matters—are all established by training a fixed 1.2B model on 21B tokens and measuring performance on 12 external English benchmarks. The experimental design isolates each axis (prompt, generator model, source/mix-in data) by holding the others constant, and the comparisons use standard public benchmarks and publicly released datasets (DCLM, Cosmopedia, Nemotron-HQ-Synth, REWIRE, etc.). No parameter is fitted to the claimed outcome; the reported scores are direct measurements. The use of self-developed artifacts such as SmolLM2, FineWeb, DataTrove, Nanotron, and LightEval is a material choice, but the central claims do not depend on these artifacts being correct: the generator-scale finding is observed across both the external Gemma 3 family and SmolLM2, and the mix-in analysis includes the external DCLM corpus. The few self-citations (e.g., Yamaguchi et al. 2026 for pedagogical benefit, Kydlíček et al. 2025 for the evaluation suite) are supporting references rather than load-bearing derivation steps; the paper's own ablations constitute the evidence. The Limitations statement honestly concedes that 'results may vary at larger scales,' which is a scope caveat and not a hidden circular dependency. No equation or claim reduces to its own input by construction, and no fitted parameter is renamed as a prediction. Therefore the derivation chain is self-contained and non-circular.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

No new physical or mathematical entities are postulated. The load-bearing assumptions are methodological: the benchmark suite, the small-scale training proxy, and the acceptance of unfiltered generator outputs as training signal. The 50/50 mix ratio and equal-weight benchmark averaging are hand-set choices that directly shape the conclusions.

free parameters (3)
  • synthetic mix ratio = 50/50
    Chosen by hand in §3; the necessity-of-mixing and mix-in selection conclusions are measured at this ratio and are not swept.
  • training token budget = 21B tokens
    Selected in §3; all performance claims are measured at this budget. The authors note in Limitations that results may vary at larger scales.
  • benchmark macro-average weights = equal weights over 12 tasks (or 6 categories)
    The headline claim of 'consistently outperform' is driven by equal weighting over the chosen benchmark set. NLU tasks where synthetic data underperforms receive equal or lower weight. The weighting is not justified.
axioms (3)
  • domain assumption The 12-benchmark macro-average is a sufficient proxy for general pretraining capability.
    §3 Evaluation; the claim 'consistently outperform' relies on this proxy, and the Limitations acknowledge the results may be benchmark-specific.
  • domain assumption A 1.2B model trained on 21B tokens yields relative rankings of data configurations that transfer to larger scales.
    §3 and Limitations: 'results may vary at larger scales'. This is load-bearing for all prescriptive conclusions about synthetic data at production scale.
  • domain assumption Instruction-tuned generator outputs are usable as pretraining tokens without filtering for hallucination or factuality.
    The pipeline rephrases web text but does not verify factual consistency; the math prompt explicitly creates problems from data, which may invent numerical relationships.

pith-pipeline@v1.3.0-alltime-deepseek · 33465 in / 17012 out tokens · 135948 ms · 2026-08-02T16:14:21.786207+00:00 · methodology

0 comments
read the original abstract

Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop \textbf{\textsc{FinePhrase}}, a 486-billion-token open dataset of rephrased web text. We show that \textsc{FinePhrase} outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community.

Figures

Figures reproduced from arXiv: 2604.13977 by Atsuki Yamaguchi, Colin Raffel, Edward Emanuel Beeching, Elie Bakouch, Guilherme Penedo, Hynek Kydl\'i\v{c}ek, Joel Niklaus, Leandro Von Werra, Lewis Tunstall, Michal \v{S}tef\'anik, Thibaud Frere, Thomas Wolf.

Figure 1
Figure 1. Figure 1: Overview of our experi￾mental methodology. To address the scaling limits of crawlable web data, we systematically evaluate methods for generating synthetic pretraining data by rephrasing web cor￾pora. Unlike prior work evaluating isolated rephras￾ing methodologies (Maini et al., 2024; Su et al., 2025; Nguyen et al., 2025, inter alia.), we conduct a con￾trolled ablation of the key components within the gen￾… view at source ↗
Figure 2
Figure 2. Figure 2: Macro-averaged scores for different Gemma 3 and SmolLM2 model scales. Full results are in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Fine-grained evaluation of pedagogical formats against the DCLM baseline. The [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: GPU cost vs. performance. Symbols distinguish model variants; adjacent charac [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: FINEPHRASE prompts vs. baselines. The best result in each task is marked with ⋆. All FINEPHRASE runs use SmolLM2 1.7B on FineWeb, mixed with FineWeb-HQ. Investing in prompt architecture provides a higher return on compute than simply increasing the parameter count of the generator. 6 FinePhrase We apply our empirical findings to build FINEPHRASE, a large-scale synthetic dataset. The construction of the dat… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

    cs.CL 2026-07 conditional novelty 6.0

    A learned orchestrator builds per-example drop/untouch/clean pipelines over noise pruning and instruction-conditioned rewriting, improving from-scratch and math continued pretraining over fixed curation methods.

Reference graph

Works this paper leans on

20 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Ask diverse questions that require different cognitive skills or cover different aspects of the text.,→

  2. [2]

    Number of minutes = 5

  3. [3]

    SmolLM2 Representative Output

    Total revolutions = 120×5 $$120 \\times 5 = 600$$ The disc makes 600 revolutions in 5 minutes. SmolLM2 Representative Output

  4. [6]

    Blog post

    URL https://huggingface.co/spaces/nanotron/ultrascale-playbook. Blog post. Yudong Wang, Zixuan Fu, Jie Cai, Peijun Tang, Hongya Lyu, Yewei Fang, Zhi Zheng, Jie Zhou, Guoyang Zeng, Chaojun Xiao, Xu Han, and Zhiyuan Liu. Ultra-FineWeb: Efficient data filtering and verification for high-quality LLM training data.arXiv preprint, arXiv:2505.05427, 2025. URLhtt...

    Pith/arXiv arXiv 2025

  5. [7]

    This approach underperforms every rephrasing approach in our analysis

    generates content from scratch rather than rephrasing. This approach underperforms every rephrasing approach in our analysis. EntiGraph (Yang et al., 2025b) targets continued pretraining with entity-centric augmentation; their diversity scaling complements our finding that prompt-level diversity saturates at approximately 20B tokens. Understanding Synthet...

    2025

  6. [9]

    Ask questions in various forms such as: - Yes/No questions that require determining whether a statement is true or false.,→ - Open-ended questions that begin with words like what, how, when, where, why and who.,→ - Multi-choice questions that offers two or more options to choose from. Include the options in the question.,→ - Comparison questions that comp...

  7. [10]

    Focus on asking questions about factual information, important knowledge, or concrete details in the text.,→

  8. [11]

    Write questions and answers using clear and concise language

  9. [12]

    Do not use Markdown

    Use plain text. Do not use Markdown

  10. [13]

    Question:

    Each question and answer pair should be on a separate line. Tag the question with "Question:" and the answer with "Answer:".,→ Text: [TEXT] Task: 20 Preprint. Under review. After reading the above text, ask up to 8 questions and provide the correct answers following the instructions. Give your response in this format:,→ Here are the questions and answers ...

  11. [14]

    Revolutions per minute = 120

  12. [19]

    Paul R. Williams: Classic Hollywood Style,

    Noninvasive Monitoring Techniques: Other alternatives include the LiDCOplus System for pulsed waveforms for monitoring and Arterial Pulse Contour Analysis for heart function and blood flow measurement. ,→ ,→ Answer: The NICOM, NICO2, and Respironic systems are noninvasive methods for monitoring cardiac output. The NICOM uses BIOREACTANCE technology, the N...

    arXiv 2013

  13. [20]

    It provides real-time data in seconds and can be done painlessly on the patient

    Continuous Wave Doppler Monitor (USCOM): The monitor uses a handheld probe that can measure stroke volume and fluid input to avoid overloading patients and diagnose problems earlier. It provides real-time data in seconds and can be done painlessly on the patient. ,→ ,→ ,→

  14. [21]

    Noninvasive Cardiac Monitoring: This includes monitoring cardiac output, stroke volume, and fluid input using tools like the USCOM Noninvasive Cardiac Output Monitor. ,→ ,→

  15. [22]

    ,→ ,→ Answer: The NICOM, NICO2, and Respironic systems are noninvasive methods for monitoring cardiac output

    Noninvasive Monitoring Techniques: Other alternatives include the LiDCOplus System for pulsed waveforms for monitoring and Arterial Pulse Contour Analysis for heart function and blood flow measurement. ,→ ,→ Answer: The NICOM, NICO2, and Respironic systems are noninvasive methods for monitoring cardiac output. The NICOM uses BIOREACTANCE technology, the N...

  16. [102]

    Ilya Loshchilov and Frank Hutter

    URLhttps://aclanthology.org/2021.acl-long.102/. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InProceedings of the Seventh International Conference on Learning Representations, 2019. URL https:// openreview.net/forum?id=Bkg6RiCqY7. Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephras...

    2021

  17. [262]

    URLhttps://aclanthology.org/2025.naacl-long.262/. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël...

    Pith/arXiv arXiv 2025

  18. [2021]

    doi: 10.1145/3474381

    ISSN 0001-0782. doi: 10.1145/3474381. URLhttps://doi.org/10.1145/3474381. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. AI models collapse when trained on recursively generated data.Nature, 631: 755–759, 2024. doi: 10.1038/s41586-024-07566-y. URL https://doi.org/10.1038/s41586- 024-07566-y. Varun Singh, Luca...

    arXiv 2024

  19. [2024]

    doi: https://doi.org/10.1016/j.neucom.2023.127063

    ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2023.127063. URL https: //www.sciencedirect.com/science/article/pii/S0925231223011864. Nouamane Tazi, Ferdinand Mom, Haojun Zhao, Phuc Nguyen, Mohamed Mekkouri, Lean- dro Werra, and Thomas Wolf. The ultra-scale playbook: Training LLMs on GPU clusters,

    arXiv 2023

  20. [2025]

    URLhttps://openreview.net/forum?id=lkjhBdz3rn. NVIDIA, Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khat- tar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, Aleksandr Shaposhnikov, Alex Kondratenko, Alexander Bukharin, Alexandre Milesi, Ali Taghibakhshi, Alisa Liu, Amelia Barton, Ameya Sun...

    Pith/arXiv arXiv 2025