Pith. sign in

REVIEW 3 major objections 5 minor 32 references

Organizing synthetic textbooks as coherent books—not just rewriting the same content—improves mid-training by about one point on a 28-benchmark mean.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 17:35 UTC pith:LYZ7YVAN

load-bearing objection Clean controlled evidence that book packaging and planned adjacency matter for synthetic textbooks—real contribution, single-run ~1-pt gaps are the main caveat. the 3 major comments →

arxiv 2607.28109 v1 pith:LYZ7YVAN submitted 2026-07-30 cs.AI

Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training

Jiawen Tao , Miao Peng , Yaoming Li , Xiaokun Yuan , Mengzhou Wu , Wenhan Yu , Guoan Wang , Nuo Chen
show 2 more authors
Tong Yang Maxm Pan
This is my paper
classification cs.AI
keywords synthetic textbooksmid-trainingbook-level organizationretrieval-grounded generationTOC planningdocument boundariesdata curationlanguage model pre-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the value of synthetic textbook data is not only what is written, but how related material is packaged into book-length documents. The authors build a retrieval-grounded pipeline that clusters corpus passages by topic, plans hierarchical tables of contents, and writes source-grounded sections into complete books—686K textbooks and 32B tokens across more than 15,000 disciplines. Replacing the natural-book slice of a mid-training mix with this corpus raises average downstream scores by +1.09. Matched controls then show why: keeping the same text but splitting each section into its own document costs about a point; randomly concatenating sections to match book length does not recover the gain; and independently rephrasing the same retrieved pool without clustering or TOC planning also falls short. The same Full-over-RandomConcat and Full-over-Natural ordering appears on Llama3-8B, so the claim is that book-level organization is a real design axis for synthetic pre-training data.

Core claim

Book-level organization adds value beyond generated content, document length, and local rewriting. Holding text and tokens fixed, training on complete TOC-planned books (Full) beats treating each section as an independent document (Split) by +1.02 mean; a length-matched scramble of sections from different books (RandomConcat) stays near Split and below Full; and a retrieval-pool-matched per-document rewrite (Rephrase) trails Full by +1.17. Replacing natural books with Full yields +1.09 overall, with transfer to Llama3-8B.

What carries the argument

The Full setting: a five-stage pipeline that retrieves corpus material, clusters it into topical units, plans hierarchical audience×style TOCs with a quality gate, generates source-grounded sections, and assembles them into complete training documents—so planned adjacent sections share continuous positions and intra-document attention.

Load-bearing premise

That one mid-training run per condition is enough to credit roughly one-point mean gaps to book organization rather than ordinary training-run noise, which the paper does not measure.

What would settle it

Retrain Full, Split, RandomConcat, and Rephrase several times with different seeds under the same mix and show that the ~1-point Full advantage over Split/RandomConcat/Rephrase disappears or reverses once run variance is estimated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Synthetic data design should treat document packaging and planned section order as first-class knobs, not only style or factual content.
  • Mid-training book slices can be improved by replacing natural books with TOC-planned, retrieval-grounded textbooks at fixed token budget.
  • Length-matched packing of unrelated sections is not a substitute for keeping planned book boundaries during training.
  • Local rewriting of retrieved documents is insufficient; clustering plus hierarchical TOC planning is needed for the reported gains.
  • The organization effect is expected to transfer across architectures when content and length controls are held fixed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If planned adjacency is the active ingredient, other long-form synthetic genres (lab manuals, legal treatises, codebases) may gain from the same cluster–TOC–assemble pattern.
  • Sequence-packing research that only concatenates related web docs may understate the benefit of generating the long document under one global plan rather than packing after the fact.
  • A cheap practical test for other labs: keep generation fixed and only toggle whether sections are one document or many—if the gap vanishes, packaging was not the driver in that stack.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that book-level organization is a distinct, useful axis for synthetic textbook data in mid-training, beyond content quality or local rewriting. It presents a retrieval-grounded pipeline (taxonomy-guided retrieval, clustering, hierarchical TOC planning with a quality gate, and source-grounded section assembly) that yields 686K textbooks (32B tokens) across 15,000+ disciplines. In a fixed mid-training mix, replacing natural books with this Full corpus improves a 28-benchmark mean by +1.09. Matched controls isolate packaging (content-identical Split: Full +1.02), length (RandomConcat remains below Full), and structured synthesis (retrieval-pool-matched Rephrase: Full +1.17). The Full > RandomConcat > Natural Books ordering also holds on Llama3-8B. Component ablations attribute complementary roles to hierarchical generation, retrieval grounding, and cluster-informed TOCs.

Significance. If the result holds, the paper cleanly elevates document organization from an incidental packaging choice to a first-class design variable for synthetic pre-training data, with practical implications for mid-training mixtures. Strengths include carefully constructed content-, length-, and retrieval-pool-matched controls (Fig. 2; §4–5), transfer across a 3B-active MoE and Llama3-8B, decontamination reporting (~0.07% flagged tokens), fixed mixture budgets and intra-document masking, and planned release of synthesis/control code plus a research-licensed corpus subset. The control taxonomy (Full/Split/RandomConcat/Rephrase) is a reusable experimental template for future data-design work.

major comments (3)
  1. [§5 Table 3; §8; Appendix G] §5 Table 3 and §8 / Appendix G: each condition is a single mid-training run with fixed order and optimizer; training-run variance is explicitly unmeasured, and bootstrap/sign tests only resample benchmarks. Reported overall gaps are ~1 point (Full−Split +1.02, Full−RandomConcat ~+1.01, Full−Rephrase +1.17), while individual swings are large (e.g., GSM8K 88.02 vs 81.64). Without at least a second seed, multi-checkpoint comparison, or a quantitative noise bound, causal attribution of these gaps to book organization versus run noise remains incompletely secured. This is the main load-bearing soft spot for the central claim; either limited replication or a stronger, explicit sensitivity analysis should be added before treating the deltas as definitive.
  2. [§5 Table 3 (Reasoning / GSM8K)] Table 3, Reasoning block: the Full−Split reasoning mean gain (+1.35) is dominated by GSM8K (+6.38); the other four reasoning tasks are mixed (−0.07, +1.23, −0.30, −0.50). Category and overall means are unweighted, so one outlier can materially shape the narrative that packaging helps “reasoning.” Please report leave-one-benchmark-out overall/category deltas (or median deltas) and discuss whether the packaging story remains intact without GSM8K, so the claim is not over-read from a single task.
  3. [§4 (Rephrase); Figure 2] §4 Rephrase control: the comparison is central to “structured synthesis beyond local rewriting,” but the manuscript does not fully specify how Rephrase documents are length-budgeted, packed, and sampled to the same 16B-token book slice, nor how audience×style is applied at document (vs book) granularity while matching the retrieval pool. Small mismatches in effective document-length distribution or style coverage could confound Full’s +1.17. A short protocol paragraph (and, if possible, length-distribution overlay vs Full) would make this control as airtight as Split/RandomConcat.
minor comments (5)
  1. [§4; Figure 3] Figure 3 caption and §4: clarify whether the 50K-token TOC target is training tokens after tokenization of the final English text, and how that maps to the reported character counts in the running example (~257K characters).
  2. [§4 Table 2] Table 2: high-school Wiki pass rate (60.2%) is much lower than other cells; a one-sentence note on whether failed high-school plans are simply dropped (reducing that audience’s share) would help interpret corpus composition.
  3. [§6.2; Appendix H] §6.2 / Table 9: intrinsic ablations use Gemini-3.1-Pro on 100 books; state whether the judge saw condition labels or was blinded, to bound demand-characteristic risk in diagnostic scores.
  4. [§2] Related Work: ACER, LiteLong, and Cosmopedia v2 are well cited; a tighter sentence contrasting “post-hoc long-context packing of related docs” (SPLiCe / In-Context Pretraining) with “generation-time TOC-planned books” would sharpen the positioning already sketched in §2.
  5. [Title page; Abstract] Minor polish: abstract and intro repeat the control deltas almost verbatim; condensing one occurrence would improve flow. Also fix “Maxm Pan” if that is a typographical error in the author list.

Circularity Check

0 steps flagged

No circularity: gains are measured on external benchmarks under matched controls, not defined by the intervention.

full rationale

This is an empirical mid-training study, not a first-principles derivation. The load-bearing claims are comparative downstream means (Full vs Split/RandomConcat/Rephrase/Natural Books; MoE and Llama3-8B) on a fixed 28-benchmark harness. Split holds generated text and tokens fixed; RandomConcat matches length while breaking planned adjacency; Rephrase matches the retrieval pool and audience×style scheme without book assembly. None of these outcomes is algebraically or definitionally forced by the packaging variable: the metric is external task accuracy, not a fitted parameter renamed as a prediction. Component ablations use a separate LLM judge and are explicitly diagnostic. Related-work citations (Cosmopedia, WRAP, packing papers, etc.) are background, not uniqueness theorems or self-citation chains that force the result. Residual concerns (single run per condition, unmeasured training variance) are statistical validity issues, not circularity. Score 0; steps empty.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 2 invented entities

This is an empirical systems/ML paper. The claim rests on standard LM training assumptions, a fixed mid-training mixture design, retrieval/clustering/generation tooling choices, and the premise that single-run benchmark deltas identify the organization factor. No new physical entities; free parameters are engineering knobs fixed in pilots rather than fitted to the downstream claim.

free parameters (6)
  • book slice fraction in mid-training mix = 8%
    Book portion fixed at 8% / 16B tokens (MoE) or 8B tokens (Llama) to match the original mix; not swept.
  • KMeans cluster count K = 40
    Clustering uses K=40 after pilots; affects topical units feeding TOC planning.
  • source chunks per section k = 10
    Section generation conditions on k=10 chunks selected via intrinsic judge curve, not downstream training sweeps.
  • TOC target book length = ~50K tokens
    Planning targets ~50K training tokens per book; realized median ~44.2K.
  • audience × style variant grid = 16 variants
    4 audiences × 4 styles = 16 TOC variants per discipline, chosen as diversity scheme.
  • TOC quality-gate threshold = score≥4 / 91.8% pass
    LLM judge continues only when recommend_continue is true (typically score ≥4); discards 8.2%.
axioms (6)
  • domain assumption Holding optimizer, schedule, batch, tokenizer, and data order fixed while swapping only the book slice isolates the effect of book data organization.
    Stated in §4 Experimental Setup; standard causal assumption in data-ablation LM studies.
  • domain assumption Intra-document attention masking with position resets at document boundaries makes Split vs Full a test of document packaging rather than mere concatenation artifacts.
    §4 Sequence packing; load-bearing for interpreting the +1.02 Full−Split gap.
  • domain assumption A 28-benchmark unweighted mean, with bootstrap over benchmarks and sign tests, is a valid summary of mid-training quality.
    §5 and Appendix G; does not capture training-run variance.
  • ad hoc to paper Keyword retrieval top-100 + embedding KMeans + LLM theme/relevance filtering yields topical units adequate for textbook planning.
    §3.3 pipeline design; methodology claimed retrieval-agnostic but results use this stack.
  • ad hoc to paper LLM judges (DeepSeek for gates; Gemini for intrinsic ablations) are sufficiently reliable diagnostic instruments for TOC/section/book quality.
    §4, §6; authors note ablations are diagnostic, not downstream replacements.
  • domain assumption 13-gram decontamination with mixed Chinese/English tokenization adequately bounds verbatim benchmark leakage as an explanation of gains.
    §4 and Appendix I; ~0.07% tokens flagged; paraphrased leakage not ruled out.
invented entities (2)
  • Full / Split / RandomConcat / Rephrase control taxonomy for synthetic textbooks independent evidence
    purpose: Disentangle document packaging, length, and structured synthesis from generated content.
    Operational experimental constructs, not physical entities; defined in §4 and Figure 2.
  • Four-level 15,387-discipline taxonomy (GB/T 13745 + LLM L4 expansion) no independent evidence
    purpose: Provide generation targets spanning 15k+ domains for scalable book synthesis.
    Constructed artifact used to drive queries; coverage claims depend on this taxonomy.

pith-pipeline@v1.2.0-daily-grok45 · 32053 in / 3756 out tokens · 69941 ms · 2026-07-31T17:35:21.266866+00:00 · methodology

0 comments
read the original abstract

Synthetic textbook data has improved language model pre-training, but prior work largely treats the benefit as a property of generated content or local rewriting style. We study a different factor: whether related content is organized into coherent book-level documents. We contribute both a scalable synthesis pipeline and controlled evidence that this organization matters. The pipeline retrieves source material from a pre-training corpus, clusters it into topical units, plans hierarchical tables of contents, and assembles source-grounded sections into complete books (our Full setting), yielding 686K textbooks (32B tokens) across 15,000+ disciplines. Replacing natural books in a mid-training mix with this corpus improves downstream performance by +1.09 on average. Controlled comparisons then disentangle the relevant design factors. A content-matched Split condition holds generated text and tokens fixed but treats each section as an independent document; Full's +1.02 mean gain isolates document packaging. A length-matched RandomConcat control that joins sections from different books remains below Full, ruling out document length alone. A retrieval-pool-matched Rephrase condition independently rewrites individual retrieved documents under the same audience-by-style scheme, without clustering, TOC planning, or book assembly; Full's +1.17 gain demonstrates the value of structured synthesis. On Llama3-8B, Full likewise outperforms both RandomConcat and Natural Books, supporting book-level organization as a useful axis for synthetic pre-training data design.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 1 linked inside Pith

  1. [1]

    Use at most three hierarchy levels; every node has a title and description

  2. [2]

    Leaf nodes contain only title, description, clusters, and token_count

    Non-leaf nodes contain only title, description, and children. Leaf nodes contain only title, description, clusters, and token_count

  3. [3]

    A leaf without suitable evidence must be merged or removed

    Every leaf must cite at least one supporting cluster and an integer word budget. A leaf without suitable evidence must be merged or removed

  4. [4]

    Prefer not to reuse one cluster more than three times

    Cite at most three clusters per leaf, selected for both topical fit and the target audience. Prefer not to reuse one cluster more than three times. [B. Coverage and Budget]

  5. [5]

    The sum of leaf budgets should generally not exceed {max_total_words}; narrower or weakly supported topics should be shorter

  6. [6]

    Omit weak, incoherent, overly advanced, or overly basic themes when this improves the book structure

    Do not force every cluster into the TOC. Omit weak, incoherent, overly advanced, or overly basic themes when this improves the book structure. [C. Readability and Naming]

  7. [7]

    Make the hierarchy clear and navigable, with consistent local numbering or labels that expose parent-child relationships

  8. [8]

    score”: 1-5 (5 = excellent, 4 = continue, 3 = borderline, 2 = do not continue, 1 = severely unfit), “recommend_continue

    Do not assume images or figures. [Output] Return one JSON tree. Top-level and intermediate nodes use {title, description, children}; leaves use {title, description, clusters, token_count}. The audience/style briefs at this stage control the organization of the book, not merely its eventual prose: for example, the college/textbook profile requests a layere...

  9. [9]

    Write the final content in English. [User]provides the following section identity and structural context, followed by the filled audience/style profile and thek=10sampled source chunks: Discipline path: {taxonomy path} Section position: {full title path from book root to current leaf} Section title: {leaf title} Section overview: {leaf description} [Struc...

  10. [10]

    Use only the strongly relevant parts of the reference materials; discard noise and do not force in irrelevant material

  11. [11]

    Do not merely paraphrase; synthesize and reorganize into a logically rigorous, step-by-step narrative fitting the requested audience and style

  12. [12]

    Preserve/normalize formulas in LaTeX ($...$, $$...$$) and use Markdown tables/lists where helpful; avoid walls of plain text

  13. [13]

    Be faithful to the materials; donotinvent specific numbers, years, names, papers, DOIs, theorem names, or experimental conclusions; correct obvious OCR/transcription errors or omit them

  14. [14]

    reference material

    Produce clean reader-facing prose: never mention “reference material”, “retrieved text”, snippet/source labels, or image-dependent phrasing; no HTML/XML tags

  15. [15]

    Do not repeat the section title (added at assembly) and do not invent global chapter numbering that conflicts with the book

  16. [16]

    Treat the requested length as a target for detail, not a hard lower bound; prioritize density and accuracy over padding

  17. [17]

    Use the supplied TOC context to differentiate this section from adjacent sections, avoid repetition, and add natural transitions where helpful

  18. [19]

    The multilayer perceptron is a foundational type of feedforward neural network architecture

  19. [20]

    20 Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training

    Universal approximation theorems provide the theoretical justification for the representational power of feedforward neural networks. 20 Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training

  20. [21]

    The forward pass in a feedforward network computes activations layer by layer from input to output

  21. [22]

    The backpropagation algorithm efficiently computes gradients of a loss function with respect to all network parameters

  22. [23]

    The chain rule of calculus forms the mathematical foundation for deriving the backpropagation algo- rithm

  23. [24]

    Stochastic gradient descent is commonly used to update weights during the training of multilayer perceptrons

  24. [25]

    Vanishing and exploding gradients are significant challenges in training deep feedforward networks

  25. [26]

    Activation functions introduce non-linearity into neural networks, enabling them to learn complex mappings

  26. [27]

    Feedforward neural networks serve as the backbone for many deep learning models in computer vision and natural language processing

  27. [28]

    Standard feedforward networks are stateless and cannot inherently model sequential dependencies

  28. [29]

    Weight initialization strategies, such as He or Xavier initialization, are critical for effective training of deep feedforward networks

  29. [30]

    Research on feedforward architectures explores alternatives to backpropagation, such as synthetic gradients or layer-wise training

  30. [31]

    The computational graph is a useful abstraction for visualizing and implementing the forward and backward passes in neural networks

  31. [32]

    learned by memory,

    The loss function quantifies the discrepancy between the network’s predictions and the true target values. C.2. Step 1: Retrieval The 14 statements produced 1,400 retrieval results and 1,346 unique documents after deduplication. For example, the universal-approximation query retrieved the following source: Query.Universal approximation theorems provide th...

  32. [2024]

    {current}

    URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 19e4ea30dded58259665db375885e412-Abstract-Datasets_and_Benchmarks_ Track.html. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, et al. Textbooks are all you need II: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is y...

    Pith/arXiv arXiv 2024