Pith. sign in

REVIEW 4 major objections 7 minor 65 references

A small orchestrator can pick a different cleaning pipeline for every pretraining chunk and beat fixed pipelines while spending less compute.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 06:54 UTC pith:OQCB2IIS

load-bearing objection Solid systems paper: learned per-chunk routing over NP/SR/PA with instruction generation beats fixed pipelines and saves cleaning FLOPs; gains are real in direction but thin at small scale and teacher-tied. the 4 major comments →

arxiv 2607.24717 v1 pith:OQCB2IIS submitted 2026-07-27 cs.CL cs.AI

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

Zhen Huang , Yikun Wang , Shijie Xia , Pengfei Liu This is my paper
classification cs.CL cs.AI
keywords pretraining data curationper-example processingorchestrator modelnoise pruningsurface rectificationpedagogical augmentationLLM rewritingcontinued pretraining
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Pretraining data is usually cleaned with one fixed recipe applied to whole corpora or domains. This paper argues that different chunks need different treatment: some should be dropped, some left alone, and some cleaned with a tailored mix of light line pruning, surface rewriting, and educational expansion. DataOrchestra trains a small orchestrator that makes those per-chunk choices and writes concrete rewrite instructions for the tools that actually edit the text. Models trained from scratch on the resulting data, from 0.5B to 7B, improve on average over single methods across eleven benchmarks; the same approach helps math continued pretraining and beats stronger fixed multi-stage and rewrite-and-mix baselines while skipping unnecessary work.

Core claim

Orchestrating an example-specific pipeline—drop, untouch, or a selected sequence of noise pruning, surface rectification, and pedagogical augmentation, each rewrite driven by a chunk-specific instruction—produces pretraining data that yields stable average gains over individual processing methods and fixed multi-stage pipelines, while reducing curation compute by routing chunks only to the stages they need.

What carries the argument

The orchestrator: a fine-tuned small model that, given a chunk, outputs drop/untouch/clean, which of NP/SR/PA to run, and chunk-specific rewrite instructions; trained on plans that a teacher proposes and that are then evolved by executing real tool models and verifier feedback.

Load-bearing premise

The load-bearing premise is that plans evolved from one large teacher and a fixed pair of small tool models generalize beyond that teacher–tool stack and the quality scorer used to build the training set.

What would settle it

Curate the same web corpus with DataOrchestra versus the strongest fixed multi-stage and rewrite-plus-fastText-mix baselines, pretrain matched 0.5B–7B models from scratch on equal token budgets, and check whether the average gain across the eleven reported benchmarks and the claimed FLOP savings disappear when the orchestrator is trained without plan evolution or against a different teacher and tool stack.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Per-example routing can replace one-size-fits-all web cleaning for general pretraining without forcing every document through full LLM rewrite.
  • The same orchestrator pattern transfers to domain continued pretraining (math) when mixed with general data.
  • Leaving high-quality chunks untouched and skipping unused stages cuts curation FLOPs relative to full NP→SR→PA or end-to-end rewrite.
  • Chunk-specific rewrite instructions reduce content loss and factual errors versus generic rewrite prompts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If routing quality is mostly distilled preference from the teacher’s rubric, swapping the teacher or quality scorer could reshuffle which chunks get PA versus untouch without changing the framework.
  • Silent deletion of dense high-value blocks (sequences, tables, code) during surface rewrite is a concrete failure mode that would cap gains on technical domains unless verifiers explicitly protect those spans.
  • A natural next test is whether a frozen orchestrator still helps when the downstream rewrite model is swapped for a different family or size.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes DataOrchestra, a framework in which a small (1.7B) orchestrator model decides, per 1024-token chunk of pretraining data, whether to drop it, leave it untouched, or route it through an ordered subset of three cleaning stages — programmatic noise pruning (NP, 0.6B), surface rectification (SR, 4B), and pedagogical augmentation (PA, 4B) — additionally generating a chunk-specific instruction for each rewriting stage. The orchestrator is trained via SFT on ~300K plans produced by a Qwen3-235B teacher and refined through an execute–verify–retry evolution loop against the actual tool models. From-scratch pretraining at 0.5B/1.5B/7B on four web corpora shows average gains over rule-based, ProX, RePro, and ReWire baselines across 11 benchmarks; the framework also improves math continued pretraining, beats fixed multi-stage and rewrite+mix baselines, and reduces curation compute relative to a full NP→SR→PA pipeline by skipping unnecessary stages.

Significance. If the results hold, the paper makes a practical contribution to pretraining-data curation: it demonstrates that per-example routing across operations of different intervention levels can match or beat uniform multi-stage pipelines at roughly 60% of the curation FLOPs, and that the framework transfers across four web corpora, two math corpora, and three model scales. Specific strengths worth naming: from-scratch multi-scale evaluation (0.5B/1.5B/7B) rather than proxy-signal-only validation; fair baseline retraining (ProX retrained on the same base model and teacher; ReWire given the same 4B rewriter); explicit compute accounting including the orchestrator's own inference cost; disjoint orchestrator-training and pretraining splits; an unusually candid bad-case appendix; and a promised release of code, scripts, and the orchestrator. The margins are modest, so the contribution is an efficiency/robustness one rather than a large quality leap.

major comments (4)
  1. [§4.1–4.2, Table 2, Fig. 3] Every (method, corpus, scale) cell rests on a single pretraining run, and the key margins are small: at 0.5B DataOrchestra beats ProX by +0.90 (RedPajama-V2), +0.52 (DCLM), +0.98 (C4), +0.54 (FineWeb), while losing to ProX on 4 of 11 benchmarks in Table 2 (HellaSwag, WinoGrande, PIQA, SIQA); the average is carried by ARC-E (+2.3) and SciQ (+2.4). Averaging the last three checkpoints (§4.1) reduces eval/checkpoint noise but not seed-level variance in data sampling, stochastic SR/PA decoding (temperature 0.7, Table 7), and training. The sign consistency across 4 corpora and 3 scales and the +1.92 margin at 7B are genuinely mitigating, but the settings are correlated (same orchestrator, teacher, corpus family), so they are not independent replications. Please add 2–3 seeds at 0.5B for the headline comparison (DataOrchestra vs. ProX vs. the strongest fixed pipeline) and report std, or explic
  2. [§5.2, Fig. 6, Appendix F] The fidelity numbers in Fig. 6 (no content loss / no factual error) are produced by the same Qwen3-235B-A22B-Instruct teacher that generated the coarse plans, verified the rewrites during plan evolution, and wrote the corrective instructions (footnote 2, App. F). Judging the outputs with the same model and rubric family that shaped them is a correctness-risk: shared blind spots (e.g., the silent sequence deletion in Fig. 21, which the pipeline itself failed to flag) will be invisible to the judge. The >86% human-agreement check is reassuring but the sample size and protocol are unspecified. Please re-run the fidelity comparison with an independent judge (a different model family and/or a human audit on a stated sample), and report inter-annotator details.
  3. [Abstract, §4.5, Table 3, Table 17] The abstract's 'reducing processing compute' holds only relative to the full multi-stage pipeline (1236 EFLOPs) and end-to-end rewrite (824): DataOrchestra at 782 EFLOPs is ~8x ProX (97), the strongest single-method baseline. Moreover, the best fixed pipeline, Drop+NP+SR+PA with fastText mixing, reaches 39.83 vs. DataOrchestra+fastText at 40.13 — a 0.3-point quality margin — so the orchestrator's real advantage over the strongest fixed alternative is primarily compute, not quality. The abstract and §4.5 should state this precisely. Separately, the DataOrchestra row in Table 3/Table 17 (AVG 40.03, MMLU 28.00) disagrees with Table 2 (AVG 39.99, MMLU 27.60) for what appears to be the identical 0.5B RedPajama configuration; please reconcile.
  4. [§3.3, Appendix J (Figs. 20–21)] Verification (NP verifier, rewrite verifier, retry loop) exists only while constructing orchestrator training data; at deployment the orchestrator and tool models run unverified. The honest bad-case study shows this can fail badly: NP deletes all 15 lines including substantive geography text (Fig. 20), and SR silently removes the 245-residue amino-acid sequence — the most valuable content in a UniProt entry — while producing more fluent output that no downstream filter flags (Fig. 21). Since these failure modes bear directly on the claimed quality gains, please quantify deployment-time failure rates (e.g., fraction of chunks emptied by NP, silent-deletion rate on a sampled audit with a verifier or human check) and state whether any guard (e.g., revert-on-empty) is applied.
minor comments (7)
  1. [§A.5, Table 6] The regression of the 4B orchestrator below even the 0.6B one is explained only by a conjecture ('over-emphasizes preserving structure'); the differences (~0.1–0.5 points) are within the same unquantified noise as the main results. Please temper the interpretation or add support.
  2. [Table 2] CSQA sits at ~19.5–21.2 for all methods and scales — essentially chance for a 5-choice task — so it contributes noise rather than signal to the 11-benchmark average; SIQA is similarly flat. Consider noting this or reporting a subset average.
  3. [§4.4] DataOrchestra's output already contains untouched raw chunks (10.8% untouch, Fig. 5a), so adding 10B more raw tokens changes its effective raw:rewrite ratio differently than for RePro/ReWire. A sentence clarifying this asymmetry would help interpret Fig. 4.
  4. [§4.5, Table 3] It is unclear which model implements the 'Drop' step in the progressive pipelines (the orchestrator? the teacher? a DataMan-style scorer?); the 52 EFLOPs entry suggests a small model but this should be stated explicitly.
  5. [§5.1, Fig. 5b] PA inflates chunks ~2.4x (670 to 1,594 tokens, Fig. 5b). The fixed 20B-token budget correctly controls for token count, but it means PA-heavy methods train on fewer unique sources; a brief discussion of this trade-off is warranted.
  6. [Fig. 3] The y-axis ranges differ across the six panels of Fig. 3, which visually exaggerates some gaps and compresses others; consistent scaling (or explicit notice) would improve readability.
  7. [§3.1–3.3, §1] τ and N are used in §3.3 before being defined in Algorithm 1; the footnote for W=1024 renders as '10241'. The contribution list promises release of the orchestrator and scripts — please clarify whether curated datasets and tool-model weights (NP-0.6B) will also be released.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose central claims are external benchmark averages, not identities forced by construction or self-citation.

full rationale

DataOrchestra is an engineering/empirical paper. The load-bearing claims are (i) from-scratch and continued-pretraining gains on held-out benchmarks (Tables 2–3, Fig. 3, Appendices H–I) and (ii) lower curation FLOPs via selective routing (Table 3). Neither claim is a mathematical derivation that reduces to its inputs. The orchestrator is SFT’d on teacher+verifier plans (Sec. 3.3, Alg. 1); those plans are not the evaluation target. Downstream metrics (ARC, MMLU, HellaSwag, science generative suites, etc.) are external and independent of the SFT objective and of DataMan seed upsampling. Same-lab citations (ProX, DataMan, MegaMath, Data Darwinism) supply tools, baselines, or corpora, not uniqueness theorems or ansatzes that force the reported averages. Mixing/ablation controls (Secs. 4.4–4.5, 5.2) further compare against fixed pipelines and rewrite+mix setups rather than renaming a fitted identity as a prediction. Mild shared-stack dependence (Qwen3 teacher/tools, DataMan scoring of seeds) is methodological coupling, not circular reduction. Per the rubric this is the normal non-finding: score 0, no circular steps.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 2 invented entities

Load-bearing commitments are methodological choices and tooling assumptions, not mathematical axioms. The central empirical claim rests on chunking, the NP→SR→PA stage inventory and order, teacher/verifier judgments as plan labels, specific tool models, DataMan-based seed rebalancing, and standard next-token pretraining evaluation. No new physical entities; free parameters are training/design knobs (thresholds, retries, sizes) rather than fits that define the reported benchmark numbers.

free parameters (7)
  • Chunk token limit W = 1024
    Documents are split into chunks of at most W tokens (W=1024, Qwen3 tokenizer); all orchestration decisions are at this granularity.
  • Drop-gate score threshold = drop if score < 2
    Teacher rates 0–5 under a custom rubric; chunk dropped only if score < 2—hand-chosen salvage threshold, not learned.
  • Rewrite no-op similarity threshold τ = 98%
    Rewriting stage dropped if Levenshtein similarity to input > 98% (or repeated verifier failure).
  • Max rewrite retries N = 5
    Failed SR/PA rewrites retry under corrective instructions up to N times during plan evolution.
  • Orchestrator model size / base = Qwen3-1.7B-Base SFT
    Default 1.7B after size sweep (0.6B/1.7B/4B); 4B underperformed—selection is empirical on their 0.5B pretrain proxy.
  • Seed quality upsampling via DataMan = equal 1–3 vs 4–5 groups
    Documents scored 1–5; scores 1–3 and 4–5 forced to equal mass in 160K seed docs—hand policy shaping training signal.
  • Math CPT mixture ratios = 50/35/15
    50% curated math / 35% DCLM web / 15% MegaMath synthetic QA fixed for all methods.
axioms (4)
  • ad hoc to paper The useful cleaning inventory is exhausted by ordered stages NP (whole-line remove_lines only), SR (meaning-preserving rewrite), and PA (pedagogical expansion), plus drop and untouch.
    Section 3.1–3.2 define this taxonomy from prior work clusters; other ops (dedup, toxicity, domain classifiers as stages, in-line substring edits) are excluded by design.
  • domain assumption A large instruction-tuned teacher and verifier LLMs can label salvageability, stage necessity, over-pruning, content loss, and factual error well enough to supervise a smaller orchestrator.
    Section 3.3 and A.2–A.3; authors report ~83–86% human agreement on sampled verifier/eval judgments, not formal guarantees.
  • domain assumption Average accuracy on the chosen 11 PPL-style benchmarks (and 9 generative science sets for CPT), smoothed over last three checkpoints, is a valid proxy for pretraining data quality differences under matched token budgets.
    Standard in the subfield (Section 4.1, Appendix D) but known to be noisy at small scale; paper does not claim causal isolation from all confounds.
  • standard math Inference FLOPs approximated as 2N(Din+Dout) per Kaplan et al. suffice to rank curation compute across pipelines.
    Appendix G; standard scaling-law accounting, ignores system effects.
invented entities (2)
  • DataOrchestra orchestrator (decision+instruction policy) no independent evidence
    purpose: Map each chunk to drop/untouch/clean, stage subset, and SR/PA instructions executed by tool models.
    Core learned artifact; existence is operational (SFT model), not a hidden physical postulate. Independent evidence is downstream pretrain metrics and released-model promise, not an external natural phenomenon.
  • Fine-grained plan evolution loop (execute–verify–retry–annotate) no independent evidence
    purpose: Convert coarse teacher plans into tool-grounded pipelines and chunk-specific instructions for SFT data.
    Procedural construct in Algorithm 1; validated only inside this training pipeline.

pith-pipeline@v1.2.0-grok45-kimik3 · 39312 in / 4222 out tokens · 92072 ms · 2026-07-31T06:54:56.729708+00:00 · methodology

0 comments
read the original abstract

Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 31 linked inside Pith

  1. [1]

    ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models , year=

    Scaling laws for downstream task performance of large language models , author=. ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models , year=

    2024

  2. [2]

    arXiv preprint arXiv:2605.11978 , year=

    On Predicting the Post-training Potential of Pre-trained LLMs , author=. arXiv preprint arXiv:2605.11978 , year=

    Pith/arXiv arXiv

  3. [3]

    arXiv preprint arXiv:2502.03387 , year=

    Limo: Less is more for reasoning , author=. arXiv preprint arXiv:2502.03387 , year=

    Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2506.20512 , year=

    Octothinker: Mid-training incentivizes reinforcement learning scaling , author=. arXiv preprint arXiv:2506.20512 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2306.11644 , year=

    Textbooks are all you need , author=. arXiv preprint arXiv:2306.11644 , year=

    Pith/arXiv arXiv

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    The fineweb datasets: Decanting the web for the finest text data at scale , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    Advances in Neural Information Processing Systems , volume=

    Datacomp-lm: In search of the next generation of training sets for language models , author=. Advances in Neural Information Processing Systems , volume=

  8. [8]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  9. [9]

    arXiv preprint arXiv:2112.11446 , year=

    Scaling language models: Methods, analysis & insights from training gopher , author=. arXiv preprint arXiv:2112.11446 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2402.09739 , year=

    Qurating: Selecting high-quality data for training language models , author=. arXiv preprint arXiv:2402.09739 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2502.19363 , year=

    Dataman: Data manager for pre-training large language models , author=. arXiv preprint arXiv:2502.19363 , year=

    Pith/arXiv arXiv

  12. [12]

    Advances in Neural Information Processing Systems , volume=

    Mates: Model-aware data selection for efficient pretraining with data influence models , author=. Advances in Neural Information Processing Systems , volume=

  13. [13]

    arXiv preprint arXiv:2409.17115 , year=

    Programming every example: Lifting pre-training data quality like experts at scale , author=. arXiv preprint arXiv:2409.17115 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2507.03253 , year=

    Refinex: Learning to refine pre-training data at scale from expert-guided programs , author=. arXiv preprint arXiv:2507.03253 , year=

    Pith/arXiv arXiv

  15. [15]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Rephrasing the web: A recipe for compute and data-efficient language modeling , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  16. [16]

    arXiv preprint arXiv:2506.04689 , year=

    Recycling the web: A method to enhance pre-training data quality and quantity for language models , author=. arXiv preprint arXiv:2506.04689 , year=

    arXiv

  17. [17]

    arXiv preprint arXiv:2510.10681 , year=

    RePro: Training Language Models to Faithfully Recycle the Web for Pretraining , author=. arXiv preprint arXiv:2510.10681 , year=

    arXiv

  18. [18]

    arXiv preprint arXiv:2507.20534 , year=

    Kimi k2: Open agentic intelligence , author=. arXiv preprint arXiv:2507.20534 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2505.02881 , year=

    Rewriting pre-training data boosts llm performance in math and code , author=. arXiv preprint arXiv:2505.02881 , year=

    arXiv

  20. [20]

    arXiv preprint arXiv:2602.07824 , year=

    Data darwinism part i: Unlocking the value of scientific data for pre-training , author=. arXiv preprint arXiv:2602.07824 , year=

    arXiv

  21. [21]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2602.15763 , year=

    Glm-5: from vibe coding to agentic engineering , author=. arXiv preprint arXiv:2602.15763 , year=

    Pith/arXiv arXiv

  23. [23]

    arXiv preprint arXiv:2603.14420 , year=

    Data Darwinism Part II: DataEvolve--AI can Autonomously Evolve Pretraining Data Curation , author=. arXiv preprint arXiv:2603.14420 , year=

    arXiv

  24. [24]

    ACM Transactions on Information Systems , volume=

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions , author=. ACM Transactions on Information Systems , volume=. 2025 , publisher=

    2025

  25. [25]

    arXiv preprint arXiv:2604.13977 , year=

    How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data , author=. arXiv preprint arXiv:2604.13977 , year=

    Pith/arXiv arXiv

  26. [26]

    Advances in neural information processing systems , volume=

    Redpajama: an open dataset for training large language models , author=. Advances in neural information processing systems , volume=

  27. [27]

    International Conference on Learning Representations , volume=

    Openwebmath: An open dataset of high-quality mathematical web text , author=. International Conference on Learning Representations , volume=

  28. [28]

    arXiv preprint arXiv:2504.02807 , year=

    Megamath: Pushing the limits of open math corpora , author=. arXiv preprint arXiv:2504.02807 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2508.10975 , year=

    Beyondweb: Lessons from scaling synthetic data for trillion-scale pretraining , author=. arXiv preprint arXiv:2508.10975 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2401.12926 , year=

    Dsdm: Model-aware dataset selection with datamodels , author=. arXiv preprint arXiv:2401.12926 , year=

    Pith/arXiv arXiv

  31. [31]

    Proceedings of the twelfth language resources and evaluation conference , pages=

    CCNet: Extracting high quality monolingual datasets from web crawl data , author=. Proceedings of the twelfth language resources and evaluation conference , pages=

  32. [32]

    arXiv preprint arXiv:2306.01116 , year=

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only , author=. arXiv preprint arXiv:2306.01116 , year=

    Pith/arXiv arXiv

  33. [33]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Deduplicating training data makes language models better , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  34. [34]

    arXiv preprint arXiv:2303.09540 , year=

    Semdedup: Data-efficient learning at web-scale through semantic deduplication , author=. arXiv preprint arXiv:2303.09540 , year=

    Pith/arXiv arXiv

  35. [35]

    Proceedings of the 15th conference of the European chapter of the association for computational linguistics: volume 2, short papers , pages=

    Bag of tricks for efficient text classification , author=. Proceedings of the 15th conference of the European chapter of the association for computational linguistics: volume 2, short papers , pages=

  36. [36]

    arXiv preprint arXiv:2605.19407 , year=

    A Bitter Lesson for Data Filtering , author=. arXiv preprint arXiv:2605.19407 , year=

    Pith/arXiv arXiv

  37. [37]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  38. [38]

    arXiv preprint arXiv:2502.04235 , year=

    Reformulation for pretraining data augmentation , author=. arXiv preprint arXiv:2502.04235 , year=

    Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:2409.07431 , year=

    Synthetic continued pretraining , author=. arXiv preprint arXiv:2409.07431 , year=

    Pith/arXiv arXiv

  40. [40]

    Ben Allal, Loubna and Lozhkov, Anton and Penedo, Guilherme and Wolf, Thomas and von Werra, Leandro , title =

  41. [41]

    arXiv preprint arXiv:2404.01413 , year=

    Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data , author=. arXiv preprint arXiv:2404.01413 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2404.07503 , year=

    Best practices and lessons learned on synthetic data , author=. arXiv preprint arXiv:2404.07503 , year=

    Pith/arXiv arXiv

  43. [43]

    2025 , month =

    Context Rot: How Increasing Input Tokens Impacts LLM Performance , author =. 2025 , month =

    2025

  44. [44]

    Qwen2.5: A Party of Foundation Models , url =

    Qwen Team , month =. Qwen2.5: A Party of Foundation Models , url =

  45. [45]

    arXiv preprint arXiv:1803.05457 , year=

    Think you have solved question answering? try arc, the ai2 reasoning challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    Pith/arXiv arXiv 2009

  47. [47]

    Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

    Hellaswag: Can a machine really finish your sentence? , author=. Proceedings of the 57th annual meeting of the association for computational linguistics , pages=

  48. [48]

    Proceedings of the 2017 conference on empirical methods in natural language processing , pages=

    Race: Large-scale reading comprehension dataset from examinations , author=. Proceedings of the 2017 conference on empirical methods in natural language processing , pages=

    2017

  49. [49]

    Communications of the ACM , volume=

    Winogrande: An adversarial winograd schema challenge at scale , author=. Communications of the ACM , volume=. 2021 , publisher=

    2021

  50. [50]

    Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    Can a suit of armor conduct electricity? a new dataset for open book question answering , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    2018

  51. [51]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Piqa: Reasoning about physical commonsense in natural language , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  52. [52]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge , author=. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages=

    2019

  53. [53]

    Social IQa: Commonsense reasoning about social interactions , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages=

    2019

  54. [54]

    Proceedings of the 3rd Workshop on Noisy User-generated Text , pages=

    Crowdsourcing multiple choice science questions , author=. Proceedings of the 3rd Workshop on Noisy User-generated Text , pages=

  55. [55]

    doi:10.5281/zenodo.12608602 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang...

    doi:10.5281/zenodo.12608602

  56. [56]

    arXiv preprint arXiv:1909.08053 , year=

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism , author=. arXiv preprint arXiv:1909.08053 , year=

    Pith/arXiv arXiv 1909

  57. [57]

    IEEE Transactions on Audio, Speech and Language Processing , year=

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning , author=. IEEE Transactions on Audio, Speech and Language Processing , year=

  58. [58]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2103.03874 , year=

    Measuring mathematical problem solving with the math dataset , author=. arXiv preprint arXiv:2103.03874 , year=

    Pith/arXiv arXiv

  60. [60]

    International Conference on Learning Representations , volume=

    Let's verify step by step , author=. International Conference on Learning Representations , volume=

  61. [61]

    arXiv preprint arXiv:2311.12022 , year=

    Gpqa: A graduate-level google-proof q&a benchmark , author=. arXiv preprint arXiv:2311.12022 , year=

    Pith/arXiv arXiv

  62. [62]

    Advances in Neural Information Processing Systems , volume=

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines , author=. Advances in Neural Information Processing Systems , volume=

  63. [63]

    arXiv preprint arXiv:2307.10635 , year=

    Scibench: Evaluating college-level scientific problem-solving abilities of large language models , author=. arXiv preprint arXiv:2307.10635 , year=

    Pith/arXiv arXiv

  64. [64]

    2024 , eprint=

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark , author=. 2024 , eprint=

    2024

  65. [65]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    Pith/arXiv arXiv 2001