Pith. sign in

REVIEW 4 major objections 5 minor 6 cited by

How to Synthesize Text Data without Model Collapse?

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Token-level editing of human text—resampling only the tokens a model already finds predictable—keeps test error bounded across iterative training rounds, preventing model collapse while improving downstream performance.

desk verdict The empirical non-iterative collapse finding and the ToEdit method are worth a look, but the theorem proving collapse avoidance has a hidden assumption that does not hold, so the theoretical guarantee is unsupported. read the letter →

arxiv 2412.14689 v3 pith:VMFX7YHV submitted 2024-12-19 cs.CL cs.AIcs.LG

Xuekai Zhu , Daixuan Cheng , Hengli Li , Kaiyan Zhang , Ermo Hua , Xingtai Lv , Ning Ding , Zhouhan Lin
show 2 more authors
Zilong Zheng Bowen Zhou
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords modelcollapsesyntheticdatatoken-leveleditingsemi-syntheticpre-trainingdistributionn-gramanalysislanguagetraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how AI-generated text can be used for training without the self-consuming degradation known as model collapse, and answers with a method it calls token-level editing (ToEdit). Instead of generating whole sequences, ToEdit keeps each token unless a pre-trained model assigns it very high probability; such easy tokens are resampled, leaving the human corpus mostly intact. The paper first shows that even a single, non-iterative mixture of synthetic and human text hurts pre-training, and that synthetic text lacks long-tail coverage and over-concentrates n-gram features. It then argues, in a linear-model idealization, that its editing process keeps test error below a fixed bound independent of the number of iterations, and reports consistent small performance gains over source data in pre-training, continual pre-training, and fine-tuning. If the claims hold, cheap token resampling rather than full synthesis is a way to make recursive training on model-influenced data safe.

What carries the argument

The paper's central object is a diagonal, idempotent edit matrix $M_i$ (with 1s indicating tokens to replace and 0s to keep) inside the recursion $\tilde Y_n^\top = M_{n-1}\hat Y_n + (1-M_{n-1})\tilde Y_{n-1}$. Theorem 1 claims that this editing yields the closed form $\hat w_{n+1} = w^* + (X^\top X)^{-1}X^\top(E_1 + \sum_{i=1}^n M_i E_{i+1})$, where the $E_i$ are independent label noises; the test-error bound then follows by taking the trace of the covariance and using the fact that the edited noises are weighted by the masks. In practice the same mechanism is instantiated through a U-shaped token-probability distribution: tokens with conditional probability above a threshold $p$ are resampled from a prior language model, while the rest of the human text is kept.

What would settle it

Compute the closed form for $\hat w_2$ in a two-sample, one-dimensional dataset with a single edited point and compare it with Eq. (5); a mismatch would show the induction step in Theorem 1 relies on an unstated condition. A second check is to run the editing recursion over many generations on a controlled corpus and measure whether the test-error bound $2\sigma^2 d/(T-d-1)$ actually holds.

Watch

Extended reading notes

Core claim

The central discovery the paper asserts is that replacing pure data synthesis with token-level editing changes the error dynamics of iterative training: under the paper's linear-model setting, any sequence of increasingly mild edits yields test error bounded by $E_{\mathrm{test}}(\hat w_{n+1}) \le 2\sigma^2 d/(T-d-1)$, independent of the number of generations, whereas pure synthesis gives $E_{\mathrm{test}}(\hat w_n) = \sigma^2 d/(T-d-1)\times n$. The same operation is shown empirically to improve over the source data across the three stages. The paper frames this as a bridge between human and synthetic data: semi-synthetic data preserves the original distribution's coverage while resampling the portions the model already knows, so it can improve performance without the collapse that pure synthetic data brings.

Load-bearing premise

The bound in Theorem 2 stands on a hidden algebraic condition in the proof, located in the induction step of Theorem 1 in Appendix A.1: the edit mask must commute with the least-squares projection, which the paper neither states nor justifies, and which fails for a generic dataset.

Editorial extensions

If this is right

  • Mixing synthetic text into pre-training harms model performance even without iterative retraining, and the harm grows with the fraction of synthetic data.
  • Under the paper's bound, recursive use of edited data should not accumulate test error, so future models could reuse their own edited outputs without collapse.
  • ToEdit improves average downstream performance over source data in pre-training from scratch, continual pre-training, and supervised fine-tuning, without increasing data size.
  • Because the method needs only a single forward pass per token, the data-editing cost stays far below autoregressive synthesis, making it practical on a single consumer GPU.
  • If the theoretical result transfers beyond linear models, token editing becomes a cheap data-regularization tool for any training pipeline that wants to stay close to a trusted human distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof's hidden algebraic condition suggests a sharper target: characterize exactly which edit masks commute with the least-squares projection, and test whether natural token-resampling masks approximately satisfy that condition in practice.
  • ToEdit can be read as a distributional regularizer that nudges the empirical token distribution toward higher entropy; if so, the threshold $p$ could be tuned per domain or per difficulty level rather than globally.
  • The same single-forward-pass editing idea could be extended to other units of data, such as spans, sentences, or embedding coordinates, with the same claim of preserving coverage while reducing predictability.
  • If the theoretical bound does not hold without the unstated condition, the practical gains may still survive, but the collapse-resistance guarantee would need a different proof or a restricted class of edit masks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies the effect of synthetic data on language model pretraining and proposes ToEdit, a token-level editing method that replaces high-confidence tokens using a prior language model. The authors report that mixing synthetic data degrades pretraining (Section 3.1), characterize distributional narrowing and n-gram over-concentration (Section 3.2), and claim a theoretical guarantee (Theorem 2) that iterative token editing keeps test error bounded, thereby avoiding model collapse. Experiments compare ToEdit against source data in pretraining, continual pretraining, and fine-tuning (Tables 2-4). The theoretical guarantee is not established: the proof of Theorem 1 relies on unstated conditions on the edit masks and the projection matrix, and the bound in Theorem 2 does not follow from the given inequalities.

Significance. The question addressed is timely and relevant, and the non-iterative mixture experiments with the distributional analyses in Section 3 provide useful observations. The paper is transparent about experimental settings and provides a code repository, which are strengths. If the n-independent test-error bound were correct, it would be a meaningful extension of the model-collapse literature. However, the central proof is invalid as written and the empirical evidence does not directly test iterative collapse. The contribution reduces to a modest data-augmentation heuristic; the advertised guarantee of preventing model collapse is unsupported.

major comments (4)
  1. [Appendix A.1, Eqs. (29)-(31)] The derivation of Theorem 1 silently assumes M_i P = M_i and M_i M_j = 0 for i != j, where P = X(X^T X)^{-1} X^T. The disjointness condition is listed among 'properties of M_i' in the proof, but it is not implied by the recursive definition (3) nor stated in Theorem 1; the condition M_i P = M_i is not stated anywhere and is false for generic design matrices. A concrete counterexample: take T=2, d=1, X=(1,2)^T, M_1=diag(1,0). Then M_1 P is not equal to M_1, and recursion (3) yields w_2 = w* + (X^T X)^{-1}X^T [E_1 + M_1 E_2 + M_1(P-I)E_1], whereas Eq. (5) omits the term M_1(P-I)E_1. Since Theorem 2's bound is derived from Eq. (5), the main theoretical guarantee is unsupported.
  2. [Appendix A.2, Eqs. (36)-(43)] Even if Eq. (5) were granted, the passage from Eq. (36) to Eq. (43) is invalid. Eq. (36) is sigma^2 E[tr((X^T X)^{-1})] + sigma^2 sum_i E[tr((X^T X)^{-1} M_i)], and the second sum is nonnegative. Neglecting it cannot produce the upper bound 2 sigma^2 d/(T-d-1); it can only give the one-term estimate. The Cauchy-Schwarz bound in Eq. (41), together with the geometric decay assumption, yields the additional term sigma^2 sqrt(E[tr((X^T X)^{-2})]) sqrt(E[tr(M_1)])/(1-eta), which is not bounded by sigma^2 d/(T-d-1) under the stated hypotheses. Thus Eq. (6) is not a consequence of the proof.
  3. [Appendix G.3, Table 15] The assumed geometric decay ||M_i|| = ||M_{i-1}|| eta with eta in (0,1), introduced in Theorem 2 to obtain an n-independent bound, is supported by only three observations on one dataset: 12.5%, 11.76%, and 11.08%. Three points cannot establish a geometric ratio; no confidence intervals, no multiple seeds, and no measurements on the pretraining or domain-specific corpora used in Section 5 are provided. Since the boundedness of the sum in Eq. (42) is exactly what prevents the n-dependence, citing this table as 'supporting evidence' for the assumption is largely circular. The empirical support for the key assumption is therefore inadequate.
  4. [Section 5, Tables 2-4] The experiments do not validate the collapse-avoidance claim. All main comparisons are one-shot uses of the edited data; they compare ToEdit against source data in a single training run. The theoretical claim concerns behavior as n grows, and no experiment tracks test error over many generations of the editing process. The only iterative result, Table 16 in Appendix G.3, covers two generations and reports average scores rather than a collapse trajectory. Thus the paper's advertised conclusion that token-level editing prevents model collapse is not directly tested.
minor comments (5)
  1. [Appendix A.2] The heading 'Proof of Theorem 4.3' should read 'Proof of Theorem 2'; the numbering appears to be leftover from an earlier version.
  2. [Section 4.2, Eq. (3)] The notation \tilde X_n^T = X conflicts with the use of \tilde X_n as the design matrix in \hat w_n = \tilde X_n^\dagger \tilde Y_n; the transpose convention should be harmonized.
  3. [Section 4.1, Algorithm 1] Algorithm 1 describes resampling from the prior distribution but does not mention top-k; the choice k=8 appears only in Section 5.1, so the method description should be self-contained.
  4. [Figure 5] The histogram lacks axis labels and units; adding them and specifying whether counts are over token positions or sequences would help the reader interpret the U-shape claim.
  5. [Appendix G.3, Table 16] Task-level results and error bars are not reported, making it difficult to judge whether the two-generation gains (e.g., average 69.01 to 69.39) are meaningful.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the proof has hidden assumptions, but the derivation is not equivalent to its inputs by construction; self-citations are not load-bearing.

full rationale

The paper's central claims—non-iterative collapse from pure synthetic data and improvements from token editing—are supported by independent experiments on external benchmarks (Paloma, Pile, and the listed downstream tasks). The theoretical result is built on the standard linear-regression collapse framework of Dohmatob et al. and Gerstgrasser et al., not on the authors' own prior results. The self-citations to Cheng et al. and Zhu et al. appear only in experimental setup, dataset construction, and related-work context; they are not load-bearing for the collapse-avoidance theorem. The geometric-decay assumption ||M_i|| = ||M_{i-1}||η is stated as an explicit additional assumption in Theorem 2, and Appendix G.3 provides three-generation measurements (12.5%, 11.76%, 11.08%) as evidence; this is an assumption with empirical support, not a fitted parameter renamed as a prediction. The main proof does contain a serious hidden assumption: the step from Eq. (29) to Eq. (30) requires M_i P = M_i, and Eq. (16) asserts M_i M_j = 0 without derivation from the definition of M_i as a diagonal 0/1 matrix. These are unstated conditions that make the clean closed form hold; this is a correctness gap in the derivation, but it is not a circular reduction of the conclusion to the inputs. The advertised bound Eq. (6) is therefore unsupported as written, but that is a validity concern, not a circularity concern. No step in the claimed derivation chain is equivalent by construction to its own input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces one free threshold p and one sampling size k, both selected via ablation. The theory rests on strong, partly unstated assumptions about the edit masks and their decay; the most serious is the implicit M_iP=M_i condition, which is false for generic designs. No new physical or conceptual entities are postulated.

free parameters (3)
  • resampling threshold p = 0.99
    Set after ablations on the biomedicine domain (Table 5). It controls how many tokens are replaced; the paper chooses a high threshold to edit only the most predictable tokens.
  • top-k sampling size k = 8
    Chosen from ablations (Table 8); larger k gives similar performance with more compute.
  • decay ratio η in ||M_i|| = ||M_{i-1}||η = not estimated; assumed ∈ (0,1)
    Theoretical assumption used to make the bound in Theorem 2 finite; only supported by a 3-generation trend (Table 15) that does not establish geometric decay.
assumptions (5)
  • domain assumption Data are generated by a linear model y = x·w* + ε with x ~ N(0,Σ) and ε ~ N(0,σ²).
    The theory in §4.2 is derived for least-squares on Gaussian features, not for language models.
  • ad hoc to paper Edit masks M_i are diagonal 0/1 matrices that are idempotent, mutually orthogonal (M_iM_j=0 for i≠j), and commute with the hat matrix P (M_iP=M_i).
    The first two properties are stated; the commutativity is silently assumed in Eq. (29)-(31) and is false in general.
  • ad hoc to paper The editing operation decays geometrically, ||M_i|| = ||M_{i-1}||η for η∈(0,1).
    Introduced to turn the sum over i into a geometric series; the empirical evidence is a 3-generation percentage drop, not a measured η.
  • standard math Inverse second moment of Gaussian design: E[(X⊤X)^{-1}] = Σ^{-1}/(T−d−1) for T≥d+2.
    Imported from Dohmatob et al. (2024a) via Lemma 3.
  • domain assumption Token editing in LLMs corresponds to label editing in the linear model.
    The algorithm edits input tokens (Eq. 2) while the theory replaces target labels (Eq. 3); no mapping is proved.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How to Synthesize Text Data without Model Collapse?." pith.science (2026). https://pith.science/paper/VMFX7YHV

@misc{pith2026241214689,
  author       = {Pith},
  title        = {Pith review of: How to Synthesize Text Data without Model Collapse?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VMFX7YHV}},
  note         = {Machine review of arXiv:2412.14689}
}
abstract

Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance. With the proliferation of AI models, synthetic data will fundamentally reshape the web data ecosystem. Future GPT-$\{n\}$ models will inevitably be trained on a blend of synthetic and human-produced data. In this paper, we focus on two questions: what is the impact of synthetic data on language model training, and how to synthesize data without model collapse? We first pre-train language models across different proportions of synthetic data, revealing a negative correlation between the proportion of synthetic data and model performance. We further conduct statistical analysis on synthetic data to uncover distributional shift phenomenon and over-concentration of n-gram features. Inspired by the above findings, we propose token editing on human-produced data to obtain semi-synthetic data. As a proof of concept, we theoretically demonstrate that token-level editing can prevent model collapse, as the test error is constrained by a finite upper bound. We conduct extensive experiments on pre-training from scratch, continual pre-training, and supervised fine-tuning. The results validate our theoretical proof that token-level editing improves model performance.

Figures

Figures reproduced from arXiv: 2412.14689 by the authors.

Figure 1
Figure 1. Model collapse of synthetic data. ① The model continuously trains on its previously generated data, leading to a gradual decline in model performance, i.e., model collapse. Starting from real data Data0, the test error Etest increases as f0 undergoes iterative training on synthetic data Data>0. ② ToEdit (ours), we use a trained model for token-level editing rather than purely synthesizing data. Leveraging f0 and an … view at source ↗
Figure 2
Figure 2. Non-iterative model collapse. Training language models from scratch on AI-synthesized data or a mixture of human and synthetic data leads to performance degradation. This degradation is negatively correlated with the proportion of synthetic data used in training. Setting: We pre-train GPT-2 Small (124M) on human data (Dolma (Soldaini et al., 2024)) and synthetic data (Cosmopedia (Ben Allal et al., 2024)) and evaluat… view at source ↗
Figure 3
Figure 3. PPL distribution of human and synthetic data estimated by Llama-3-8B. The synthetic data lacks the long tail of the human￾produced data and is also concentrated within the first 25% of the human-produced data distribution. A. Distribution of human-produced data is sharp with a long tail, spanning a wide range from 0 to over 100. B. The values are concentrated within a much narrower range, mostly between 0 and 14. Th… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: A. Pre-training results for selected synthetic data and other data mixtures on OLMo-237M. B. Embedding visualization between human-produced, synthetic, and DSIR-selected data using sentence-transformer [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: U-shape token probability distribution of Dolma-sampled V6 estimated by Qwen-0.5B-Instruct (qwe, 2024). editing, which leverages a prior distribution to adjust the data. Our method can maintain the source distribution while improving the source data, called semi-synthe…
Figure 6
Figure 6. Figure 6: Token distribution across different probability ranges in BioMed dataset. C. More Discussion of Main Results As shown in [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Pre-training loss of GPT-2 Small (124M) on human (Dolma (Soldaini et al., 2024)) and synthetic (Cosmopedia (Ben Allal et al., 2024)) data. As the proportion of synthetic data increases, the model’s loss decreases. Specifically, both Rephrasing the Web (Maini et al., 20…
Figure 8
Figure 8. Figure 8: OLMo-237M pretraining with mixed human and syn￾thetic data proportions. We pretrain the OLMo-237M model using a mixture of human data (Dolma (Soldaini et al., 2024)) and syn￾thetic data (Cosmopedia (Ben Allal et al., 2024)) [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 10
Figure 10. Figure 10: PPL distribution of human and synthetic data estimated by StabLM-Zephyr-3B. This indicates that different prior distributions yielded the same result, which is consistent with [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Uni/Bi-gram feature distribution across 10,000 hash buckets. Lessons from our work: The key to improving the quality of synthetic data lies in balancing long-tail distribution preservation and optimizing synthetic data approaches. In other words, we should focus on tw…
Figure 12
Figure 12. Figure 12: The top 40 bi-grams from separately sampled 1M subsets of Dolma, Cosmopedia, and DSIR-selected datasets. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: The top 64 bi-grams from separately sampled 1M subsets of Dolma, Cosmopedia, and DSIR-selected datasets [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Density sampling response values. This result further confirms the issue of feature collapse in synthetic data. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Fairness collapse: gender-occupation bias worsens in the first iterations of recursive synthetic-data training, ahead of visible model-collapse indicators.

  2. What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Moderately diverse LLM-generated data can improve fine-tuned model performance in low-data settings when distribution shift is minimal, while high diversity or large distribution shift hurts.

  3. Reformulation for Pretraining Data Augmentation

    cs.CL 2025-02 conditional novelty 6.0 of 10

    MGA reformulates existing high-quality text into diverse genre-audience variants, producing a 770B-token corpus that improves LLM pretraining under data-constrained, high-repetition conditions.

  4. Hermes 4 Technical Report

    cs.AI 2025-08 conditional novelty 5.0 of 10

    Hermes 4 releases three open-weight reasoning models (14B, 70B, 405B) trained with synthetic data and a length-control SFT stage, evaluated on mathematics, code, knowledge, and alignment benchmarks.

  5. Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Training on LLM text embeddings can cause tabular classifiers to collapse to single-class predictions, which spuriously inflates Accuracy-on-the-Line correlations.

  6. Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set

    cs.CL 2024-12 reject novelty 3.0 of 10

    The paper's proof of inevitable language-model collapse reduces to a definitional identity, since the error terms whose accumulation drives the result are chosen to fit the model outputs rather than derived from train...

Reference graph

Works this paper leans on

70 extracted references · 18 canonical work pages · cited by 6 Pith papers

  1. [1]

    Qwen2 technical report. 2024

    2024

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  4. [4]

    M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., et al

    Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., et al. A survey on data selection for language models. arXiv preprint arXiv:2402.16827, 2024

    arXiv 2024

  5. [5]

    I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R

    Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. G. Self-consuming generative models go mad. arXiv preprint arXiv:2307.01850, 4: 0 14, 2023

    arXiv 2023

  6. [6]

    L., and Paul, M

    Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541, 2024

    arXiv 2024

  7. [7]

    Picor: Multi-task deep reinforcement learning with policy correction

    Bai, F., Zhang, H., Tao, T., Wu, Z., Wang, Y., and Xu, B. Picor: Multi-task deep reinforcement learning with policy correction. Proceedings of the AAAI Conference on Artificial Intelligence, 37 0 (6): 0 6728--6736, Jun. 2023

    work page 2023

  8. [8]

    Efficient model-agnostic alignment via bayesian persuasion

    Bai, F., Wang, M., Zhang, Z., Chen, B., Xu, Y., Wen, Y., and Yang, Y. Efficient model-agnostic alignment via bayesian persuasion. arXiv preprint arXiv:2405.18718, 2024

    arXiv 2024

Show all 70 references
  1. [9]

    Rat: Adversarial attacks on deep reinforcement agents for targeted behaviors

    Bai, F., Liu, R., Du, Y., Wen, Y., and Yang, Y. Rat: Adversarial attacks on deep reinforcement agents for targeted behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.\ 15453--15461, 2025

    2025

  2. [10]

    Stable lm 2 1.6 b technical report

    Bellagente, M., Tow, J., Mahan, D., Phung, D., Zhuravinskyi, M., Adithyan, R., Baicoianu, J., Brooks, B., Cooper, N., Datta, A., et al. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834, 2024

    2024 arXiv

  3. [11]

    Cosmopedia, 2024

    Ben Allal, L., Lozhkov, A., Penedo, G., Wolf, T., and von Werra, L. Cosmopedia, 2024. URL https://huggingface.co/datasets/HuggingFaceTB/cosmopedia

    2024

  4. [12]

    J., Duplessis, A., Jiralerspong, M., and Gidel, G

    Bertrand, Q., Bose, A. J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data. arXiv preprint arXiv:2310.00429, 2023

    2023 arXiv

  5. [13]

    Large language models suffer from their own output: An analysis of the self-consuming training loop

    Briesch, M., Sobania, D., and Rothlauf, F. Large language models suffer from their own output: An analysis of the self-consuming training loop. arXiv preprint arXiv:2311.16822, 2023

    2023 arXiv

  6. [14]

    Instruction pre-training: Language models are supervised multitask learners

    Cheng, D., Gu, Y., Huang, S., Bi, J., Huang, M., and Wei, F. Instruction pre-training: Language models are supervised multitask learners. In Conference on Empirical Methods in Natural Language Processing, 2024 a . URL https://api.semanticscholar.org/CorpusID:270620509

    2024

  7. [15]

    Adapting large language models via reading comprehension

    Cheng, D., Huang, S., and Wei, F. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://openreview.net/forum?id=y886UXPEZ0

    2024

  8. [16]

    X., Luan, Z., Dai, B., and Zhang, Z

    Cheng, D., Huang, S., Zhu, Z., Zhang, X., Zhao, W. X., Luan, Z., Dai, B., and Zhang, Z. On domain-specific post-training for multimodal large language models. arXiv preprint arXiv:2411.19930, 2024 c

    2024 arXiv

  9. [17]

    Model collapse demystified: The case of regression

    Dohmatob, E., Feng, Y., and Kempe, J. Model collapse demystified: The case of regression. arXiv preprint arXiv:2402.07712, 2024 a

    2024 arXiv

  10. [18]

    Strong model collapse

    Dohmatob, E., Feng, Y., Subramonian, A., and Kempe, J. Strong model collapse. arXiv preprint arXiv:2410.04840, 2024 b

    2024 arXiv

  11. [19]

    A tale of tails: Model collapse as a change of scaling laws

    Dohmatob, E., Feng, Y., Yang, P., Charton, F., and Kempe, J. A tale of tails: Model collapse as a change of scaling laws. arXiv preprint arXiv:2402.07043, 2024 c

    2024 arXiv

  12. [20]

    and Li, Y

    Eldan, R. and Li, Y. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023

    2023 arXiv

  13. [21]

    Beyond model collapse: Scaling up with syn-thesized data requires verification

    Feng, Y., Dohmatob, E., Yang, P., Charton, F., Kempe, J., and Meta, F. Beyond model collapse: Scaling up with syn-thesized data requires verification. arXiv preprint arXiv:2406.07515, 2024

    2024 arXiv

  14. [22]

    J., and Gidel, G

    Ferbach, D., Bertrand, Q., Bose, A. J., and Gidel, G. Self-consuming generative models with curated data provably optimize human preferences. arXiv preprint arXiv:2407.09499, 2024

    2024 arXiv

  15. [24]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020 b

    2020 arXiv

  16. [25]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  17. [26]

    Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data

    Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413, 2024

    2024 arXiv

  18. [27]

    R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J

    Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J. D., Muennigh...

    2024

  19. [28]

    L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al

    Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023

    2023 arXiv

  20. [29]

    Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023

    2023 arXiv

  21. [30]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  22. [31]

    Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving

    Jia, X., Yang, Z., Li, Q., Zhang, Z., and Yan, J. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. arXiv preprint arXiv:2406.03877, 2024

    2024 arXiv

  23. [32]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  24. [33]

    L., and Koyejo, S

    Kazdan, J., Schaeffer, R., Dey, A., Gerstgrasser, M., Rafailov, R., Donoho, D. L., and Koyejo, S. Collapse or thrive? perils and promises of synthetic data in a self-generating world, 2025. URL https://openreview.net/forum?id=Xr5iINA3zU

    2025

  25. [34]

    R., Stevens, K., Barhoum, A., Duc, N

    Kopf, A., Kilcher, Y., von Rutte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., Shahul, E., Suri, S., Glushkov, D., Dantuluri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations - democrati...

    2023 arXiv

  26. [35]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  27. [36]

    Normalization enhances generalization in visual reinforcement learning

    Li, L., Lyu, J., Ma, G., Wang, Z., Yang, Z., Li, X., and Li, Z. Normalization enhances generalization in visual reinforcement learning. arXiv preprint arXiv:2306.00656, 2023

    2023 arXiv

  28. [37]

    Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2)

    Li, Q., Jia, X., Wang, S., and Yan, J. Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2). arXiv preprint arXiv:2402.16720, 2024

    2024 arXiv

  29. [38]

    Face detection based on receptive field enhanced multi-task cascaded convolutional neural networks

    Li, X., Yang, Z., and Wu, H. Face detection based on receptive field enhanced multi-task cascaded convolutional neural networks. IEEE access, 8: 0 174922--174930, 2020

    2020

  30. [39]

    Rho-1: Not all tokens are what you need

    Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., et al. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965, 2024

    2024 arXiv

  31. [40]

    Best practices and lessons learned on synthetic data for language models

    Liu, R., Wei, J., Liu, F., Si, C., Zhang, Y., Rao, J., Zheng, S., Peng, D., Yang, D., Zhou, D., et al. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503, 2024

    2024 arXiv

  32. [41]

    J., and Liu, J

    Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sampling improves preference optimization. ArXiv, abs/2309.06657, 2023. URL https://api.semanticscholar.org/CorpusID:261705578

    2023 arXiv

  33. [42]

    W., Tay, Y., Zhou, D., Le, Q

    Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023

    2023 arXiv

  34. [43]

    A., Richardson, K., and Dodge, J

    Magnusson, I., Bhagia, A., Hofmann, V., Soldaini, L., Jha, A., Tafjord, O., Schwenk, D., Walsh, P., Elazar, Y., Lo, K., Groeneveld, D., Beltagy, I., Hajishirzi, H., Smith, N. A., Richardson, K., and Dodge, J. Paloma: A benchmark for evaluating language model fit. ArXiv, abs/23...

    2023 arXiv

  35. [44]

    H., Grangier, D., Zhang, Y., and Jaitly, N

    Maini, P., Seto, S., Bai, R. H., Grangier, D., Zhang, Y., and Jaitly, N. Rephrasing the web: A recipe for compute and data-efficient language modeling. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://api.semanticscholar.org/CorpusID:267312030

    2024

  36. [45]

    A., Juarez, M., and Sarkar, R

    Mart \' nez, G., Watson, L., Reviriego, P., Hern \'a ndez, J. A., Juarez, M., and Sarkar, R. Towards understanding the interplay of generative artificial intelligence and the internet. In International Workshop on Epistemic Uncertainty in Artificial Intelligence, pp.\ 59--73. ...

    2023

  37. [46]

    M., Razzak, M

    Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., H \"o ltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learnin...

    2022

  38. [47]

    Self-distillation amplifies regularization in hilbert space

    Mobahi, H., Farajtabar, M., and Bartlett, P. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33: 0 3351--3361, 2020

    2020

  39. [48]

    Lightzero: A unified benchmark for monte carlo tree search in general sequential decision scenarios

    Niu, Y., Pu, Y., Yang, Z., Li, X., Zhou, T., Ren, J., Hu, S., Li, H., and Liu, Y. Lightzero: A unified benchmark for monte carlo tree search in general sequential decision scenarios. Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [49]

    Unizero: Generalized and efficient planning with scalable latent world models

    Pu, Y., Niu, Y., Yang, Z., Ren, J., Li, H., and Liu, Y. Unizero: Generalized and efficient planning with scalable latent world models. arXiv preprint arXiv:2406.10667, 2024

    2024 arXiv

  41. [50]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019

    2019

  42. [51]

    High-resolution image synthesis with latent diffusion models, 2021

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, 2021

    2021

  43. [52]

    Principles of Mathematical Analysis

    Rudin, W. Principles of Mathematical Analysis. McGraw-Hill, New York, 3rd edition, 1976

    1976

  44. [53]

    Seddik, M. E. A., Chen, S.-W., Hayou, S., Youssef, P., and Debbah, M. How bad is training on synthetic data? a statistical analysis of language model collapse. arXiv preprint arXiv:2404.05090, 2024

    2024 arXiv

  45. [54]

    Ai models collapse when trained on recursively generated data

    Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., and Gal, Y. Ai models collapse when trained on recursively generated data. Nature, 631 0 (8022): 0 755--759, 2024

    2024

  46. [55]

    D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P

    Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023

    2023 arXiv

  47. [56]

    H., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M

    Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A. H., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravich...

    2024 arXiv

  48. [57]

    Large language models for data annotation and synthesis: A survey

    Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhattacharjee, A., Karami, M., Li, J., Cheng, L., and Liu, H. Large language models for data annotation and synthesis: A survey. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empiri...

    2024 doi

  49. [58]

    Solving olympiad geometry without human demonstrations

    Trinh, T., Wu, Y., Le, Q., He, H., and Luong, T. Solving olympiad geometry without human demonstrations. Nature, 2024. doi:10.1038/s41586-023-06747-5

    2024 doi

  50. [59]

    Bootstrapping llm-based task-oriented dialogue agents via self-talk

    Ulmer, D., Mansimov, E., Lin, K., Sun, J., Gao, X., and Zhang, Y. Bootstrapping llm-based task-oriented dialogue agents via self-talk. arXiv preprint arXiv:2401.05033, 2024

    2024 arXiv

  51. [60]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022

    2022 arXiv

  52. [61]

    H., Xia, F., Le, Q., and Zhou, D

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., hsin Chi, E. H., Xia, F., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903, 2022. URL https://api.semanticscholar.org/CorpusID:246411621

    2022 arXiv

  53. [62]

    Less: Selecting influential data for targeted instruction tuning

    Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: Selecting influential data for targeted instruction tuning. ArXiv, abs/2402.04333, 2024. URL https://api.semanticscholar.org/CorpusID:267522839

    2024 arXiv

  54. [63]

    M., Santurkar, S., Ma, T., and Liang, P

    Xie, S. M., Santurkar, S., Ma, T., and Liang, P. S. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023

    2023

  55. [64]

    Llm4drive: A survey of large language models for autonomous driving

    Yang, Z., Jia, X., Li, H., and Yan, J. Llm4drive: A survey of large language models for autonomous driving. arXiv e-prints, pp.\ arXiv--2311, 2023

    2023

  56. [65]

    Ultramedical: Building specialized generalists in biomedicine

    Zhang, K., Zeng, S., Hua, E., Ding, N., Chen, Z.-R., Ma, Z., Li, H., Cui, G., Qi, B., Zhu, X., et al. Ultramedical: Building specialized generalists in biomedicine. arXiv preprint arXiv:2406.03949, 2024

    2024 arXiv

  57. [66]

    Gobigger: A scalable platform for cooperative-competitive multi-agent interactive simulation

    Zhang, M., Zhang, S., Yang, Z., Chen, L., Zheng, J., Yang, C., Li, C., Zhou, H., Niu, Y., and Liu, Y. Gobigger: A scalable platform for cooperative-competitive multi-agent interactive simulation. In The Eleventh International Conference on Learning Representations, 2023

    2023

  58. [67]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., and Ma, Y. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Th...

    2024 arXiv

  59. [68]

    Storytrans: Non-parallel story author-style transfer with discourse representations and content enhancing

    Zhu, X., Guan, J., Huang, M., and Liu, J. Storytrans: Non-parallel story author-style transfer with discourse representations and content enhancing. arXiv preprint arXiv:2208.13423, 2022

    2022 arXiv

  60. [69]

    Critical data size of language models from a grokking perspective

    Zhu, X., Fu, Y., Zhou, B., and Lin, Z. Critical data size of language models from a grokking perspective. arXiv preprint arXiv:2401.10463, 2024 a

    2024 arXiv

  61. [70]

    P a D : Program-aided distillation can teach small models reasoning better than chain-of-thought fine-tuning

    Zhu, X., Qi, B., Zhang, K., Long, X., Lin, Z., and Zhou, B. P a D : Program-aided distillation can teach small models reasoning better than chain-of-thought fine-tuning. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Cha...

    2024 doi

  62. [71]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.