Pith. sign in

REVIEW 9 cited by

Strong Model Collapse

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.04840 v2 pith:6FR5EJAU submitted 2024-10-07 cs.LG stat.ML

Elvis Dohmatob , Yunzhen Feng , Arjun Subramonian , Julia Kempe This is my paper
classification cs.LGstat.ML
keywords collapsemodeltraininglargermodelslargenetworksneural
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the existance of a strong form of the model collapse phenomenon, a critical performance degradation due to synthetic data in the training corpus. Our results show that even the smallest fraction of synthetic data (e.g., as little as 1\% of the total training dataset) can still lead to model collapse: larger and larger training sets do not enhance performance. We further investigate whether increasing model size, an approach aligned with current trends in training large language models, exacerbates or mitigates model collapse. In a simplified regime where neural networks are approximated via random projections of tunable size, we both theoretically and empirically show that larger models can amplify model collapse. Interestingly, our theory also indicates that, beyond the interpolation threshold (which can be extremely high for very large datasets), larger models may mitigate the collapse, although they do not entirely prevent it. Our theoretical findings are empirically verified through experiments on language models and feed-forward neural networks for images.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibration

    cs.LG 2026-07 conditional novelty 6.0 of 10

    In adaptive OOD detection, bank impurity follows a mean-field urn law whose kernel slope acts as a reproduction number; a frozen-reserve gate removes the supercritical collapse, and a two-world theorem caps label-free...

  2. Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A new method, Partial Model Collapse, iteratively fine-tunes an LLM on its own self-generated responses to conditionally collapse its output distribution on forget queries, removing private answers without the true la...

  3. What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Moderately diverse LLM-generated data can improve fine-tuned model performance in low-data settings when distribution shift is minimal, while high diversity or large distribution shift hurts.

  4. Epistemic diversity across language models mitigates knowledge collapse

    cs.LG 2025-12 reject novelty 5.0 of 10

    In repeated self-training loops on Wikitext2, ecosystems of four small language models show lower average perplexity than one, two, or sixteen models, but the paper's broader claims about monotonic optima, robustness,...

  5. Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillation

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    SKD-CAG erases adversarial text triggers from diffusion models by distilling the model's own clean outputs through cross-attention guidance, claiming 100% and 93% removal for pixel and style backdoors.

  6. Benchmarking and Rethinking Knowledge Editing for Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Under autoregressive and sequential editing, parameter-based knowledge editing methods perform poorly, while the retrieval-based SCR baseline consistently outperforms them across datasets and models.

  7. Generating on Generated: An Approach Towards Self-Evolving Diffusion Models

    cs.CV 2025-02 conditional novelty 5.0 of 10

    A recursive self-training loop that filters prompts, selects preferred images, and reweights out-of-distribution samples improves Stable Diffusion models over multiple rounds.

  8. Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Training on LLM text embeddings can cause tabular classifiers to collapse to single-class predictions, which spuriously inflates Accuracy-on-the-Line correlations.

  9. LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Under a shared retrieval-augmented memory, multiple LLMs' outputs converge to near-identical semantic answers, and the analogous Gaussian mixture system is proven to collapse.

Pith tools