REVIEW 12 cited by
Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLM) are already widely used to generate content for a variety of online platforms. As we are not able to safely distinguish LLM-generated content from human-produced content, LLM-generated content is used to train the next generation of LLMs, giving rise to a self-consuming training loop. From the image generation domain we know that such a self-consuming training loop reduces both quality and diversity of images finally ending in a model collapse. However, it is unclear whether this alarming effect can also be observed for LLMs. Therefore, we present the first study investigating the self-consuming training loop for LLMs. Further, we propose a novel method based on logic expressions that allows us to unambiguously verify the correctness of LLM-generated content, which is difficult for natural language text. We find that the self-consuming training loop produces correct outputs, however, the output declines in its diversity depending on the proportion of the used generated data. Fresh data can slow down this decline, but not stop it. Given these concerning results, we encourage researchers to study methods to negate this process.
Forward citations
Cited by 12 Pith papers
-
Lost in Retraining: Roaming the Parameter Space of Exponential Families Under Closed-Loop Learning
Repeated maximum-likelihood retraining on self-generated data collapses exponential-family models, but a single fresh external data point or a prior stabilizes them.
-
Model Misalignment and Language Change: Traces of AI-Associated Language in Unscripted Spoken English
After ChatGPT's release, science and tech podcast speakers used AI-associated words like 'surpass' and 'align' more often, while control synonyms showed no average shift.
-
Exploring the Structure of AI-Induced Language Change in Scientific English
In PubMed abstracts, AI-associated 'spiking' words rise together with their synonyms rather than replacing them, and declining words show less systematic, more organic patterns.
-
What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning
Moderately diverse LLM-generated data can improve fine-tuned model performance in low-data settings when distribution shift is minimal, while high diversity or large distribution shift hurts.
-
Transformer Semantic Genetic Programming for Symbolic Regression
A transformer trained on synthetic function pairs with similar behavior can act as a semantic variation operator for genetic programming, improving symbolic regression accuracy and solution size.
-
Are We in the AI-Generated Text World Already? Quantifying and Monitoring AIGT on Social Media
A detector trained on a new multi-LLM social media benchmark estimates that AI-generated text on Medium and Quora grew from about 2% to roughly 37 to 39 percent between 2022 and 2024, while Reddit stayed near 2%.
-
How to Synthesize Text Data without Model Collapse?
Token-level editing of human text with a high-confidence threshold yields modest performance gains over the original data and avoids the sharp degradation seen with purely synthetic data.
-
Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models
A formal three-step corpus method identifies 21 words whose recent spike in scientific abstracts tracks ChatGPT overuse, and tests of training data, architecture, and human feedback leave RLHF as a plausible but unpro...
-
Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
A controlled, multi-family study shows the relative generation-verification gap grows with pretraining flops for stable verification methods, and iterative self-improvement saturates quickly.
-
The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
AI safety should be measured by whether deployed systems keep errors visible, contestable, containable, and recoverable across five integrity layers, not only by whether individual model outputs look safe.
-
Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
Iterative self-training on a model's own correct outputs, with simple length and voting filters, lets transformers generalize to far longer arithmetic and path-finding problems than they saw in training.
-
Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set
The paper's proof of inevitable language-model collapse reduces to a definitional identity, since the error terms whose accumulation drives the result are chosen to fit the model outputs rather than derived from train...
Discussion (0). Continue with ORCID to comment.