In repeated self-training loops on Wikitext2, ecosystems of four small language models show lower average perplexity than one, two, or sixteen models, but the paper's broader claims about monotonic optima, robustness, and scaling are not supported by the reported experiments.
LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
The increasing use of synthetic data from the public Internet has enhanced data usage efficiency in large language model (LLM) training. However, the potential threat of model collapse remains insufficiently explored. Existing studies primarily examine model collapse in a single model setting or rely solely on statistical surrogates. In this work, we introduce LLM Web Dynamics (LWD), an efficient framework for investigating model collapse at the network level. By simulating the Internet with a retrieval-augmented generation (RAG) database, we analyze the convergence pattern of model outputs. Furthermore, we provide theoretical guarantees for this convergence by drawing an analogy to interacting Gaussian Mixture Models.
fields
cs.LG 1years
2025 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
Epistemic diversity across language models mitigates knowledge collapse
In repeated self-training loops on Wikitext2, ecosystems of four small language models show lower average perplexity than one, two, or sixteen models, but the paper's broader claims about monotonic optima, robustness, and scaling are not supported by the reported experiments.