Pith. sign in

REVIEW 1 major objections 5 minor 37 references

Lost in Retraining: Roaming the Parameter Space of Exponential Families Under Closed-Loop Learning

T0 review · 1 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that closed-loop maximum-likelihood training of exponential-family models on their own data drives the model to collapse onto absorbing states, and that a single external data point or a prior can prevent that collapse.

desk verdict A genuinely new and mostly right framework for closed-loop learning in exponential families, but the main theorem overreaches and two continuous-time illustrations contain fixable errors. read the letter →

arxiv 2506.20623 v2 pith:LFYA4KSG submitted 2025-06-25 cs.LG cond-mat.dis-nnphysics.data-anstat.ML

Fariba Jangjoo , Matteo Marsili , Yasser Roudi This is my paper
classification cs.LGcond-mat.dis-nnphysics.data-anstat.ML
keywords closed-looplearningmodelcollapseexponentialfamilymartingaleFokker-Planckequationmaximumlikelihoodestimationaposterioriself-consuminggenerativemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies what happens when a statistical model is repeatedly fitted to data it generated itself, a closed loop that becomes relevant as AI-generated text and images increasingly fill training corpora. For models in the exponential family, it argues that maximum-likelihood estimation turns the empirical sufficient statistics into a martingale: the expected value of the next generation's statistic equals this generation's value. Because bounded martingales tend to hit their extremes, the authors conclude that the model's output distribution collapses onto absorbing states that amplify whatever bias the initial data contained. The paper then shows collapse is not inevitable: a Bayesian prior, regularization, or even a single data point drawn from an external ground-truth distribution makes the continuous-time dynamics possess a normalizable stationary distribution, so diversity is retained. This is the paper's central message: in closed-loop learning, the usual large-sample equivalence between maximum likelihood and regularized estimation breaks down.

What carries the argument

The machinery is the martingale property of the sufficient statistics, Eq. (5), combined with the Fokker-Planck equation of the continuous-time limit. A martingale is a process whose conditional expectation of the next value equals the current value; the paper shows maximum likelihood imposes this on the empirical averages $\bar\phi_a(\hat s(t))$ because ML matching, Eq. (4), makes the model mean equal the observed mean. In the large-$M$ limit the parameter moves by a Gaussian step whose covariance is the inverse Fisher information matrix $J^{-1}(\theta)$, and the drift is governed by $U(\theta)=\tfrac12\log\det J - u\log p_0(\theta)+mD_{\rm KL}(q\|\theta)$; integrating the zero-current condition of the associated Fokker-Planck equation gives the stationary density $p_{\rm st}(\theta)=c\,p_0(\theta)^{2u}e^{-2mD_{\rm KL}(q\|\theta)}$. Normalizability of this density is the criterion that separates collapse ($u=m=0$) from survival.

What would settle it

Run the closed-loop ML dynamics for an exponential family with a continuous bounded sufficient statistic, such as $f(s|\theta)\propto e^{\theta s}$ on $s\in[-1,1]$, from many initial parameters with large $M$; the paper predicts the empirical statistic eventually reaches $\pm1$ with probability linear in the initial statistic. A trajectory whose empirical statistic converges to an interior value, or a distribution of final values deviating from that linear absorption probability, would contradict the generality of the collapse claim.

Watch

Extended reading notes

Core claim

The central claim is that closed-loop learning in exponential families is generically unstable under maximum likelihood. The empirical averages of the sufficient statistics $\bar\phi_a(\hat s(t))$ satisfy the martingale condition $\langle \bar\phi_{a,t+1}\mid\bar\phi_{a,t}\rangle_{\theta_t}=\bar\phi_{a,t}$, so by a theorem on bounded martingale Markov chains they converge to absorbing states; the limiting distribution $f(s|\theta_\infty)$ is a delta-function on the values of $\phi_a(s)$, meaning the model generates only the few configurations consistent with a frozen statistic. In the complementary continuous-time description, the parameters follow a Langevin equation whose drift and diffusion are controlled by the Fisher information matrix, and the stationary density is $p_{\rm st}(\theta)\propto p_0(\theta)^{2u} e^{-2m D_{\rm KL}(q\|\theta)}$. For pure maximum likelihood with no external data ($u=m=0$) this stationary density is not normalizable and the entropy falls as $\langle S\rangle_{\phi_\tau} = \langle S\rangle_{\phi_0} - \tau D$, formalizing collapse; with $u=1$ or $m\ge1$ the density can become normalizable and collapse is avoided, even in the $M\to\infty$ limit where such perturbations would be negligible in ordinary learning.

Load-bearing premise

The central conclusion depends on the assumption that a quantity whose expected next value equals its current value and which stays within fixed bounds must eventually settle at one of the two extreme values; the appendix proves this only for discrete-valued quantities, not for continuous ones.

Editorial extensions

If this is right

  • Closed-loop ML training of an exponential family on purely self-generated data drives sufficient statistics to absorbing values, so the model's output diversity shrinks to configurations compatible with the frozen statistic.
  • The probability of absorbing at the upper extreme is linear in the initial statistic, $(\bar\phi_{1,0}-\phi^-)/(\phi^+-\phi^-)$, so small initial biases are amplified into complete polarization.
  • Adding one external ground-truth sample per generation ($m=1$), or switching from ML to MAP or regularized estimation ($u=1$), can render the stationary density normalizable and prevent collapse, with no vanishing effect as $M\to\infty$.
  • For the Ising model with zero coupling, adding a single external magnetization sample produces a finite stationary distribution for the field; for the Poisson model, the stationary mean becomes a Gamma distribution with shape $2m\vartheta_0$.
  • Because the entropy decreases at rate $D$ (the number of parameters) under pure ML, higher-dimensional exponential families collapse faster.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same martingale mechanism operates in over-parameterized neural networks, initial algorithmic and architectural biases would be amplified rather than corrected by self-training, making the choice of prior or regularizer a matter of which bias to conserve rather than a small correction.
  • The continuous-time analysis predicts a measurable signature: along a sufficient-statistic direction under pure ML, the variance of model-generated samples should grow diffusively with no drift before snapping to the boundary; this could be tested in synthetic exponential-family experiments before studying deep networks.
  • The one-external-sample result suggests a minimal 'data anchor' protocol for generative model training, but it relies on the external sample being drawn from the same fixed ground-truth distribution $q$ at every generation; if the external stream drifts, the stationary distribution would inherit that drift.
  • The proof gap for continuous sufficient statistics matters in practice: continuous bounded statistics are common, and if such a martingale can converge to an interior value, collapse would be less universal than stated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper studies closed-loop learning, i.e., repeated maximum-likelihood (or MAP) retraining of an exponential-family model on data generated in part by the model itself. The authors show that when inference is by ML with no external data (u = m = 0), the empirical sufficient statistics form a martingale (Eq. 5), and they argue that this implies convergence to absorbing states and hence model collapse (Eq. 6). They then derive a continuous-time diffusion approximation (Eqs. 14–15) and, for the general case with a prior (u = 1) or a fixed number m of external ground-truth samples, the stationary distribution p_st(θ) ∝ p0(θ)^{2u} exp(−2m D_KL(q|θ)) (Eq. 18). The results are illustrated on Ising, REM, and Poisson models, with numerical simulations confirming the stationary distribution.

Significance. If the central claims hold, the paper provides an analytically tractable mechanism for model collapse in closed-loop learning: ML estimation makes sufficient statistics a martingale, and absorption at the boundary explains the loss of diversity. The stationary-distribution result is the most valuable contribution, showing that a single external data point or a prior can act as a persistent anchor even in the M→∞ limit, a genuinely non-classical effect. The paper has clear strengths: the martingale property follows directly from the ML moment-matching equations, the Fokker-Planck calculation is explicit, and the numerical checks of Eq. (18) are convincing. The main weakness is the overgeneralization of the absorption theorem to continuous state spaces, which is load-bearing for the abstract's claim about general exponential families.

major comments (1)
  1. [Sec. "Closed-loop learning under ML" (Eq. 6) and Appendix A] The claim that a bounded martingale Markov chain converges to absorbing states is proven only for discrete-valued chains. Appendix A uses the fact that if φ_t = φ^- then any transition to φ' > φ^- would violate the martingale inequality; in a continuous state space this argument does not rule out convergence to an interior limit. For example, Pólya's urn is a bounded martingale that converges almost surely to a random interior limit, not to an absorbing state. Since the sufficient statistics of a general exponential family can be continuous, the conclusion in Eq. (6) that the dynamics converges to absorbing states is not established for continuous state spaces. The authors should either restrict the statement to models with discrete sufficient statistics or add explicit regularity conditions (e.g., Feller property plus positive conditional variance at every interior point, which would follow from positive-definiteness of the Fisher information) and then prove absorption in that setting. As written, the general claim in the abstract and Eq. (6) is unsupported.
minor comments (5)
  1. [Eq. (17)] The Ito correction in the differential of the entropy appears to be off by a factor of 1/2. From dφ_a dφ_b = J_ab dτ and ∂²S/∂φ_b∂φ_c = −J^{-1}_{bc}, the second-order term is −(1/2) Tr(J^{-1}J) dτ = −(D/2) dτ, not −D dτ. Consequently ⟨S⟩ should decay as −τ D/2, not −τ D. This does not affect the qualitative conclusion but should be corrected.
  2. [Paragraph after Eq. (17)] The displayed SDE for the Ising magnetization per spin, dµ = √(1−µ²)dW, omits a factor 1/√N. Since φ = Nµ and the diffusion coefficient for φ is √(N(1−µ²)), the per-spin variable obeys dµ = √((1−µ²)/N)dW. The subsequent relaxation rate ⟨µ²⟩ = 1 − (1−⟨µ²⟩0)e^{-τ} should likewise have exponent −τ/N. This is a concrete error in a main-text equation, though it does not change the absorption picture.
  3. [Eq. (6)] For discrete state spaces, the Dirac delta in Eq. (6) should be a Kronecker delta (or the equation should clarify that δ denotes the appropriate discrete measure). This is a notation issue, not a substantive one.
  4. [Appendix C] There are several typographical errors in the transition-probability derivation: (mϑ0)^S should be (mϑ0)^T, the second e^{−Mϑ} should be e^{−mϑ0}, and "M+n+λ" should be "M+m+λ". Also, δ(ϑ′ − S+T/M+n+λ) should read δ(ϑ′ − (S+T)/(M+m+λ)).
  5. [Throughout] Minor typos include "close-loop" in the Introduction, "phonomena" in the Discussion, "aritificial" in the Discussion, "the the" in the Discussion, and "infinite times" in Appendix A (should be "infinitely many times").

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the martingale property and stationary distribution are derived from the likelihood/prior structure, not assumed or fitted.

full rationale

The paper's load-bearing ingredients are (i) the martingale property of sufficient statistics under ML estimation and (ii) the stationary distribution p_st(theta) = c p0(theta)^{2u} exp(-2m D_KL(q|theta)). Neither reduces to an assumed conclusion. Eq. (5) follows directly from the ML moment-matching equations Eq. (4): the predictive expectation of the empirical sufficient statistics conditional on theta_t equals the empirical sufficient statistics that defined theta_t. This is a property of maximum likelihood, not an input. The stationary state, Eq. (18), is obtained from an explicit Fokker-Planck calculation in Appendix B with the zero-current condition, using the drift and diffusion terms derived from the likelihood, prior, and external-data KL divergence. The analytic predictions for the Ising and Poisson models are compared with simulations (Figs. 2 and 3) as independent numerical checks, not used to determine constants or fitted parameters. There is no fitted-input-called-prediction step: no parameter is calibrated on a subset of data and then reported as a prediction. Self-citations are absent in the load-bearing argument; the martingale definition is cited to an external review, and the absorption argument cites Feller, not the authors' own prior work. One genuine concern is noted in the manuscript itself: in Appendix A the absorption statement is proved only for a discrete-valued chain ('For a single variable phi taking discrete values in [phi^-, phi^+]'), while the main text applies it to continuous bounded sufficient statistics. This is a proof gap or correctness risk, not circularity: the conclusion does not equal its input by construction. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper adds no fitted parameters. All computed quantities follow from the model definition. The key external inputs are the distribution q and prior p0, treated as given. The main assumptions are bounded sufficient statistics, fresh external samples, and large-M asymptotics.

assumptions (5)
  • domain assumption A bounded Markov-chain martingale converges to absorbing states.
    Invoked to prove collapse after Eq. (5). Appendix A only demonstrates it for discrete state spaces, so for continuous sufficient statistics this premise is unproven.
  • domain assumption The sufficient statistics are bounded functions on the sample space.
    Stated after Eq. (1); bounds the martingale and is needed for the absorption argument. Models with unbounded sufficient statistics (e.g., Gaussian means) are outside the collapse proof.
  • domain assumption External data points are drawn independently from a fixed distribution q at every iteration, with constant m.
    Step 1 of the closed-loop dynamics. The remedy depends on the external points being refreshed each generation, so their influence persists as M tends to infinity.
  • domain assumption The Fisher information matrix is invertible and M is large enough for Laplace/saddle-point expansions.
    Appendix B. The authors explicitly mention singular models as future work in the Discussion.
  • standard math Ito calculus and Fokker-Planck equations apply to the continuous-time limit.
    Used to derive Eqs. (14)-(15) and the stationary distribution Eq. (18).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lost in Retraining: Roaming the Parameter Space of Exponential Families Under Closed-Loop Learning." pith.science (2026). https://pith.science/paper/LFYA4KSG

@misc{pith2026250620623,
  author       = {Pith},
  title        = {Pith review of: Lost in Retraining: Roaming the Parameter Space of Exponential Families Under Closed-Loop Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LFYA4KSG}},
  note         = {Machine review of arXiv:2506.20623}
}
read the original abstract

Closed-loop learning is the process of repeatedly estimating a model from data generated from the model itself. It is receiving great attention due to the possibility that large neural network models may, in the future, be primarily trained with data generated by artificial neural networks themselves. We study this process for models that belong to exponential families, deriving equations of motions that govern the dynamics of the parameters. We show that maximum likelihood estimation of the parameters endows sufficient statistics with the martingale property and that as a result the process converges to absorbing states that amplify initial biases present in the data. However, we show that this outcome may be prevented if the data contains at least one data point generated from a ground truth model, by relying on maximum a posteriori estimation or by introducing regularisation.

Figures

Figures reproduced from arXiv: 2506.20623 by the authors.

Figure 1
Figure 1. FIG. 1. Closed-loop dynamics and model collapse for [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. FIG. 2 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. FIG. 3 [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Note that with m = 0 the distribution is not normalizable, i.e. ϑ = 0 is an absorbing state for all λ > 0 when m = 0. Finally we note that when the Poisson distribution is expressed in terms of the canonical parameter θ = log⟨s⟩θ, Eq. (B33) leads to a stationary state …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 33 canonical work pages

  1. [1]

    Alemohammad et al

    S. Alemohammad et al. , Self-consuming generative models go MAD, in International Con- ference on Learning Representations (ICLR) , 2024

    work page 2024

  2. [2]

    Bertrand, A

    Q. Bertrand, A. J. Bose, A. Duplessis, M. Jiralerspong, and G. Gidel, On the stability of 11 iterative retraining of generative models on their own data, in International Conference on Learning Representations (ICLR), 2024

    work page 2024

  3. [3]

    Marchi, S

    M. Marchi, S. Soatto, P. Chaudhari, and P. Tabuada, Heat death of generative models in closed-loop learning, in 2024 IEEE 63rd Conference on Decision and Control (CDC) , pp. 1524–1530, 2024

    work page 2024

  4. [4]

    Shumailov et al

    I. Shumailov et al. , Nature 631, 755 (2024)

    work page 2024

  5. [5]

    Y. Guo, G. Shang, M. Vazirgiannis, and C. Clavel, The curious decline of linguistic diversity: Training language models on synthetic text, in Findings of the Association for Computational Linguistics: NAACL 2024 , edited by K. Duh, H. Gomez, and S. Bethard, pp. 3589–3604, 2024

    work page 2024

  6. [6]

    Briesch, D

    M. Briesch, D. Sobania, and F. Rothlauf, Large language models suffer from their own output: An analysis of the self-consuming training loop, 2024, 2311.16822

    arXiv 2024

  7. [7]

    Dohmatob, Y

    E. Dohmatob, Y. Feng, and J. Kempe, Model collapse demystified: The case of regression, in Advances in Neural Information Processing Systems, edited by A. Globerson et al., volume 37, pp. 46979–47013, Curran Associates, Inc., 2024

    work page 2024

  8. [8]

    H. A. Vu, G. Reeves, and E. Wenger, arXiv preprint arXiv:2505.21677 (2025)

    arXiv 2025

Show all 37 references
  1. [9]

    Villalobos et al

    P. Villalobos et al. , arXiv preprint arXiv:2211.04325 3 (2024)

    2024 arXiv

  2. [10]

    Thanh-Tung and T

    H. Thanh-Tung and T. Tran, Catastrophic forgetting and mode collapse in gans, in 2020 international joint conference on neural networks (ijcnn) , pp. 1–10, IEEE, 2020

    2020

  3. [11]

    Biroli, T

    G. Biroli, T. Bonnaire, V. De Bortoli, and M. M´ ezard, Nature Communications 15, 9957 (2024)

    2024

  4. [12]

    E. M. Witzel, The origins of the world’s mythologies (Oxford University Press, 2012)

    2012

  5. [13]

    Bikhchandani, D

    S. Bikhchandani, D. Hirshleifer, and I. Welch, Journal of political Economy 100, 992 (1992)

    1992

  6. [14]

    C. M. Bishop and H. Bishop, Deep learning: Foundations and concepts (Springer Nature, 2023)

    2023

  7. [15]

    Casella and R

    G. Casella and R. Berger, Statistical inference (CRC press, 2024)

    2024

  8. [16]

    Tibshirani, Journal of the Royal Statistical Society Series B: Statistical Methodology 58, 267 (1996)

    R. Tibshirani, Journal of the Royal Statistical Society Series B: Statistical Methodology 58, 267 (1996)

    1996

  9. [17]

    Rold´ anet al

    ´E. Rold´ anet al. , Advances in Physics 72, 1 (2023)

    2023

  10. [18]

    Feller, An introduction to probability theory and its applications, Vol

    W. Feller, An introduction to probability theory and its applications, Vol. 1 (John Wiley & Sons, New York, 1968). 12

    1968

  11. [19]

    Derrida, Physical Review B 24, 2613 (1981)

    B. Derrida, Physical Review B 24, 2613 (1981)

    1981

  12. [20]

    D. R. Cox and D. V. Hinkley, Theoretical statistics (CRC Press, 1979)

    1979

  13. [21]

    C. W. Gardiner, Handbook of Stochastic Methods for Physics, Chemistry, and the Natural Sciences, Springer Series in Synergetics, 3rd ed. (Springer, Berlin, Heidelberg, 2009)

    2009

  14. [22]

    Amari, Information geometry and its applications volume 194 (Springer, 2016)

    S.-i. Amari, Information geometry and its applications volume 194 (Springer, 2016)

    2016

  15. [23]

    J. C. Cox et al. , Econometrica 53, 385 (1985)

    1985

  16. [24]

    P. L. Bartlett, A. Montanari, and A. Rakhlin, Acta numerica 30, 87 (2021)

    2021

  17. [25]

    Rahaman et al

    N. Rahaman et al. , On the spectral bias of neural networks, in International conference on machine learning, pp. 5301–5310, PMLR, 2019

    2019

  18. [26]

    Francazi, A

    E. Francazi, A. Lucchi, and M. Baity-Jesi, arXiv preprint arXiv:2306.00809 (2023)

    2023 arXiv

  19. [27]

    A. Jain, R. Nobahari, A. Baratin, and S. Sarao Mannelli, Advances in Neural Information Processing Systems 37, 24435 (2024)

    2024

  20. [28]

    Hayase and R

    T. Hayase and R. Karakida, The spectrum of fisher information of deep networks achieving dynamical isometry, in International Conference on Artificial Intelligence and Statistics , pp. 334–342, PMLR, 2021

    2021

  21. [29]

    W. Cao, A. Dytso, M. Fauß, H. V. Poor, and G. Feng, arXiv preprint arXiv:2005.03622 (2020)

    2020 arXiv

  22. [30]

    Watanabe, The Journal of Machine Learning Research 14, 867 (2013)

    S. Watanabe, The Journal of Machine Learning Research 14, 867 (2013)

    2013

  23. [31]

    C. L. Baker, R. Saxe, and J. B. Tenenbaum, Cognition 113, 329 (2009)

    2009

  24. [32]

    Najar, E

    A. Najar, E. Bonnet, B. Bahrami, and S. Palminteri, PLoS biology 18, e3001028 (2020)

    2020

  25. [33]

    C. J. Charpentier, K. Iigaya, and J. P. O’Doherty, Neuron 106, 687 (2020)

    2020

  26. [34]

    Boyer, Religion Explained: The Evolutionary Origins of Religious Thought (Basic Books, New York, 2001)

    P. Boyer, Religion Explained: The Evolutionary Origins of Religious Thought (Basic Books, New York, 2001)

    2001

  27. [35]

    F. J. Martini, T. Guillam´ on-Vivancos, V. Moreno-Juan, M. Valdeolmillos, and G. L´ opez- Bendito, Neuron 109, 2519 (2021)

    2021

  28. [36]

    R. O. Wong, Annual review of neuroscience 22, 29 (1999). Appendix A: Absorption probabilities in martingales In the dynamical process described in the main text, ˆs(t) determines ¯ϕa(ˆs(t)) and thus determines ˆθt and the distribution p(·| ˆθt). Consequently, given ˆs(t) and t...

    1999

  29. [37]

    The stationary state As discussed in Gardiner [21], the Fokker-Planck equation associated to the stochastic differential equation (14) in the Letter can be written as ∂p ∂t = X a ∂Fa ∂θa (B22) where the probability current F is given by Fa = −Aap + 1 2 X b ∂ ∂θb J −1 a,b p = −...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.