Pith. sign in

REVIEW 3 major objections 4 minor 43 references

Self-Consuming Generative Models with Adversarially Curated Data

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The central claim is that the fate of a self-consuming generative model under adversarial data curation is decided by the sign of the covariance between the true user reward and the adversarial reward: positive covariance means the loop…

desk verdict New and practical attack framing, but the central theoretical lemmas are false: Lemma 3.3's covariance bound uses the wrong extremum and a two-point example violates it, and Lemma 3.4's geometric sum is also off. read the letter →

arxiv 2505.09768 v1 pith:6OXUEVBR submitted 2025-05-14 cs.LG

Xiukun Wei , Xueru Zhang This is my paper · ORCID
classification cs.LG
keywords self-consuminggenerativemodelsadversarialdatacurationpreferencealignmentrewardcovariancelabelflippingattackpoisoningdiffusiongeneralizedBradley-Terrymodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper extends the study of self-consuming generative models to settings where the data used for retraining is curated by adversarial users with a limited budget, rather than only by genuine user preferences. Its central theoretical result is a covariance condition: the long-run reward experienced by users increases when the adversarial reward is non-negatively correlated with the true user reward, and can collapse when the correlation is negative. The paper also shows that mixing real data into training is not by itself a defense against adversarial curation. Building on this condition, it proposes attack algorithms that strategically flip preference labels to force the covariance negative, and validates them on synthetic data and diffusion models over image datasets.

What carries the argument

The load-bearing object is the covariance $\operatorname{Cov}_{p_t}\left(e^{r(x)}, e^{\tilde r_t(x)}\right)$ between the exponentialized true user reward and exponentialized adversarial reward at the current model distribution. In words, it measures whether malicious curation tends to pick the same samples that genuine users prefer. It appears in Lemma 3.3: the lower bound on the next expected reward is $\mathbb{E}_{p_t}[e^r] + (1-\phi_t)\frac{K-1}{K}\frac{\operatorname{Var}_{p_t}[e^r]}{e^{r_{t,\max}}} + \phi_t\frac{K-1}{K}\frac{\operatorname{Cov}_{p_t}[e^r, e^{\tilde r_t}]}{e^{\tilde r_{t,\min}}}$. The sign of the covariance, multiplied by the malicious-user fraction $\phi_t$ and the sample-selection factor $(K-1)/K$, determines whether the expected reward is pushed up by the benign variance term or dragged down by the adversarial term. The argument carries through the closed-form update $p_{t+1}(x)=p_t(x)\left[(1-\phi_t)\frac{e^{r(x)}}{\mathbb{E}_{p_t}[e^r]}+\phi_t\frac{e^{\tilde r_t(x)}}{\mathbb{E}_{p_t}[e^{\tilde r_t}]}\right]$ that arises in the $K\to\infty$ limit, turning the self-consuming loop into a multiplicative reweighting step whose drift is read off from this covariance.

What would settle it

Fix a retraining loop on CIFAR-10 with an adversarial reward function chosen so that $\operatorname{Cov}_{p_t}(e^r, e^{\tilde r})<0$ at every round and a 20% malicious fraction, as in the paper's experiments. If the measured $\mathbb{E}_{p_t}[e^{r(x)}]$ stays at or above its initial level for ten rounds, the Lemma 3.3 sign criterion is violated. Conversely, a run with positive covariance in which the expected reward falls would falsify the robustness direction.

Watch

Extended reading notes

Core claim

The paper argues that in a self-consuming retraining loop where each generation's synthetic data is curated by a mixture of benign and malicious users, the expected user reward from the model obeys a bound with two correction terms. Under the generalized Bradley-Terry curation model, the change in $\mathbb{E}_{p_t}[e^{r(x)}]$ between rounds is controlled by the variance of the true reward and by $\operatorname{Cov}_{p_t}[e^{r(x)}, e^{\tilde r_t(x)}]$, the covariance between the true reward and the adversarial reward evaluated on the current model. As long as that covariance is non-negative, the expected reward grows toward its maximum and the model aligns with user preferences despite the attackers; when it is negative, the adversarial term can dominate, and the expected reward can fall, oscillate, and drift away from the preference-optimizing distribution. The same covariance sign appears when real data are mixed into training: the lower bound becomes $\mathbb{E}_{p_{\text{data}}}[e^{r(x)}] + \phi^\star(\lambda+1)\min_{i\le t}\operatorname{Cov}_{p_i}[e^r, e^{\tilde r_i}]$, so fresh real data does not automatically rescue alignment. On top of this, the paper designs label-flipping attacks that make the covariance negative under a limited flip budget and shows empirically on diffusion models over CIFAR-10 and CIFAR-100 that these attacks move generation away from user-preferred classes.

Load-bearing premise

The entire theory assumes that every retraining round computes the exact maximum-likelihood distribution over all achievable models, so real models with finite capacity, optimization noise, or limited training budgets do not exactly satisfy the closed-form update; if that assumption fails, the covariance bound need not describe what actually happens.

Editorial extensions

If this is right

  • If the adversarial reward is positively correlated with user preferences, the retraining loop is robust: expected user reward keeps increasing toward its maximum even while malicious users curate a fraction of the data.
  • If the correlation is negative, the attack term can outweigh the benign variance term; expected reward decreases, oscillates, and the generated distribution drifts to low-preference samples.
  • Adding real data at any fixed ratio does not restore preference alignment once the negative-covariance condition holds; it only pulls the model toward the real data distribution, not toward the user-preferred distribution.
  • A platform with a budget-limited set of malicious users can push the covariance negative by flipping only a fraction $\kappa$ of preference labels, using the proposed gradient-based or heuristic algorithms.
  • Because the attack is realized through public curation or voting rather than backend data access, detection by outlier filtering is partially countered by a distance penalty in the attack objective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable early-warning for practitioners: platforms can monitor $\operatorname{Cov}_{p_t}(e^r, e^{\tilde r})$ in production, and a persistent drop below zero should trigger defensive action before the expected reward visibly falls.
  • The covariance criterion may transfer to human-feedback alignment pipelines outside self-consuming loops, such as preference learning with poisoned comparison data, where the same geometry between reward models could determine drift; the paper does not make this claim.
  • In finite-capacity models, the covariance sign may become a necessary but not sufficient condition, since optimization noise and capacity limits could create plateaus not predicted by Lemma 3.3; this is testable by ablating model size in the same retraining setup.
  • The result suggests a defensive design principle not explored in the paper: curate or reweight synthetic data so that the adversarial reward correlation stays non-negative, for example by filtering samples that maximize the negative covariance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies self-consuming generative models trained on data curated by a mixture of benign users and adversarial malicious users, under a generalized Bradley-Terry choice model. The authors derive recursions for the output distribution, prove bounds on the evolution of expected user reward (Lemmas 3.3 and 3.4), and use the covariance between true and adversarial rewards to characterize when the model remains aligned with user preferences. Based on this theory, they propose gradient-based and heuristic label-flipping attacks to misalign a competitor's model, and validate them on synthetic and CIFAR datasets.

Significance. If the theoretical results were correct, the paper would provide a novel characterization of adversarial data curation in self-consuming generative loops, extending the benign-curation analysis of Ferbach et al. (2024). The proposed attack algorithms address a realistic competitive scenario and the experimental results on CIFAR-10/CIFAR-100 show clear misalignment under the gradient-based attack. However, the central lemmas are false as stated, and the qualitative conclusions are not established by the given proofs. The empirical findings may still be interesting, but the paper's main theoretical contribution is not sound.

major comments (3)
  1. [Section 3, Lemma 3.3] The lower bound in Lemma 3.3 is false as stated. Consider p_t with two equiprobable points where e^r is 1 and 10 and e^{er} is 1 and 2, with ϕ=1 and K→∞. By Lemma 3.1, p_{t+1} ∝ p_t e^{er}, so the exact next expected reward is (0.5·1·1 + 0.5·10·2)/(0.5·1+0.5·2) = 10.5/1.5 = 7.0. The claimed lower bound is E_{p_t}[e^r] + Cov[e^r,e^{er}]/e^{er_min} = 5.5 + 2.25/1 = 7.75, which is violated. The proof in Appendix B.3 replaces the random denominator Σ e^{er(x_j)} by K e^{er_min} to bound B from below when the covariance is positive; for a positive ratio, the correct extremal denominator is K e^{er_max} (the largest possible denominator minimizes the ratio). This error invalidates the quantitative bound and the claimed implication that Cov ≥ 0 guarantees E_{p_{t+1}}[e^r] ≥ E_{p_t}[e^r] for finite K.
  2. [Section 3, Lemma 3.4] The geometric series in the proof of Lemma 3.4 is mis-summed. With q = λ/(1+λ), Σ_{j=1}^{t} q^j = q(1-q^t)/(1-q) = λ(1 − (λ/(1+λ))^t). The paper instead obtains ϕ*(1+λ)(1 − (λ/(1+λ))^t) Cov_min, which is too large by a factor of (1+λ)/λ. The asymptotic bound should be ϕ*λ Cov_min, not ϕ*(1+λ) Cov_min. Additionally, the proof relies on Lemma B.2, which is itself false: the same two-point example gives E_{p_{t+1}}[e^r] = 7.0 < E_{p_t}[e^r] + ϕ Cov = 7.75, contradicting Eq. (13). The error in the proof of Lemma B.2 is the step E[e^r e^{er}]/E[e^{er}] = E[e^r] + Cov, which drops the denominator E[e^{er}].
  3. [Section 3] Because Lemma 3.3 is false, the paper's central statements about robustness and vulnerability are not proven. In the K→∞, ϕ=1 special case, the exact increment is Cov[e^r,e^{er}]/E[e^{er}], so the sign of the covariance does control whether expected reward increases; however, the paper claims the stronger finite-K bound E_{p_{t+1}}[e^r] ≥ E_{p_t}[e^r] + (K−1)/K · Cov/e^{er_min}, which is not supported. Consequently, the qualitative interpretations in Section 3 and the theoretical motivation for the attack objective in Eq. (10) rest on incorrect mathematics, even though the attack algorithms may still be effective empirically.
minor comments (4)
  1. [Throughout] There are several typos and inconsistent spellings, e.g., "bengiun" (Figure 7), "CIRFAR" (Appendix C.1), and "curation" vs "curation".
  2. [Lemma 3.4] The definition of Cov_min in the main text uses i ∈ [t], but the proof of Lemma B.4 takes the minimum over i ∈ {0,1,...,t}. Please align the index sets.
  3. [Section 5 and Table 1] The experimental results, especially Table 1 and the reward curves, are reported without confidence intervals or multiple-seed statistics. Given the large variability of the random attack shown in Figure 6, reporting error bars would substantially strengthen the empirical claims.
  4. [Algorithm 1] The algorithm clips δ_i to [-1,1] while the original labels o_i are in {0,0.5,1}; the resulting o_i+δ_i may fall outside {0,0.5,1}. Please clarify how the perturbed labels are represented in the loss of Eq. (7).

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the theoretical derivation and attack validation are self-contained; the only self-citations are peripheral literature references.

full rationale

The central claim, Lemma 3.3, is derived inside the paper from the Bradley-Terry curation update in Eq. (4) together with the bounded-reward Assumption 3.2; it does not assume the covariance conclusion it derives. The proof in Appendix B.3 starts from the exact one-step expectation and bounds the variance and cross terms, so there is no fitted parameter or renamed input. Lemma 3.4 is a recursive application of the same argument with the mixed real/synthetic objective in Eq. (3), again an internal derivation rather than a reuse of the paper's own conclusions. Section 4's attack objective in Eq. (10) is motivated by the covariance term in Lemma 3.3, but the experiments in Section 5 are independent empirical checks: the reward functions and DDPM retraining are specified externally on CIFAR-10, CIFAR-100, and Gaussian data, and the attack is not fit to the measured rewards. The only self-references are peripheral: Xie and Zhang (2024) and Suya et al. (2023) are cited in related-work discussions of bias amplification and label-flipping robustness, and neither citation is used to justify Lemma 3.3, Lemma 3.4, or the attack design. The algebraic objections summarized by the skeptic concern the correctness of the bounds (for example, the use of e^{ert,min} versus e^{ert,max} and a geometric-series sum), which is a correctness risk, not circularity. Overall, no step of the derivation reduces by construction to its own inputs, so the paper receives a low circularity score.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The theoretical claim rests on standard-but-strong modeling assumptions (Bradley-Terry choice, unconstrained density updates, bounded rewards, K to infinity) and on hand-set experimental parameters (kappa, alpha, Gaussian reward constants). The main liability is not the ledger itself but the invalid bounding step in Lemma 3.3, which fails even under these assumptions.

free parameters (4)
  • kappa (flip budget) = 0.2 (CIFAR), 0.5 (Gaussian stress test)
    Flip budget as a fraction of preference pairs; chosen by hand, and the paper notes in Section 6 that the attacker may not know kappa in practice.
  • alpha (attack trade-off parameter) = not reported
    Weight of the detection-avoidance penalty dist(R_theta, R_tilde_theta) in Eq. (10); never stated in the experiments, and the distance metric d is left unspecified.
  • Gaussian reward constants (gamma, tau, mu*) = gamma = -10, tau = 3, mu* = (2,0)
    Hand-chosen for the synthetic experiments in Appendix C.2; no sensitivity analysis is given.
  • beta and iteration counts = 25k curated from 50k generated; 10 iterations
    Experimental design choices with no ablation or justification for these particular numbers.
assumptions (6)
  • domain assumption The model class P is unrestricted and each update reaches the global maximum-likelihood solution.
    Eqs. (3)-(4) are solved exactly so p_{t+1} equals the filtered density; the paper's Limitations section admits this may not hold in practice.
  • domain assumption Reward functions are bounded (Assumption 3.2).
    Required for replacing the Bradley-Terry denominators with extremal constants in Lemma 3.3; the sign-sensitive bounding step is still invalid.
  • domain assumption User choices follow the generalized Bradley-Terry model (Eqs. 1-2).
    Standard behavioral model in this line of work, but a strong assumption; all theory inherits it.
  • domain assumption K to infinity sample limit in Lemma 3.1.
    The closed-form update is derived for an infinite number of candidate samples per prompt; experiments use finite filtering (25k of 50k).
  • domain assumption The adversary has access to the target's generated samples and to an identically distributed preference dataset.
    Assumed in Section 4 so the adversary can learn R_theta and guide malicious users; stated, not argued.
  • standard math Hessian invertibility for the implicit function theorem (Eq. 11).
    Needed for the gradient-based attack; standard in influence-function methods but not verified experimentally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Consuming Generative Models with Adversarially Curated Data." pith.science (2026). https://pith.science/paper/6OXUEVBR

@misc{pith2026250509768,
  author       = {Pith},
  title        = {Pith review of: Self-Consuming Generative Models with Adversarially Curated Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6OXUEVBR}},
  note         = {Machine review of arXiv:2505.09768}
}
read the original abstract

Recent advances in generative models have made it increasingly difficult to distinguish real data from model-generated synthetic data. Using synthetic data for successive training of future model generations creates "self-consuming loops", which may lead to model collapse or training instability. Furthermore, synthetic data is often subject to human feedback and curated by users based on their preferences. Ferbach et al. (2024) recently showed that when data is curated according to user preferences, the self-consuming retraining loop drives the model to converge toward a distribution that optimizes those preferences. However, in practice, data curation is often noisy or adversarially manipulated. For example, competing platforms may recruit malicious users to adversarially curate data and disrupt rival models. In this paper, we study how generative models evolve under self-consuming retraining loops with noisy and adversarially curated data. We theoretically analyze the impact of such noisy data curation on generative models and identify conditions for the robustness of the retraining process. Building on this analysis, we design attack algorithms for competitive adversarial scenarios, where a platform with a limited budget employs malicious users to misalign a rival's model from actual user preferences. Experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithms.

Figures

Figures reproduced from arXiv: 2505.09768 by the authors.

Figure 1
Figure 1. An example of adversarial data curation in a competitive setting: the adversarial platform and the target platform serve the same population of users. The adversarial platform can access the data generated by the target platform and obtain real user preferences through its own preference collection mechanisms. Using the attack algorithm, the adversarial platform employs malicious users to adversarially curate data o… view at source ↗
Figure 2
Figure 2. The proportion of each class generated by the self-consumption model retrained with different data curation methods on CIFAR-10: benign curation based on actual user preferences (left), adversarial curation using the proposed gradient-based attack algorithm (middle), and adversarial curation via a random attack (right). The results show that the proposed gradient-based attacks are the most effective in deviating the… view at source ↗
Figure 3
Figure 3. The proportion of each ten classes generated by the self-consumption model retrained with different data curation methods on CIFAR-100: benign curation based on actual user preferences (left), adversarial curation using the proposed gradient-based attack algorithm (middle). And empirical estimate of Ept [r(x)] from samples generated by the model over iterations (right). dataset is defined as follows: 1. Synthetic Ga… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Empirical estimate of Ept [r(x)] from model-generated samples over iterations on CIFAR-10: it increases (resp. decreases) over iterations under benign (resp. adversarial) data curation. preference labels uniformly at random) algorithms to target 20% of data pairs [PIT…
Figure 5
Figure 5. Figure 5: The proportion of each class generated by the self-consumption model retrained with a mix of adversarially curated synthetic and real CIFAR-10 data. It shows that adding real data only helps the model align with the real data distribution pdata but does not defend agai…
Figure 6
Figure 6. Figure 6: Iterative retraining of the self-consumption model on the CIFAR-10 dataset with two independent experiments employing a randomized adversarial curation (20% random): (1) Left side: proportion of each class for the two independent runs. (2) Right side: average reward va…
Figure 7
Figure 7. Figure 7: Samples generated by self-consumption model with different curation on CIFAR-10 dataset:(1) Top: bengiun curation, filtered by r(x). (2) Middle: adversarial curation with 20% malicious data injected by gradient algorithm. (3) Bottom: mixed dataset created by combining …
Figure 8
Figure 8. Figure 8: The proportion of each class on at t = 1 under different attack algorithms on the same CIFAR-10 dataset. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Samples generated by self-consumption model with different curation on synthetic Gaussian dataset:(1) Top: bengiun curation, filtered by r(x). (2) Middle: adversarial curation with 20% malicious data injected by gradient algorithm. (3) Bottom: adversarial curation with…
Figure 10
Figure 10. Figure 10: Samples generated by self-consumption model on different mixed Gaussian dataset:(1) Top: adversarially curated synthetic dataset with 50% malicious data injected by gradient algorithm (sever attack). (2) Second to Bottom: mixed dataset created by combining real data w…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 27 canonical work pages

  1. [1]

    Resemble AI - AI voice generation and cloning platform

    AI, R. Resemble AI - AI voice generation and cloning platform. https://www.resemble.ai/, 2025. Accessed: 2025-01-10

    work page 2025

  2. [2]

    I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R

    Alemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A. I., Babaei, H., LeJeune, D., Siahkoohi, A., and Baraniuk, R. Self-consuming generative models go MAD . In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.net/forum?id=ShjMHfmPs0

    work page 2024

  3. [3]

    I., Agarwal, S., Collomosse, J., and Baraniuk, R

    Alemohammad, S., Humayun, A. I., Agarwal, S., Collomosse, J., and Baraniuk, R. Self-improving diffusion models with synthetic data, 2024 b . URL https://arxiv.org/abs/2408.16333

    arXiv 2024

  4. [4]

    Claude - A next-generation AI assistant by anthropic

    Anthropic. Claude - A next-generation AI assistant by anthropic. https://www.anthropic.com/index/claude-introduction, 2025. Accessed: 2025-01-10

    work page 2025

  5. [5]

    Best-of-venom: Attacking RLHF by injecting poisoned preference data

    Baumg \"a rtner, T., Gao, Y., Alon, D., and Metzler, D. Best-of-venom: Attacking RLHF by injecting poisoned preference data. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=v74mJURD1L

    work page 2024

  6. [6]

    On the stability of iterative retraining of generative models on their own data

    Bertrand, Q., Bose, J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=JORAfH2xFd

    2024

  7. [7]

    Support vector machines under adversarial label noise

    Biggio, B., Nelson, B., and Laskov, P. Support vector machines under adversarial label noise. In Proceedings of the Asian Conference on Machine Learning, volume 20, pp.\ 97--112, South Garden Hotels and Resorts, Taoyuan, Taiwain, 2011. PMLR. URL https://proceedings.mlr.press/v20/biggio11.html

    work page 2011

  8. [8]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39: 0 324--345, 1952

    work page 1952

Show all 43 references
  1. [9]

    A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramèr, F

    Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramèr, F. Poisoning web-scale training datasets is practical. In 2024 IEEE Symposium on Security and Privacy (SP), pp.\ 407--425, 2024. doi:10.1109/SP54263.2024.00179

    2024

  2. [10]

    Would deep generative models amplify bias in future models? In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 10833--10843, 2024

    Chen, T., Hirota, Y., Otani, M., Garcia, N., and Nakashima, Y. Would deep generative models amplify bias in future models? In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 10833--10843, 2024. doi:10.1109/CVPR52733.2024.01030

    2024

  3. [11]

    A fast and elitist multiobjective genetic algorithm: Nsga-ii

    Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation, 6 0 (2): 0 182--197, 2002. doi:10.1109/4235.996017

    2002

  4. [12]

    Stable diffusion - open-source ai for creating images from text

    Diffusion, S. Stable diffusion - open-source ai for creating images from text. https://stabledifffusion.com/, 2025. Accessed: 2025-01-10

    2025

  5. [13]

    J., and Gidel, G

    Ferbach, D., Bertrand, Q., Bose, A. J., and Gidel, G. Self-consuming generative models with curated data provably optimize human preferences. In Advances in Neural Information Processing Systems, volume 37, pp.\ 102531--102567. Curran Associates, Inc., 2024

    2024

  6. [14]

    B., Gromov, A., Roberts, D., Yang, D., Donoho, D

    Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Korbak, T., Sleight, H., Agrawal, R., Hughes, J., Pai, D. B., Gromov, A., Roberts, D., Yang, D., Donoho, D. L., and Koyejo, S. Is model collapse inevitable? Breaking the curse of recursion by accumulating real and synthet...

    2024

  7. [15]

    Self-correcting self-consuming loops for generative model training

    Gillman, N., Freeman, M., Aggarwal, D., HSU, C.-H., Luo, C., Tian, Y., and Sun, C. Self-correcting self-consuming loops for generative model training. In Scaling Self-Improving Foundation Models without Human Supervision, 2025. URL https://openreview.net/forum?id=O1B95aIlFn

    2025

  8. [16]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 770--778, 2016. doi:10.1109/CVPR.2016.90

    2016 doi

  9. [17]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp.\ 6840--6851. Curran Associates, Inc., 2020

    2020

  10. [18]

    Forcing generative models to degenerate ones: The power of data poisoning attacks

    Jiang, S., Kadhe, S., Zhou, Y., Cai, L., and Baracaldo, N. Forcing generative models to degenerate ones: The power of data poisoning attacks. In NeurIPS 2023 Workshop on Backdoors in Deep Learning - The Good, the Bad, and the Ugly, 2024. URL https://openreview.net/forum?id=8R4z3XZt5J

    2023

  11. [19]

    Koh, P. W. and Liang, P. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp.\ 1885--1894. PMLR, 2017. URL https://proceedings.mlr.press/v70/koh17a.html

    2017

  12. [20]

    Learning multiple layers of features from tiny images

    Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. URL https://www.cs.toronto.edu/ kriz/cifar.html

    2009

  13. [21]

    Pika labs - AI -generated videos from text

    Labs, P. Pika labs - AI -generated videos from text. https://www.pikalabs.com/, 2025. Accessed: 2025-01-10

    2025

  14. [22]

    Robust linear regression against training data poisoning

    Liu, C., Li, B., Vorobeychik, Y., and Oprea, A. Robust linear regression against training data poisoning. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec '17, pp.\ 91–102, New York, NY, USA, 2017. Association for Computing Machinery. doi:...

    2017

  15. [23]

    Aligning with human judgement: The role of pairwise preference in large language model evaluators

    Liu, Y., Zhou, H., Guo, Z., Shareghi, E., Vuli \'c , I., Korhonen, A., and Collier, N. Aligning with human judgement: The role of pairwise preference in large language model evaluators. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=9gdZI7c6yr

    2024

  16. [24]

    and Zhu, X

    Mei, S. and Zhu, X. Using machine teaching to identify optimal training-set attacks on machine learners. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, pp.\ 2871–2877. AAAI Press, 2015

    2015

  17. [25]

    Midjourney - AI -generated art platform

    MidJourney. Midjourney - AI -generated art platform. https://www.midjourney.com/, 2025. Accessed: 2025-01-10

    2025

  18. [26]

    Runway ML - AI tools for creators

    ML, R. Runway ML - AI tools for creators. https://runwayml.com/, 2025. Accessed: 2025-01-10

    2025

  19. [27]

    Pareto multi objective optimization

    Ngatchou, P., Zarei, A., and El-Sharkawi, A. Pareto multi objective optimization. In Proceedings of the 13th International Conference on, Intelligent Systems Application to Power Systems, pp.\ 84--91, 2005. doi:10.1109/ISAP.2005.1599245

    2005

  20. [28]

    GPT -4 technical report, 2024

    OpenAI. GPT -4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  21. [29]

    ChatGPT - Conversational AI by OpenAI

    OpenAI. ChatGPT - Conversational AI by OpenAI . https://openai.com/chatgpt, 2025. Accessed: 2025-01-10

    2025

  22. [30]

    Journeydb: A benchmark for generative image understanding, 2023

    Pan, J., Sun, K., Ge, Y., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y., Dai, J., Qiao, Y., and Li, H. Journeydb: A benchmark for generative image understanding, 2023

    2023

  23. [31]

    Paudice, A., Mu \ n oz-Gonz \'a lez, L., and Lupu, E. C. Label sanitization against label flipping poisoning attacks. In ECML PKDD 2018 Workshops, pp.\ 5--15. Springer International Publishing, 2019

    2018

  24. [32]

    AI models collapse when trained on recursively generated data

    Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., and Gal, Y. AI models collapse when trained on recursively generated data. Nature, 631 0 (8022): 0 755--759, Jul 2024. URL https://doi.org/10.1038/s41586-024-07566-y

    2024 doi

  25. [33]

    and Zisserman, A

    Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In The 3rd International Conference on Learning Representations, 2015. URL http://arxiv.org/abs/1409.1556

    2015 arXiv

  26. [34]

    What distributions are robust to indiscriminate poisoning attacks for linear learners? In Advances in Neural Information Processing Systems, volume 36, pp.\ 34942--34980

    Suya, F., Zhang, X., Tian, Y., and Evans, D. What distributions are robust to indiscriminate poisoning attacks for linear learners? In Advances in Neural Information Processing Systems, volume 36, pp.\ 34942--34980. Curran Associates, Inc., 2023

    2023

  27. [35]

    and Hashimoto, T

    Taori, R. and Hashimoto, T. Data feedback loops: Model-driven amplification of dataset biases. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp.\ 33883--33920. PMLR, 2023. URL https://proceedings.mlr.press/v202/taori23a.html

    2023

  28. [36]

    and Kantarcioglu, M

    Vorobeychik, Y. and Kantarcioglu, M. Data Poisoning Attacks, pp.\ 77--98. Springer International Publishing, 2018. URL https://doi.org/10.1007/978-3-031-01580-9_6

    2018 doi

  29. [37]

    Preference poisoning attacks on reward model learning

    Wu, J., Wang, J., Xiao, C., Wang, C., Zhang, N., and Vorobeychik, Y. Preference poisoning attacks on reward model learning. In 2025 IEEE Symposium on Security and Privacy (SP), pp.\ 1622--1640, 2025. URL https://doi.ieeecomputersociety.org/10.1109/SP61157.2025.00094

    2025

  30. [38]

    Fairness feedback loops: Training on synthetic data amplifies bias

    Wyllie, S., Shumailov, I., and Papernot, N. Fairness feedback loops: Training on synthetic data amplifies bias. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, pp.\ 2113–2147. Association for Computing Machinery, 2024. URL ht...

    2024

  31. [39]

    and Zhang, X

    Xie, T. and Zhang, X. Automating data annotation under strategic human agents: Risks and potential solutions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=2UJLv3KPGO

    2024

  32. [40]

    Meta-Sift : How to sift out a clean subset in the presence of data poisoning? In 32nd USENIX Security Symposium, pp.\ 1667--1684

    Zeng, Y., Pan, M., Jahagirdar, H., Jin, M., Lyu, L., and Jia, R. Meta-Sift : How to sift out a clean subset in the presence of data poisoning? In 32nd USENIX Security Symposium, pp.\ 1667--1684. USENIX Association, 2023. URL https://www.usenix.org/conference/usenixsecurity23/p...

    2023

  33. [41]

    Practical data poisoning attack against next-item recommendation

    Zhang, H., Li, Y., Ding, B., and Gao, J. Practical data poisoning attack against next-item recommendation. Proceedings of The Web Conference 2020, 2020. URL https://api.semanticscholar.org/CorpusID:215416122

    2020

  34. [42]

    RMB: comprehensively benchmarking reward models in LLM alignment

    Zhou, E., Zheng, G., Wang, B., Xi, Z., Dou, S., Bao, R., Shen, W., Xiong, L., Fan, J., Mou, Y., Zheng, R., Gui, T., Zhang, Q., and Huang, X. RMB: comprehensively benchmarking reward models in LLM alignment. In The Thirteenth International Conference on Learning Representations...

    2025

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.