{"id":"e0418df7-061e-4079-9a6f-92e34c257310","arxiv_id":"2505.09768","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Under adversarial data curation, a self-consuming generative model's alignment with user preferences is claimed to depend on the covariance between true and malicious reward functions; the proposed attack algorithms aim to make that covariance negative.","lead":"This paper studies what happens when a generative model is retrained on its own synthetic output after curation by users, including malicious users paid by a competitor. It claims that whether such attacks derail the model depends on how the attackers' preferences correlate with real users' preferences, and proposes label-flipping attacks that exploit negative correlation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 3.3's lower bound is false as stated: the covariance term uses e^{er_min} where e^{er_max} is required, and a two-point example violates the bound; Lemma 3.4 also mis-sums the geometric series.","rationale":"The reader's declared weakest assumption is that the iterative update in Eqs. (3)-(4) realizes the exact maximum-likelihood solution, so finite capacity and optimization error break the closed-form recursion. That is a legitimate concern, but it is not the most load-bearing one: even under the paper's own idealized K=infty update, the central lemma is false. The reader's rationale does identify the same Lemma 3.3 counterexample and the Lemma 3.4 factor error, so the overall reject verdict is well supported. My stress test confirms the rejection and therefore does not change the verdict; I mark agreement as partial because the stated weakest_assumption differs from the more decisive internal inconsistency found in Appendix B.3 and B.4. The qualitative intuition that positive correlation protects alignment may still be recoverable in special cases, but the quantitative theorems that justify the attack algorithms and robustness discussion are not correct as written.","tokens_in":23645,"tokens_out":7255,"duration_ms":71688,"concrete_test":"Recompute the K=infty, phi=1 two-point example from Appendix B.3: let p_t put equal mass on two points with e^r in {1,10} and e^{er} in {1,2}. The exact update gives E_{p_{t+1}}[e^r] = E_p[e^r e^{er}] / E_p[e^{er}] = 10.5 / 1.5 = 7.0, while Lemma 3.3's lower bound is E_p[e^r] + Cov[e^r, e^{er}] / e^{er_min} = 5.5 + 2.25 / 1 = 7.75. If these numbers reproduce, the lemma's covariance denominator must be replaced by e^{er_max} for positive covariance and the stated bound cannot stand. Separately verify Lemma 3.4 using the identity sum_{j=1}^t (lambda/(1+lambda))^j = lambda(1-(lambda/(1+lambda))^t), which yields phi^* lambda(1-(lambda/(1+lambda))^t) Covmin, not the paper's phi^*(1+lambda)(1-(lambda/(1+lambda))^t) Covmin.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical support is Lemma 3.3, which is supposed to prove that positive covariance between true and adversarial rewards guarantees growth of expected reward. The proof in Appendix B.3 bounds the cross term B by replacing the random denominator sum with K e^{er_min} when Cov >= 0. This is the wrong extremum: for a positive numerator, the ratio numerator/denominator is minimized by the largest denominator, e^{er_max}; using e^{er_min} makes the claimed lower bound too large. The error is not cosmetic. With two equiprobable points having e^r in {1,10} and e^{er} in {1,2}, phi = 1, K -> infinity, Lemma 3.1 gives p_{t+1}(x) proportional to p_t(x) e^{er(x)}, so the exact next expected reward is E_p[e^r e^{er}] / E_p[e^{er}] = 10.5 / 1.5 = 7.0. Lemma 3.3 claims a lower bound of E_p[e^r] + Cov[e^r, e^{er}] / e^{er_min} = 5.5 + 2.25 / 1 = 7.75, which is violated. Lemma 3.4 has an independent algebra error: the sum of (lambda/(1+lambda))^j from j=1 to t equals lambda(1-(lambda/(1+lambda))^t), not (1+lambda)(1-(lambda/(1+lambda))^t), so the asymptotic mixed-data bound should involve phi^* lambda Covmin rather than phi^*(1+lambda) Covmin. These are failures of the paper's own idealized update, not artifacts of finite model capacity. The qualitative sign claim can survive in the K=infty, phi=1 case because the exact increment equals Cov[e^r,e^{er}]/E[e^{er}], but the quantitative bounds and the derived conditions in the theorems are not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies self-consuming generative models trained on data curated by a mixture of benign users and adversarial malicious users, under a generalized Bradley-Terry choice model. The authors derive recursions for the output distribution, prove bounds on the evolution of expected user reward (Lemmas 3.3 and 3.4), and use the covariance between true and adversarial rewards to characterize when the model remains aligned with user preferences. Based on this theory, they propose gradient-based and heuristic label-flipping attacks to misalign a competitor's model, and validate them on synthetic and CIFAR datasets.","tokens_in":23881,"tokens_out":7987,"duration_ms":76754,"significance":"If the theoretical results were correct, the paper would provide a novel characterization of adversarial data curation in self-consuming generative loops, extending the benign-curation analysis of Ferbach et al. (2024). The proposed attack algorithms address a realistic competitive scenario and the experimental results on CIFAR-10/CIFAR-100 show clear misalignment under the gradient-based attack. However, the central lemmas are false as stated, and the qualitative conclusions are not established by the given proofs. The empirical findings may still be interesting, but the paper's main theoretical contribution is not sound.","major_comments":[{"comment":"The lower bound in Lemma 3.3 is false as stated. Consider p_t with two equiprobable points where e^r is 1 and 10 and e^{er} is 1 and 2, with ϕ=1 and K→∞. By Lemma 3.1, p_{t+1} ∝ p_t e^{er}, so the exact next expected reward is (0.5·1·1 + 0.5·10·2)/(0.5·1+0.5·2) = 10.5/1.5 = 7.0. The claimed lower bound is E_{p_t}[e^r] + Cov[e^r,e^{er}]/e^{er_min} = 5.5 + 2.25/1 = 7.75, which is violated. The proof in Appendix B.3 replaces the random denominator Σ e^{er(x_j)} by K e^{er_min} to bound B from below when the covariance is positive; for a positive ratio, the correct extremal denominator is K e^{er_max} (the largest possible denominator minimizes the ratio). This error invalidates the quantitative bound and the claimed implication that Cov ≥ 0 guarantees E_{p_{t+1}}[e^r] ≥ E_{p_t}[e^r] for finite K.","section":"Section 3, Lemma 3.3"},{"comment":"The geometric series in the proof of Lemma 3.4 is mis-summed. With q = λ/(1+λ), Σ_{j=1}^{t} q^j = q(1-q^t)/(1-q) = λ(1 − (λ/(1+λ))^t). The paper instead obtains ϕ*(1+λ)(1 − (λ/(1+λ))^t) Cov_min, which is too large by a factor of (1+λ)/λ. The asymptotic bound should be ϕ*λ Cov_min, not ϕ*(1+λ) Cov_min. Additionally, the proof relies on Lemma B.2, which is itself false: the same two-point example gives E_{p_{t+1}}[e^r] = 7.0 < E_{p_t}[e^r] + ϕ Cov = 7.75, contradicting Eq. (13). The error in the proof of Lemma B.2 is the step E[e^r e^{er}]/E[e^{er}] = E[e^r] + Cov, which drops the denominator E[e^{er}].","section":"Section 3, Lemma 3.4"},{"comment":"Because Lemma 3.3 is false, the paper's central statements about robustness and vulnerability are not proven. In the K→∞, ϕ=1 special case, the exact increment is Cov[e^r,e^{er}]/E[e^{er}], so the sign of the covariance does control whether expected reward increases; however, the paper claims the stronger finite-K bound E_{p_{t+1}}[e^r] ≥ E_{p_t}[e^r] + (K−1)/K · Cov/e^{er_min}, which is not supported. Consequently, the qualitative interpretations in Section 3 and the theoretical motivation for the attack objective in Eq. (10) rest on incorrect mathematics, even though the attack algorithms may still be effective empirically.","section":"Section 3"}],"minor_comments":[{"comment":"There are several typos and inconsistent spellings, e.g., \"bengiun\" (Figure 7), \"CIRFAR\" (Appendix C.1), and \"curation\" vs \"curation\".","section":"Throughout"},{"comment":"The definition of Cov_min in the main text uses i ∈ [t], but the proof of Lemma B.4 takes the minimum over i ∈ {0,1,...,t}. Please align the index sets.","section":"Lemma 3.4"},{"comment":"The experimental results, especially Table 1 and the reward curves, are reported without confidence intervals or multiple-seed statistics. Given the large variability of the random attack shown in Figure 6, reporting error bars would substantially strengthen the empirical claims.","section":"Section 5 and Table 1"},{"comment":"The algorithm clips δ_i to [-1,1] while the original labels o_i are in {0,0.5,1}; the resulting o_i+δ_i may fall outside {0,0.5,1}. Please clarify how the perturbed labels are represented in the loss of Eq. (7).","section":"Algorithm 1"}],"recommendation":"reject","confidential_remarks":"The paper's central theoretical results (Lemmas 3.3 and 3.4) are false as stated, and the proofs contain clear errors. The counterexamples are simple and directly refute the claimed bounds. Although the qualitative sign intuition about covariance may survive in a corrected K→∞ analysis, the current manuscript does not provide a sound basis for its main claims. The empirical attack results might be publishable in a separate empirical study, but the theoretical contribution of this paper is not valid. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The paper's problem framing is genuinely new: modeling self-consuming retraining under a mixture of benign and malicious Bradley-Terry curators, and designing attacks that flip preference labels through public feedback. That is a real gap relative to Ferbach et al. 2024 and Wu et al. 2025. The qualitative intuition—positive covariance between true and adversarial rewards preserves alignment, negative covariance breaks it—is plausible and consistent with the K→∞ limit.\n\nBut the theory as stated is false. Lemma 3.3's lower bound is violated by a two-point example with e^r in {1,10}, e^{er} in {1,2}, equal weights, phi=1, K→∞. Using their own Lemma 3.1, the next expected reward is 7.0, while the claimed lower bound is 7.75. The proof in Appendix B.3 bounds the covariance term by e^{er_min} for positive covariance; for a positive numerator, the ratio is minimized by the largest denominator, e^{er_max}. This is not cosmetic—the bounds in both Lemma 3.3 and the derived conditions in the theorems rest on it. Lemma 3.4 has an independent algebra error: the geometric sum of (lambda/(1+lambda))^j is lambda(1-(lambda/(1+lambda))^t), not (1+lambda)(1-(lambda/(1+lambda))^t). So the mixed-data bound is also unsupported.\n\nI should give credit where it's due. The paper is honest about the global-convergence assumption in its Limitations section. The attack algorithms are reasonable, and the CIFAR experiments do show the gradient attack shifting the generated distribution away from preferred classes. But the experiments are single-run, have no error bars or code, never compare against the closest prior attack (Wu et al. 2025), and use hand-designed rewards. The theory, which is the main claimed contribution, does not hold up.\n\nWho is this for? Someone looking for a new attack scenario might skim Section 4, but the paper cannot be cited for its theoretical claims. It deserves a serious referee only if the authors can repair Lemma 3.3; as posted, the central bounds are false. I would not accept it for peer review in this state.","headline":"New and practical attack framing, but the central theoretical lemmas are false: Lemma 3.3's covariance bound uses the wrong extremum and a two-point example violates it, and Lemma 3.4's geometric sum is also off.","tokens_in":24650,"tokens_out":4680,"would_cite":false,"duration_ms":41979,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The central claim is that the fate of a self-consuming generative model under adversarial data curation is decided by the sign of the covariance between the true user reward and the adversarial reward: positive covariance means the loop…","keywords":["self-consuming generative models","adversarial data curation","preference alignment","reward covariance","label flipping attack","data poisoning","diffusion models","generalized Bradley-Terry model"],"falsifier":"Fix a retraining loop on CIFAR-10 with an adversarial reward function chosen so that $\\operatorname{Cov}_{p_t}(e^r, e^{\\tilde r})<0$ at every round and a 20% malicious fraction, as in the paper's experiments. If the measured $\\mathbb{E}_{p_t}[e^{r(x)}]$ stays at or above its initial level for ten rounds, the Lemma 3.3 sign criterion is violated. Conversely, a run with positive covariance in which the expected reward falls would falsify the robustness direction.","tokens_in":23204,"feed_emoji":"⚖️","tokens_out":7269,"duration_ms":67243,"temperature":0.7,"pith_summary":"This paper extends the study of self-consuming generative models to settings where the data used for retraining is curated by adversarial users with a limited budget, rather than only by genuine user preferences. Its central theoretical result is a covariance condition: the long-run reward experienced by users increases when the adversarial reward is non-negatively correlated with the true user reward, and can collapse when the correlation is negative. The paper also shows that mixing real data into training is not by itself a defense against adversarial curation. Building on this condition, it proposes attack algorithms that strategically flip preference labels to force the covariance negative, and validates them on synthetic data and diffusion models over image datasets.","feed_headline":"Retraining survives attacks only when reward covariances line up","feed_subtitle":"Adversarial curation flips the sign of one covariance and drives generative models away from user preferences.","key_machinery":"The load-bearing object is the covariance $\\operatorname{Cov}_{p_t}\\left(e^{r(x)}, e^{\\tilde r_t(x)}\\right)$ between the exponentialized true user reward and exponentialized adversarial reward at the current model distribution. In words, it measures whether malicious curation tends to pick the same samples that genuine users prefer. It appears in Lemma 3.3: the lower bound on the next expected reward is $\\mathbb{E}_{p_t}[e^r] + (1-\\phi_t)\\frac{K-1}{K}\\frac{\\operatorname{Var}_{p_t}[e^r]}{e^{r_{t,\\max}}} + \\phi_t\\frac{K-1}{K}\\frac{\\operatorname{Cov}_{p_t}[e^r, e^{\\tilde r_t}]}{e^{\\tilde r_{t,\\min}}}$. The sign of the covariance, multiplied by the malicious-user fraction $\\phi_t$ and the sample-selection factor $(K-1)/K$, determines whether the expected reward is pushed up by the benign variance term or dragged down by the adversarial term. The argument carries through the closed-form update $p_{t+1}(x)=p_t(x)\\left[(1-\\phi_t)\\frac{e^{r(x)}}{\\mathbb{E}_{p_t}[e^r]}+\\phi_t\\frac{e^{\\tilde r_t(x)}}{\\mathbb{E}_{p_t}[e^{\\tilde r_t}]}\\right]$ that arises in the $K\\to\\infty$ limit, turning the self-consuming loop into a multiplicative reweighting step whose drift is read off from this covariance.","core_discovery":"The paper argues that in a self-consuming retraining loop where each generation's synthetic data is curated by a mixture of benign and malicious users, the expected user reward from the model obeys a bound with two correction terms. Under the generalized Bradley-Terry curation model, the change in $\\mathbb{E}_{p_t}[e^{r(x)}]$ between rounds is controlled by the variance of the true reward and by $\\operatorname{Cov}_{p_t}[e^{r(x)}, e^{\\tilde r_t(x)}]$, the covariance between the true reward and the adversarial reward evaluated on the current model. As long as that covariance is non-negative, the expected reward grows toward its maximum and the model aligns with user preferences despite the attackers; when it is negative, the adversarial term can dominate, and the expected reward can fall, oscillate, and drift away from the preference-optimizing distribution. The same covariance sign appears when real data are mixed into training: the lower bound becomes $\\mathbb{E}_{p_{\\text{data}}}[e^{r(x)}] + \\phi^\\star(\\lambda+1)\\min_{i\\le t}\\operatorname{Cov}_{p_i}[e^r, e^{\\tilde r_i}]$, so fresh real data does not automatically rescue alignment. On top of this, the paper designs label-flipping attacks that make the covariance negative under a limited flip budget and shows empirically on diffusion models over CIFAR-10 and CIFAR-100 that these attacks move generation away from user-preferred classes.","pith_inferences":["A testable early-warning for practitioners: platforms can monitor $\\operatorname{Cov}_{p_t}(e^r, e^{\\tilde r})$ in production, and a persistent drop below zero should trigger defensive action before the expected reward visibly falls.","The covariance criterion may transfer to human-feedback alignment pipelines outside self-consuming loops, such as preference learning with poisoned comparison data, where the same geometry between reward models could determine drift; the paper does not make this claim.","In finite-capacity models, the covariance sign may become a necessary but not sufficient condition, since optimization noise and capacity limits could create plateaus not predicted by Lemma 3.3; this is testable by ablating model size in the same retraining setup.","The result suggests a defensive design principle not explored in the paper: curate or reweight synthetic data so that the adversarial reward correlation stays non-negative, for example by filtering samples that maximize the negative covariance."],"forward_implications":["If the adversarial reward is positively correlated with user preferences, the retraining loop is robust: expected user reward keeps increasing toward its maximum even while malicious users curate a fraction of the data.","If the correlation is negative, the attack term can outweigh the benign variance term; expected reward decreases, oscillates, and the generated distribution drifts to low-preference samples.","Adding real data at any fixed ratio does not restore preference alignment once the negative-covariance condition holds; it only pulls the model toward the real data distribution, not toward the user-preferred distribution.","A platform with a budget-limited set of malicious users can push the covariance negative by flipping only a fraction $\\kappa$ of preference labels, using the proposed gradient-based or heuristic algorithms.","Because the attack is realized through public curation or voting rather than backend data access, detection by outlier filtering is partially countered by a distance penalty in the attack objective."],"supporting_citations":[{"why":"Supplies the benign-curation baseline and the framework of self-consuming training on curated synthetic data that this paper extends to adversarial curators.","marker":"Ferbach et al., 2024"},{"why":"Provides the choice model in Eq. (1) that the mixture curation distribution in Eq. (2) builds on.","marker":"Bradley & Terry, 1952"},{"why":"Supplies the mixed real/synthetic retraining setup and the stabilization result that Lemma 3.4 contrasts with the negative-covariance case.","marker":"Bertrand et al., 2024"},{"why":"Gives the preference-poisoning attack on reward models that the proposed label-flipping attack generalizes to the self-consuming loop, along with the implicit-derivative technique.","marker":"Wu et al., 2025"},{"why":"Provides the influence-function machinery used to compute the gradient of the attack objective with respect to label perturbations.","marker":"Koh & Liang, 2017"},{"why":"Provides the DDPM architecture used in the experiments measuring reward evolution under adversarial curation.","marker":"Ho et al., 2020"},{"why":"Supplies CIFAR-10 and CIFAR-100 as the image datasets for the retraining and attack experiments.","marker":"Krizhevsky, 2009"}],"fun_headline_variants":["Covariance sign decides if retraining resists attacks","Adversarial curation flips covariance, derails model alignment","Self-consuming loops: covariance sign governs attack resilience","When reward covariance turns negative, retraining collapses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire theory assumes that every retraining round computes the exact maximum-likelihood distribution over all achievable models, so real models with finite capacity, optimization noise, or limited training budgets do not exactly satisfy the closed-form update; if that assumption fails, the covariance bound need not describe what actually happens.","fun_headline_variants_meta":{"raw":{"variants":["Covariance sign decides if retraining resists attacks","Adversarial curation flips covariance, derails model alignment","Self-consuming loops: covariance sign governs attack resilience","When reward covariance turns negative, retraining collapses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1388,"prompt_tokens":1066,"completion_tokens":322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":258}},"tokens_in":682,"tokens_out":322,"duration_ms":3436,"temperature":1.0,"reasoning_tokens":258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:27:23.540943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix a retraining loop on CIFAR-10 with an adversarial reward function chosen so that $\\operatorname{Cov}_{p_t}(e^r, e^{\\tilde r})<0$ at every round and a 20% malicious fraction, as in the paper's experiments. If the measured $\\mathbb{E}_{p_t}[e^{r(x)}]$ stays at or above its initial level for ten rounds, the Lemma 3.3 sign criterion is violated. Conversely, a run with positive covariance in which the expected reward falls would falsify the robustness direction.","supporting_citations":[{"cited_title":"J., and Gidel, G","cited_arxiv_id":null,"evidence_quote":"Supplies the benign-curation baseline and the framework of self-consuming training on curated synthetic data that this paper extends to adversarial curators."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the choice model in Eq. (1) that the mixture curation distribution in Eq. (2) builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the influence-function machinery used to compute the gradient of the attack objective with respect to label perturbations."},{"cited_title":"Learning multiple layers of features from tiny images","cited_arxiv_id":null,"evidence_quote":"Supplies CIFAR-10 and CIFAR-100 as the image datasets for the retraining and attack experiments."}],"review_version":1}