{"id":"9f04cf8b-6a0a-42e5-96b0-c3353eeace23","arxiv_id":"2505.21848","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Probabilistically adding high-intensity noise to individual token embeddings during fine-tuning reduces replication in Stable Diffusion by up to 28.78% in the paper's experiments, with unchanged or improved FID.","lead":"This paper proposes FPAN, a fine-tuning trick that adds random noise to individual word-token embeddings during diffusion model training, cutting how often generated images copy training images by up to about 29% in the authors' tests. It matters because image replication is a privacy and copyright risk in text-to-image models, and the method can be stacked on top of existing safeguards.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline R/FID gains are selected from the same evaluation set they are reported on; with no held-out split or error bars, the 28.78% reduction may be tuning noise, and the abstract's 'average' conflicts with Table 2.","rationale":"The reader's weakest-assumption analysis correctly identifies the central risk: hyperparameters are tuned and evaluated on the same 10,000-image set, with no held-out data or error bars, so the reported improvements may not generalize. My read agrees with the CONDITIONAL verdict: the method is plausible, the reported numbers are internally consistent in their rough magnitudes, and the noise-injection mechanism is a reasonable extension of prior work, but the empirical support is not yet strong enough to accept the headline claim as robust. I additionally note a specific reporting inconsistency between the abstract's 'average of 28.78%' and the body's 'up to 28.78%', which should be corrected or clarified. Because this does not change the overall assessment, the reader's CONDITIONAL verdict remains appropriate and no adjustment is needed.","tokens_in":14512,"tokens_out":5715,"duration_ms":59192,"concrete_test":"Split the 10,000-sample LAION subset into two disjoint halves. On half A, run the full hyperparameter sweep (W in {1.5, 1.6, 1.7}, P in {0, 0.1, ..., 1.0}) with at least three random seeds; freeze the selected (W, P) on half A; then evaluate only on half B. The selection is robust only if the half-A-selected settings remain on the Pareto frontier for half B and if the relative R reduction's bootstrap confidence interval excludes zero. Separately, recompute the average reduction from all four P values in Table 2 and state whether 28.78% is a mean or a maximum.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is not robustly supported because the operating points are chosen from the same evaluation protocol that produces the reported numbers. In Section 4.2, W=1.7 and P in {0.3,...,0.6} are selected by inspecting R-FID curves (Figure 3) computed on the same 10,000-image generation set later used for Table 2. The 'well-fitting' stage boundaries in Figure 1 are inferred from third-order polynomial fits to those same runs. With no held-out partition, no multiple seeds, and no error bars, the reported R differences (e.g., 0.452 vs. 0.438) are within plausible evaluation noise, so the claimed 28.78% reduction may reflect selection rather than a stable property of FPAN. In addition, the abstract's 'average of 28.78%' is not derivable from Table 2: the four P values give relative reductions of about 20.2%, 26.5%, 26.5%, and 28.8%, whose mean is approximately 25.5%, not 28.78%. The body text itself says 'up to 28.78%', so the headline number conflates a maximum with an average. These issues do not disprove the method, but they make the central claim's magnitude and generalizability unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FPAN is a fine-tuning-time intervention for text-to-image diffusion models: at each training iteration, each token embedding independently receives Gaussian noise of intensity W with probability P, otherwise remains unchanged. The paper first studies deterministic noise intensity W, identifies overfitting/well-fitting/underfitting regimes from polynomial fits of R and FID, and then proposes probabilistic injection with W=1.7 and P in {0.3,...,0.6}. On a 10,000-image LAION subset with Stable Diffusion 2.1, the authors report replication score reductions relative to baseline (R=0.438 at P=0.6 versus 0.615) with FID slightly improved, and further reductions when FPAN is combined with existing methods. The main quantitative claim in the abstract is an average 28.78% replication reduction.","tokens_in":14813,"tokens_out":6924,"duration_ms":63430,"significance":"If validated, FPAN is an attractive mitigation method because it is simple, architecture-agnostic, and combines with existing techniques; the three-stage FID/R characterization and the probabilistic mechanism provide a plausible explanation for why occasional high-intensity noise can outperform uniform small noise. The ablation against CPAN and RM is thoughtful, and the Appendix A.3 mean/variance derivation gives a concrete mechanistic rationale. However, the paper's central quantitative evidence is currently single-run, and the operating point is chosen from the same evaluation data, so the magnitude of the effect and its generalizability are not yet established.","major_comments":[{"comment":"The headline result is not robustly supported because W=1.7 and P in {0.3,...,0.6} are selected by inspecting R-FID curves computed from the same 10,000 generated images later used to report the final numbers, and no held-out split, multiple seeds, or error bars are provided. The reported R values at the four selected operating points (0.491, 0.452, 0.452, 0.438) differ by at most 0.053, which is comparable to the run-to-run variation observed in fine-tuning studies of this scale; selection on the evaluation set can therefore account for a material part of the claimed 28.78% reduction. Similarly, the stage boundaries in Figure 1 are inferred from third-order polynomial fits to the same runs. Please add a validation protocol (separate search and evaluation sets, several seeds, and confidence intervals) or explicitly present the numbers as single-run, exploratory findings.","section":"Sec. 4.2 / Table 2 / Fig. 3"},{"comment":"The abstract's 'average of 28.78%' is not derivable from the reported data. For W=1.7 with P=0.3, 0.4, 0.5, and 0.6, the relative reductions in R are approximately 20.2%, 26.5%, 26.5%, and 28.8%, whose mean is about 25.5%; the 28.78% figure is the maximum, obtained only at P=0.6. The body text correctly says 'up to 28.78%' in Section 4.3, so the abstract overstates the result. Please correct the abstract and any other summary statements.","section":"Abstract / Sec. 4.3 / Table 2"},{"comment":"The classification into overfitting, well-fitting, and underfitting stages relies on a single threshold tau_CLIP computed from one pretrained reference model, and the boundaries W=0.7 and W=1.7 are asserted from fitted curves without uncertainty quantification. Since W=1.7 is then used as the fixed intensity for FPAN, the stage analysis is load-bearing for the method's design. A sensitivity analysis of the boundary locations, or an independent criterion for the well-fitting regime, would strengthen this part of the paper.","section":"Sec. 3.2 / Fig. 1"}],"minor_comments":[{"comment":"There are typographical errors: 'embeddigns' in Section 3.1, 'exprimental' in Section 3.2, and an unwanted space in 'F ine-Grained' in the title.","section":"Sec. 3.1 / Sec. 3.2 / Title"},{"comment":"The sentence beginning 'The above findings suggest that our method outperforms most other methods, when deployed as a standalone strategy, offers similar if not improved trade-off' has a grammatical break and should be rephrased.","section":"Sec. 4.3"},{"comment":"Figure 1 would benefit from showing raw data points with confidence intervals or error bars in addition to the polynomial fits, because the stage boundaries are read off the fitted curves.","section":"Fig. 1"},{"comment":"The reference list contains a formatting error in [13], where page numbers '3, 1' appear; please check all citation metadata for similar issues.","section":"References"},{"comment":"The labels A1 through A4 are defined only in the caption of Figure 3, but Table 1 references them first; please define the labels in Table 1 as well, or add a note to Table 1.","section":"Table 1 / Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The evaluation protocol is the main obstacle: the operating point is selected from the same evaluation set that produces the headline numbers, and no variance information is provided. If the authors add multi-seed experiments with a proper selection/evaluation split and fix the abstract's 'average' claim, I would be willing to reconsider. The paper is within the journal's scope as a methods contribution, but the current evidence does not justify the quantitative headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the method is a sensible extension of noise-injection mitigation, and the mean-variance analysis in the appendix is a genuinely nice formal touch. But the abstract's 'average of 28.78%' is not what the data show—it's the best case at P=0.6. And the operating points are selected from the same R-FID curves that produce the headline numbers, with no held-out split or error bars. So the size of the effect, not its direction, is unverified.\n\nWhat's new: FPAN adds high-intensity noise to each token independently with probability P. That is a direct descendant of Somepalli et al.'s consistent small-magnitude Gaussian noise on the full embedding, but the per-token Bernoulli scheme is a real twist. The paper also documents a three-stage behavior in FID as noise intensity grows (over/well/underfitting), and uses a CLIP-similarity threshold to support that reading. The Appendix A.3 derivation shows FPAN preserves the mean token embedding while increasing variance, whereas random masking shrinks both. That is a clean, testable distinction and the empirical distribution plot supports it.\n\nSoft spots, in proportion. The biggest is the headline number. In Table 2 the relative reductions for P=0.3,0.4,0.5,0.6 are about 20%, 27%, 27%, 29%; the mean is about 25.5%, not 28.78%. The body text says 'up to 28.78%', so the abstract simply overstates it. Second, hyperparameters W=1.7 and the P range are chosen by inspecting the same R-FID curves that later produce the reported results (Figure 3, Table 2). Without a separate validation split, multiple seeds, or error bars, the claimed superiority over GN and the synergy numbers could be selection noise. I don't think the method is fake—the direction is consistent with known noise-injection effects—but the magnitude is not nailed down. Minor: the function-word explanation in Appendix A.2 is a story, not evidence.\n\nWho benefits: researchers working on memorization/privacy in text-to-image diffusion. It's worth a serious referee. My recommendation: send it to review, but the authors should fix the abstract, add seeds/error bars, and ideally re-tune on a held-out split. That's a revision, not a rejection.","headline":"Plausible incremental mitigation method, but the headline 28.78% reduction is a max, not an average, and hyperparameters are tuned on the same evaluation set used for the final numbers.","tokens_in":15325,"tokens_out":2694,"would_cite":false,"duration_ms":24957,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Probabilistic per-token Gaussian noise injected during fine-tuning reduces replication in text-to-image diffusion models by 28.78% on average while keeping image quality at or better than baseline.","keywords":["diffusion models","replication mitigation","token embeddings","probabilistic noise injection","fine-tuning","text-to-image generation","memorization","image quality"],"falsifier":"Fine-tune the same Stable Diffusion 2.1 setup on a held-out split of the same 10,000-image LAION subset with W=1.7, P=0.6 and several random seeds, and measure R and FID on prompts not used in hyperparameter selection; if the mean replication score is not clearly below the baseline's 0.615 (for instance, not below roughly 0.46) or the FID is not within a small margin of 18.24, the central claim fails.","tokens_in":14335,"feed_emoji":"🖼️","tokens_out":8693,"duration_ms":73333,"temperature":0.7,"pith_summary":"To curb the tendency of text-to-image diffusion models to copy training images, this paper proposes a fine-tuning intervention called Fine-grained Probabilistic Addition of Noise (FPAN). The method adds Gaussian noise of a chosen intensity to each token embedding independently, with a fixed probability, during every training step, rather than adding noise to the whole caption or to the image. The paper's central claim is that this simple intervention lowers the replication score by an average of 28.78% relative to the baseline, beats the prior consistent-magnitude noise-addition approach by 26.51%, and preserves or improves FID. A sympathetic reader would care because FPAN is a training-time-only change that leaves the model architecture untouched and composes with existing mitigation methods to reach replication scores as low as 0.357.","feed_headline":"Probabilistic token noise cuts diffusion replication by 28.78%","feed_subtitle":"Per-token random noise during fine-tuning keeps image quality flat while dropping replication from 0.615 to 0.438.","key_machinery":"The load-bearing object is the FPAN noise distribution $\\xi_i \\sim z_i \\cdot \\mathcal{N}(0, W^2 I)$ with $z_i \\sim \\mathrm{Bernoulli}(P)$, applied to each token embedding $\\tau_i$ of the CLIP text embedding. The paper shows that this scheme preserves the mean of the token-embedding distribution (keeping caption semantics intact) while inflating its variance (reducing the frequency of identical embeddings the model sees), and contrasts it with Random Masking, which shrinks both mean magnitude and variance. The R-FID curve, fitted to measured (R, FID) pairs as the probability P varies, is the instrument used to select W=1.7 and the favorable P range 0.3-0.6.","core_discovery":"The paper's central claim is that the effect of token-embedding noise on a fine-tuned diffusion model is not a monotonically degrading trade-off; it has three stages. At low noise intensities the model overfits, at intermediate intensities it is 'well-fitting' with the lowest FID local minimum, and at high intensities it underfits. FPAN takes the largest noise intensity that still belongs to the well-fitting stage (W=1.7) and applies it per-token with a Bernoulli probability P, so that a fraction of tokens get strong noise while the rest are unchanged. In the paper's experiments, this yields a replication score of R=0.438 at W=1.7, P=0.6 versus R=0.615 for the baseline, with FID improving from 18.24 to 17.96; the same procedure combines with LoyalDiffusion to reach R=0.357.","pith_inferences":["The headline 28.78% reduction is the best of four reported probabilities; the average over P=0.3, 0.4, 0.5, 0.6 is about 25.9%, so a conservative restatement is 'roughly a quarter reduction'.","If the three-stage overfitting/well-fitting/underfitting curve is real, then W and P need not be fixed: a training curriculum that anneals noise intensity downward as training progresses could reach the same replication suppression with even better final FID.","A natural next experiment is token-level importance weighting, perturbing the tokens most responsible for replication more aggressively; the paper notes this would require token-level importance estimation.","The mean-preserving property suggests FPAN should be tested on datasets with heavy caption duplication; on already-deduplicated data the achievable replication reduction may be smaller."],"forward_implications":["At W=1.7 with P=0.6, replication falls from 0.615 to 0.438 while FID improves from 18.24 to 17.96, so the method has a setting that is better than baseline on both metrics.","Because FPAN beats the consistent-magnitude Gaussian noise baseline by 26.51%, the probabilistic per-token mechanism, not simply adding more noise, is what drives the improvement.","Combining FPAN with prior mitigation methods yields further replication reductions (up to 16.82%) and reaches R=0.357 when paired with LoyalDiffusion, showing the interventions are complementary.","FPAN requires no architectural change and touches only the text-embedding input during fine-tuning, so it can be layered onto existing checkpoints and training pipelines.","The well-fitting noise window (0.7 < W <= 1.7) offers a concrete operating range for practitioners who want replication mitigation without FID degradation."],"supporting_citations":[{"why":"Defines the replication-score framework, supplies the Gaussian Noise (GN) baseline, and contributes the caption-mitigation baselines MC, RC, and CWR that FPAN is compared against.","marker":"[28]"},{"why":"Provides LoyalDiffusion, the strongest combined-method comparison, and the R-FID curve methodology used for hyperparameter selection.","marker":"[15]"},{"why":"Supplies the Dual Fusion comparison and the evidence that highly specific captions drive replication, which motivates token-level noise.","marker":"[14]"},{"why":"Provides the TMAA cross-attention masking method that FPAN composes with and compares against.","marker":"[21]"},{"why":"Supplies the SimCLIP overfitting metric used to partition the noise-intensity axis into overfitting, well-fitting, and underfitting stages.","marker":"[17]"},{"why":"Stable Diffusion 2.1 is the pretrained model that FPAN fine-tunes; the method's behavior is defined relative to this backbone.","marker":"[22]"},{"why":"LAION is the source of the 10,000-sample fine-tuning dataset used for all experiments.","marker":"[26]"},{"why":"Establishes the replication phenomenon in diffusion models and the fine-tuning protocol that the experiments follow.","marker":"[27]"}],"fun_headline_variants":["Per-token noise cuts diffusion replication by 28.78%","Probabilistic token noise trims diffusion duplication to 0.438","FPAN: fine-grained noise cuts diffusion copy-paste without quality loss","Random token noise preserves image quality while reducing replication"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the W=1.7 and P in {0.3, 0.4, 0.5, 0.6} settings, chosen by inspecting R-FID curves computed with the same evaluation that produces the headline results, generalize beyond those curves; the paper provides no held-out validation, error bars, or multiple-seed variation.","fun_headline_variants_meta":{"raw":{"variants":["Per-token noise cuts diffusion replication by 28.78%","Probabilistic token noise trims diffusion duplication to 0.438","FPAN: fine-grained noise cuts diffusion copy-paste without quality loss","Random token noise preserves image quality while reducing replication"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2760,"prompt_tokens":962,"completion_tokens":1798,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1725}},"tokens_in":578,"tokens_out":1798,"duration_ms":11790,"temperature":1.0,"reasoning_tokens":1725,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:21:29.870057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same Stable Diffusion 2.1 setup on a held-out split of the same 10,000-image LAION subset with W=1.7, P=0.6 and several random seeds, and measure R and FID on prompts not used in hyperparameter selection; if the mean replication score is not clearly below the baseline's 0.615 (for instance, not below roughly 0.46) or the FID is not within a small margin of 18.24, the central claim fails.","supporting_citations":[{"cited_title":"LoyalDiffusion: A Diffusion Model Guarding Against Data Replication","cited_arxiv_id":"2412.01118","evidence_quote":"Provides LoyalDiffusion, the strongest combined-method comparison, and the R-FID curve methodology used for hyperparameter selection."},{"cited_title":"Understanding and mitigating copying in diffusion models","cited_arxiv_id":null,"evidence_quote":"Defines the replication-score framework, supplies the Gaussian Noise (GN) baseline, and contributes the caption-mitigation baselines MC, RC, and CWR that FPAN is compared against."},{"cited_title":"Mitigate replication and copying in diffusion mod- els with generalized caption and dual fusion enhancement","cited_arxiv_id":null,"evidence_quote":"Supplies the Dual Fusion comparison and the evidence that highly specific captions drive replication, which motivates token-level noise."},{"cited_title":"Unveiling and mitigating mem- orization in text-to-image diffusion models through cross at- tention","cited_arxiv_id":null,"evidence_quote":"Provides the TMAA cross-attention masking method that FPAN composes with and compares against."},{"cited_title":"Enhancing DreamBooth with LoRA for generating unlimited characters with Stable Dif- fusion","cited_arxiv_id":null,"evidence_quote":"Supplies the SimCLIP overfitting metric used to partition the noise-intensity axis into overfitting, well-fitting, and underfitting stages."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Stable Diffusion 2.1 is the pretrained model that FPAN fine-tunes; the method's behavior is defined relative to this backbone."},{"cited_title":"LAION-5B: An open large-scale dataset for train- ing next generation image-text models","cited_arxiv_id":null,"evidence_quote":"LAION is the source of the 10,000-sample fine-tuning dataset used for all experiments."},{"cited_title":"Diffusion art or digital forgery? Investigating data replication in diffusion models","cited_arxiv_id":null,"evidence_quote":"Establishes the replication phenomenon in diffusion models and the fine-tuning protocol that the experiments follow."}],"review_version":1}