{"id":"fa700430-ba4b-4549-bcf6-4e93e8565cd0","arxiv_id":"2506.18428","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning text-to-image diffusion models generally undoes prior model edits, with DoRA causing the strongest reversal and UCE edits being most robust.","lead":"This paper tests whether edits made to text-to-image models survive later fine-tuning, and finds that fine-tuning usually reverses or weakens them. It matters for AI safety because fine-tuning could be used to remove harmful edits, but it also means beneficial edits need to be reapplied after adaptation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gender-debiasing evidence for edit loss is confounded: Table 1's Δ is absolute, so moves toward parity are counted as degradation; signed re-reading shows roughly equal improvements and reversals.","rationale":"The reader's weakest assumption (CLIP under style shift) is worth testing, but the absolute-difference metric is a more direct threat: the paper's own Table 1 shows that gender-debiasing edits are not 'generally' degraded after fine-tuning. Since the central claim and the safety implications depend on direction, the verdict should be conditional on a signed reanalysis. This is an internal measurement validity issue, not a disagreement with consensus or an ad hominem. The paper contains other evidence (qualitative style shifts, human-annotated unsafe content, ReFACT efficacy plots) that may support a weaker version of the claim, so outright rejection before reanalysis would be premature. However, without the signed reanalysis the abstract's 'generally fail to persist' and 'DoRA strongest reversal' statements are not supported by the data as reported. I therefore keep the reader's CONDITIONAL verdict but for a stronger reason than the style-shift concern.","tokens_in":16061,"tokens_out":10571,"duration_ms":103790,"concrete_test":"Recompute Table 1 with signed Δ = δ(Med_ft) − δ(Med) for all 24 cells and report the sign distribution, mean, and a bootstrap confidence interval over the six professions. Also recompute the unsafe-content Flip and Δ counts separately for safe→unsafe and unsafe→safe flips. If the signed distribution is roughly balanced (the printed decimals suggest it is), the claim that fine-tuning degrades or reverses edits must be withdrawn or substantially qualified; if it is predominantly positive, the central claim survives this objection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 defines δ = |F_p − 50|/50 (lower is better) and evaluates edit persistence with Δ = |δ(Med) − δ(Med_ft)|. Because Δ is unsigned, a fine-tuning run that moves a profession closer to gender parity contributes the same Δ as one that moves away from parity. The paper nevertheless concludes that 'all four fine-tuning methods lead to a degradation of the gender debiasing effect across all six professions.' The printed Table 1 contradicts this: at least 11 of the 24 cells move toward parity, e.g. Teacher+LoRA 0.53→0.24, Hairdresser+DoRA 0.63→0.43, Housekeeper+Full 0.58→0.43. A signed count of the table yields roughly 13 positive (away from parity) and 11 negative (toward parity) cells, so the central claim that edits 'generally fail to persist' is not established by the gender-debiasing experiment. The same sign confusion enters Table 3: a flip from safe to unsafe and a flip from unsafe to safe are pooled in a single Flip count. Additionally, the abstract's 'DoRA exhibits the strongest edit reversal effect' is unsupported by Table 1, where full-size, DoRA, and LoRA all show the same average absolute Δ = 0.115. The reader's style-shift worry is real, but this directional ambiguity is more fundamental: it is internal to the metric and does not depend on whether CLIP classifies anime-style images correctly.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether concept edits applied to text-to-image diffusion models survive a subsequent fine-tuning step, and it reports an empirical study spanning two editing methods (UCE, ReFACT), three model families (SD1.4, SDXL, FLUX), and four fine-tuning approaches (DreamBooth, full fine-tuning, LoRA, DoRA). The authors measure edit persistence using gender-debiasing statistics (δ), appearance/role efficacy, unsafe-content annotation counts, and FID/CLIP scores. Their central conclusion is that edits generally fail to persist through fine-tuning, that DoRA reverses edits most aggressively, and that UCE is more robust than ReFACT. The paper also draws safety implications: fine-tuning could remediate malicious edits, while beneficial edits may need to be reapplied after fine-tuning.","tokens_in":16309,"tokens_out":4199,"duration_ms":42027,"significance":"If the central claim is correct, the paper addresses an important and understudied question in the model-editing deployment lifecycle: whether edited behavior survives subsequent adaptation. The experimental breadth—multiple editing tasks, model families, and fine-tuning methods—is a genuine strength, as is the inclusion of human annotation for unsafe content with reported inter-annotator agreement. The paper is also largely reproducible in principle, since it uses public datasets and open-source methods. However, the quantitative evidence as presented is internally inconsistent in several load-bearing places: the key gender-debiasing table uses an unsigned metric that mixes improvements with degradations, the DoRA 'strongest reversal' claim is not supported by the reported averages, and the unsafe-content tables pool opposite directions of label flips. These issues mean the headline claims are not currently established by the data, although they may be restorable with a corrected, signed analysis.","major_comments":[{"comment":"The claim that 'all four fine-tuning methods lead to a degradation of the gender debiasing effect across all six professions' is contradicted by the paper's own table. Δ is defined as the absolute difference |δ(Med) − δ(Med_ft)|, so a move toward parity is scored identically to a move away from parity. Reading Table 1 with signs, at least 11 of the 24 cells move toward parity (e.g., Teacher+LoRA: δ goes 0.53→0.24; Housekeeper+Full: 0.58→0.43; Housekeeper+DoRA: 0.58→0.43), while the remaining cells move away. A signed count gives roughly 13 away-from-parity and 11 toward-parity changes. The conclusion that edits 'generally fail to persist' is therefore not established by the gender-debiasing experiment as reported. Please report signed changes and per-profession significance tests, and recompute any aggregate claims.","section":"4.2, Table 1"},{"comment":"The claim that 'DoRA exhibits the strongest edit reversal effect' is unsupported by the reported data. In Table 1, full-size fine-tuning, DoRA, and LoRA all have the same average absolute Δ of 0.115, with only DreamBooth lower (0.057). In Table 3, the unsafe-content results are also mixed: full-size fine-tuning increases the number of safe images from 36 in Med to 39 in Med_ft, and DoRA increases it from 36 to 42, while LoRA decreases it to 31. No single metric or aggregate in the paper shows DoRA as the strongest reversal across tasks. If the claim is based on other results (e.g., Table 2 variance or qualitative step analysis), please specify the exact evidence and the chosen metric.","section":"Abstract; 4.2, Table 1; 4.4, Table 3"},{"comment":"The unsafe-content analysis pools opposite directions of label changes. The Flip score counts any label change regardless of direction, and Δ is an unsigned net change, so a safe→unsafe flip and an unsafe→safe flip are treated as equally damaging. In Table 3, for full-size fine-tuning, safe images increase from 36 to 39 and black images drop from 7 to 0, which is not obviously 'weakening the safety level' as claimed. Similarly, the FLUX result in Table 4 reports 12 flips away from safe but does not report how many unsafe/can't-decide images became safe. Please report signed transitions (e.g., a confusion matrix between Med and Med_ft labels) and define which transitions count as degradation.","section":"3.4, 4.4, Table 3"},{"comment":"The evaluation metrics are applied across a large style shift without validation. CLIP-based gender classification and concept-similarity scores are used on anime-style images produced after Naruto-style fine-tuning, but the edits and baselines are validated on photorealistic outputs. If the CLIP classifier's gender decisions or the ReFACT similarity measures are biased by the style shift, the observed 'reversal' could be partly a measurement artifact. The paper should either validate the metrics on style-shifted images (e.g., human-annotated gender labels on a sample of Med_ft images, or per-style classifier accuracy) or explicitly report sensitivity of the main conclusions to this shift.","section":"3.4, 4.2"}],"minor_comments":[{"comment":"There are typos: 'supplimentary' should be 'supplementary' (Section 3.2), and 'attampt' should be 'attempt' (Section 3.3).","section":"3.2, 3.3"},{"comment":"The definition of δ in Section 3.4 relies on F_p, the percentage of generated female-presenting images, but F_p is only defined explicitly in Section 10.1. Please define F_p at first use.","section":"3.4, 10.1"},{"comment":"Figure 3 presents efficacy, generality, and specificity differences only as bar charts without numeric values or error bars; adding the underlying numbers would make the claimed degradation pattern in Section 4.3 verifiable.","section":"Figure 3"},{"comment":"The 'Avg.(std.)' row is formatted confusingly: it mixes standard deviations of δ values with average Δ values, and the standard deviations are reported without explaining that they summarize per-profession δ variation. Please separate the summary statistics for δ and Δ.","section":"Table 1"},{"comment":"The explanation that DoRA overrides edits because of Prodigy's aggressive learning rate schedule is presented as a conclusion but is only a hypothesis. Please label it as speculative or provide supporting evidence.","section":"4.2"},{"comment":"References [40] and [41] are the same paper (Safe Latent Diffusion); please merge them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely and practically important question, and the experimental setup is valuable. However, the headline claims are not currently supported by the paper's own tables, particularly because the unsigned Δ in Table 1 mixes improvements with degradations and because the DoRA 'strongest reversal' claim contradicts the reported averages. These issues are fixable: a signed re-analysis, per-condition confidence intervals, and a clarified unsafe-content transition analysis could restore the central claim or appropriately weaken it. I recommend major revision rather than rejection, as the underlying question and experimental scope warrant another round."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper asks a question worth asking—whether model edits in text-to-image models survive a later fine-tuning step—and it runs a wider study than anyone else has so far. But its headline claim, that edits generally fail to persist, is not actually supported by the gender-debiasing experiment, because the Δ metric there is unsigned and counts movement toward parity as degradation.\n\nThe good parts: the setup is broad—two editing methods (UCE, ReFACT), four fine-tuning recipes (full, LoRA, DoRA, DreamBooth), three model families (SD1.4, SDXL, FLUX). The ReFACT appearance/role results (efficacy, generality, specificity) do consistently show the edit weakening after fine-tuning, and the unsafe-content removal results go in the same direction. The qualitative sequence in Figure 5, where the safety edit deteriorates as the Naruto style comes in, is a genuinely useful observation. The paper also flags its own limitations, including the small Naruto dataset (1,220 images) and the excluded UCE-on-SDXL case.\n\nThe soft spots are real and load-bearing. In Table 1, Δ = |δ(Med)−δ(Med_ft)|. Look at Teacher+LoRA: δ goes from 0.53 to 0.24—closer to parity, i.e. the fine-tuned model is less biased, not more—yet the paper counts this as a degradation of the edit. A signed reading of the table gives roughly 13 cells moving away from parity and 11 moving toward it. The conclusion that all four fine-tuning methods degrade the debiasing edit is not established. The abstract's claim that DoRA shows the strongest reversal is also contradicted by the paper's own average Δ values: Full, DoRA, and LoRA are all 0.115, while DreamBooth is 0.057. Table 3 has the same direction-pooling problem: flipping from safe to unsafe and from unsafe to safe are both counted as \"Flip.\" No confidence intervals, no significance tests, and only six professions, 30 images each.\n\nNone of this kills the general thesis. The ReFACT results and the qualitative evidence suggest fine-tuning does tend to override edits. But the paper has to fix the metric to signed changes, report both directions, and soften the DoRA claim before the headline can be trusted. It also should release code and data.\n\nMy recommendation: send it out for peer review—the question is important and the empirical coverage is unique—but expect a major revision. I'd bring it to a reading group only after the authors respond to the directional-metric issue.","headline":"A broad empirical study of edit persistence after fine-tuning, but the gender-debiasing metric is unsigned and undermines the headline claim, so the paper needs major revision before the conclusion is trustworthy.","tokens_in":16841,"tokens_out":2871,"would_cite":false,"duration_ms":28274,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning erases prior edits in image-generation models","keywords":["model editing","text-to-image diffusion","fine-tuning","edit persistence","gender debiasing","unsafe content removal","parameter-efficient fine-tuning","concept editing"],"falsifier":"Rerun the gender-debiasing and unsafe-removal evaluations on the edited-then-fine-tuned models using human annotators or a style-invariant classifier trained on anime-style images; if the edited behaviors are found intact under those measures, the reversal claim is a measurement artifact rather than a genuine loss.","tokens_in":15836,"feed_emoji":"🖼️","tokens_out":7186,"duration_ms":64960,"temperature":0.7,"pith_summary":"The paper asks whether a behavior injected into a text-to-image model by model editing survives a later fine-tuning step. Across two editing methods, three model families, and four fine-tuning approaches, the authors report that it generally does not: fine-tuning degrades or reverses edits even when the fine-tuning task is stylistically unrelated to the edited behavior. The finding matters because deployed models are often edited for bias mitigation or unsafe-content removal and then fine-tuned for style or domain adaptation. If the paper is right, beneficial edits must be re-applied after fine-tuning, and fine-tuning becomes a plausible way to scrub hidden malicious edits.","feed_headline":"Fine-tuning erases prior edits in image-generation models","feed_subtitle":"Even unrelated style tuning degrades safety and debiasing fixes; DoRA wipes them out most.","key_machinery":"The load-bearing setup is a four-way model comparison: $M$ (original), $M_{ed}$ (edited), $M_{ft}$ (fine-tuned), and $M_{ed\\text{-}ft}$ (edited then fine-tuned). Persistence is quantified as $\\Delta(\\psi; M_{ed}, M_{ed\\text{-}ft})$, the normalized behavioral discrepancy on prompts relevant to the edit $\\psi$; a small $\\Delta$ means the edit survived. To measure it, the paper uses CLIP-based concept similarity, the gender-ratio deviation $\\delta$, human safety annotations on I2P prompts, FID, and CLIP Score. The candidate explanation for reversal is polysemantic neurons, units that encode several unrelated concepts at once, whose stored version of the unwanted concept can be re-activated while fine-tuning strengthens other features.","core_discovery":"The central claim is that the cascade edit-then-fine-tune collapses the edit: an edited model that is later fine-tuned behaves more like a model that was only fine-tuned than like the edited model. The evidence comes from a four-way comparison of base, edited, fine-tuned, and edited-then-fine-tuned variants of Stable Diffusion v1.4, SDXL, and FLUX. Across gender-debiasing, appearance, role, and unsafe-concept-removal tasks, fine-tuning weakens the edit; full-size fine-tuning and DoRA produce the largest reversal, DreamBooth the smallest, and UCE-based edits survive noticeably better than ReFACT-based edits. The authors conclude that editing alone is not durable through the deployment lifecycle, which makes fine-tuning both a potential defense against malicious edits and a hazard to beneficial safety and alignment edits.","pith_inferences":["A testable extension is to vary how semantically close the fine-tuning data is to the edited concept: the reversal should be stronger when the two overlap, and weaker when they are entirely disjoint.","Editing methods could be made lifecycle-aware by adding a regularization term that penalizes drift under simulated fine-tuning updates, turning edit persistence into an explicit optimization objective.","Part of the observed reversal may be a measurement artifact, since CLIP-based gender and concept classifiers are validated on photorealistic images but applied to anime-style outputs; human re-annotation on style-shifted images would separate true edit loss from metric bias.","The results suggest treating edit persistence as a standard evaluation dimension for editing methods, analogous to how continual learning tracks forgetting, rather than evaluating edits in isolation."],"forward_implications":["Safety edits that remove nudity or violence can be partially undone by later style fine-tuning, so deployments should re-audit safety after any fine-tuning.","Gender-debiasing edits degrade across all tested fine-tuning methods, so maintaining parity after adaptation requires re-editing.","Full-size fine-tuning and DoRA are the most effective at erasing prior edits, making them the practical choice for remediating hidden or malicious edits.","DreamBooth preserves edits best and LoRA is more stable than DoRA, so lightweight tuning methods are preferable when edit retention is the priority; among editors, UCE is more durable than ReFACT."],"supporting_citations":[{"why":"Supplies the UCE editing method whose debiasing and unsafe-removal edits are tested for persistence.","marker":"[14]"},{"why":"Provides the ReFACT editing method and the RoAD benchmark used to measure appearance and role edit efficacy.","marker":"[1]"},{"why":"Defines LoRA, one of the fine-tuning methods whose effect on edit persistence is measured.","marker":"[18]"},{"why":"Defines DoRA, the fine-tuning method found to reverse edits most aggressively.","marker":"[27]"},{"why":"Defines DreamBooth and supplies its official subject dataset, the fine-tuning setup that preserves edits best.","marker":"[39]"},{"why":"Provides the Naruto-style image dataset used as the downstream fine-tuning data in the edit-then-fine-tune cascade.","marker":"[5]"},{"why":"Supplies the I2P benchmark of unsafe prompts used for human safety evaluation.","marker":"[41]"},{"why":"Supplies the Winobias profession prompts used to measure gender debiasing and its persistence.","marker":"[48]"}],"fun_headline_variants":["Fine-tuning erases prior edits in image models","DoRA fine-tuning reverses edits most; UCE most robust","Fine-tuning can strip safety edits from diffusion models","Edit-then-fine-tune: model edits don't survive","Fine-tuning reverses edits in T2I models, study says"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metrics used to decide whether an edit is still present remain valid after fine-tuning shifts the image style, so the reported reversals reflect genuine loss of the edit rather than artifacts of scoring anime-style images with CLIP-based classifiers.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning erases prior edits in image models","DoRA fine-tuning reverses edits most; UCE most robust","Fine-tuning can strip safety edits from diffusion models","Edit-then-fine-tune: model edits don't survive","Fine-tuning reverses edits in T2I models, study says"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000806,"raw_usage":{"total_tokens":3579,"prompt_tokens":1024,"completion_tokens":2555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":2474}},"tokens_in":640,"tokens_out":2555,"duration_ms":19280,"temperature":1.0,"reasoning_tokens":2474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:48:19.979712+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the gender-debiasing and unsafe-removal evaluations on the edited-then-fine-tuned models using human annotators or a style-invariant classifier trained on anime-style images; if the edited behaviors are found intact under those measures, the reversal claim is a measurement artifact rather than a genuine loss.","supporting_citations":[{"cited_title":"Unified concept editing in diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the UCE editing method whose debiasing and unsafe-removal edits are tested for persistence."},{"cited_title":"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation","cited_arxiv_id":null,"evidence_quote":"Defines DreamBooth and supplies its official subject dataset, the fine-tuning setup that preserves edits best."},{"cited_title":"Gender bias in coreference resolution: Evaluation and debiasing methods","cited_arxiv_id":null,"evidence_quote":"Supplies the Winobias profession prompts used to measure gender debiasing and its persistence."}],"review_version":1}