Pith. sign in

REVIEW 3 major objections 5 minor 33 references

DuMo: Dual Encoder Modulation Network for Precise Concept Erasure

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read DuMo erases concepts by tuning only the U-Net skip connections with the backbone frozen, preserving non-target images.

desk verdict DuMo's TLMO modulation is a genuine improvement for concept erasure preservation, but the SOTA claim rests on an incomplete baseline comparison and an internal contradiction about the preservation loss. read the letter →

arxiv 2501.01125 v1 pith:HGATVBWC submitted 2025-01-02 cs.CV

classification cs.CV
keywords concepterasurediffusionmodelstext-to-imagegenerationskipconnectionsU-NetmodulationLPIPSdaI2P
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the best way to erase an unwanted concept from a text-to-image diffusion model is to leave the U-Net backbone untouched and put all the erasing power into the skip connections, which carry the high-frequency details of the image. It proposes DuMo, a two-part network: an Eraser with PRior Knowledge (EPR) module that adds a trainable branch onto each skip connection, and a Time-Layer MOdulation (TLMO) process that learns per-layer and per-timestep scaling factors so the erasure is applied exactly where it helps. On three benchmarks—explicit content erasure, cartoon character removal, and artistic style erasure—DuMo reports the best trade-off between removing the target concept and preserving other concepts. If the claim holds, it gives model providers a way to scrub unsafe or copyrighted content without degrading the model's general generative ability.

What carries the argument

The Eraser with PRior Knowledge (EPR) module is a copy of the U-Net encoder blocks, connected to the original skip connection features through zero-initialized 1x1 convolutions (ZeroConv); the original U-Net parameters stay frozen and the original skip features are added to the EPR output. The Timestep-Layer MOdulation process (TLMO) introduces per-layer and per-timestep scalar factors $M_t^l$ that scale each EPR output, with the final skip feature computed as $x_t^l + M_t^l S_{t,\mathrm{cera}}^l$. The modulation factors are learned with the erasing loss plus a preservation loss on empty-prompt outputs. The key mechanism is the combination: freezing the backbone preserves low-frequency structure, while the prior-knowledge skip features and learned scaling localize the erasure to the high-frequency details that carry style and concept identity.

What would settle it

On a set of concepts that require structural change (e.g., changing a person's posture or removing an object while keeping the scene), measure whether DuMo's frozen backbone leaves the structure untouched while the target concept remains; if erasing such a concept forces visible structural corruption of non-target content, the high-frequency-only assumption is shown to fail.

Watch

Extended reading notes

Core claim

The central discovery is that concept erasure can be achieved almost entirely on the high-frequency detail path of a U-Net, not on the structural backbone. The paper shows that a plug-in EPR module, trained with the backbone frozen and the original skip features kept as prior knowledge, removes a target concept while preserving the structure of non-target objects. The subsequent TLMO analysis reveals that erasure effectiveness varies strongly by skip-connection layer and denoising timestep, so a set of learned modulation factors per layer and timestep yields the final trade-off. On the I2P nudity benchmark DuMo detects 34 nude body parts versus 41 for the next-best method, and on cartoon and style erasure it reports LPIPSda of 0.428 and 0.387 respectively, the highest among compared methods. The paper presents these results as state-of-the-art for the three tasks on Stable Diffusion v1.4.

Load-bearing premise

The method assumes that the concept to be erased lives entirely in the high-frequency detail stream (the skip connections) and that the low-frequency backbone structure can be left untouched; if erasing a concept like nudity requires structural changes, such as putting clothes on a character, the preservation advantage weakens.

Editorial extensions

If this is right

  • Removing a concept leaves the model's general image-generation ability largely intact, because the backbone parameters never change.
  • Multiple concepts can be erased simultaneously by summing the outputs of several EPR modules on the skip connections.
  • The per-layer, per-timestep modulation factors provide a controllable way to dial the erasure strength without retraining the whole model.
  • The two-stage training procedure (EPR then TLMO) gives a practical recipe for extending the method to new concepts: train the eraser branch first, then calibrate its scaling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The FreeU-based separation of high/low frequency is approximate; DuMo's success suggests that the skip-connection path is a natural 'style knob', so the same architectural intervention could be used for style transfer or attribute editing, not just erasure.
  • Because the modulation factors are learned per layer and timestep, their final values (e.g., zero on layer group 4 for 'Van Gogh') could be interpreted as a signature of where a concept lives in the network, potentially serving as a diagnostic for concept localization.
  • The method's second-stage preservation loss is applied only on empty prompts; extending it to a set of frequent non-target prompts could further improve preservation, especially for concepts that overlap with common visual categories.
  • The reported 34 nude parts vs 41 for the next best is on a single detector (Nudenet) at 0.6 threshold; the ranking could shift if detector or threshold changes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DuMo, a two-stage concept erasure method for text-to-image diffusion models. In the first stage, an Eraser with Prior Knowledge (EPR) module is trained to modify only U-Net skip-connection features while the backbone is frozen; in the second stage, a Timestep-Layer Modulation (TLMO) process learns per-timestep and per-layer scaling factors for the EPR outputs. The method is evaluated on explicit content erasure using the I2P benchmark and Nudenet detection, on cartoon concept removal, and on artistic style erasure, where it reports the lowest number of detected nude body parts and the highest LPIPSda scores in most comparisons.

Significance. DuMo is an attractive plug-and-play design: it leaves the backbone parameters untouched, intervenes only on skip connections, and the authors provide code. If the reported numbers hold, the method would be a useful contribution to concept erasure because it offers a better preservation of non-target concepts while still erasing the target concept. The use of external metrics (Nudenet, CLIP Score, FID, LPIPS) on held-out prompts is a strength. However, the evidence for the headline state-of-the-art claim is weakened by the omission of the RECE baseline from two of the three benchmarks and by an ablation result that appears to contradict the claimed benefit of the TLMO module, so the current version does not fully substantiate the strongest conclusions.

major comments (3)
  1. [Sec. 4 (Tables 2 and 3)] The paper states that it compares with 10 baseline methods, including RECE (Gong et al. 2024), but RECE appears only in Table 1 and is absent from both the Cartoon Concept Removal and Artistic Style Erasure tables. Since RECE is a general concept-erasure method and is listed among the baselines, its omission leaves the state-of-the-art claim for those two benchmarks unsupported. The authors should include RECE in Tables 2 and 3 or explain explicitly why RECE cannot be applied to those tasks.
  2. [Sec. 4.4, Table 5] Table 5 reports LPIPSda values of 0.426 for EPR alone and 0.358 for EPR+TLMO on the 'Van Gogh' concept, so adding the TLMO modulation stage decreases the composite trade-off metric by 0.068. The decrease comes from a drop in LPIPSe from 0.459 to 0.383, i.e., weaker erasure, even though LPIPSu improves from 0.033 to 0.025. This contradicts the narrative that TLMO improves the erasure-preservation balance and that the joint use of timestep and layer factors attains the optimal trade-off. The authors need to reconcile the quantitative result with the claim or revise the claim.
  3. [Sec. 4.1 and Eq. (6)] The 'Results of Explicit Content Erasure' paragraph says that 'our method achieves superior preservation effects without incorporating any preservation loss,' but Eq. (6) defines the TLMO training loss as L = L_era2 + lambda * L_pre. Unless the authors used lambda = 0 for the nudity experiments, the statement is inconsistent with the method description. Please clarify whether and where the preservation loss was used, and how this affects the interpretation of the explicit-content results.
minor comments (5)
  1. [Sec. 4, first paragraph] The sentence says 'we compare ours with 10 baseline methods' but only nine baseline methods are enumerated: ESD, UCE, SLD-Med, SA, CA, SDD, RECE, MACE, and SPM. Please correct the count or add the missing method.
  2. [Throughout] There are several typographical errors: 'noval' should be 'novel', 'briefy' should be 'briefly', 'protential' should be 'potential', 'our gold' should be 'our goal' in the Method section, and 'MOdulation' should be 'Modulation' in the module name.
  3. [Sec. 3.1, Eq. (1)] Eq. (1) and Fig. 2(a) do not clearly specify the internal structure of the EPR module, particularly how the copied encoder block relates to the function E_cera and the zero-convolution layers. A short architectural description would improve reproducibility.
  4. [Sec. 3.3] The text introduces 'Timestep-Layer MOdulation process (TLMO)' but the abstract and some later passages use 'Time-Layer MOdulation'; please unify the terminology.
  5. [Sec. 4.2] The definition of LPIPSe and LPIPSu is implicit; please state explicitly that LPIPSe is computed against images generated by the unmodified model for erased concepts and LPIPSu against images for unerased concepts, and specify which images are used as references.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; DuMo is evaluated on external benchmarks with parameters trained from stated losses.

full rationale

The paper's central claims (lowest Nudenet count, highest LPIPSda on cartoon and style erasure) are supported by evaluations on I2P, COCO-30K, and artist-prompt datasets that are external to the training losses in Eqs. (3)-(6). The EPR module and TLMO factors are trained to minimize noise-prediction losses, not fitted to the reported Nudenet or LPIPS values. The design premise that skip connections carry high-frequency detail and backbone features carry low-frequency structure is imported from FreeU (Si et al. 2024), an external prior work by different authors, so it is independent support rather than a self-citation chain. The only adjacent concern is that the authors' own RECE baseline appears in Table 1 but not in Tables 2-3; this is a missing-comparison / selective-reporting issue, not a case where a prediction reduces to its input by construction. No equation in the paper defines a reported outcome in terms of the benchmark quantity it is said to predict, and no load-bearing step relies on an unverified self-citation. Therefore the derivation is self-contained and non-circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the FreeU frequency decomposition, the ESD loss, and the unstated hyperparameters eta and lambda. The TLMO factors are fitted parameters rather than derived quantities. No new physical or conceptual entities are postulated.

free parameters (4)
  • TLMO modulation factors M_l^t = learned, shown in Fig. 5
    Per-concept scalars (20 timestep groups x 13 layers, initialized to 1) trained with Eq. 6; they directly set the erasure scale and hence the reported trade-off.
  • Erasure strength eta = not reported
    Appears in Eqs. 3 and 5; controls the strength of the erasure gradient, but its value is not given in the reviewed text.
  • Preservation weight lambda = not reported
    Appears in Eq. 6; balances erasure and preservation losses, but its value is not given in the reviewed text.
  • Nudenet detection threshold = 0.6
    Sets the operating point for the explicit-content evaluation; the reported '34 nude parts' is threshold-dependent.
assumptions (4)
  • domain assumption U-Net backbone features carry low-frequency structural information, while skip connection features carry high-frequency detail and style (FreeU).
    Invoked in Sec. 1 and Sec. 3.1 to justify modifying only skip connections. The method's preservation guarantee depends on this frequency decomposition holding for the target concept.
  • domain assumption The ESD erasing loss (Eq. 3) is a valid training objective for concept erasure even without tuning the backbone.
    Borrowed from Gandikota et al. 2023; the paper does not re-derive or validate its optimality in this architecture.
  • domain assumption LPIPSda = Avg(LPIPSe) - Avg(LPIPSu) is a meaningful trade-off metric for erasure vs preservation.
    Used in Tables 2-5; it is an ad-hoc difference of means, and no calibration against human judgment is provided.
  • ad hoc to paper The preservation loss Lpre with weight lambda adequately regularizes non-target concepts during TLMO training.
    Introduced in Eq. 4 for this paper; the choice of lambda and its effect are not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DuMo: Dual Encoder Modulation Network for Precise Concept Erasure." pith.science (2026). https://pith.science/paper/HGATVBWC

@misc{pith2026250101125,
  author       = {Pith},
  title        = {Pith review of: DuMo: Dual Encoder Modulation Network for Precise Concept Erasure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HGATVBWC}},
  note         = {Machine review of arXiv:2501.01125}
}
read the original abstract

The exceptional generative capability of text-to-image models has raised substantial safety concerns regarding the generation of Not-Safe-For-Work (NSFW) content and potential copyright infringement. To address these concerns, previous methods safeguard the models by eliminating inappropriate concepts. Nonetheless, these models alter the parameters of the backbone network and exert considerable influences on the structural (low-frequency) components of the image, which undermines the model's ability to retain non-target concepts. In this work, we propose our Dual encoder Modulation network (DuMo), which achieves precise erasure of inappropriate target concepts with minimum impairment to non-target concepts. In contrast to previous methods, DuMo employs the Eraser with PRior Knowledge (EPR) module which modifies the skip connection features of the U-NET and primarily achieves concept erasure on details (high-frequency) components of the image. To minimize the damage to non-target concepts during erasure, the parameters of the backbone U-NET are frozen and the prior knowledge from the original skip connection features is introduced to the erasure process. Meanwhile, the phenomenon is observed that distinct erasing preferences for the image structure and details are demonstrated by the EPR at different timesteps and layers. Therefore, we adopt a novel Time-Layer MOdulation process (TLMO) that adjusts the erasure scale of EPR module's outputs across different layers and timesteps, automatically balancing the erasure effects and model's generative ability. Our method achieves state-of-the-art performance on Explicit Content Erasure, Cartoon Concept Removal and Artistic Style Erasure, clearly outperforming alternative methods. Code is available at https://github.com/Maplebb/DuMo

Figures

Figures reproduced from arXiv: 2501.01125 by the authors.

Figure 1
Figure 1. (a) Compared to previous methods, our DuMo framework can precisely erase target concepts to safeguard the models [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Framework overview. Given a target concept [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. The final modulation factors of different timesteps [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of the erasing effect of different [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Qualititave results of Explicit Content Erasure. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Image visualization of Artistic Style Erasure. Up [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Image visualization of our Component verification [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 12 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Dhariwal, P.; and Nichol, A. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 8780--8794

  4. [4]

    Fan, C.; Liu, J.; Zhang, Y.; Wei, D.; Wong, E.; and Liu, S. 2023. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. arXiv preprint arXiv:2310.12508

  5. [5]

    Gandikota, R.; Materzynska, J.; Fiotto-Kaufman, J.; and Bau, D. 2023. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2426--2436

  6. [6]

    Gandikota, R.; Orgad, H.; Belinkov, Y.; Materzy \'n ska, J.; and Bau, D. 2024. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 5111--5120

  7. [7]

    Gong, C.; Chen, K.; Wei, Z.; Chen, J.; and Jiang, Y.-G. 2024. Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models. arXiv preprint arXiv:2407.12383

  8. [8]

    Heng, A.; and Soh, H. 2024. Selective amnesia: A continual learning approach to forgetting in deep generative models. Advances in Neural Information Processing Systems, 36

Show all 33 references
  1. [9]

    L.; and Choi, Y

    Hessel, J.; Holtzman, A.; Forbes, M.; Bras, R. L.; and Choi, Y. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718

  2. [10]

    Huang, C.-P.; Chang, K.-P.; Tsai, C.-T.; Lai, Y.-H.; and Wang, Y.-C. F. 2023. Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. arXiv preprint arXiv:2311.17717

  3. [11]

    Hunter, T. 2023. AI porn is easy to make now. For women, that's a nightmare. The Washington Post, NA--NA

  4. [12]

    H.; Brown, L.; Cheng, J.; Khan, M.; Gupta, A.; Workman, D.; Hanna, A.; Flowers, J.; and Gebru, T

    Jiang, H. H.; Brown, L.; Cheng, J.; Khan, M.; Gupta, A.; Workman, D.; Hanna, A.; Flowers, J.; and Gebru, T. 2023. AI Art and its Impact on Artists. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 363--374

  5. [13]

    Kim, S.; Jung, S.; Kim, B.; Choi, M.; Shin, J.; and Lee, J. 2023. Towards safe self-distillation of internet-scale text-to-image diffusion models. arXiv preprint arXiv:2307.05977

  6. [14]

    Kumari, N.; Zhang, B.; Wang, S.-Y.; Shechtman, E.; Zhang, R.; and Zhu, J.-Y. 2023. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 22691--22702

  7. [15]

    Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 1...

  8. [16]

    Lu, S.; Wang, Z.; Li, L.; Liu, Y.; and Kong, A. W.-K. 2024. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6430--6440

  9. [17]

    Luo, W.; Hu, T.; Zhang, S.; Sun, J.; Li, Z.; and Zhang, Z. 2024. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36

  10. [18]

    Lyu, M.; Yang, Y.; Hong, H.; Chen, H.; Jin, X.; He, Y.; Xue, H.; Han, J.; and Ding, G. 2024. One-dimensional Adapter to Rule Them All: Concepts Diffusion Models and Erasing Applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7559--7568

  11. [19]

    Nichol, A.; Dhariwal, P.; Ramesh, A.; Shyam, P.; Mishkin, P.; McGrew, B.; Sutskever, I.; and Chen, M. 2021. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741

  12. [20]

    Q.; and Dhariwal, P

    Nichol, A. Q.; and Dhariwal, P. 2021. Improved denoising diffusion probabilistic models. In International conference on machine learning, 8162--8171. PMLR

  13. [21]

    Parmar, G.; Zhang, R.; and Zhu, J.-Y. 2022. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11410--11420

  14. [22]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

  15. [23]

    Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2): 3

  16. [24]

    Rando, J.; Paleka, D.; Lindner, D.; Heim, L.; and Tram \`e r, F. 2022. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610

  17. [25]

    L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al

    Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing syste...

  18. [26]

    Schramowski, P.; Brack, M.; Deiseroth, B.; and Kersting, K. 2023. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22522--22531

  19. [27]

    Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems,...

  20. [28]

    Setty, R. 2023. Ai art generators hit with copyright suit over artists’ images. Bloomberg Law. Accessed on February, 1: 2023

  21. [29]

    Shi, Z.; Zhou, X.; Qiu, X.; and Zhu, X. 2020. Improving image captioning with better use of captions. arXiv preprint arXiv:2006.11807

  22. [30]

    Si, C.; Huang, Z.; Jiang, Y.; and Liu, Z. 2024. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4733--4743

  23. [31]

    Smith, M. 2022. Tutorial: How to remove the safety filter in 5 seconds

  24. [32]

    Zhang, L.; Rao, A.; and Agrawala, M. 2023. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3836--3847

  25. [33]

    Zhang, Y.; Jia, J.; Chen, X.; Chen, A.; Zhang, Y.; Liu, J.; Ding, K.; and Liu, S. 2023. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now. arXiv preprint arXiv:2310.11868

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.