REVIEW 3 major objections 5 minor 33 references
Don't Forget your Inverse DDIM for Image Editing
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Inversion-time self-attention maps alone can guide high-fidelity text edits in pre-trained diffusion models.
desk verdict A simple, efficient self-attention guidance idea that is unfortunately oversold by an unsupported central claim and a false abstract claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the collection of self-attention maps the diffusion U-Net computes during DDIM inversion; for 512x512 images the paper uses the 32x32 maps from the second and third encoder blocks, and 16x16 cross-attention maps for blending. These stored maps are the reference signal: a reconstruction loss compares them with the maps produced during DDIM sampling under the target prompt, and its gradient steers the latent trajectory back to the input's structure. The mechanism works because self-attention captures global interactions among all image tokens, so matching these maps preserves layout and identity without requiring the latent itself to match the input. A decreasing schedule for $\lambda$ lets early steps perform the edit and later steps refine reconstruction.
What would settle it
A direct test is to run SAGE twice on the same image and prompt, once with the real inversion-time self-attention maps and once with maps recorded from a different image's inversion; if structure and background preservation scores (LPIPS, structure distance) stay the same, the maps are not the carrier of the reconstruction signal and the method's explanation is wrong.
Extended reading notes
Core claim
The discovery is that reconstruction guidance for editing does not need to operate in the pixel or latent space, nor require optimizing a null embedding: the self-attention maps from a single deterministic DDIM inversion are a sufficient and stable target. During generation, SAGE applies classifier-free guidance between the input prompt and the target prompt, and simultaneously minimizes $L_{\text{self}}^t = \sum_i \|S_{\text{in}, t,i} - S_{\text{out}, t,i}\|_1$, where $S_{\text{in},t,i}$ are the self-attention maps recorded during inversion and $S_{\text{out},t,i}$ are the maps from the U-Net when conditioned on the target prompt at the same timestep. The gradient, scaled by a decreasing weight $\lambda$, is added to the noise update. The paper argues this stabilizes the reverse process and preserves unedited regions because self-attention encodes global spatial relationships, and it combines this with cross-attention local blending and replacement to refine where edits occur.
Load-bearing premise
The load-bearing empirical premise is that the self-attention maps recorded during DDIM inversion remain the correct targets while the model generates under the edited prompt, and that minimizing their L1 distance pulls the latent toward the input without suppressing the intended edit.
Editorial extensions
If this is right
- SAGE completes an edit in about 12.6 seconds on an A100 with 7.4 GB peak memory, versus over a minute for null-text inversion, while keeping reconstruction metrics competitive.
- Because reconstruction is anchored in attention-map space, the method transfers across PieBench and MagicBrush with the same hyperparameter set.
- Prompt-based operations such as word swaps, object removal, and style transfer all reduce to the same pipeline: store inversion maps, then apply self-attention guidance plus cross-attention local blending.
- The ablation shows that guidance from self-attention maps outperforms guidance from cross-attention maps and from latent-space ($z_t$) targeting on structure distance and LPIPS.
Reading between the lines
- Editorial inference: because the reference maps come from an arbitrary inversion, the same guidance could be applied on top of exact or coupled inversion schemes by simply replacing the source of the recorded maps; the mechanism itself does not depend on DDIM's update rule.
- Editorial inference: the reported failure mode in object removal, where removed objects are replaced by structurally similar content, suggests that masking self-attention guidance in the removed region is a concrete next test; the paper itself flags this direction.
- Editorial inference: the fixed layer selection (32x32 maps from encoder blocks 2 and 3) is an empirical choice, so a systematic sweep over layers and resolutions could either harden the method or reveal that only a subset carries the reconstruction signal.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SAGE, a prompt-based image editing method that combines DDIM inversion with a self-attention guidance loss and cross-attention manipulation (local blending, cross-attention replacement, reweighting) to preserve unedited regions while applying text-driven edits. The central claim is that the self-attention maps recorded during DDIM inversion contain sufficient information to steer generation toward the input without per-image optimization, yielding high-fidelity editing with lower memory and compute than optimization-based baselines. The authors provide quantitative benchmarks on PieBench and MagicBrush, a user study, ablation studies, and public code.
Significance. If the central claim is established, SAGE would be a valuable practical contribution: it avoids per-image text-embedding optimization, runs in comparable time to Direct Inversion, and uses less memory in FP16. The paper ships code, provides ablations, and evaluates on two benchmark datasets, which are clear strengths. However, the current evidence does not yet isolate the effect of self-attention guidance from the local-blending component, and the manuscript contains internal inconsistencies about which CFG branch is guided. These issues are load-bearing for the claimed attribution of SAGE's fidelity to self-attention guidance, so the contribution, while promising, needs strengthening before publication.
major comments (3)
- [§III-A, Fig. 3, §V] There is a direct contradiction about which branch provides the self-attention maps S_out in Eq. (3). §III-A states that "during synthesis, corresponding maps S_out_i,t are collected from ε_θ(z_t, P_out)", whereas the caption of Fig. 3 and the Conclusion state that guidance is applied to the branch conditioned on the initial prompt P_in. The mechanism changes entirely depending on which branch is used: guiding toward P_in inversion maps is a reconstruction mechanism, while guiding toward P_out maps would push the edited image away from the input. The authors must clarify the exact branch and justify it, because the method's definition and the interpretation of all ablations depend on this choice.
- [Supp. §VII-C, Table VI; §III-B, Eq. (5)] The reconstruction-only experiment in Supp. Table VI tests the load-bearing claim that self-attention guidance alone preserves the input. With P_in = P_out and no local blending or cross-attention replacement, SAGE achieves structure distance 12.0, LPIPS 65.8, and MSE 65.1, markedly worse than Direct Inversion (3.0, 51.7, 28.9) and Null-Text Inversion (3.3, 54.8, 31.1). Yet the headline results in Tables II and IV come from the full pipeline, which includes local blending (Eq. (5)) that directly replaces z_{t-1} with the inversion latent z_in_{t-1} outside the edit mask. That direct latent copying is an explicit reconstruction mechanism that can explain the background-preservation gains independently of self-attention guidance. To support the paper's attribution, the authors should add an ablation that runs local blending without self-attention guidance and self-attention guidance without local blending on the same benchmark, and report both structure and background metrics.
- [Abstract; §IV-D, Table IV] The abstract's claim that "all 47 surveyed users preferred SAGE over competing methods" is contradicted by Table IV. The user study recruited 22 participants for the PieBench comparisons and 25 additional participants for the MagicBrush DI comparison (total 47), but SAGE's global preference over DI is 52.7% on both, and its global preference over NT is 54.5%. Many of the 47 users did not prefer SAGE in those comparisons. The same internal contradiction appears between the abstract and the Introduction's "preferred in over 60% of cases." The claims in the abstract and Sec. IV-D must be corrected to report preference rates per comparison, not a universal-preference statement.
minor comments (5)
- [Supp. §VII-B] Typo: "An appropiate balance" should be "An appropriate balance".
- [Supp. §VI] The dataset name is written inconsistently as "PIE-Bench" in the supplementary material and "PieBench" elsewhere; please unify.
- [Table V] The footnote states "FP32 best results are marked with *", but in the table the asterisks mark Plug-n-Play time and Direct Inversion memory, which are not clearly the best in their columns (e.g., SAGE FP16 ties at 7.4 GB memory). Please clarify what the asterisk denotes.
- [Eq. (3)] The summation index N and the subscript i are used without defining the set of blocks or the spatial resolution of the maps; please specify that the sum ranges over selected U-Net blocks and state the map sizes (e.g., 32x32 from blocks 2 and 3, as in Supp. VI-A) in the main text.
- [Fig. 9 caption] The phrase "'a cat tiger sitting next to a mirror" contains an extra leading apostrophe; please fix the typo in the prompt string.
Circularity Check
No significant circularity: SAGE's editing fidelity is evaluated against external benchmarks, and the single self-cited λ schedule is not load-bearing.
full rationale
The paper's forward chain is a constructive method, not a derivation of a quantity from a fitted input. The reconstruction signal L_self (Eq. 3) is defined as an L1 distance between self-attention maps recorded during DDIM inversion and maps produced during generation, and it is used as an optimization objective for the latent update (Eq. 4); nothing in the reported metrics is fitted or predicted from L_self. The hyperparameter schedule 'Following [24], factor λ progressively decreases with t' is borrowed from prior work by two of the present authors, but it is a scalar schedule, not a load-bearing derivation, and external benchmarks (PieBench, MagicBrush) and ablations (Table I, Supp. Table VI) are used for evaluation. The fact that local blending (Eq. 5) directly copies inversion latents in masked regions is a mechanism-attribution concern, not a circularity: the method's advertised claims would still be empirically testable. No step reduces by construction to its own input, and the central comparison is against external baselines and user judgments.
Assumptions & free parameters
free parameters (7)
- Classifier-free guidance scale w =
7.5
- Self-attention guidance scale lambda =
200
- Cross-attention reweighting factor =
2.0
- Local blending schedule =
first 40 of 50 steps
- Cross-attention replacement length =
first 5 of 50 steps
- Selected self-attention map resolution and layers =
32x32 maps from 2nd/3rd encoder blocks; 16x16 cross-attention; 24x24 for 768px
- FP16 loss scaling factor =
500
assumptions (5)
- domain assumption DDIM inversion with Pin yields a noise trajectory that, when sampled under CFG with Pout, can be redirected to produce a valid edited image.
- domain assumption Self-attention maps in the pre-trained Stable Diffusion U-Net encode spatial structure that is comparable across inversion and generation trajectories.
- domain assumption Minimizing L1 distance between attention maps via gradient on zt improves reconstruction without suppressing the intended edit.
- domain assumption Stable Diffusion 1.4 and 2.1 checkpoints behave as published, and the chosen attention layers and map sizes are representative.
- domain assumption Standard metrics, LPIPS, SSIM, CLIP-T, DINO, and structure distance, provide meaningful measures of editing quality and background preservation.
Cite this review
Pith. "Pith review of Don't Forget your Inverse DDIM for Image Editing." pith.science (2026). https://pith.science/paper/BPVLQH6V
@misc{pith2026250509571,
author = {Pith},
title = {Pith review of: Don't Forget your Inverse DDIM for Image Editing},
year = {2026},
howpublished = {\url{https://pith.science/paper/BPVLQH6V}},
note = {Machine review of arXiv:2505.09571}
}
read the original abstract
The field of text-to-image generation has undergone significant advancements with the introduction of diffusion models. Nevertheless, the challenge of editing real images persists, as most methods are either computationally intensive or produce poor reconstructions. This paper introduces SAGE (Self-Attention Guidance for image Editing) - a novel technique leveraging pre-trained diffusion models for image editing. SAGE builds upon the DDIM algorithm and incorporates a novel guidance mechanism utilizing the self-attention layers of the diffusion U-Net. This mechanism computes a reconstruction objective based on attention maps generated during the inverse DDIM process, enabling efficient reconstruction of unedited regions without the need to precisely reconstruct the entire input image. Thus, SAGE directly addresses the key challenges in image editing. The superiority of SAGE over other methods is demonstrated through quantitative and qualitative evaluations and confirmed by a statistically validated comprehensive user study, in which all 47 surveyed users preferred SAGE over competing methods. Additionally, SAGE ranks as the top-performing method in seven out of 10 quantitative analyses and secures second and third places in the remaining three.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Photorealistic text-to-image diffusion models with deep language understanding,
C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., “Photorealistic text-to-image diffusion models with deep language understanding,” NeurIPS, vol. 35, pp. 36 479–36 494, 2022. 1, 3
work page 2022
-
[2]
High- resolution image synthesis with latent diffusion models,
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in CVPR, 2022, pp. 10 684–10 695. 1, 2, 3
work page 2022
-
[3]
Null- text inversion for editing real images using guided diffusion models,
R. Mokady, A. Hertz, K. Aberman, Y . Pritch, and D. Cohen-Or, “Null- text inversion for editing real images using guided diffusion models,” in CVPR, 2023, pp. 6038–6047. 1, 2, 3, 4, 5, 6, 7
work page 2023
-
[4]
Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models,
D. Miyake, A. Iohara, Y . Saito, and T. Tanaka, “Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models,” arXiv preprint arXiv:2305.16807 , 2023. 1, 2, 3, 4, 6, 7
arXiv 2023
-
[5]
Zero-shot image-to-image translation,
G. Parmar, K. Kumar Singh, R. Zhang, Y . Li, J. Lu, and J.-Y . Zhu, “Zero-shot image-to-image translation,” in ACM SIGGRAPH, 2023, pp. 1–11. 1, 2, 4, 5, 6, 7
work page 2023
-
[6]
Classifier-free diffusion guidance,
J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applica- tions, 2021. 2, 4, 1
work page 2021
-
[7]
Denoising diffusion implicit models,
J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in ICLR, 2021. 1, 2, 3
work page 2021
-
[8]
U-net: Convolutional networks for biomedical image segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in MICAI, 2015, pp. 234–241. 1
work page 2015
Show all 33 references
-
[9]
SDEdit: Guided image synthesis and editing with stochastic differential equations,
C. Meng, Y . He, Y . Song, J. Song, J. Wu, J.-Y . Zhu, and S. Ermon, “SDEdit: Guided image synthesis and editing with stochastic differential equations,” in ICLR, 2022. 2
2022
-
[10]
An image is worth one word: Personalizing text-to- image generation using textual inversion,
R. Gal, Y . Alaluf, Y . Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-or, “An image is worth one word: Personalizing text-to- image generation using textual inversion,” in ICLR, 2023. 2
2023
-
[11]
Dreambooth: Fine tuning text-to-image diffusion models for subject- driven generation,
N. Ruiz, Y . Li, V . Jampani, Y . Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject- driven generation,” in CVPR, 2023, pp. 22 500–22 510. 2
2023
-
[12]
Diffusionclip: Text-guided diffusion models for robust image manipulation,
G. Kim, T. Kwon, and J. C. Ye, “Diffusionclip: Text-guided diffusion models for robust image manipulation,” in CVPR, 2022, pp. 2426–2435. 2
2022
-
[13]
Adding conditional control to text-to-image diffusion models,
L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in ICCV, 2023, pp. 3836–3847. 2
2023
-
[14]
Plug-and-play diffusion features for text-driven image-to-image translation,
N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, “Plug-and-play diffusion features for text-driven image-to-image translation,” in CVPR, 2023, pp. 1921–1930. 2, 3, 6, 1
2023
-
[15]
Direct inversion: Boosting diffusion-based editing with 3 lines of code,
X. Ju, A. Zeng, Y . Bian, S. Liu, and Q. Xu, “Direct inversion: Boosting diffusion-based editing with 3 lines of code,” arXiv preprint arXiv:2304.04269, 2023. 2, 4, 5, 6, 7, 1
2023 arXiv
-
[16]
Prompt-to-prompt image editing with cross-attention control,
A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y . Pritch, and D. Cohen-or, “Prompt-to-prompt image editing with cross-attention control,” in ICLR, 2023. 2, 4, 5, 1
2023
-
[17]
Edict: Exact diffusion inversion via coupled transformations,
B. Wallace, A. Gokul, and N. Naik, “Edict: Exact diffusion inversion via coupled transformations,” in CVPR, 2023, pp. 22 532–22 541. 2
2023
-
[18]
Exact diffusion inversion via bidirectional integration approximation,
G. Zhang, J. P. Lewis, and W. B. Kleijn, “Exact diffusion inversion via bidirectional integration approximation,” in ECCV. Springer, 2024, pp. 19–36. 2
2024
-
[19]
Inversion-free image editing with natural language,
S. Xu, Y . Huang, J. Pan, Z. Ma, and J. Chai, “Inversion-free image editing with natural language,” arXiv preprint arXiv:2312.04965 , 2023. 3
2023 arXiv
-
[20]
Denoising diffusion probabilistic models,
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in NeurIPS, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 6840–6851. 3
2020
-
[21]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, vol. 30, 2017. 3
2017
-
[22]
Diffusion autoencoders: Toward a meaningful and decodable represen- tation,
K. Preechakul, N. Chatthee, S. Wizadwongsa, and S. Suwajanakorn, “Diffusion autoencoders: Toward a meaningful and decodable represen- tation,” in CVPR, 2022. 3
2022
-
[23]
Improving tuning-free real image editing with proximal guidance,
L. Han, S. Wen, Q. Chen, Z. Zhang, K. Song, M. Ren, R. Gao, A. Stathopoulos, X. He, Y . Chen, D. Liu, Q. Zhangli, J. Jiang, Z. Xia, A. Srivastava, and D. Metaxas, “Improving tuning-free real image editing with proximal guidance,” in WACV, 2024. 4, 7
2024
-
[24]
Zero- shot spatial layout conditioning for text-to-image diffusion models,
G. Couairon, M. Careil, M. Cord, S. Lathuili `ere, and J. Verbeek, “Zero- shot spatial layout conditioning for text-to-image diffusion models,” in ICCV, 2023, pp. 2174–2183. 4
2023
-
[25]
Magicbrush: A manually annotated dataset for instruction-guided image editing,
K. Zhang, L. Mo, W. Chen, H. Sun, and Y . Su, “Magicbrush: A manually annotated dataset for instruction-guided image editing,” in NeurIPS,
-
[26]
Splicing vit features for semantic appearance transfer,
N. Tumanyan, O. Bar-Tal, S. Bagon, and T. Dekel, “Splicing vit features for semantic appearance transfer,” in CVPR, 2022, pp. 10 748–10 757. 5
2022
-
[27]
The unreasonable effectiveness of deep features as a perceptual metric,
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018. 5
2018
-
[28]
Image quality assessment: from error visibility to structural similarity,
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing , vol. 13, no. 4, pp. 600–612, 2004. 5
2004
-
[29]
GODIV A: generating open-domain videos from natural descriptions,
C. Wu, L. Huang, Q. Zhang, B. Li, L. Ji, F. Yang, G. Sapiro, and N. Duan, “GODIV A: generating open-domain videos from natural descriptions,” CoRR, vol. abs/2104.14806, 2021. 5
2021 arXiv
-
[30]
Emerging properties in self-supervised vision transformers,
M. Caron, H. Touvron, I. Misra, H. J ´egou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in ICCV, 2021, pp. 9650–9660. 5
2021
-
[31]
On distillation of guided diffusion models,
C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans, “On distillation of guided diffusion models,” in CVPR,
-
[32]
Flow straight and fast: Learning to generate and transfer data with rectified flow,
X. Liu, C. Gong, and qiang liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” in ICLR, 2023. 8
2023
-
[33]
Fireflow: Fast inversion of rectified flow for image semantic editing,
Y . Deng, X. He, C. Mei, P. Wang, and F. Tang, “Fireflow: Fast inversion of rectified flow for image semantic editing,” arXiv preprint arXiv:2412.07517, 2024. 8 1 Don’t Forget your Inverse DDIM for Image Editing Supplementary Material The supplementary materials provide additi...
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.