Pith. sign in

REVIEW 3 major objections 5 minor 93 references

Image Editing As Programs with Diffusion Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that decomposing any image-editing instruction into a program of five atomic operations executed sequentially on a diffusion-transformer backbone outperforms end-to-end editors, especially on layout-changing and…

desk verdict A well-engineered modular editing framework whose headline gains rest on a self-referential GPT-4o judge; the core idea is sound, but the evaluation overclaims. read the letter →

arxiv 2506.04158 v1 pith:6SUYVLX4 submitted 2025-06-04 cs.CV

Yujia Hu , Songhua Liu , Zhenxiong Tan , Xingyi Yang , Xinchao Wang This is my paper
classification cs.CV
keywords instruction-drivenimageeditingdiffusiontransformeratomicoperationschain-of-thoughtreasoningneuralprograminterpreterlayout-changingeditsRoIinpaintingbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Image-editing models follow instructions reliably when an edit keeps the layout intact, such as changing colors, styles, or lighting, but fall apart when an instruction demands moving, resizing, adding, or removing things, because those edits require reorganizing the whole scene. This paper's central proposal is to stop treating each instruction as one monolithic generation task and instead write a program for it: a vision-language model parses the instruction into a sequence from a small set of atomic operations, such as locating a region, inpainting, editing attributes, compositing edges, and applying a global transformation, and each operation is executed by a lightweight adapter sharing one diffusion-transformer backbone. On the MagicBrush and AnyEdit benchmarks, the paper reports consistent state-of-the-art gains, with the largest margins on complex multi-step instructions. If this decomposition result holds, modular and inspectable editing pipelines would be a better architecture for instruction-driven editing than single-pass end-to-end models.

What carries the argument

The central machinery is the program formulation: an editing instruction $T$ is rewritten as a sequential combination of atomic operations $A_k$ over a small alphabet, and a chain-of-thought prompt on a vision-language model classifies each sub-instruction into one of thirteen directive categories (add, remove, replace, action change, move, resize, color, material, background, appearance, expression, tone, style) that map onto the five atomic executables. A neural program interpreter then runs them in order on a shared DiT backbone fine-tuned with LoRA adapters. The other load-bearing pieces are a segmentation-based RoI localizer, a layout reconfiguration step in which an LLM rewrites bounding boxes for move, resize, and add, and an annular-mask fusion network that blends edited regions back into the image.

What would settle it

Run the same AnyEdit comparisons with a judge that did not participate in planning or data curation, such as a panel of human raters or a different strong vision-language model evaluating the same outputs, and compare scores on the local-semantic and complex multi-step subsets. The central claim stands only if the advantage over the strongest baseline survives when the judge is decoupled from the planner.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the hard part of instruction-driven editing is structural: models trained end-to-end handle attribute and style edits but consistently fail at edits that alter spatial layout, such as add, remove, replace, move, resize, and action change. The paper's response is the claim that any such instruction can be reduced to a composition of five atomic operations executed as a program: RoI localization, RoI inpainting, RoI editing, RoI compositing, and global transformation. The author's claim is that this reduction, together with a neural program interpreter, turns a diffusion transformer into an editor that outperforms previous methods on standard benchmarks and rivals or beats large proprietary models on complex multi-step instructions, while preserving consistency between the edited object and its surroundings.

Load-bearing premise

The results on the AnyEdit benchmark are scored by GPT-4o, the same model that decomposes the instructions, proposes bounding boxes, and filters the training data; if that judge simply prefers edits that match its own planning, the reported gains could shrink or disappear under an independent judge.

Editorial extensions

If this is right

  • Any edit expressible as a sequence of the five atomic operations becomes tractable on the same DiT backbone, so the framework's coverage is set by the planner's ability to parse instructions, not by retraining for each new edit type.
  • Complex multi-step instructions are executed in a traceable order, so failures can be attributed to one operation rather than to the whole model.
  • Each atomic adapter is lightweight and shares one backbone, so adding a new operation type costs one LoRA fine-tune on that operation's data instead of a full end-to-end retrain.
  • For layout-changing edits, the gap to end-to-end baselines is largest, implying that decomposition, not scale of the base model, is what fixes structural editing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The action-change pipeline (inpaint the old region, edit the object, composite) is a general recipe for any change-the-pose instruction; we would expect it to transfer to video editing, where the same decomposition could be applied frame-wise with temporal constraints.
  • Because the judge and planner overlap, an independent human-rated evaluation of the same outputs is the cheapest decisive check of the claimed advantage; we do not interpret the published GPT-4o scores as settled evidence.
  • The taxonomy of consistent versus inconsistent layout edits could be reused as a diagnostic: benchmark designers can partition any editing test set along this axis and report scores separately, since aggregate scores hide the dichotomy that motivates this method.
  • The per-operation latency reported by the paper suggests that for interactive use the bottleneck is the number of operations, so optimizing RoI localization or fusing adjacent operations is an obvious extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Image Editing As Programs (IEAP), a modular instruction-driven image editing framework built on the FLUX.1-dev Diffusion Transformer. The central idea is to parse free-form editing instructions with a VLM (GPT-4o) via Chain-of-Thought reasoning into a sequence of atomic operations: RoI localization, RoI inpainting, RoI editing, RoI compositing, and global transformation, each implemented as a lightweight LoRA adapter sharing the same backbone. An LLM-based layout reconfiguration step handles move/resize/add operations that require spatial changes. The paper reports a taxonomy study showing that DiT-based editors degrade on layout-changing edits, and then presents quantitative comparisons on MagicBrush and AnyEdit, qualitative comparisons with open-source and proprietary models, per-category results, ablations, and a discussion of limitations.

Significance. The paper's core architectural claim—that decomposing instructions into a small set of atomic operations executed sequentially is a better design than monolithic end-to-end editing for layout-changing instructions—is plausible and potentially influential. The ablation study (Table 3) is informative and generally supports the contribution of each module. The authors also ship code, which is valuable for reproducibility. However, the main evidence for 'significantly outperforming state-of-the-art' rests on GPT-4o ratings on AnyEdit, while GPT-4o is also the model used inside the pipeline for planning and layout. The objective MagicBrush numbers are marginal and mixed, and several per-category AnyEdit results actually lag the best baseline. These issues weaken the central empirical claim as it currently stands, although they are addressable with additional evaluation.

major comments (3)
  1. [Sec. 5.1 / Tables 1–2 / Appendix B] The AnyEdit scores that drive the 'significantly outperforms state-of-the-art' claim are GPT-4o ratings (Sec. 5.1), but GPT-4o is also the model used for instruction decomposition (Sec. 4.2, Eqs. 4–6), layout proposal and add-object bounding-box prediction (Appendix B), and training-data filtering (Sec. 5.1). Because the judge is the same system that generates the plans and layout decisions, it may systematically prefer outputs that conform to its own decomposition, inflating IEAP's AnyEdit gaps (e.g., GPT 4.41 vs 4.13 in Table 1; 4.42/4.54/4.51 vs 4.08/4.21/4.45 in Table 2). This is a measurement-validity confound, not an internal inconsistency. I request a human evaluation or the use of a judge model that is not used anywhere in the pipeline, together with inter-annotator agreement or equivalent reliability statistics.
  2. [Table 1 / Appendix C / Tables 4–20] The claim of 'significant and comprehensive improvements' is not fully supported by the reported numbers. On MagicBrush the gains are marginal and not uniform: L1 is slightly worse than ICEdit (0.060 vs 0.058) while other metrics improve by small margins. In Appendix C, IEAP is not the best on all AnyEdit categories: on Remove (Table 5) the GPT average is 3.91 vs ICEdit's 4.29, and on Textual Change (Table 20) it is 4.20 vs ICEdit's 4.43. No error bars or significance tests are reported in any of Tables 1, 2, or 4–20, so the word 'significantly' is an overstatement. Please add per-category variance/confidence intervals, significance tests where possible, and a discussion of categories where IEAP does not achieve state-of-the-art.
  3. [Sec. 4.2, Eq. (3) / Appendix B] Equation (3) asserts that every editing instruction can be represented as a sequence of the five atomic operations, but this is an unverified assumption. Appendix B's CoT prompt actually enumerates 13 concrete instruction categories, and the formal mapping from these categories to the five primitives is not specified. The correctness of this reduction is load-bearing for the 'unified framework' claim; the paper currently provides no decomposition-accuracy measurement or analysis of failure cases where the planner misclassifies or omits steps. Please clarify the mapping explicitly and report a decomposition evaluation, for example by having human raters judge whether the VLM's atomic plan is faithful to the original instruction.
minor comments (5)
  1. [Sec. 4.2, Eq. (3)] The notation in Eq. (3) appears garbled ('KM' with a subscript, and 'L' is referenced as the sequential program combination but does not appear in the displayed equation); please define the composition operator and the variables used.
  2. [Sec. 5.1 / Appendix C] The text states that 16 categories are selected from AnyEdit, but Appendix C contains 17 per-category tables (Tables 4–20); please reconcile the count or explain the discrepancy.
  3. [Appendix B] Both layout-prompt templates contain the typo 'coordinante' instead of 'coordinate', and the example in the move prompt has an incomplete formatting line; please proofread the appendix.
  4. [Sec. 3.2 / Sec. 5.2] The macro-task name in Sec. 5.2 is 'overall semantic editing,' while Sec. 3.2 uses 'overall content editing'; please keep terminology consistent throughout.
  5. [Table 1] The table reports CLIPout on AnyEdit even though Sec. 5.1 says some AnyEdit categories lack reference captions; please clarify how CLIPout is computed for the AnyEdit subset and whether the metric is available for all categories.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation in IEAP reduces to its own inputs; the central decomposition claim is an empirically tested design hypothesis, and the GPT-4o judge/planner overlap is a measurement-validity confound rather than a circular step.

full rationale

IEAP's central claim is that complex instructions can be decomposed into five atomic operations executed sequentially. This is stated as a modeling reduction (Sec. 4.2, Eq. 3) rather than derived from the data, and the paper's evidence for it comes from ablations (Table 3), qualitative comparisons, and objective MagicBrush metrics (CLIP, L1, DINO). None of Eqs. 4-11 fits a parameter to the same quantity that is later reported as a prediction; each specialized model is trained on its own task and evaluated on held-out AnyEdit/MagicBrush splits. The reliance on OminiControl [59] for conditioning is a same-group citation, but it is used as a backbone and ablation baseline, and the central contribution does not reduce to it. The only concern is that AnyEdit's GPT-4o ratings are issued by the same model used to decompose instructions, propose bounding boxes, and filter training data (Sec. 5.1; Appendix B). That is a potential preference/self-evaluation confound, but the ratings are not definitionally determined by the planner, and the paper also reports objective metrics on MagicBrush, so the derivation chain is not circular. This concern belongs to measurement validity, not circularity.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the strength of the pretrained FLUX.1-dev DiT, the accuracy of segmentation and VLM planning, and the representativeness of the benchmarks. The only hand-chosen parameters that visibly affect the pipeline are the annular mask radii and the LoRA hyperparameters. No new physical entities are introduced.

free parameters (2)
  • Annular mask radii k1, k2 = 3, 3 (defaults)
    Used in Eq. (10) to construct the annular mask for boundary blending. The values are hand-chosen defaults; the ablation removes the mask entirely but does not sweep the radii.
  • LoRA rank and alpha = 128/128
    Training settings for all four adapters (Sec. 5.1). These standard hyperparameters affect adapter capacity and edit quality, and are chosen by hand, not fitted.
assumptions (6)
  • domain assumption FLUX.1-dev pretrained DiT is a strong base for generation and editing
    All adapters fine-tune FLUX.1-dev; if the base model were weak, the pipeline would fail.
  • domain assumption Grounded-SAM and SAM-based segmentation can accurately localize RoIs in arbitrary images
    RoI localization (Sec. 4.2, Eq. 5) depends on segmentation quality; failure to segment leads to wrong edits.
  • domain assumption GPT-4o can reliably decompose instructions and predict valid bounding boxes
    The CoT planner and layout editor (Sec. 4.2, Appendix B) assume the VLM produces executable atomic programs; no human verification is reported.
  • domain assumption AnyEdit and MagicBrush test sets represent diverse editing scenarios
    Evaluation claims rely on benchmark representativeness; AnyEdit is used as a partial subset of 16 categories.
  • ad hoc to paper The five atomic operations cover all editing instructions
    The framework defines all editing as combinations of the five primitives (Sec. 4.2, Eq. 3). If an instruction cannot be decomposed or executed sequentially, the central claim fails.
  • domain assumption The OminiControl unified conditioning framework provides correct spatial alignment for the DiT
    The method appends condition tokens to the token sequence based on [59]; if this conditioning is flawed, all adapters inherit the flaw.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Image Editing As Programs with Diffusion Models." pith.science (2026). https://pith.science/paper/6SUYVLX4

@misc{pith2026250604158,
  author       = {Pith},
  title        = {Pith review of: Image Editing As Programs with Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SUYVLX4}},
  note         = {Machine review of arXiv:2506.04158}
}
read the original abstract

While diffusion models have achieved remarkable success in text-to-image generation, they encounter significant challenges with instruction-driven image editing. Our research highlights a key challenge: these models particularly struggle with structurally inconsistent edits that involve substantial layout changes. To mitigate this gap, we introduce Image Editing As Programs (IEAP), a unified image editing framework built upon the Diffusion Transformer (DiT) architecture. At its core, IEAP approaches instructional editing through a reductionist lens, decomposing complex editing instructions into sequences of atomic operations. Each operation is implemented via a lightweight adapter sharing the same DiT backbone and is specialized for a specific type of edit. Programmed by a vision-language model (VLM)-based agent, these operations collaboratively support arbitrary and structurally inconsistent transformations. By modularizing and sequencing edits in this way, IEAP generalizes robustly across a wide range of editing tasks, from simple adjustments to substantial structural changes. Extensive experiments demonstrate that IEAP significantly outperforms state-of-the-art methods on standard benchmarks across various editing scenarios. In these evaluations, our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions. Codes are available at https://github.com/YujiaHu1109/IEAP.

Figures

Figures reproduced from arXiv: 2506.04158 by the authors.

Figure 1
Figure 1. Visual results of our IEAP. Rows 1 and 3 showcase complex multi-step edits (Row 1 is [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Results of our preliminary experiments. Figure (a) shows the GPT-4o scores for three [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Our pipeline. The original instruction is first parsed by a VLM into atomic operations, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Example procedure. Figure (a) and Figure (b) illustrate the procedures of action change [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison results of ours with baseline methods on representative editing cases. Others [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparisons on Complex Instructions with Leading Multimodal Models. Our method [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Qualitative ablation of action change operation. Module-wise Ablation Studies. To quantify the impact of each key component in our framework, we perform a series of ablation studies on the AnyEdit local semantic editing test set as we split in Sec. 5.2. As shown in Tab…
Figure 8
Figure 8. Figure 8: More Visualization Results. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: More Visualization Results. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: More Visualization Results. Instruction: “Make the cat have a floral pattern, put a vase on the chair. Then, alter the color of the spoon to yellow and erase the pink decoration on the wall. Replace the biggest blue cof ee cup with a cake and zoom in the pink cup next…
Figure 11
Figure 11. Figure 11: More Detailed Visualization Processes of the pipeline. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

93 extracted references · 34 canonical work pages

  1. [1]

    Gpt-4 technical report, 2024

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, et al. Gpt-4 technical report, 2024

    2024

  2. [2]

    Blended latent diffusion.ACM transactions on graphics (TOG), 42(4):1–11, 2023

    Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion.ACM transactions on graphics (TOG), 42(4):1–11, 2023

    2023

  3. [3]

    Spatext: Spatio-textual representation for controllable image generation

    Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18370– 18380, 2023

    2023

  4. [4]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18208–18218, 2022

    2022

  5. [5]

    Patchmatch: A randomized correspondence algorithm for structural image editing.ACM Trans

    Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing.ACM Trans. Graph., 28(3):24, 2009

    2009

  6. [6]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18392–18402, 2023

    2023

  7. [7]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021

    2021

  8. [8]

    Learning to Follow Object-Centric Image Editing Instructions Faithfully

    Tuhin Chakrabarty, Kanishk Singh, Arkadiy Saakyan, and Smaranda Muresan. Learning to follow object- centric image editing instructions faithfully.arXiv preprint arXiv:2310.19145, 2023

    work page Pith review arXiv 2023

Show all 93 references
  1. [9]

    Unireal: Universal image generation and editing via learning real-world dynamics.arXiv preprint arXiv:2412.07774, 2024

    Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, et al. Unireal: Universal image generation and editing via learning real-world dynamics.arXiv preprint arXiv:2412.07774, 2024

    2024 arXiv

  2. [10]

    Diffedit: Diffusion-based semantic image editing with mask guidance.arXiv preprint arXiv:2210.11427, 2022

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance.arXiv preprint arXiv:2210.11427, 2022

    2022 arXiv

  3. [11]

    Comprehensive dataset of face manipulations for development and evaluation of forensic tools, 2022

    Brian DeCann and Kirill Trapeznikov. Comprehensive dataset of face manipulations for development and evaluation of forensic tools, 2022

    2022

  4. [12]

    Diffusion self-guidance for controllable image generation.Advances in Neural Information Processing Systems, 36:16222–16239, 2023

    Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation.Advances in Neural Information Processing Systems, 36:16222–16239, 2023

    2023

  5. [13]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine l...

    2024

  6. [14]

    Dit4edit: Diffusion transformer for image editing

    Kunyu Feng, Yue Ma, Bingyuan Wang, Chenyang Qi, Haozhe Chen, Qifeng Chen, and Zeyu Wang. Dit4edit: Diffusion transformer for image editing. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2969–2977, 2025

    2025

  7. [15]

    Guiding instruction- based image editing via multimodal large language models.arXiv preprint arXiv:2309.17102, 2023

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction- based image editing via multimodal large language models.arXiv preprint arXiv:2309.17102, 2023

    2023 arXiv

  8. [16]

    Make-a-scene: Scene-based text-to-image generation with human priors

    Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. InEuropean Conference on Computer Vision, pages 89–106. Springer, 2022

    2022

  9. [17]

    Instructdiffusion: A generalist modeling interface for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al. Instructdiffusion: A generalist modeling interface for vision tasks. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognit...

    2024

  10. [18]

    Pair diffusion: A comprehensive multimodal object-level image editor

    Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Xingqian Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair diffusion: A comprehensive multimodal object-level image editor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  11. [19]

    Experiment with gemini 2.0 flash native image generation

    Google. Experiment with gemini 2.0 flash native image generation. Technical report, Google AI Studio, 2025

    2025

  12. [20]

    Focus on your instruction: Fine-grained and multi-instruction image editing by attention modulation

    Qin Guo and Tianwei Lin. Focus on your instruction: Fine-grained and multi-instruction image editing by attention modulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6986–6996, 2024

    2024

  13. [21]

    Visual programming: Compositional visual reasoning without training, 2022

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training, 2022

    2022

  14. [22]

    Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  15. [23]

    Denoising diffusion probabilistic models, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020

    2020

  16. [24]

    Cascaded diffusion models for high fidelity image generation.Journal of Machine Learning Research, 23(47):1–33, 2022

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation.Journal of Machine Learning Research, 23(47):1–33, 2022

    2022

  17. [25]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  18. [26]

    Composer: Creative and controllable image synthesis with composable conditions.arXiv preprint arXiv:2302.09778, 2023

    Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions.arXiv preprint arXiv:2302.09778, 2023

    2023 arXiv

  19. [27]

    Diffusion model-based image editing: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1–27, 2025

    Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Liangliang Cao, and Shifeng Chen. Diffusion model-based image editing: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1–27, 2025

    2025

  20. [28]

    Smartedit: Exploring complex instruction-based image editing with multimodal large language models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large language models. InProceedings of the IEEE/CVF Conference ...

    2024

  21. [29]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, et al

    Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, et al. Gpt-4o system card, 2024

    2024

  22. [30]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013

    2013

  23. [31]

    Flux.https://github.com/black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024

    2024

  24. [32]

    One diffusion to generate them all.arXiv preprint arXiv:2411.16318, 2024

    Duong H Le, Tuan Pham, Sangho Lee, Christopher Clark, Aniruddha Kembhavi, Stephan Mandt, Ranjay Krishna, and Jiasen Lu. One diffusion to generate them all.arXiv preprint arXiv:2411.16318, 2024

    2024 arXiv

  25. [33]

    Instructany2pix: Flexible visual editing via multimodal instruction following.arXiv preprint arXiv:2312.06738, 2023

    Shufan Li, Harkanwar Singh, and Aditya Grover. Instructany2pix: Flexible visual editing via multimodal instruction following.arXiv preprint arXiv:2312.06738, 2023

    2023 arXiv

  26. [34]

    Moecontroller: Instruction-based arbitrary image manipulation with mixture-of-expert controllers.arXiv preprint arXiv:2309.04372, 2023

    Sijia Li, Chen Chen, and Haonan Lu. Moecontroller: Instruction-based arbitrary image manipulation with mixture-of-expert controllers.arXiv preprint arXiv:2309.04372, 2023

    2023 arXiv

  27. [35]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22511–22521, 2023

    2023

  28. [36]

    Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models, 2024

    Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models, 2024

    2024

  29. [37]

    Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

    Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

    2025 arXiv

  30. [38]

    Ace++: Instruction-based image creation and editing via context-aware content filling, 2025

    Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, and Jingren Zhou. Ace++: Instruction-based image creation and editing via context-aware content filling, 2025

    2025

  31. [39]

    Prodigy: An expeditiously adaptive parameter-free learner

    Konstantin Mishchenko and Aaron Defazio. Prodigy: An expeditiously adaptive parameter-free learner. arXiv preprint arXiv:2306.06101, 2023

    2023 arXiv

  32. [40]

    T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 4...

    2024

  33. [41]

    Instruction-guided editing controls for images and multimedia: A survey in llm era

    Thanh Tam Nguyen, Zhao Ren, Trinh Pham, Thanh Trung Huynh, Phi Le Nguyen, Hongzhi Yin, and Quoc Viet Hung Nguyen. Instruction-guided editing controls for images and multimedia: A survey in llm era. arXiv preprint arXiv:2411.09955, 2024

    2024 arXiv

  34. [42]

    Image-based modeling and photo editing

    Byong Mok Oh, Max Chen, Julie Dorsey, and Frédo Durand. Image-based modeling and photo editing. InProceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 433–442, 2001

    2001

  35. [43]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  36. [44]

    Multi-modal attention for speech emotion recognition

    Zexu Pan, Zhaojie Luo, Jichen Yang, and Haizhou Li. Multi-modal attention for speech emotion recognition. arXiv preprint arXiv:2009.04107, 2020

    2009 arXiv

  37. [45]

    Precisecon- trol: Enhancing text-to-image diffusion models with fine-grained attribute control

    Rishubh Parihar, VS Sachidanand, Sabariswaran Mani, Tejan Karmali, and R Venkatesh Babu. Precisecon- trol: Enhancing text-to-image diffusion models with fine-grained attribute control. InEuropean Conference on Computer Vision, pages 469–487. Springer, 2024

    2024

  38. [46]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  39. [47]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  40. [48]

    Unicontrol: A unified diffusion model for controllable visual generation in the wild.arXiv preprint arXiv:2305.11147, 2023

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild.arXiv preprint arXiv:2305.11147, 2023

    2023 arXiv

  41. [49]

    Neural programmer-interpreters.arXiv preprint arXiv:1511.06279, 2015

    Scott Reed and Nando De Freitas. Neural programmer-interpreters.arXiv preprint arXiv:1511.06279, 2015

    2015 arXiv

  42. [50]

    Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    2024

  43. [51]

    Morphological gradients.Journal of Electronic Imaging, 2(4):326–336, 1993

    Jean-Francois Rivest, Pierre Soille, and Serge Beucher. Morphological gradients.Journal of Electronic Imaging, 2(4):326–336, 1993

    1993

  44. [52]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  45. [53]

    High-resolution image synthesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022

    2022

  46. [54]

    Analysis of image processing using morphological erosion and dilation

    Khairul Anuar Mat Said and Asral Bahari Jambek. Analysis of image processing using morphological erosion and dilation. InJournal of Physics: Conference Series, volume 2071, page 012033. IOP Publishing, 2021

    2021

  47. [55]

    Emu edit: Precise image editing via recognition and generation tasks

    Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2024

  48. [56]

    Seededit: Align image re-generation to image editing.arXiv preprint arXiv:2411.06686, 2024

    Yichun Shi, Peng Wang, and Weilin Huang. Seededit: Align image re-generation to image editing.arXiv preprint arXiv:2411.06686, 2024

    2024 arXiv

  49. [57]

    Dragdiffusion: Harnessing diffusion models for interactive point-based image editing

    Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  50. [58]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  51. [59]

    Ominicontrol: Minimal and universal control for diffusion transformer.arXiv preprint arXiv:2411.15098, 2024

    Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for diffusion transformer.arXiv preprint arXiv:2411.15098, 2024. 12

    2024 arXiv

  52. [60]

    Ominicontrol2: Efficient conditioning for diffusion transformers.arXiv preprint arXiv:2503.08280, 2025

    Zhenxiong Tan, Qiaochu Xue, Xingyi Yang, Songhua Liu, and Xinchao Wang. Ominicontrol2: Efficient conditioning for diffusion transformers.arXiv preprint arXiv:2503.08280, 2025

    2025 arXiv

  53. [61]

    Click to grasp: Zero-shot precise manipulation via visual diffusion descriptors

    Nikolaos Tsagkas, Jack Rome, Subramanian Ramamoorthy, Oisin Mac Aodha, and Chris Xiaoxuan Lu. Click to grasp: Zero-shot precise manipulation via visual diffusion descriptors. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11610–11617. ...

    2024

  54. [62]

    High- resolution image synthesis and semantic manipulation with conditional gans

    Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    2018

  55. [63]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  56. [64]

    Less-to-more gener- alization: Unlocking more controllability by in-context generation.arXiv preprint arXiv:2504.02160, 2025

    Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, and Qian He. Less-to-more gener- alization: Unlocking more controllability by in-context generation.arXiv preprint arXiv:2504.02160, 2025

    2025 arXiv

  57. [65]

    Dreamomni: Unified image generation and editing.arXiv preprint arXiv:2412.17098, 2024

    Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, and Jiaya Jia. Dreamomni: Unified image generation and editing.arXiv preprint arXiv:2412.17098, 2024

    2024

  58. [66]

    Omnigen: Unified image generation.arXiv preprint arXiv:2409.11340, 2024

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shut- ing Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation.arXiv preprint arXiv:2409.11340, 2024

    2024 arXiv

  59. [67]

    Prompt-free diffusion: Taking" text" out of text-to-image diffusion models

    Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, and Humphrey Shi. Prompt-free diffusion: Taking" text" out of text-to-image diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8682–8692, 2024

    2024

  60. [68]

    Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model

    Shiyuan Yang, Xiaodong Chen, and Jing Liao. Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. InProceedings of the 31st ACM International Conference on Multimedia, pages 3190–3199, 2023

    2023

  61. [69]

    Complex-Edit: Cot-like instruction generation for complexity-controllable image editing benchmark, 2025

    Siwei Yang, Mude Hui, Bingchen Zhao, Yuyin Zhou, Nataniel Ruiz, and Cihang Xie. Complex-Edit: Cot-like instruction generation for complexity-controllable image editing benchmark, 2025

    2025

  62. [70]

    Anyedit: Mastering unified high-quality image editing for any idea

    Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. arXiv preprint arXiv:2411.15738, 2024

    2024 arXiv

  63. [71]

    Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos, 2025

    Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos, 2025

    2025

  64. [72]

    Nexus-gen: A unified model for image understanding, generation, and editing.arXiv preprint arXiv:2504.21356, 2025

    Hong Zhang, Zhongjie Duan, Xingjun Wang, Yingda Chen, Yuze Zhao, and Yu Zhang. Nexus-gen: A unified model for image understanding, generation, and editing.arXiv preprint arXiv:2504.21356, 2025

    2025 arXiv

  65. [73]

    Magicbrush: A manually annotated dataset for instruction-guided image editing.Advances in Neural Information Processing Systems, 36:31428–31449, 2023

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing.Advances in Neural Information Processing Systems, 36:31428–31449, 2023

    2023

  66. [74]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InProceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023

    2023

  67. [75]

    Tie: Revolutionizing text-based image editing for complex-prompt following and high-fidelity editing, 2024

    Xinyu Zhang, Mengxue Kang, Fei Wei, Shuang Xu, Yuhe Liu, and Lin Ma. Tie: Revolutionizing text-based image editing for complex-prompt following and high-fidelity editing, 2024

    2024

  68. [76]

    Easycontrol: Adding efficient and flexible control for diffusion transformer, 2025

    Yuxuan Zhang, Yirui Yuan, Yiren Song, Haofan Wang, and Jiaming Liu. Easycontrol: Adding efficient and flexible control for diffusion transformer, 2025

    2025

  69. [77]

    In-context edit: Enabling instructional image editing with in-context generation in large scale diffusion transformer, 2025

    Zechuan Zhang, Ji Xie, Yu Lu, Zongxin Yang, and Yi Yang. In-context edit: Enabling instructional image editing with in-context generation in large scale diffusion transformer, 2025

    2025

  70. [78]

    Ultraedit: Instruction-based fine-grained image editing at scale

    Haozhe Zhao, Xiaojian Shawn Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained image editing at scale. Advances in Neural Information Processing Systems, 37:3058–3093, 2024

    2024

  71. [79]

    Invalid category:

    Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models.Advances in Neural Information Processing Systems, 36:11127–11150, 2023. 13 Technical Appendices and Supplem...

    2023

  72. [80]

    [Add] add a car on the road

  73. [81]

    [Color Change] change the color of the shoes to blue

  74. [82]

    bed", [50, 300, 450, 450]), (

    [Move] move the lamp to the left Do not include any extra text, explanations, JSON or markdown, just the list. Below are the detailed prompts used to adjust the layout of move and resize operations: You are an intelligent bounding box editor. I will provide you with the curren...

  75. [83]

    [Appearance Change] make the cat have a floral pattern

  76. [84]

    [Add] add a vase on the chair

  77. [85]

    [Color Change] change the color of the spoon to yellow

  78. [86]

    [Remove] erase the pink decoration on the wall

  79. [87]

    [Replace] replace the biggest blue coffee cup with a cake

  80. [88]

    [Resize] zoom in the pink cup next to the cat

  81. [89]

    [Tone Transfer] change the time to the evening RoI Localization RoI Inpainting RoI Editing RoI Compositioning Global Transformation CoT Neural Program Interpreter

  82. [90]

    [Background Change] Change the background to the forest

  83. [91]

    [Appearance Change] Make the lady wear a white dress

  84. [92]

    [Add] Add a fox beside the lady

  85. [93]

    What would it be like if we placed the lady in a forest during autumn, and she was wearing a white dress, with a fox beside her?

    [Tone Transfer] Change the time to autumn Neural Program Interpreter Instruction: “What would it be like if we placed the lady in a forest during autumn, and she was wearing a white dress, with a fox beside her?” CoT Figure 11: More Detailed Visualization Processes of the pipe...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.