Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Instruction-based Image Manipulation by Watching How Things Move

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper trains an image editor on 6M real video frame pairs with MLLM-written instructions, enabling non-rigid pose, layout, and viewpoint edits that preserve identity.

desk verdict A genuinely useful dataset idea with a plausible new conditioning trick, but the evidence is too thin to back the SOTA claim. read the letter →

arxiv 2412.12087 v1 pith:7BF3W54Y submitted 2024-12-16 cs.CV

Mingdeng Cao , Xuaner Zhang , Yinqiang Zheng , Zhihao Xia This is my paper · ORCID
classification cs.CV
keywords instruction-basedimageeditingvideoframepairsmultimodallargelanguagemodelsdiffusionspatialconditioningnon-rigidviewpointchangedatasetconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the bottleneck in instruction-based image editing—the lack of large, realistic source-target-instruction training data—can be solved by harvesting pairs of frames from ordinary videos. The authors build a dataset of 6 million triplets by sampling video frame pairs, filtering them with optical flow to keep moderate, background-stable motion, and asking multimodal LLMs to write the editing instruction that turns the first frame into the second. They then fine-tune a pretrained text-to-image diffusion model with a spatial conditioning scheme that concatenates the source image beside the noisy target, allowing the model to attend to any part of the reference while restructuring the output. If the central claim is right, instruction-based editors can handle non-rigid pose changes, element rearrangement, and viewpoint changes that synthetic-data models fail at, while preserving identity.

What carries the argument

Two components carry the argument. First, the data pipeline: fixed-interval frame sampling, RAFT optical-flow magnitude filtering to discard pairs with too little or too much movement, a background-occlusion mask ratio to discard large background changes, and MLLM-generated instructions with the option to reject pairs the model cannot describe. Second, Spatial Conditioning: instead of concatenating the reference latent with the noisy target along the channel dimension (which locks pixel alignment), the source latent and noisy target latent are placed side by side along the width dimension, the doubled-width tensor is fed into the denoising U-Net, and the right half of the output is cropped to compute the denoising loss. Because every cross-attention layer can attend from any target region to any source region, the network can apply non-rigid transformations and viewpoint changes while still copying fine appearance details from the reference.

What would settle it

Take a random sample of the 6 million triplets and give a held-out text-only editor only the source image and the generated instruction, then measure whether humans or a vision-language model can pick the true target frame from a set of distractors such as frames from other videos or other edits; if pick rates approach chance on a substantial fraction, the frame pairs or instructions are too noisy to support the claimed mapping. A complementary experiment would shuffle instructions across frame pairs and train the same model: if edited-image quality does not drop, the instructions are not the causal driver the paper claims.

Watch

Extended reading notes

Core claim

The paper's central claim is that real video frames, not synthetic targets, are the supervision signal that lets an instruction-following editor perform structural edits. From internet videos it constructs a 6-million-triplet dataset: pairs of frames about three seconds apart, filtered with optical-flow magnitude and background-occlusion checks to keep moderate, describable motion, and then labeled by multimodal LLMs with short action-verb instructions that explain how the first frame becomes the second. The model, InstructMove, is a pretrained text-to-image diffusion model fine-tuned with Spatial Conditioning, in which the source-image latent and the noisy target latent are concatenated along the width dimension, fed through the denoising U-Net, and the right half of the predicted noise is cropped for the diffusion loss. The paper reports that this approach outperforms prior instruction-based and tuning-free methods on a 50-image benchmark for non-rigid edits and viewpoint changes, with the highest instruction-alignment scores and 87.62% human preference, because it can change structure while preserving the source subject's identity.

Load-bearing premise

The load-bearing premise is that a pair of video frames separated by a few seconds, filtered by optical flow and described by an MLLM, is a faithful training signal for an image-edit instruction—the time gap must correspond to one describable edit, and the generated instruction must match what actually changed.

Editorial extensions

If this is right

  • Instruction-based editing models can be trained at scale from unlabeled internet video, avoiding the cost and artifact problems of synthetic target generation.
  • Models trained this way should handle non-rigid pose and expression changes, object repositioning, and camera-perspective shifts that structure-preserving methods leave mostly unchanged.
  • Because the conditioning scheme keeps the base text-to-image architecture intact, the trained editor can be combined with masks, ControlNet, or other spatial adapters for localized and skeleton- or sketch-guided edits.
  • Using this video-derived dataset alongside existing synthetic datasets can cover both realistic motion edits and artistic edits like style transfer, as the paper demonstrates in the supplementary material.
  • The proposed CLIP-Inst metric, which asks an MLLM to regenerate instructions from input and output images and measures CLIP distance to the original instruction, gives a way to evaluate instruction adherence without human labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the video-supervision premise holds, the scaling path for instruction editing is video abundance: larger and more diverse video corpora should keep improving edit quality without any manual annotation, which prior synthetic-data approaches cannot offer.
  • The MLLM instruction step imposes a ceiling: frame-pair changes that cannot be verbalized are systematically excluded, so the dataset implicitly defines 'edit' as language-describable change; recovering the excluded transformations would require human-in-the-loop labeling or a richer instruction interface.
  • A testable extension is to train on longer-range frame pairs or multi-view videos: if viewpoint transformations are learned from natural video, the same pipeline may yield camera-control abilities without explicit 3D supervision.
  • The spatial-conditioning trick may transfer to other conditional generation tasks where the reference and output are not pixel-aligned, such as reference-based synthesis or structure-guided generation from a different viewpoint.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a data-construction pipeline that samples frame pairs from internet videos, filters them by optical flow magnitude and background occlusion, and uses multimodal LLMs (GPT-4o, Pixtral) to generate editing instructions. The resulting 6M triplet dataset is used to fine-tune Stable Diffusion with a spatial-conditioning scheme in which the source image latent is concatenated to the noisy target latent along the width dimension. The model, InstructMove, is evaluated on a 50-image benchmark of non-rigid manipulations, viewpoint changes, and rearrangements, against six zero-shot and instruction-based baselines, with a human study. The paper also demonstrates mask-based local editing and integration with ControlNet.

Significance. The central idea—using real video pairs as supervision for instruction-based editing—is timely and potentially addresses the scalability and realism bottleneck of synthetic editing datasets. If the evidence were stronger, the paper could open a new data avenue for the community. The spatial conditioning is simple and effective, and the ablation in Table 4 supports the contribution of both the real data and the conditioning scheme. However, the current evidence for the state-of-the-art claim is limited: the evaluation is a small, non-released benchmark without error bars, and the consistency of the video-triplet supervision is not quantified.

major comments (3)
  1. [Sec. 3.2/3.3 and Eq. (1)] The consistency of the triplet (I_s, I_e, C) is load-bearing for the central claim, but the pipeline does not measure whether C completely describes the changes between I_s and I_e. The assertion in Sec. 3.3 that 'small discrepancies do not significantly impact the training process' is not supported by any experiment. Since the denoising loss in Eq. (1) trains the model to reproduce all changes between the frames, unmodeled lighting shifts, background motion, or identity drift become part of the supervision, which can cause the model to hallucinate edits or ignore the instruction (as acknowledged in Sec. 6). Please provide quantitative evidence of instruction completeness (e.g., a human or LLM-based verification on a random sample, or an ablation with a completeness filter) and report the rejection rate of the MLLM.
  2. [Sec. 5.1, Table 2] The evaluation benchmark is only 50 self-curated images and the metrics are reported as point estimates without error bars or significance tests. The differences between Ours and the strongest baselines in CLIP-Inst (0.8724 vs. 0.8648) and CLIP-I (0.9275 vs. 0.9380) are small, and we cannot tell if they are statistically meaningful. The user study (Table 3) reports 87.62% preference for Ours but does not describe participant variance, the number of independent choices per example, or inter-annotator agreement. Please release the benchmark, report confidence intervals or significance tests, and provide more details on the user study.
  3. [Sec. 3.1] The frame sampling parameters are not reported: the flow-magnitude thresholds for 'moderate movement', the occlusion ratio threshold, and the fixed time interval (stated as 3 seconds) are not given, and the number of source videos and the MLLM rejection rate are omitted. Without these, the dataset construction is not reproducible and the '6M pairs' claim cannot be verified. Please specify the thresholds, the filtering statistics, and ideally release the dataset or a sample.
minor comments (5)
  1. [Sec. 5.1] The new CLIP-Inst metric is not validated. It would strengthen the paper to show that CLIP-Inst correlates with human judgments or to compare it against the existing CLIP-D on a set where ground-truth instructions are known.
  2. [Sec. 1] The text contains a formatting artifact: 'LLaV A' should be 'LLaVA' (also in Sec. 2.2, the reference to LLaVA is cited as 'Liu et al.' with a space).
  3. [Table 1] Table 1 is visually unclear: the columns 'Real Target?' and the numbers appear misaligned, and it is not specified whether '6M' refers to the final filtered triplet count. Please clarify.
  4. [Sec. 5.1] The paper refers to 'the supplementary material' for details of the benchmark, but the supplementary material is not provided; please include it in the revision.
  5. [Fig. 8] The limitation example on the left shows a 'slight viewpoint change', but it is unclear whether this is a common failure mode; a frequency estimate would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training pipeline, evaluation metric, and baseline comparisons are logically independent.

full rationale

The paper's derivation chain is a standard supervised learning setup: it samples real video frame pairs, uses MLLMs to generate editing instructions, and fine-tunes a pretrained text-to-image model with the denoising objective in Eq. (1). The evaluation is performed on a separately created 50-image benchmark with human-curated instructions, and the proposed CLIP-Inst metric uses an MLLM to regenerate instructions from input/output images and compares CLIP distances to the original instructions. This metric is an independent proxy for instruction adherence; it is not the training loss and does not define the target image in terms of the metric. The Sec. 6 admission that MLLMs 'may generate inaccurate instructions or fail to capture all transformations between frames' is a data-quality limitation, not a circular reduction. The only self-citation of note (MasaCtrl [6]) appears as a baseline method in the comparison, not as load-bearing justification for the central claim. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no equation reduces to its own input by construction. The central claim is an empirical, externally checkable result, so no significant circularity is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

All the training data is produced by the authors' pipeline, so the ledger includes the hand-chosen sampling interval, the unreported motion filter thresholds, and the MLLM rejection rule. The core assumptions are video-frame identity preservation and MLLM instruction accuracy, both acknowledged as limitations.

free parameters (3)
  • Frame sampling interval = 3 seconds (fixed by hand)
    Sec 3.1 sets a fixed 3 s interval between source and target frames; no ablation or sensitivity analysis is reported, though this determines the amount of motion in every training pair.
  • Motion filtering thresholds = not reported
    Sec 3.1 uses RAFT flow magnitude and occlusion mask ratio cutoffs to retain 'moderate' movement, but the exact ranges are never given. These thresholds define the 6M-pair dataset.
  • MLLM rejection criteria = not specified
    Sec 3.2 instructs MLLMs to reject pairs whose edits are too complex to describe, but no criteria or measured rejection rate is provided.
assumptions (3)
  • domain assumption Video frame pairs preserve subject and scene identity.
    Sec 1 and Sec 3.1 assume a 3 s video gap yields a source and target with the same identity. Lighting changes, occlusions, motion blur, and identity drift across frames violate this for some pairs.
  • domain assumption MLLM-generated instructions accurately and completely describe the visual difference.
    The training supervision is entirely from GPT-4o and Pixtral outputs; Sec 6 concedes MLLMs 'may generate inaccurate instructions or fail to capture all transformations.'
  • ad hoc to paper Optical flow and occlusion filters select useful training pairs.
    These heuristics are not standard and their thresholds are unreported; the quality of the whole dataset depends on their correctness (Sec 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Instruction-based Image Manipulation by Watching How Things Move." pith.science (2026). https://pith.science/paper/7BF3W54Y

@misc{pith2026241212087,
  author       = {Pith},
  title        = {Pith review of: Instruction-based Image Manipulation by Watching How Things Move},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BF3W54Y}},
  note         = {Machine review of arXiv:2412.12087}
}
read the original abstract

This paper introduces a novel dataset construction pipeline that samples pairs of frames from videos and uses multimodal large language models (MLLMs) to generate editing instructions for training instruction-based image manipulation models. Video frames inherently preserve the identity of subjects and scenes, ensuring consistent content preservation during editing. Additionally, video data captures diverse, natural dynamics-such as non-rigid subject motion and complex camera movements-that are difficult to model otherwise, making it an ideal source for scalable dataset construction. Using this approach, we create a new dataset to train InstructMove, a model capable of instruction-based complex manipulations that are difficult to achieve with synthetically generated datasets. Our model demonstrates state-of-the-art performance in tasks such as adjusting subject poses, rearranging elements, and altering camera perspectives.

Figures

Figures reproduced from arXiv: 2412.12087 by the authors.

Figure 1
Figure 1. We propose InstructMove, an instruction-based image editing model trained on frame pairs from videos with instructions gen￾erated by Multimodal LLMs. Our model excels at non-rigid editing, such as adjusting subject poses, expressions, and altering viewpoints, while maintaining content consistency. Additionally, our method supports precise, localized edits through the integration of masks, human poses, and other cont… view at source ↗
Figure 2
Figure 2. Existing methods struggle with complex edits on real [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Our data construction pipeline. (a) We begin by sam￾pling suitable frame pairs from videos, ensuring realistic and mod￾erate transformations. (b) These frame pairs are used to prompt Multimodal Large Language Models (MLLMs) to generate de￾tailed editing instructions. (c) This process results in a large-scale dataset with realistic image pairs and precise editing instructions. and replacement. EmuEdit [33] improves d… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Overview of the proposed model architecture for instruction-based image editing. The source and target images are first encoded into latent representations z s and z e using a pre￾trained encoder. The target latent z e is then transformed into a noisy latent z e t thro…
Figure 5
Figure 5. Figure 5: Qualitative comparison with state-of-the-art image editing methods, including both description-based and instruction-based approaches. Existing methods struggle with complex edits such as non-rigid transformations (e.g., changes in pose and expression), object repositi…
Figure 6
Figure 6. Figure 6: Qualitative results of our method with additional controls. (a) Our model can utilize a mask to specify which part of the image to edit, enabling localized adjustments and resolving ambiguities in the instructions. (b) When combined with ControlNet, our model can accep…
Figure 7
Figure 7. Figure 7: Qualitative ablation on the training dataset and con￾ditioning approach. Compared to models trained on the Instruct￾Pix2Pix dataset (IP2P) [4] or using the standard Channel Con￾ditioning (CC) method, our model trained on our dataset with the Spatial Conditioning (SC) a…
Figure 8
Figure 8. Figure 8: Limitations. Left: Occasional unintended slight view￾point changes. Right: Inability to accurately isolate specific ob￾jects based on instructions. proposed Spatial Conditioning approach significantly im￾proves the model’s ability to apply complex edits while achieving…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Edicho: Consistent Image Editing in the Wild

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Edicho makes edits consistent across in-the-wild image pairs by injecting explicit pixel correspondences into the attention and classifier-free guidance steps of a pretrained diffusion model, with no training.

  2. ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motions

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A released 6.4 million pair dataset and 613 sample benchmark for instruction-guided image editing of non-rigid motions, plus a Flux.1-dev based baseline that outperforms open-source methods on the new benchmark.

Reference graph

Works this paper leans on

44 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Am ´elie H ´eliou, Paul Ja- cob, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073 ,

    arXiv

  3. [3]

    Ledits++: Limitless image editing using text-to-image models

    Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolin´ario Passos. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 8861–8870, 2024. 2

    work page 2024

  4. [4]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 1, 2, 3, 4, 6, 8

    work page 2023

  5. [5]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 2

    arXiv 2005

  6. [6]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 22560–22570, 2023. 1, 2, 6

    work page 2023

  7. [7]

    Diffedit: Diffusion-based seman- tic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based seman- tic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022. 1, 2

    arXiv 2022

  8. [8]

    Turboedit: Text-based image editing using few-step diffusion models

    Gilad Deutch, Rinon Gal, Daniel Garibi, Or Patashnik, and Daniel Cohen-Or. Turboedit: Text-based image editing using few-step diffusion models. arXiv preprint arXiv:2408.00735, 2024. 2

    arXiv 2024

Show all 44 references
  1. [9]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 1

    2021

  2. [10]

    Guiding instruction-based im- age editing via multimodal large language models

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction-based im- age editing via multimodal large language models. arXiv preprint arXiv:2309.17102, 2023. 2, 4

    2023 arXiv

  3. [11]

    Instructdiffusion: A generalist modeling inter- face for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al. Instructdiffusion: A generalist modeling inter- face for vision tasks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Reco...

    2024

  4. [12]

    Proxedit: Improving tuning-free real image editing with proximal guidance

    Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopou- los, Xiaoxiao He, Yuxiao Chen, et al. Proxedit: Improving tuning-free real image editing with proximal guidance. In Proceedings of the IEEE/CVF Winter Conference on Appl...

    2024

  5. [13]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 1, 2, 6

    2022 arXiv

  6. [14]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  7. [15]

    Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large lan- guage models. In Proceedings of the IEEE/CVF Conferen...

    2024

  8. [16]

    An edit friendly ddpm noise space: Inversion and manipulations

    Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12469– 12478, 2024. 2

    2024

  9. [17]

    Direct inversion: Boosting diffusion-based edit- ing with 3 lines of code

    Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct inversion: Boosting diffusion-based edit- ing with 3 lines of code. arXiv preprint arXiv:2310.01506,

    arXiv

  10. [18]

    Imagic: Text-based real image editing with diffusion models

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017, 2023. 2, 6

    2023

  11. [19]

    Diffu- sionclip: Text-guided diffusion models for robust image ma- nipulation

    Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffu- sionclip: Text-guided diffusion models for robust image ma- nipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2426–2435,

  12. [20]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 4

    2013 arXiv

  13. [21]

    Diffusion models already have a semantic latent space

    Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210.10960, 2022. 2

    2022 arXiv

  14. [22]

    Layerdiffusion: Layered controlled image editing with dif- fusion models

    Pengzhi Li, Qinxuan Huang, Yikang Ding, and Zhiheng Li. Layerdiffusion: Layered controlled image editing with dif- fusion models. In SIGGRAPH Asia 2023 Technical Commu- nications, pages 1–4. 2023. 2

    2023

  15. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  16. [24]

    Sdedit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equa- tions. arXiv preprint arXiv:2108.01073, 2021. 1, 2

    2021 arXiv

  17. [25]

    Null-text inversion for editing real im- ages using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real im- ages using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023. 1, 2, 6

    2023

  18. [26]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image 9 diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image 9 diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, ...

    2024

  19. [27]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 1

    2021 arXiv

  20. [28]

    Zero-shot image-to-image translation

    Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceed- ings, pages 1–11, 2023. 2

    2023

  21. [29]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  22. [30]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 1, 2

    2022 arXiv

  23. [31]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 5

    2022

  24. [32]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  25. [33]

    Emu edit: Precise image editing via recognition and gen- eration tasks

    Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and gen- eration tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pa...

    2024

  26. [34]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 2

    2015

  27. [35]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2, 5

    2010 arXiv

  28. [36]

    Generative modeling by esti- mating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 2

    2019

  29. [37]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 402–419. Springer,

    2020

  30. [38]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  31. [39]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1921–1930, 2023. 1, 2

    1921

  32. [40]

    Turboedit: Instant text-based image editing

    Zongze Wu, Nicholas Kolkin, Jonathan Brandt, Richard Zhang, and Eli Shechtman. Turboedit: Instant text-based image editing. arXiv preprint arXiv:2408.08332, 2024. 2

    2024 arXiv

  33. [41]

    Magicbrush: A manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 2, 4, 5, 6

    2024

  34. [42]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 4, 5

    2023

  35. [43]

    Forgedit: Text guided image editing via learning and forgetting

    Shiwen Zhang, Shuai Xiao, and Weilin Huang. Forgedit: Text guided image editing via learning and forgetting. arXiv preprint arXiv:2309.10556, 2023. 2

    2023 arXiv

  36. [44]

    Ultraedit: Instruction-based fine-grained im- age editing at scale

    Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Ru- jie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained im- age editing at scale. arXiv preprint arXiv:2407.05282, 2024. 4, 6 10

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.