Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Action-based image editing guided by human instructions

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EditAction fine-tunes a static image editor with a contrastive action loss so a text instruction like 'flip the tofu' changes position or posture while preserving the object's appearance and the scene.

desk verdict Sensible idea and useful new datasets, but the experimental protocol leaks video identity across train/test, so the reported gains over the fine-tuned baseline are not yet convincing. read the letter →

arxiv 2412.04558 v2 pith:77R654JF submitted 2024-12-05 cs.CV

Maria Mihaela Trusca , Mingxiao Li , Marie-Francine Moens This is my paper · ORCID
classification cs.CV
keywords action-basedimageeditingdiffusionmodelscontrastivelearninginstructionfollowingvideoframesactionrecognitionInstructPix2Pixegocentric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes EditAction, a model that extends text-prompted image editing from static changes such as adding or removing objects to dynamic action changes, where the output shows an object or person in a new position or posture while keeping its visual identity and the surrounding scene. The idea is to fine-tune InstructPix2Pix on pairs of first and last frames extracted from videos of human actions, using a contrastive loss that makes the model prefer the correct action caption over a randomly chosen one. The authors report that this learns action discrepancies well enough to implement actions, including verbs unseen during training, and that it outperforms prior editors on action-recognition accuracy and human evaluation. If correct, action-based image editing becomes a single-frame generation problem rather than a video-generation problem, and edited images can serve as goal frames for embodied agents and virtual-scene creation.

What carries the argument

The load-bearing mechanism is a contrastive action loss built on the diffusion model's evidence lower bound. For an output image, an input image, the correct action caption, and a randomly chosen caption sharing no tokens with the correct one, the model is trained to maximize the probability of the correct caption by comparing expected denoising errors for the two captions, following the diffusion-as-classifier construction. This loss is combined with the original static editing loss and a regularization term that penalizes drift between the fine-tuned U-Net and the frozen pretrained U-Net, and the cross-attention layers are frozen throughout training. The mechanism lets the model suppress unrelated actions while learning to implement the specified one, and it is what carries the claimed improvements in action accuracy and appearance preservation.

What would settle it

Train EditAction and the TimeSformer evaluator on the same video corpora as in the paper, then test on a held-out set of action segments from unrelated video datasets (for example, sports or assembly tasks) with fresh text annotations. If action-recognition accuracy does not improve over a static-editing baseline on those unseen actions, the claimed learning of generalizable action discrepancies fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a diffusion-based editor trained for static instruction-following can be converted into an action-aware editor by fine-tuning it to maximize the likelihood of the correct action text relative to a random action text, given the same input image and edited image. The authors build EditAction on InstructPix2Pix, freeze the cross-attention layers to preserve text-to-visual alignment, and add a regularization term that keeps the fine-tuned denoising network close to the frozen pretrained network. Evaluated on two new datasets built from video-recognition corpora, EditAction reaches the highest action-recognition accuracy among all compared models (71.4 on the fixed-camera LC test set, 54.41 on the flexible-camera HC test set, and 46.29 on the long-distance HC test subset) while maintaining the best or near-best FID scores. Human annotators also rate it highest for action implementation, posture and position editing, and reasoning about the final scene. The broader claim is that the task of action-based image editing can be treated as producing one final frame that depicts the action while preserving the objects and environment of the input frame.

Load-bearing premise

The training data assumption: each action segment in MPII-Cooking and EPIC-Kitchen is represented by taking the first and last frames, and the text annotation for the segment is assumed to describe the change between those two frames well enough that the model can learn appearance preservation and action implementation from these pairs.

Editorial extensions

If this is right

  • Action instructions can be applied to a single input image without generating a video sequence, because the method learns to jump directly from the before-frame to the after-frame.
  • Appearance and background preservation can be inherited from a static editor by freezing its cross-attention layers while learning action directions from video-frame pairs.
  • New actions described by unseen verbs can be implemented, as demonstrated by 'wipe' on the LC dataset and 'place' on the HC dataset.
  • The model can reason beyond the input frame, as shown on the HC test100 subset where it produces a plausible new viewpoint of the scene after a long-distance action.
  • Inference runs at InstructPix2Pix speed, roughly the same cost as the base static editor and much faster than several training-free editing baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The before/after frame extraction recipe could generalize to other instruction-following editors, since large action-recognition video datasets already provide abundant labeled transitions.
  • Because the action-accuracy metric uses classifiers fine-tuned on the same MPII-Cooking and EPIC-Kitchen videos that supply the training pairs, an independent test set of videos from other activities would be needed to confirm that the learned action discrepancies generalize beyond kitchen and egocentric actions.
  • A natural extension is to combine the contrastive action loss with optical flow or trajectory information, which would likely address the reported failure cases involving nested objects or long-distance actions where the wrong instance is moved.
  • If the diffusion-as-classifier interpretation is correct, the same fine-tuning recipe could apply to temporal tasks such as action-specified keyframe prediction or instruction-guided video editing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces EditAction, a method for text-guided image editing that implements actions (e.g., changing object positions or postures) rather than static edits. The authors fine-tune InstructPix2Pix on pairs of first/last frames extracted from action segments in MPII-Cooking (LC) and EPIC-Kitchen (HC), using a contrastive action loss (Eq. 5) that contrasts the correct action caption with a random caption, plus a regularization loss (Eq. 6) and freezing of cross-attention layers. Evaluation uses TimeSformer action recognition accuracy between input and output images, FID scores, and human ratings (R1-R5). The paper reports that EditAction outperforms several baselines, including a fine-tuned InstructPix2Pix, on both datasets and on a reasoning-focused HC test100 subset.

Significance. The task of editing images according to action instructions is a timely and useful extension of text-based image editing, and the proposed contrastive training on video-derived pairs is a plausible way to inject action awareness into a pretrained diffusion editor. The datasets (LC and HC) are potentially reusable resources. If the reported results survive a cleaner experimental protocol, the method would be a solid contribution; the paper also includes human evaluation and some analysis of unseen verbs and failure cases.

major comments (4)
  1. [3.3, Eq. (5)] Equation (5) is not a log-likelihood as labeled; the right-hand side is a softmax probability of the form exp(a)/sum_j exp(a_j), and the exponent uses +E[||epsilon - epsilon_theta||^2] whereas the diffusion ELBO estimate in Eq. (2) is proportional to -E[||epsilon - epsilon_theta||^2]. As written, minimizing Laction would push the model to increase the noise-prediction error for the correct caption, which is the opposite of the intended objective. Please correct the sign and the logarithm, and state explicitly whether the implementation optimizes the negative log-likelihood.
  2. [4.1 and 4.2] The test sets are created by randomly sampling 500 image pairs from each dataset after training on the remaining pairs. Because each source video yields many segments and hence many first/last-frame pairs, this pair-level split can put segments from the same video (same actor, kitchen, and often same background and objects) into both training and test. This is especially problematic because the reported gains over the strongest baseline are small (e.g., LC Acc 71.4 vs 69.2; HC R1 4.28 vs 4.25 in Tables 1 and 2). Furthermore, the TimeSformer evaluator is fine-tuned on the same source videos used to construct the training pairs. A video-disjoint split is necessary to support the central generalization claim; please report such results or provide a leakage analysis.
  3. [Table 5] The HC test500 row for 'w/o Laction' reports Acc = 104.68 ± 0.22 and FIDinput = 0.99 ± 1.23. An accuracy exceeding 100% is impossible, and an FID input near 1.0 is implausible when all other HC rows are near 95-105. As printed, this row also contradicts the text in Section 7.3, which states that omitting Laction has the largest negative effect; the row instead appears to show a huge positive effect. Please correct the table or explain the anomaly; the ablation conclusions cannot be evaluated without this row.
  4. [Tables 1 and 2, Section 5] The human evaluation differences between EditAction and Fine-tuned InstructPix2Pix are very small (e.g., R1 4.37 vs 4.23 on LC, 4.28 vs 4.25 on HC), yet the abstract and Section 5 claim 'substantial improvements.' No significance tests or confidence intervals are reported for the human ratings, and the Acc metric is reported as mean ± variance over three seeds without a statistical test. Please add appropriate significance testing or temper the claims.
minor comments (6)
  1. [3.3, Eq. (4)] Equation (4) is typeset incorrectly; the denominator should be a proper fraction. Please fix the formatting.
  2. [Table 5] Table 5, HC test500 row 'w/o Lreg' lists a standard deviation of ±17 for Acc, which is unusually large; please verify the variance calculation.
  3. [Section 5] In Section 5, the statement that the HC test100 result is a 'substantial difference' is not supported by Table 3, where EditAction's Acc is 46.29 vs 45.89 for Fine-tuned InstructPix2Pix; consider a more measured description.
  4. [4.3 and Appendix 7.2] The notation for hyperparameters (e.g., '50−4', '30−2') is ambiguous; please clarify whether these are powers of ten or multiplication factors (e.g., 50×10^-4).
  5. [Section 3.2] The assumption that the first and last frames of a segment represent the start and end of a single action, and that the text annotation describes the change, should be discussed, especially for segments with multiple sub-actions or camera motion; consider adding a statistical analysis of segment lengths and labels.
  6. [Figures 2 and 3] In Figure 2, the text is small and the red/green/blue annotations from Figure 3 are not reproduced; please improve readability.

Circularity Check

2 steps flagged · score 4.0 of 10

No derivation collapse, but two evaluation choices make parts of the reported numbers self-referential: hyperparameters are selected on FID_output which is then reported as a result, and the TimeSformer accuracy evaluator is fine-tuned on the same videos from which the edit pairs are sampled.

  1. fitted input called prediction [Appendix 7.2 and Tables 1, 3, 5]
    "To select the hyperparameters λ1 and λ2, we use F IDoutput score to rank models trained on 1000 training steps with λ1, λ2 ∈ {10−4, 30−4, 50−4, 10−3, 30−3, 50−3, 10−2, 30−2, 50−2}."

    The reported evaluation includes FID_output (Tables 1, 3, 5), but the model hyperparameters λ1 and λ2 were chosen by ranking models on exactly FID_output. Unless a separate, undisclosed validation split was used, the FID_output numbers are a selected quantity rather than an independent prediction. No validation split is described in Section 4.1. This does not affect the action-accuracy claim, which was not used for selection, but it makes the FID-based preservation claims partly self-referential.

  2. other [Sections 4.1-4.2]
    "To define the test datasets, we randomly selected 500 image pairs from the LC and HC datasets, dubbing them LCtest500 and HC test500. ... To identify the actions, we fine-tune TimeSformer separately on videos from the MPII-Cooking and EPIC-Kitchens datasets, which are used to define the LC and HC datasets."

    EditAction is trained on first/last-frame pairs extracted from the MPII-Cooking and EPIC-Kitchens video segments (Section 3.2), and the test pairs are random samples from the same LC/HC datasets. The accuracy evaluator is then fine-tuned on videos from those same datasets. The Acc metric therefore measures how well the model reproduces the dataset-specific before/after transitions that both the training pairs and the evaluator are derived from, rather than providing an independent test of action generalization. The human evaluations (R1, R4, R5) are independent and keep the central claim from being wholly circular.

full rationale

The derivation chain itself is not circular: EditAction is a standard fine-tuned diffusion model trained with a static reconstruction loss, a contrastive action loss, and a regularization loss (Eq. 7); no output quantity is defined in terms of the model's own predictions, and no load-bearing uniqueness theorem or self-citation is invoked. The two concerns above are evaluation-level circularity. First, hyperparameters λ1 and λ2 are selected using FID_output and then FID_output is reported as a result, so the preservation numbers are partly a model-selection artifact. Second, the TimeSformer accuracy measure is fine-tuned on the same video collections used to build the training and test pairs, so high accuracy partly reflects reproducing the source dataset's annotated transitions. Because the central action-editing claim is also supported by independent human annotators on R1/R4/R5, and because the contrastive loss and architecture are not equivalent to the evaluation metric by construction, the paper is only partially circular rather than derivationally forced.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The model introduces no new physical entities. The free parameters are two loss weights selected on the evaluation metric and two guidance scales inherited from the base model. The main unproved premises are the validity of the ELBO-as-likelihood approximation, the cleanliness of the before/after frame pairs, and the hypothesis that freezing cross-attention preserves editing ability.

free parameters (4)
  • lambda_1 (action loss weight) = 5e-4 (reported as 50−4)
    Chosen by ranking models trained for 1000 steps on FID_output (Appendix 7.2). This is a hyperparameter fitted to the evaluation metric, not derived.
  • lambda_2 (regularization loss weight) = 3e-2 (reported as 30−2)
    Chosen the same way as lambda_1; the appendix states the best lambda_2 is selected by including only the regularization loss during training.
  • si (image guidance scale) = 1
    Inherited from InstructPix2Pix; set to the recommended value, not tuned on the new data.
  • sc (text guidance scale) = 7.5
    Inherited from InstructPix2Pix; set to the recommended value, not tuned on the new data.
assumptions (4)
  • domain assumption The ELBO objective in Eq. 2 is a valid proxy for the conditional log-likelihood used in the classifier conversion (Eq. 4-5).
    The paper follows reference [27] in treating the denoising error as a surrogate for p(x0|c,i); the gap between ELBO and true likelihood is not analyzed.
  • domain assumption First and last frames of an action segment in MPII-Cooking and EPIC-Kitchen represent a valid before/after pair for the action described in the text.
    Section 3.2 assumes the video segmentation and text annotations are clean enough to define the editing task; the paper does not filter segments for occlusions, multiple sub-actions, or camera motion.
  • ad hoc to paper Freezing cross-attention layers preserves the static-editing alignment from InstructPix2Pix while allowing action learning in the rest of the network.
    This is stated as a hypothesis in Section 3.3; the ablation 'w/o freezing U-Net' tests it empirically but provides no mechanistic justification.
  • ad hoc to paper Randomly selected negative action instructions with no shared tokens are a good contrastive set for teaching action discrimination.
    Section 3.3; the paper does not analyze the effect of hard negatives or the choice of the contrastive sampling distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Action-based image editing guided by human instructions." pith.science (2026). https://pith.science/paper/77R654JF

@misc{pith2026241204558,
  author       = {Pith},
  title        = {Pith review of: Action-based image editing guided by human instructions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/77R654JF}},
  note         = {Machine review of arXiv:2412.04558}
}
read the original abstract

Text-based image editing is typically approached as a static task that involves operations such as inserting, deleting, or modifying elements of an input image based on human instructions. Given the static nature of this task, in this paper, we aim to make this task dynamic by incorporating actions. By doing this, we intend to modify the positions or postures of objects in the image to depict different actions while maintaining the visual properties of the objects. To implement this challenging task, we propose a new model that is sensitive to action text instructions by learning to recognize contrastive action discrepancies. The model training is done on new datasets defined by extracting frames from videos that show the visual scenes before and after an action. We show substantial improvements in image editing using action-based text instructions and high reasoning capabilities that allow our model to use the input image as a starting scene for an action while generating a new image that shows the final scene of the action.

Figures

Figures reproduced from arXiv: 2412.04558 by the authors.

Figure 1
Figure 1. Given a text instruction describing an action, we propose [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison between EditAction and the baselines on the LC and HC datasets. Unlike the baselines, our model can implement [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Left side: HC dataset; Right side: LC dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Training of the U-Net model employed by EditAction. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustrations of EditAction’s reasoning capabilities, [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Despite being trained on a limited number of actions, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Limitations of EditAction. the foreground, as seen in the HC dataset. However, this comes at the cost of altering both the background and the objects targeted by the action. To evaluate the reasoning ability of EditAction, we use the HCtest100 dataset. Since we expect …
Figure 8
Figure 8. Figure 8: The selection of λ1 and λ2 hyperparameters based on the F IDoutput score. Considering the results reported for LC and HC datasets, the optimal values of λ1 and λ2 are 50−4 and 30−2 , respectively. dataset. This indicates that the regularization loss is par￾ticularly im…
Figure 9
Figure 9. Figure 9: Comparison between EditAction and the baselines on LC and HC datasets. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 47 canonical work pages

  1. [1]

    Black, and G¨ul Varol

    Nikos Athanasiou, Mathis Petrovich, Michael J. Black, and G¨ul Varol. SINC: spatial composition of 3d human motions for simultaneous action generation. In IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 9950–9961. IEEE, 2023. 2

    work page 2023

  2. [2]

    Text2live: Text-driven layered image and video editing

    Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kas- ten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 707–723. Springer, 2022. 1, 2

    work page 2022

  3. [3]

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the 38th International Conference on Ma- chine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 813–824. PMLR, 2021. 6

    work page 2021

  4. [4]

    Tim Brooks, Aleksander Holynski, and Alexei A. Efros. In- structpix2pix: Learning to follow image editing instructions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17- 24, 2023, pages 18392–18402. IEEE, 2023. 1, 2, 5, 6

    work page 2023

  5. [5]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing. In IEEE/CVF International Conference on Com- puter Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 22503–22513. IEEE, 2023. 3, 6

    work page 2023

  6. [6]

    A short note about kinetics-

    Jo ˜ao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-

    work page

  7. [7]

    Freeman, Michael Ru- binstein, Yuanzhen Li, and Dilip Krishnan

    Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jos ´e Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Ru- binstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to- image generation via masked generative transformers. In In- ternational Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawa...

    work page 2023

  8. [8]

    Learning variational motion prior for video-based motion capture

    Xin Chen, Zhuo Su, Lingbo Yang, Pei Cheng, Lan Xu, Bin Fu, and Gang Yu. Learning variational motion prior for video-based motion capture. CoRR, abs/2210.15134, 2022. 2

    arXiv 2022

Show all 53 references
  1. [9]

    Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. Int. J. Com...

    2022

  2. [10]

    Efros, and Aleksander Holynski

    Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, and Aleksander Holynski. Diffusion self-guidance for control- lable image generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurIPS 2023...

    2023

  3. [11]

    DPOK: reinforcement learning for fine-tuning text-to-image diffu- sion models

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, Kangwook Lee, and Kimin Lee. DPOK: reinforcement learning for fine-tuning text-to-image diffu- sion models. CoRR, abs/2305.16381, 2023. 5

    2023 arXiv

  4. [12]

    Egocentric vision language planning

    Zhirui Fang, Ming Yang, Weishuai Zeng, Boyu Li, Junpeng Yue, Ziluo Ding, Xiu Li, and Zongqing Lu. Egocentric vision language planning. CoRR, abs/2408.05802, 2024. 3, 6

    2024 arXiv

  5. [13]

    Bermano, Gal Chechik, and Daniel Cohen-Or

    Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models.ACM Trans. Graph., 42(4):150:1–150:13, 2023. 1, 2

    2023

  6. [14]

    The ”something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fr¨und, Peter Yianilos, Moritz Mueller- Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The ”something something” video da...

    2017

  7. [15]

    Generating diverse and natu- ral 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natu- ral 3d human motions from text. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5142–5151. IEE...

    2022

  8. [16]

    TM2T: stochastic and tokenized modeling for the reciprocal gener- ation of 3d human motions and texts

    Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. TM2T: stochastic and tokenized modeling for the reciprocal gener- ation of 3d human motions and texts. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV , pages 580–

    2022

  9. [17]

    Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopou- los, Xiaoxiao He, Yuxiao Chen, Di Liu, Qilong Zhangli, Jindong Jiang, Zhaoyang Xia, Akash Srivastava, and Dim- itris N. Metaxas. Proxedit: Improving tuning-free real ima...

    2024

  10. [18]

    Prompt-to-prompt image editing with cross-attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross-attention control. InThe Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, ...

    2023

  11. [19]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. In 2021 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2021, pages 7514–7528, 2021. 6

    2021

  12. [20]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In 30th Conference on Neural Information Processing Systems, NeurIPS 2016, pages 6626–6637, 2017. 6

    2016

  13. [21]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In Advances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural 9 Information Processing Systems 2020, NeurIPS 2020, De- cember 6-12, 2020, virtual, 2020. 3

    2020

  14. [22]

    KV inversion: KV embeddings learning for text-conditioned real image action editing

    Jiancheng Huang, Yifan Liu, Jin Qin, and Shifeng Chen. KV inversion: KV embeddings learning for text-conditioned real image action editing. In Pattern Recognition and Com- puter Vision - 6th Chinese Conference, PRCV 2023, Xia- men, China, October 13-15, 2023, Proceedings, Part...

    2023

  15. [23]

    Smartedit: Ex- ploring complex instruction-based image editing with mul- timodal large language models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, and Ying Shan. Smartedit: Ex- ploring complex instruction-based image editing with mul- timodal large language models. In IEEE/CVF Conference on Com...

    2024

  16. [24]

    An edit friendly DDPM noise space: Inversion and manipulations

    Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly DDPM noise space: Inversion and manipulations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 12469–12478. IEEE, 2024. 3, 6

    2024

  17. [25]

    Guided motion diffusion for controllable human motion synthesis

    Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. In IEEE/CVF Inter- national Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 2151–2162. IEEE, 2023. 2

    2023

  18. [26]

    Imagic: Text-based real image editing with diffusion mod- els

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Hui- wen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion mod- els. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2023, Vancouver, BC, Canada, Ju...

    2023

  19. [27]

    Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak

    Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 2206–2217. IEEE, 2023. 5

    2023

  20. [28]

    Dongxu Li, Junnan Li, and Steven C. H. Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to- image generation and editing. In Advances in Neural Infor- mation Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, ...

    2023

  21. [29]

    Subject- diffusion: Open domain personalized text-to-image genera- tion without test-time fine-tuning

    Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject- diffusion: Open domain personalized text-to-image genera- tion without test-time fine-tuning. In ACM SIGGRAPH 2024 Conference Papers, SIGGRAPH 2024, Denver, CO, USA, 27 July 2024- 1 August 2024, page 25. ACM, 2024. 1, 2

    2024

  22. [30]

    Dragondiffusion: Enabling drag-style manipula- tion on diffusion models

    Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipula- tion on diffusion models. In The Twelfth International Con- ference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. 2

    2024

  23. [31]

    The blessing of ran- domness: SDE beats ODE in general diffusion-based image editing

    Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, and Chongxuan Li. The blessing of ran- domness: SDE beats ODE in general diffusion-based image editing. In The Twelfth International Conference on Learn- ing Representations, ICLR 2024, Vienna, Austria, May 7-11,

    2024

  24. [32]

    Black, and G ¨ul Varol

    Mathis Petrovich, Michael J. Black, and G ¨ul Varol. Action- conditioned 3d human motion synthesis with transformer V AE. In2021 IEEE/CVF International Conference on Com- puter Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 10965–10975. IEEE, 2021. 2

    2021

  25. [33]

    Black, and G ¨ul Varol

    Mathis Petrovich, Michael J. Black, and G ¨ul Varol. TEMOS: generating diverse human motions from textual descriptions. In Computer Vision - ECCV 2022 - 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 480–497. Springer, 2022. 2

    2022

  26. [34]

    Recognizing fine-grained and composite ac- tivities using hand-centric features and script data

    Marcus Rohrbach, Anna Rohrbach, Michaela Regneri, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, and Bernt Schiele. Recognizing fine-grained and composite ac- tivities using hand-centric features and script data. Int. J. Comput. Vis., 119(3):346–373, 2016. 3

    2016

  27. [35]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 10674–...

    2022

  28. [36]

    Emu edit: Precise image editing via recognition and genera- tion tasks

    Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and genera- tion tasks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA...

    2024

  29. [37]

    Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Han- shu Yan, Wenqing Zhang, Vincent Y . F. Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for in- teractive point-based image editing. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR 20...

    2024

  30. [38]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In 9th International Con- ference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 3

    2021

  31. [39]

    Dm-align: Leveraging the power of natural language instructions to make changes to images

    Maria Mihaela Trusca, Tinne Tuytelaars, and Marie- Francine Moens. Dm-align: Leveraging the power of natural language instructions to make changes to images. CoRR, abs/2404.18020, 2024. 1, 2

    2024 arXiv

  32. [40]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Van- couver, BC, Canada, June 17-24, 2023 , pages 1921–1930. IEE...

    2023

  33. [41]

    Synthesizing long-term 3d human motion and interaction in 3d scenes

    Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiao- long Wang. Synthesizing long-term 3d human motion and interaction in 3d scenes. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 9401–9411. Computer Vision Foun...

    2021

  34. [42]

    Dynamic prompt learning: Address- ing cross-attention leakage for text-based image editing

    Kai Wang, Fei Yang, Shiqi Yang, Muhammad Atif Butt, and Joost van de Weijer. Dynamic prompt learning: Address- ing cross-attention leakage for text-based image editing. In Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processin...

    2023

  35. [43]

    In-context learning unlocked for diffusion models

    Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang (Atlas) Wang, and Mingyuan Zhou. In-context learning unlocked for diffusion models. In Advances in Neural Information Processing Sys- tems 36: Annual Conference on Neural Information Proce...

    2023

  36. [44]

    Visual chatgpt: Talking, drawing and editing with visual foundation models

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, abs/2303.04671, 2023. 1, 2

    2023 arXiv

  37. [45]

    Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms

    Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Ste- fano Ermon, and Bin Cui. Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21-27,

    2024

  38. [46]

    Zero-shot contrastive loss for text-guided diffusion image style transfer

    Serin Yang, Hyunmin Hwang, and Jong Chul Ye. Zero-shot contrastive loss for text-guided diffusion image style transfer. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 22816– 22825. IEEE, 2023. 1, 2

    2023

  39. [47]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 3813–

    2023

  40. [48]

    Learning motion priors for 4d human body capture in 3d scenes

    Siwei Zhang, Yan Zhang, Federica Bogo, Marc Pollefeys, and Siyu Tang. Learning motion priors for 4d human body capture in 3d scenes. In 2021 IEEE/CVF International Con- ference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 11323–11333. IEEE,

    2021

  41. [49]

    OpenReview.net, 2024. 1, 2

    2024

  42. [53]

    LC and HC datasets LC dataset

    Appendix 7.1. LC and HC datasets LC dataset. While 66.49% of LC text instructions specify a starting point for the action, only 34.11% indicate the end- ing point. In 29.08% of cases, the text instruction specifies only the action verb and the target object of the activity. Af...

  43. [600]

    CoRR, abs/1808.01340, 2018. 6

    2018 arXiv

  44. [2024]

    OpenReview.net, 2024. 2

    2024

  45. [5851]

    IEEE Computer Society, 2017. 6

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.