{"id":"6e068cec-41fa-402a-adf7-30b58bf39bd4","arxiv_id":"2412.04558","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EditAction fine-tunes InstructPix2Pix with a contrastive action loss and video-derived before/after frames to edit images according to action text commands while preserving object appearance and background.","lead":"The researchers trained a diffusion model to edit images by following action instructions, such as moving an object or changing a posture, while keeping the object's look and the background intact. The method uses pairs of video frames showing scenes before and after an action to teach the model, and it tests better than prior image editors on kitchen activities.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test split is at image-pair level, not video level, so train and test can contain segments of the same video; the small reported gains over Fine-tuned InstructPix2Pix could be explained by this leakage.","rationale":"The reader's weakest assumption concerns the quality of video-derived pairs and the shared source between training data and the TimeSformer evaluator. I agree that the video-derived pairs are noisy and that the evaluator shares data-source bias, but the most load-bearing issue is more specific: the test split is described only at the image-pair level, with no indication that all segments from a single source video are kept in one split. This is a standard and critical methodological requirement for video-derived image-editing datasets, because many pairs from the same video share the same scene, actor, objects, and action label. If the split is not video-disjoint, the model can exploit near-duplicate scenes at test time, and the reported gains over the Fine-tuned InstructPix2Pix baseline are small enough (roughly 1.5–2.2 accuracy points and 0.03–0.14 human-score points) that even modest overlap could account for them. The reader's requested fixes to Eq. 5, Table 5, and hyperparameter selection are also legitimate, but they concern presentation and potential optimistic bias; the video-level split is the assumption that, if violated, would directly undermine the central empirical claim. A concrete, inexpensive check is to retrain and reevaluate under a video-disjoint split. If the advantage persists, the main claim is much better supported; if it does not, the paper's headline improvement is not established. Therefore the verdict should remain conditional, with this additional condition made explicit.","tokens_in":15059,"tokens_out":6029,"duration_ms":61435,"concrete_test":"Re-run the complete training and evaluation pipeline with a strict video-level split: assign all pairs extracted from each source video entirely to train, validation, or test, and fine-tune TimeSformer only on training videos. Report Acc, FID, and the human evaluation scores on this disjoint test set, along with the number of videos in each split and a check that no test video overlaps with training. If EditAction's advantage over Fine-tuned InstructPix2Pix disappears or falls within reported variance, the claimed improvement is an artifact of train/test leakage rather than a general action-editing ability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1 describes the test sets as 'randomly selected 500 image pairs from the LC and HC datasets' with the remaining instances used for training; no video-level separation is reported. Each source video yields many first/last-frame pairs, so random pair-level assignment can place segments from the same video (same kitchen, actor, action label, and often the same background and objects) in both training and test. In that case, EditAction can succeed by memorizing dataset-specific scene transitions rather than by executing the action instruction on a genuinely new input. This is especially damaging because the reported advantage over the strongest baseline is small: on LCtest500, EditAction's Acc is 71.4 vs 69.2 for Fine-tuned InstructPix2Pix; on HC test500 it is 54.41 vs 52.84, and the human R1 differences are 0.14 and 0.03. The TimeSformer evaluator is fine-tuned on the same source videos (Section 4.2), so the Acc metric may reward reproducing those dataset-specific before-to-after transitions rather than semantically correct action editing. Without a video-disjoint split, the central generalization claim that the model edits actions in new images is not supported by the reported numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EditAction, a method for text-guided image editing that implements actions (e.g., changing object positions or postures) rather than static edits. The authors fine-tune InstructPix2Pix on pairs of first/last frames extracted from action segments in MPII-Cooking (LC) and EPIC-Kitchen (HC), using a contrastive action loss (Eq. 5) that contrasts the correct action caption with a random caption, plus a regularization loss (Eq. 6) and freezing of cross-attention layers. Evaluation uses TimeSformer action recognition accuracy between input and output images, FID scores, and human ratings (R1-R5). The paper reports that EditAction outperforms several baselines, including a fine-tuned InstructPix2Pix, on both datasets and on a reasoning-focused HC test100 subset.","tokens_in":15348,"tokens_out":6103,"duration_ms":53864,"significance":"The task of editing images according to action instructions is a timely and useful extension of text-based image editing, and the proposed contrastive training on video-derived pairs is a plausible way to inject action awareness into a pretrained diffusion editor. The datasets (LC and HC) are potentially reusable resources. If the reported results survive a cleaner experimental protocol, the method would be a solid contribution; the paper also includes human evaluation and some analysis of unseen verbs and failure cases.","major_comments":[{"comment":"Equation (5) is not a log-likelihood as labeled; the right-hand side is a softmax probability of the form exp(a)/sum_j exp(a_j), and the exponent uses +E[||epsilon - epsilon_theta||^2] whereas the diffusion ELBO estimate in Eq. (2) is proportional to -E[||epsilon - epsilon_theta||^2]. As written, minimizing Laction would push the model to increase the noise-prediction error for the correct caption, which is the opposite of the intended objective. Please correct the sign and the logarithm, and state explicitly whether the implementation optimizes the negative log-likelihood.","section":"3.3, Eq. (5)"},{"comment":"The test sets are created by randomly sampling 500 image pairs from each dataset after training on the remaining pairs. Because each source video yields many segments and hence many first/last-frame pairs, this pair-level split can put segments from the same video (same actor, kitchen, and often same background and objects) into both training and test. This is especially problematic because the reported gains over the strongest baseline are small (e.g., LC Acc 71.4 vs 69.2; HC R1 4.28 vs 4.25 in Tables 1 and 2). Furthermore, the TimeSformer evaluator is fine-tuned on the same source videos used to construct the training pairs. A video-disjoint split is necessary to support the central generalization claim; please report such results or provide a leakage analysis.","section":"4.1 and 4.2"},{"comment":"The HC test500 row for 'w/o Laction' reports Acc = 104.68 ± 0.22 and FIDinput = 0.99 ± 1.23. An accuracy exceeding 100% is impossible, and an FID input near 1.0 is implausible when all other HC rows are near 95-105. As printed, this row also contradicts the text in Section 7.3, which states that omitting Laction has the largest negative effect; the row instead appears to show a huge positive effect. Please correct the table or explain the anomaly; the ablation conclusions cannot be evaluated without this row.","section":"Table 5"},{"comment":"The human evaluation differences between EditAction and Fine-tuned InstructPix2Pix are very small (e.g., R1 4.37 vs 4.23 on LC, 4.28 vs 4.25 on HC), yet the abstract and Section 5 claim 'substantial improvements.' No significance tests or confidence intervals are reported for the human ratings, and the Acc metric is reported as mean ± variance over three seeds without a statistical test. Please add appropriate significance testing or temper the claims.","section":"Tables 1 and 2, Section 5"}],"minor_comments":[{"comment":"Equation (4) is typeset incorrectly; the denominator should be a proper fraction. Please fix the formatting.","section":"3.3, Eq. (4)"},{"comment":"Table 5, HC test500 row 'w/o Lreg' lists a standard deviation of ±17 for Acc, which is unusually large; please verify the variance calculation.","section":"Table 5"},{"comment":"In Section 5, the statement that the HC test100 result is a 'substantial difference' is not supported by Table 3, where EditAction's Acc is 46.29 vs 45.89 for Fine-tuned InstructPix2Pix; consider a more measured description.","section":"Section 5"},{"comment":"The notation for hyperparameters (e.g., '50−4', '30−2') is ambiguous; please clarify whether these are powers of ten or multiplication factors (e.g., 50×10^-4).","section":"4.3 and Appendix 7.2"},{"comment":"The assumption that the first and last frames of a segment represent the start and end of a single action, and that the text annotation describes the change, should be discussed, especially for segments with multiple sub-actions or camera motion; consider adding a statistical analysis of segment lengths and labels.","section":"Section 3.2"},{"comment":"In Figure 2, the text is small and the red/green/blue annotations from Figure 3 are not reproduced; please improve readability.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The pair-level test split is the most serious issue; if the authors cannot provide a video-disjoint split, the empirical contribution is substantially weakened. The corrupted ablation row in Table 5 should be corrected before any further review. The loss equation issue may be a notation problem, but the sign is a potential implementation concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey,\n\nRead the EditAction paper. The core idea is reasonable: take InstructPix2Pix, fine-tune it on before/after frames from action video datasets with an extra contrastive term that tells the model which action text matches the edit, plus a regularization that keeps the fine-tune from wrecking the base model. That combination is new, the two derived datasets are useful, and the human evaluation is real work. The model does beat the strongest baseline on several metrics, though the margins are small.\n\nThe paper has two fixable but real technical problems, and one experiment-design problem that matters more.\n\nFirst, Eq. 5 is simply not a log-likelihood as written; it is a softmax probability over two classes. Either the equation or the text needs correcting, and the loss should be stated as something like the negative log of that quantity. Table 5 also has a nonsensical row: HC test500 \"w/o Laction\" shows accuracy 104.68 and FID_input 0.99. That is a typo or a formatting error, but it makes the ablation table unreliable as printed.\n\nSecond, the bigger issue: the 500 test pairs are selected randomly from the whole pair pool, with no video-level separation. Each source video contributes many first/last-frame pairs, so the same kitchen, actor, and often the same background can appear in both training and test. Under those conditions, the reported Acc gains over Fine-tuned InstructPix2Pix (71.4 vs 69.2 on LC, 54.4 vs 52.8 on HC) could be explained by the model memorizing dataset-specific scene transitions rather than by genuinely executing a new action instruction. The TimeSformer evaluator is also fine-tuned on the same source videos, so it may reward those same transitions. The human R1 differences (0.14 and 0.03) are too small to resolve this. This is not a fatal flaw in the method, but as reported the experiments do not establish the generalization claim. The authors need a video-disjoint split.\n\nAlso worth asking: lambda_1 and lambda_2 were selected by ranking FID_output on models trained for 1000 steps. It is not stated whether that ranking was done on a held-out split or on the test sets. Given the small final gains, this matters.\n\nThe paper is honest about its limitations and shows failure cases, which I appreciate. The idea and datasets deserve attention, but the current numbers should not be taken at face value. I would send it to review with a clear request for a video-disjoint test split, corrected equations, and a cleaned-up Table 5. If that split still shows a consistent margin over the fine-tuned baseline, this becomes a useful paper for people working on instruction-based editing and egocentric planning.","headline":"Sensible idea and useful new datasets, but the experimental protocol leaks video identity across train/test, so the reported gains over the fine-tuned baseline are not yet convincing.","tokens_in":15843,"tokens_out":2653,"would_cite":false,"duration_ms":88071,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EditAction fine-tunes a static image editor with a contrastive action loss so a text instruction like 'flip the tofu' changes position or posture while preserving the object's appearance and the scene.","keywords":["action-based image editing","diffusion models","contrastive learning","instruction following","video frames","action recognition","InstructPix2Pix","egocentric video"],"falsifier":"Train EditAction and the TimeSformer evaluator on the same video corpora as in the paper, then test on a held-out set of action segments from unrelated video datasets (for example, sports or assembly tasks) with fresh text annotations. If action-recognition accuracy does not improve over a static-editing baseline on those unseen actions, the claimed learning of generalizable action discrepancies fails.","tokens_in":1464,"feed_emoji":"🎬","tokens_out":2539,"duration_ms":59226,"temperature":0.7,"pith_summary":"This paper proposes EditAction, a model that extends text-prompted image editing from static changes such as adding or removing objects to dynamic action changes, where the output shows an object or person in a new position or posture while keeping its visual identity and the surrounding scene. The idea is to fine-tune InstructPix2Pix on pairs of first and last frames extracted from videos of human actions, using a contrastive loss that makes the model prefer the correct action caption over a randomly chosen one. The authors report that this learns action discrepancies well enough to implement actions, including verbs unseen during training, and that it outperforms prior editors on action-recognition accuracy and human evaluation. If correct, action-based image editing becomes a single-frame generation problem rather than a video-generation problem, and edited images can serve as goal frames for embodied agents and virtual-scene creation.","feed_headline":"One model turns 'flip the tofu' into an edit","feed_subtitle":"EditAction learns action instructions from before/after video frames, keeping the scene intact.","key_machinery":"The load-bearing mechanism is a contrastive action loss built on the diffusion model's evidence lower bound. For an output image, an input image, the correct action caption, and a randomly chosen caption sharing no tokens with the correct one, the model is trained to maximize the probability of the correct caption by comparing expected denoising errors for the two captions, following the diffusion-as-classifier construction. This loss is combined with the original static editing loss and a regularization term that penalizes drift between the fine-tuned U-Net and the frozen pretrained U-Net, and the cross-attention layers are frozen throughout training. The mechanism lets the model suppress unrelated actions while learning to implement the specified one, and it is what carries the claimed improvements in action accuracy and appearance preservation.","core_discovery":"The paper's central claim is that a diffusion-based editor trained for static instruction-following can be converted into an action-aware editor by fine-tuning it to maximize the likelihood of the correct action text relative to a random action text, given the same input image and edited image. The authors build EditAction on InstructPix2Pix, freeze the cross-attention layers to preserve text-to-visual alignment, and add a regularization term that keeps the fine-tuned denoising network close to the frozen pretrained network. Evaluated on two new datasets built from video-recognition corpora, EditAction reaches the highest action-recognition accuracy among all compared models (71.4 on the fixed-camera LC test set, 54.41 on the flexible-camera HC test set, and 46.29 on the long-distance HC test subset) while maintaining the best or near-best FID scores. Human annotators also rate it highest for action implementation, posture and position editing, and reasoning about the final scene. The broader claim is that the task of action-based image editing can be treated as producing one final frame that depicts the action while preserving the objects and environment of the input frame.","pith_inferences":["The before/after frame extraction recipe could generalize to other instruction-following editors, since large action-recognition video datasets already provide abundant labeled transitions.","Because the action-accuracy metric uses classifiers fine-tuned on the same MPII-Cooking and EPIC-Kitchen videos that supply the training pairs, an independent test set of videos from other activities would be needed to confirm that the learned action discrepancies generalize beyond kitchen and egocentric actions.","A natural extension is to combine the contrastive action loss with optical flow or trajectory information, which would likely address the reported failure cases involving nested objects or long-distance actions where the wrong instance is moved.","If the diffusion-as-classifier interpretation is correct, the same fine-tuning recipe could apply to temporal tasks such as action-specified keyframe prediction or instruction-guided video editing."],"forward_implications":["Action instructions can be applied to a single input image without generating a video sequence, because the method learns to jump directly from the before-frame to the after-frame.","Appearance and background preservation can be inherited from a static editor by freezing its cross-attention layers while learning action directions from video-frame pairs.","New actions described by unseen verbs can be implemented, as demonstrated by 'wipe' on the LC dataset and 'place' on the HC dataset.","The model can reason beyond the input frame, as shown on the HC test100 subset where it produces a plausible new viewpoint of the scene after a long-distance action.","Inference runs at InstructPix2Pix speed, roughly the same cost as the base static editor and much faster than several training-free editing baselines."],"supporting_citations":[{"why":"Supplies the base InstructPix2Pix model, its static editing loss, and the inference-time guidance formula that EditAction builds on.","marker":"[4]"},{"why":"Provides the derivation that converts a conditional diffusion generator into a classifier via Bayes' theorem, which is the basis of the contrastive action loss.","marker":"[27]"},{"why":"Motivates the regularization term that keeps the fine-tuned network close to the frozen pretrained network to prevent forgetting.","marker":"[11]"},{"why":"Provides the MPII-Cooking video corpus from which the LC (fixed-camera) action editing dataset is built.","marker":"[34]"},{"why":"Provides the EPIC-Kitchen egocentric video corpus from which the HC (flexible-camera) action editing dataset is built.","marker":"[9]"},{"why":"The TimeSformer video-activity model that is fine-tuned and used as the automatic action-recognition accuracy metric.","marker":"[3]"},{"why":"Serves as a representative tuning-free baseline against which EditAction's action implementation and preservation are compared.","marker":"[5]"}],"fun_headline_variants":["New model edits images from action instructions like 'flip the tofu'","Action-based image editing: keep the scene, change the pose","EditAction: Learn action edits from before/after video frames","From static to dynamic: images edited by action phrases","Action-aware editor: 'flip the tofu' becomes a visual edit"],"cache_read_input_tokens":17920,"weakest_assumption_plain":"The training data assumption: each action segment in MPII-Cooking and EPIC-Kitchen is represented by taking the first and last frames, and the text annotation for the segment is assumed to describe the change between those two frames well enough that the model can learn appearance preservation and action implementation from these pairs.","fun_headline_variants_meta":{"raw":{"variants":["New model edits images from action instructions like 'flip the tofu'","Action-based image editing: keep the scene, change the pose","EditAction: Learn action edits from before/after video frames","From static to dynamic: images edited by action phrases","Action-aware editor: 'flip the tofu' becomes a visual edit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000915,"raw_usage":{"total_tokens":3918,"prompt_tokens":925,"completion_tokens":2993,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":2905}},"tokens_in":541,"tokens_out":2993,"duration_ms":23200,"temperature":1.0,"reasoning_tokens":2905,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:23:45.429612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train EditAction and the TimeSformer evaluator on the same video corpora as in the paper, then test on a held-out set of action segments from unrelated video datasets (for example, sports or assembly tasks) with fresh text annotations. If action-recognition accuracy does not improve over a static-editing baseline on those unseen actions, the claimed learning of generalizable action discrepancies fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the base InstructPix2Pix model, its static editing loss, and the inference-time guidance formula that EditAction builds on."},{"cited_title":"Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak","cited_arxiv_id":null,"evidence_quote":"Provides the derivation that converts a conditional diffusion generator into a classifier via Bayes' theorem, which is the basis of the contrastive action loss."},{"cited_title":"Recognizing fine-grained and composite ac- tivities using hand-centric features and script data","cited_arxiv_id":null,"evidence_quote":"Provides the MPII-Cooking video corpus from which the LC (fixed-camera) action editing dataset is built."},{"cited_title":"Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100","cited_arxiv_id":null,"evidence_quote":"Provides the EPIC-Kitchen egocentric video corpus from which the HC (flexible-camera) action editing dataset is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The TimeSformer video-activity model that is fine-tuned and used as the automatic action-recognition accuracy metric."},{"cited_title":"Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing","cited_arxiv_id":null,"evidence_quote":"Serves as a representative tuning-free baseline against which EditAction's action implementation and preservation are compared."}],"review_version":1}