{"id":"f2f3ae02-fc2d-4c6c-888a-cf565078e5ee","arxiv_id":"2412.12087","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A diffusion editing model, InstructMove, is trained on video frame pairs annotated by MLLMs using spatial conditioning, enabling non-rigid edits and viewpoint changes.","lead":"The authors train an image-editing model on pairs of frames sampled from internet videos, with editing instructions generated by multimodal language models. The model targets hard edits such as pose changes and viewpoint shifts, and the paper reports strong results on a small benchmark they created.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training signal from video frame pairs may contain unmodeled temporal changes that MLLM instructions fail to cover; Sec 3.3's dismissal is unsupported, threatening the Sec 5.2 claim of precise instruction-following.","rationale":"The reader's weakest assumption is exactly the one I identify: video frame pairs are linked by time, not by intent, so temporal differences such as lighting, motion blur, identity drift, or background motion may not be captured by the MLLM instruction. The paper itself acknowledges this in Sec 6, and Sec 3.3 provides only an unsupported assertion that the discrepancies are small. This is load-bearing because the entire training signal depends on the instruction being an accurate and complete description of the transformation; if it is not, the model learns a noisy mapping between text and image changes. The concrete annotation-and-retraining test would settle whether the noise actually degrades instruction following. I do not see an argument that the method is internally inconsistent; the concern is an empirical risk, not a logical flaw. The spatial conditioning idea is novel and the ablations (Table 4) support the importance of both the dataset and the conditioning choice, which is real evidence in the paper's favor. However, the lack of released code, data, thresholds, and error bars, combined with the unresolved supervision-noise question, keeps the central claim conditional. My read therefore does not change the reader's verdict: CONDITIONAL acceptance subject to verification of dataset quality and reproducibility remains appropriate.","tokens_in":11653,"tokens_out":8821,"duration_ms":86629,"concrete_test":"Randomly sample 200 triplets from the constructed dataset (or, if data are not released, reconstruct the pipeline on 50 videos and sample 200 pairs). Have three independent annotators list all salient differences between I_s and I_e, and check whether the MLLM-generated instruction covers every listed difference; compute the uncovered-difference rate. Then fine-tune the same architecture on the subset with zero uncovered differences and evaluate on the 50-image benchmark plus a new 200-image external benchmark. If the cleaned model improves CLIP-Inst by more than the gap between Ours and the closest baseline (0.8724 vs 0.8648, Table 2) or changes human preference, the noisy supervision is load-bearing; if performance is unchanged, the Sec 3.3 assertion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a model trained on real video-frame pairs with MLLM instructions can perform complex non-rigid edits while preserving consistency. This requires the triplet (I_s, I_e, C) to be consistent: C should describe the salient transformation from I_s to I_e, and I_e should not contain large changes absent from C. The pipeline in Sec 3.1 filters by optical-flow magnitude and background occlusion ratio only; it does not filter for editability or instruction completeness. Sec 3.2 lets the MLLM reject pairs it cannot describe, but no completeness metric is reported. Sec 3.3 asserts that 'small discrepancies do not significantly impact the training process' without evidence. Sec 6 admits MLLMs 'may generate inaccurate instructions or fail to capture all transformations between frames,' which can cause unintended edits (Fig. 8 left). If a substantial fraction of the 6M pairs has unmodeled lighting shifts, motion blur, identity drift, or background motion, the denoising objective (Eq. 1) trains the model to reproduce those unmentioned changes, so at inference the model may hallucinate edits or ignore the instruction. That would invalidate the claim that the method 'successfully handles these edits while preserving consistency.' The paper's defense rests on an unquantified assertion rather than measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data-construction pipeline that samples frame pairs from internet videos, filters them by optical flow magnitude and background occlusion, and uses multimodal LLMs (GPT-4o, Pixtral) to generate editing instructions. The resulting 6M triplet dataset is used to fine-tune Stable Diffusion with a spatial-conditioning scheme in which the source image latent is concatenated to the noisy target latent along the width dimension. The model, InstructMove, is evaluated on a 50-image benchmark of non-rigid manipulations, viewpoint changes, and rearrangements, against six zero-shot and instruction-based baselines, with a human study. The paper also demonstrates mask-based local editing and integration with ControlNet.","tokens_in":11863,"tokens_out":3861,"duration_ms":32082,"significance":"The central idea—using real video pairs as supervision for instruction-based editing—is timely and potentially addresses the scalability and realism bottleneck of synthetic editing datasets. If the evidence were stronger, the paper could open a new data avenue for the community. The spatial conditioning is simple and effective, and the ablation in Table 4 supports the contribution of both the real data and the conditioning scheme. However, the current evidence for the state-of-the-art claim is limited: the evaluation is a small, non-released benchmark without error bars, and the consistency of the video-triplet supervision is not quantified.","major_comments":[{"comment":"The consistency of the triplet (I_s, I_e, C) is load-bearing for the central claim, but the pipeline does not measure whether C completely describes the changes between I_s and I_e. The assertion in Sec. 3.3 that 'small discrepancies do not significantly impact the training process' is not supported by any experiment. Since the denoising loss in Eq. (1) trains the model to reproduce all changes between the frames, unmodeled lighting shifts, background motion, or identity drift become part of the supervision, which can cause the model to hallucinate edits or ignore the instruction (as acknowledged in Sec. 6). Please provide quantitative evidence of instruction completeness (e.g., a human or LLM-based verification on a random sample, or an ablation with a completeness filter) and report the rejection rate of the MLLM.","section":"Sec. 3.2/3.3 and Eq. (1)"},{"comment":"The evaluation benchmark is only 50 self-curated images and the metrics are reported as point estimates without error bars or significance tests. The differences between Ours and the strongest baselines in CLIP-Inst (0.8724 vs. 0.8648) and CLIP-I (0.9275 vs. 0.9380) are small, and we cannot tell if they are statistically meaningful. The user study (Table 3) reports 87.62% preference for Ours but does not describe participant variance, the number of independent choices per example, or inter-annotator agreement. Please release the benchmark, report confidence intervals or significance tests, and provide more details on the user study.","section":"Sec. 5.1, Table 2"},{"comment":"The frame sampling parameters are not reported: the flow-magnitude thresholds for 'moderate movement', the occlusion ratio threshold, and the fixed time interval (stated as 3 seconds) are not given, and the number of source videos and the MLLM rejection rate are omitted. Without these, the dataset construction is not reproducible and the '6M pairs' claim cannot be verified. Please specify the thresholds, the filtering statistics, and ideally release the dataset or a sample.","section":"Sec. 3.1"}],"minor_comments":[{"comment":"The new CLIP-Inst metric is not validated. It would strengthen the paper to show that CLIP-Inst correlates with human judgments or to compare it against the existing CLIP-D on a set where ground-truth instructions are known.","section":"Sec. 5.1"},{"comment":"The text contains a formatting artifact: 'LLaV A' should be 'LLaVA' (also in Sec. 2.2, the reference to LLaVA is cited as 'Liu et al.' with a space).","section":"Sec. 1"},{"comment":"Table 1 is visually unclear: the columns 'Real Target?' and the numbers appear misaligned, and it is not specified whether '6M' refers to the final filtered triplet count. Please clarify.","section":"Table 1"},{"comment":"The paper refers to 'the supplementary material' for details of the benchmark, but the supplementary material is not provided; please include it in the revision.","section":"Sec. 5.1"},{"comment":"The limitation example on the left shows a 'slight viewpoint change', but it is unclear whether this is a common failure mode; a frequency estimate would help.","section":"Fig. 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely from a strong industrial-academic collaboration, and the core idea is appealing. The main concern is the gap between the strength of the claims ('state-of-the-art') and the evidence (a 50-image self-curated benchmark, no released data or code, no error bars). The triplet-consistency issue in the data pipeline is the most serious technical risk; if the authors can provide quantitative completeness analysis and a more rigorous evaluation, the paper could become suitable for a top venue. I would encourage the editor to consider a revision with these changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea is worth taking seriously: instead of synthesizing target images, sample real frame pairs from video, let an MLLM write the editing instruction, and fine-tune a diffusion model with the source image concatenated alongside the noise latent rather than across channels. That spatial-conditioning trick is simple, well-motivated, and the ablations suggest it does real work. The paper is the first large-scale editing dataset built this way, and the qualitative results show pose, expression, and viewpoint edits that InstructPix2Pix and MagicBrush simply don't produce.\n\nThe weaknesses are real but mostly at the evidence level. The benchmark is 50 self-curated images, no code or data release, the motion-filter thresholds and MLLM rejection rates are unreported, and every number in Tables 2-4 lacks error bars. The new CLIP-Inst metric is a reasonable idea but needs validation. The user study is small and self-selected. That is enough to make 'state-of-the-art' a claim the paper hasn't earned.\n\nOn the stress-test: the worry about unmodeled temporal changes is legitimate, but I don't think it's a load-bearing flaw. Sec 3.3's claim that small discrepancies don't matter is indeed a hand-wave—no completeness metric, no ablation on instruction noise. But the pipeline does filter for moderate motion and background occlusion, and lets the MLLM reject pairs it can't describe. The limitation section openly shows an unintended viewpoint shift. So the risk is real, and the paper should quantify it, but the central approach isn't invalidated by it. The model is clearly learning the intended transformations on the examples shown.\n\nBottom line: the dataset idea is a genuine contribution and the conditioning trick is worth a look. What the paper needs before it can claim SOTA is a much stronger evaluation: a larger benchmark with baselines re-run, error bars, released data/code, and some measurement of instruction completeness and noise. I'd send this to peer review—it deserves referee time—and I'd probably cite it for the data pipeline, but I wouldn't trust the quantitative claims as they stand.","headline":"A genuinely useful dataset idea with a plausible new conditioning trick, but the evidence is too thin to back the SOTA claim.","tokens_in":12431,"tokens_out":2992,"would_cite":true,"duration_ms":26175,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper trains an image editor on 6M real video frame pairs with MLLM-written instructions, enabling non-rigid pose, layout, and viewpoint edits that preserve identity.","keywords":["instruction-based image editing","video frame pairs","multimodal large language models","diffusion models","spatial conditioning","non-rigid editing","viewpoint change","dataset construction"],"falsifier":"Take a random sample of the 6 million triplets and give a held-out text-only editor only the source image and the generated instruction, then measure whether humans or a vision-language model can pick the true target frame from a set of distractors such as frames from other videos or other edits; if pick rates approach chance on a substantial fraction, the frame pairs or instructions are too noisy to support the claimed mapping. A complementary experiment would shuffle instructions across frame pairs and train the same model: if edited-image quality does not drop, the instructions are not the causal driver the paper claims.","tokens_in":11402,"feed_emoji":"🎬","tokens_out":8759,"duration_ms":66272,"temperature":0.7,"pith_summary":"This paper tries to establish that the bottleneck in instruction-based image editing—the lack of large, realistic source-target-instruction training data—can be solved by harvesting pairs of frames from ordinary videos. The authors build a dataset of 6 million triplets by sampling video frame pairs, filtering them with optical flow to keep moderate, background-stable motion, and asking multimodal LLMs to write the editing instruction that turns the first frame into the second. They then fine-tune a pretrained text-to-image diffusion model with a spatial conditioning scheme that concatenates the source image beside the noisy target, allowing the model to attend to any part of the reference while restructuring the output. If the central claim is right, instruction-based editors can handle non-rigid pose changes, element rearrangement, and viewpoint changes that synthetic-data models fail at, while preserving identity.","feed_headline":"Photo editing from text gets real-world training data from video","feed_subtitle":"Trained on 6M real frame pairs, it handles pose, layout, and viewpoint edits that synthetic-data models miss.","key_machinery":"Two components carry the argument. First, the data pipeline: fixed-interval frame sampling, RAFT optical-flow magnitude filtering to discard pairs with too little or too much movement, a background-occlusion mask ratio to discard large background changes, and MLLM-generated instructions with the option to reject pairs the model cannot describe. Second, Spatial Conditioning: instead of concatenating the reference latent with the noisy target along the channel dimension (which locks pixel alignment), the source latent and noisy target latent are placed side by side along the width dimension, the doubled-width tensor is fed into the denoising U-Net, and the right half of the output is cropped to compute the denoising loss. Because every cross-attention layer can attend from any target region to any source region, the network can apply non-rigid transformations and viewpoint changes while still copying fine appearance details from the reference.","core_discovery":"The paper's central claim is that real video frames, not synthetic targets, are the supervision signal that lets an instruction-following editor perform structural edits. From internet videos it constructs a 6-million-triplet dataset: pairs of frames about three seconds apart, filtered with optical-flow magnitude and background-occlusion checks to keep moderate, describable motion, and then labeled by multimodal LLMs with short action-verb instructions that explain how the first frame becomes the second. The model, InstructMove, is a pretrained text-to-image diffusion model fine-tuned with Spatial Conditioning, in which the source-image latent and the noisy target latent are concatenated along the width dimension, fed through the denoising U-Net, and the right half of the predicted noise is cropped for the diffusion loss. The paper reports that this approach outperforms prior instruction-based and tuning-free methods on a 50-image benchmark for non-rigid edits and viewpoint changes, with the highest instruction-alignment scores and 87.62% human preference, because it can change structure while preserving the source subject's identity.","pith_inferences":["If the video-supervision premise holds, the scaling path for instruction editing is video abundance: larger and more diverse video corpora should keep improving edit quality without any manual annotation, which prior synthetic-data approaches cannot offer.","The MLLM instruction step imposes a ceiling: frame-pair changes that cannot be verbalized are systematically excluded, so the dataset implicitly defines 'edit' as language-describable change; recovering the excluded transformations would require human-in-the-loop labeling or a richer instruction interface.","A testable extension is to train on longer-range frame pairs or multi-view videos: if viewpoint transformations are learned from natural video, the same pipeline may yield camera-control abilities without explicit 3D supervision.","The spatial-conditioning trick may transfer to other conditional generation tasks where the reference and output are not pixel-aligned, such as reference-based synthesis or structure-guided generation from a different viewpoint."],"forward_implications":["Instruction-based editing models can be trained at scale from unlabeled internet video, avoiding the cost and artifact problems of synthetic target generation.","Models trained this way should handle non-rigid pose and expression changes, object repositioning, and camera-perspective shifts that structure-preserving methods leave mostly unchanged.","Because the conditioning scheme keeps the base text-to-image architecture intact, the trained editor can be combined with masks, ControlNet, or other spatial adapters for localized and skeleton- or sketch-guided edits.","Using this video-derived dataset alongside existing synthetic datasets can cover both realistic motion edits and artistic edits like style transfer, as the paper demonstrates in the supplementary material.","The proposed CLIP-Inst metric, which asks an MLLM to regenerate instructions from input and output images and measures CLIP distance to the original instruction, gives a way to evaluate instruction adherence without human labels."],"supporting_citations":[{"why":"Defines the synthetic-target dataset and instruction-following training paradigm that this paper contrasts, ablates against, and uses as a baseline.","marker":"[4]"},{"why":"Supplies the optical-flow method used to filter frame pairs by motion magnitude and background occlusion.","marker":"[37]"},{"why":"Provides the pretrained text-to-image diffusion model that InstructMove fine-tunes with spatial conditioning.","marker":"[31]"},{"why":"One of the multimodal LLMs used to generate editing instructions from frame pairs.","marker":"[1]"},{"why":"Multimodal LLM used for captioning and instruction generation, and for the MLLM-based evaluation approaches.","marker":"[2]"},{"why":"The synthetic target-image generation method used by prior datasets, representing the paradigm the paper argues is the bottleneck.","marker":"[13]"},{"why":"Provides a benchmark and baseline; its structure-preserving test set motivates the paper's new non-rigid editing benchmark.","marker":"[41]"},{"why":"Shows the trained editor can integrate external spatial controls because the base architecture is unchanged.","marker":"[42]"}],"fun_headline_variants":["Video teaches AI to edit photos with real-world motion","Real video frames train AI for complex photo edits","Learning photo edits from natural video dynamics","AI photo editor learns from real video motion","Text-to-photo editing learns from real video movement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a pair of video frames separated by a few seconds, filtered by optical flow and described by an MLLM, is a faithful training signal for an image-edit instruction—the time gap must correspond to one describable edit, and the generated instruction must match what actually changed.","fun_headline_variants_meta":{"raw":{"variants":["Video teaches AI to edit photos with real-world motion","Real video frames train AI for complex photo edits","Learning photo edits from natural video dynamics","AI photo editor learns from real video motion","Text-to-photo editing learns from real video movement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1779,"prompt_tokens":885,"completion_tokens":894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":824}},"tokens_in":501,"tokens_out":894,"duration_ms":8191,"temperature":1.0,"reasoning_tokens":824,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:16:15.310800+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 6 million triplets and give a held-out text-only editor only the source image and the generated instruction, then measure whether humans or a vision-language model can pick the true target frame from a set of distractors such as frames from other videos or other edits; if pick rates approach chance on a substantial fraction, the frame pairs or instructions are too noisy to support the claimed mapping. A complementary experiment would shuffle instructions across frame pairs and train the same model: if edited-image quality does not drop, the instructions are not the causal driver the paper claims.","supporting_citations":[{"cited_title":"In- structpix2pix: Learning to follow image editing instructions","cited_arxiv_id":null,"evidence_quote":"Defines the synthetic-target dataset and instruction-following training paradigm that this paper contrasts, ablates against, and uses as a baseline."},{"cited_title":"Magicbrush: A manually annotated dataset for instruction- guided image editing","cited_arxiv_id":null,"evidence_quote":"Provides a benchmark and baseline; its structure-preserving test set motivates the paper's new non-rigid editing benchmark."},{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Shows the trained editor can integrate external spatial controls because the base architecture is unchanged."}],"review_version":1}