{"id":"e9f09474-8626-42aa-bf51-53d4c24d1d86","arxiv_id":"2506.06006","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using an inverse dynamics model to label and verify frame transitions lets vision-language models learn forward dynamics and outperform specialized image editors on Aurora-Bench.","lead":"This paper shows that vision-language models can learn to predict the next image in a scene from a text instruction, by first training them to describe the action between two images and then using that ability to generate or verify future frames. The result is a general-purpose model that outperforms specialized image-editing systems on a benchmark of action-centric edits.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic-trajectory labels are filtered by the IDM's own likelihood and never independently verified; if those labels are systematically wrong, the reported FDM gains would reflect extra visual data, not inverse-dynamics bootstrapping.","rationale":"I agree with the reader's weakest_assumption and treat it as the decisive point. The paper's positive contribution has two dependencies: (i) IDP is learnable, supported by Table 1, and (ii) IDP labels can serve as effective FDP supervision. The second dependency is the one not secured by the paper's own evidence: the only filter applied to synthetic labels is the IDM's predictive likelihood, and the IDM's modest action-prediction scores leave room for systematic error. The human evaluation and judge-reliability checks establish that final outputs are preferred over baselines, but do not isolate whether the improvement comes from correct action semantics. A permutation control would do exactly that: if randomly paired actions preserve the gain, the effect is visual-data scaling, not bootstrapping from inverse dynamics. Since the reader already made the verdict CONDITIONAL, my read does not change that verdict; it specifies the experiment that would either clear or sink the central claim.","tokens_in":28146,"tokens_out":8398,"duration_ms":84732,"concrete_test":"Sample 300 synthetic triples uniformly from the 21K used for C-FDM and have two annotators score whether the IDM action string is correct and sufficiently specific for the visual change from source to target. Separately, retrain C-FDM on the same 21K triples with the IDM action strings randomly permuted across triples, preserving the marginal action distribution, and compare AURORA-BENCH GPT4o scores with Table 2. If human-verified label accuracy is near chance, or if the permuted-label model retains the full synthetic-data gain, the weak-supervision improvement cannot be attributed to correct inverse-dynamics annotations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 and Equation 1 train the FDM on (o_s, â, o_t) triples with â sampled from p_IDM(a|o_s, o_t), and Appendix E's Algorithm 1 then selects triples by ranking the IDM's own likelihood. No independent human or model-based check of the 21K synthetic action labels is reported. This matters because the IDM itself attains only BLEU 0.20 and ROUGE-L 0.44 on held-out action prediction (Table 1), and confidence under the IDM's own distribution is not a correctness certificate: a generic but fluent action string can receive high likelihood for many visually different transitions. If the IDM makes systematic action errors on the out-of-domain MIT/Kinetics/UCF videos, Equation 1 optimizes p_theta(o_t | â, o_s) for the wrong â, so the FDM learns spurious action-effect associations. The Table 4 ablation shows that removing synthetic data lowers scores, but a positive ablation only establishes a data-scale effect; it does not establish that the semantic content of the IDM labels is the operative ingredient. That is precisely the load-bearing condition for the claim that IDP bootstraps FDP.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether unified vision-language models (VLMs) can perform forward dynamics prediction (FDP): predicting the next image observation from a source observation and a textual action. It first shows, via a zero-shot preference test on AURORA-BENCH, that nine VLMs do not consistently prefer ground-truth (o_s, a, o_t) trajectories over adversarially manipulated ones (Section 2). It then proposes to bootstrap FDP from an inverse dynamics model (IDM) that predicts actions from observation pairs. Two strategies are presented: (i) weakly supervised training on synthetic trajectories, where the IDM annotates key-frame pairs from unlabelled videos with actions, combined with a recognition-weighted loss (Sections 3.2); and (ii) inference-time verification, where the IDM scores multiple candidate future frames generated by the FDP (Section 3.3). The resulting forward dynamics models (C-FDM on Chameleon-7B and L-FDM on Liquid-8B) are evaluated on AURORA-BENCH with GPT-4o-as-a-judge and human evaluation, as well as on two spatial reasoning benchmarks. The paper reports that the best model improves over state-of-the-art image editing models by 7%–13% according to GPT-4o, and achieves the best average human evaluation.","tokens_in":28389,"tokens_out":9666,"duration_ms":87614,"significance":"If the central claims hold, the paper offers a practical and timely recipe for equipping general-purpose VLMs with elementary forward-dynamics capabilities without new human annotation, by transferring the easier inverse-dynamics task to a weak-supervision signal. The evaluation is more extensive than is typical for a method paper: it includes ablations for synthetic data and loss weighting, a blind human evaluation with significance tests, a judge-reliability analysis including pairwise win rates, and a stratified analysis of camera motion. The release of code and models is an additional strength. The main quantitative claims and the causal interpretation of the synthetic-data ablation, however, rest on assumptions that are not fully verified in the manuscript: the correctness of the IDM-generated action labels, the disjointness of training and evaluation data, and the statistical robustness of the reported margins.","major_comments":[{"comment":"The manuscript never states whether the AURORA data used to fine-tune the IDM (Sections 3.1 and H.1) and the FDM baselines (Sections 3.2 and H.3) are disjoint from the AURORA-BENCH evaluation subsets used in Section 4.3. Section 2 reports that each AURORA-BENCH subset contains 50 trajectory triplets; if these same triplets are included in the fine-tuning set, the reported gains over the zero-shot and fine-tuned baselines would be inflated or invalid. Please specify the official train/test split, confirm that the evaluation triplets are excluded from all training data, and clarify whether the same 50 triplets are used for the preference test in Section 2 and the FDP evaluation in Section 4.3.","section":"Section 4.1, H.1–H.3"},{"comment":"The synthetic trajectory labels are produced by the IDM and filtered in Algorithm 1 by the IDM's own predicted likelihood, with no independent verification of their correctness. The ablation in Table 4 shows that adding synthetic data changes FDP scores, but it does not establish that the semantic content of the IDM's action labels is the operative ingredient; the improvement could in principle stem from the extra visual diversity of the unlabelled video frames alone. To support the claim that IDP bootstraps FDP, the paper should include a control in which the same synthetic image pairs are paired with random (or shuffled) action texts, or provide a human or strong-model evaluation of a sample of the IDM-generated action labels. This is load-bearing because Eq. (1) optimizes p_theta(o_t | a_hat, o_s) with a_hat sampled from p_IDM, so systematic label errors would train the FDM on incorrect action-effect mappings.","section":"Section 3.2, Eq. (1), Appendix E, Table 4"},{"comment":"The main results are reported as point estimates without error bars or confidence intervals, although the Limitations section acknowledges high variance across runs for Chameleon. Figure 4 shows a wide shaded standard deviation for C-FT on the average GPT-4o score, and Table 4 reports averages of 3 runs without indicating the spread. The headline improvements of 13% for Chameleon (C-FT 3.26 vs. C-FDM 3.67 in Table 2) and 7% for Liquid (L-FT 4.04 vs. L-FDM 4.32) may not be statistically significant if the underlying variance is high. Please provide confidence intervals or significance tests for the comparisons that motivate the abstract's performance claims.","section":"Tables 2 and 4, Figure 4"}],"minor_comments":[{"comment":"The text reports that Qwen2.5-VL-7B prefers the ground-truth action in 60.08% and 67.34% of cases, while Figure 2 appears to show 60.1 and 67.3; the rounding is acceptable, but please ensure the numbers are reported consistently.","section":"Section 2"},{"comment":"The distinction between AURORA and AURORA-BENCH is not defined in the main text; please state explicitly that AURORA refers to the training corpus and AURORA-BENCH to the evaluation benchmark, including the official split used.","section":"Section 4.1"},{"comment":"The model list in the implementation section includes VILA-U, but VILA-U is not used in any experiment in the paper; please remove it or indicate where it is used.","section":"Appendix H.5"},{"comment":"The caption of Figure 11 refers to the metric as \"OE\", while the text and Table 3 use \"ME\" for Minimal Editing; the abbreviation should be made consistent.","section":"Appendix G"},{"comment":"The phrase \"competitive with state-of-the-art image editing models\" should be qualified, because the comparison is on an action-centric image editing benchmark (AURORA-BENCH) and not on general image editing; consider rephrasing to avoid overgeneralization.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the ambiguous relationship between the AURORA training data and the AURORA-BENCH evaluation subsets. If the authors cannot confirm a disjoint train/test split, the paper's empirical claims collapse. The second most important issue is the lack of an independent check of the synthetic action labels; a random-label control or a small human verification study would make the bootstrapping claim credible. The missing error bars are also a concern given the acknowledged high variance. Overall, the paper presents a promising idea and a substantial experimental infrastructure, but the current evidence does not yet suffice to support the causal claim that IDP bootstraps FDP."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this one: the authors show that fine-tuning a VLM for inverse dynamics prediction (captioning the action between two frames) is markedly easier than forward dynamics prediction (generating the next frame), and that the resulting IDM can help a forward model in two ways—by auto-labeling unlabeled video pairs for extra training data, and by scoring candidate future frames at test time. On AURORA-BENCH, their best general-purpose VLM is competitive with specialized image-editing diffusion models, with human evaluation backing the trend. That is a real result worth discussing.\n\nWhat is genuinely new is the package: the IDP-bootstrapping recipe, the recognition-weighted loss that focuses the FDM on changed regions, and the inference-time verification with a learned reward model. The paper also does more than most method papers on the evaluation side—ablations, a blind human study, a judge-reliability check, and a candid limitations section. Credit where due.\n\nThe soft spots are real but not fatal. The biggest one is the stress-test concern: the synthetic trajectories are filtered by the IDM's own likelihood, so the FDM is trained on (source, action, target) triples whose action labels are never independently verified. The IDM itself gets only BLEU 0.20 and ROUGE-L 0.44 on held-out action prediction, so there is honest risk that some labels are fluent but wrong. The ablation dropping synthetic data shows a performance drop, but that only establishes a data-scale effect; it does not prove that the semantic content of the IDM labels is the operative ingredient. This weakens the strong reading of the bootstrapping claim, though the test-time verification leg does not depend on synthetic labels and still delivers gains.\n\nTwo smaller issues: the main tables report point estimates without error bars despite acknowledged high variance, especially for Chameleon; and the train/test disjointness between AURORA training data and AURORA-BENCH is not explicitly documented. Neither sinks the paper, but both should be cleaned up.\n\nOverall, I think the central claim is plausible and the paper is honest. It is for people working on world models from VLMs, action-centric image editing, and self-supervised video understanding. It deserves serious peer review; the authors should add error bars, document dataset splits, and do a small human or independent check on a sample of synthetic labels to address the self-training concern.","headline":"A useful, honest paper showing that VLMs can bootstrap forward dynamics from inverse dynamics, but the synthetic-data leg of the bootstrapping story needs tighter evidence.","tokens_in":28930,"tokens_out":1975,"would_cite":true,"duration_ms":21455,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Predicting the action between two frames can be used to train a vision-language model to predict the next frame, a task it otherwise fails at.","keywords":["world models","forward dynamics prediction","inverse dynamics prediction","vision-language models","action-centric image editing","synthetic trajectories","test-time verification","multimodal grounding"],"falsifier":"Train the same IDM on data with a random subset of action labels shuffled, use it to generate the synthetic triplets, then train the FDM on those triplets and compare GPT-4o scores on AURORA-BENCH; if the gains persist with shuffled labels, the reported bootstrap effect is not due to accurate inverse-dynamics knowledge.","tokens_in":27958,"feed_emoji":"🎬","tokens_out":11112,"duration_ms":91624,"temperature":0.7,"pith_summary":"The paper sets out to give a general-purpose vision-language model the ability to predict the next image frame from a current frame and a language instruction, a task called forward dynamics prediction, at which off-the-shelf VLMs fail, showing near-chance preference for real transitions over manipulated ones. The authors' central claim is that the reverse direction, inverse dynamics prediction, which captions the action that turns one frame into the next, is far easier to learn, and that this easier skill can bootstrap the harder one. They demonstrate two bootstrap routes in a single VLM: using the fine-tuned inverse model to annotate about 45 hours of unlabeled video with actions, turning those pairs into weak supervision for the forward model, and using the same inverse model at inference time to score candidate next frames and pick the best. On the action-centric AURORA-BENCH suite, the resulting forward model, still a general-purpose VLM, is reported to improve on state-of-the-art specialized image-editing models by between 7% and 13% under a GPT-4o judge and to achieve the best average human evaluation. If the claim holds, it offers a practical recipe for building world models without large amounts of paired action-outcome data.","feed_headline":"Naming actions between frames teaches models to predict the next frame","feed_subtitle":"This route lets a general-purpose VLM rival specialized image editors on action-centric benchmarks.","key_machinery":"The load-bearing mechanism is the inverse dynamics model (IDM), namely the same VLM fine-tuned to output $p_{\\mathrm{IDM}}(a \\mid o_s, o_t)$, which performs two jobs. In the synthetic-supervision strategy, it annotates optical-flow-selected key-frame pairs from unlabeled videos with action labels, and a stratified top-k sampling step keeps triplets with high IDM likelihood while preserving class diversity; the resulting triplets train the forward dynamics model (FDM) through a recognition-weighted loss that multiplies each next-frame image token's loss by the squared L2 distance between the pre-quantized source and target features. In the verification strategy, the same IDM assigns reward $p_{\\mathrm{IDM}}(a \\mid o_s, o_t^{(i)})$ to each of $N$ sampled candidates, and the maximum-reward candidate becomes the final prediction. The mechanism's power is that one easy-to-learn function supplies both the extra training signal and the scoring function that the hard function lacks.","core_discovery":"The central discovery is an asymmetry in how well a unified vision-language model grounds the two directions of a transition. Given an observation pair $(o_s, o_t)$, predicting the intervening action $a$ (inverse dynamics) becomes reliable after fine-tuning on a modest amount of ground-truth trajectories, whereas predicting $o_t$ from $(o_s, a)$ (forward dynamics) barely improves and remains prone to copying the source. The paper claims this asymmetry is exploitable: the inverse dynamics model can act as both a teacher and a judge for the forward dynamics model. As a teacher, it labels key-frame pairs selected by optical flow from unlabeled real-world videos, and the synthetic triplets are used to fine-tune the forward model under a recognition-weighted loss that concentrates gradients on image tokens that actually change. As a judge, it scores $N$ candidate next observations sampled from the forward model, and the candidate with the highest inverse likelihood is selected. With both strategies, the bootstrapped forward model, still a general-purpose VLM, is reported to improve over fine-tuning on supervised data alone by 13% for Chameleon and 7% for Liquid in average GPT-4o-as-judge scores across AURORA-BENCH, and to achieve the best human-evaluation average.","pith_inferences":["The same inverse-to-forward bootstrap could apply to other modalities or settings where the 'action' coupling two states is easier to verbalize than the state itself, such as audio-to-video synchronization or robot control policies.","Nothing in the method prevents closing the loop: the improved FDM could generate harder candidate futures for the IDM to score, forming a self-improving dynamics cycle, but the paper evaluates verification only for a fixed FDM and does not retrain the IDM on FDM outputs.","The recognition-weighted loss is a natural training-time analogue of the 'minimal editing' criterion used by the GPT-4o judge; making the weights depend on the action rather than only on pixel differences could reduce the over-editing the paper reports.","Because the IDM and FDM share the same backbone, a systematic IDM error (for example, misnaming a class of actions) would be invisible to the verifier; an independent verifier trained on a different backbone would provide a stronger test of the bootstrap claim."],"forward_implications":["The IDM's synthetic annotations alone raise FDM performance: removing the unlabelled-video triplets drops GPT-4o scores on most AURORA-BENCH subsets, especially Something-Something and Action-Genome.","Inference-time verification with the IDM improves forward dynamics prediction to roughly the same level as trajectory synthesis, offering a training-free way to scale prediction quality with more candidate samples.","The recognition-weighted loss produces its largest gains in high-motion scenes, where copying the source frame fails, while slightly improving instruction-following at the cost of occasional over-editing.","FDP training transfers beyond editing: the bootstrapped Chameleon and Liquid models improve accuracy on the SpatialMQA and EmbodiedSpatial-Bench spatial-reasoning benchmarks over their zero-shot baselines."],"supporting_citations":[{"why":"Supplies the AURORA ground-truth trajectory triplets that train both the IDM and FDM, and the AURORA-BENCH subsets used for all evaluation.","marker":"Krojer et al., 2024"},{"why":"Adds egocentric action clips from EPIC-Kitchen as additional IDM training data.","marker":"Damen et al., 2018"},{"why":"Source of unlabeled human-action video clips from which IDM-annotated synthetic trajectory triplets are extracted.","marker":"Monfort et al., 2019"},{"why":"Source of unlabeled Kinetics700 clips used for synthetic trajectory generation.","marker":"Kay et al., 2017; Carreira et al., 2019"},{"why":"Source of unlabeled UCF-101 clips used for synthetic trajectory generation.","marker":"Soomro et al., 2012"},{"why":"GoT serves as a state-of-the-art specialized image-editing baseline and the source of the GPT-4o-as-a-judge evaluation protocol adopted here.","marker":"Fang et al., 2025"},{"why":"Establishes the prior of training an inverse dynamics model on unlabeled video to synthesize trajectories, which this work ports to real-world frames with language actions.","marker":"Baker et al., 2022"},{"why":"PixInstruct serves as a specialized image-editing baseline the bootstrapped FDM is compared against.","marker":"Brooks et al., 2023"},{"why":"SmartEdit serves as a specialized image-editing baseline that also uses the GPT-4o judge protocol.","marker":"Huang et al., 2024"},{"why":"Motivate scaling inference-time compute by sampling multiple candidates, which the IDP-verifier strategy implements.","marker":"Snell et al., 2024; Muennighoff et al., 2025"}],"fun_headline_variants":["Action-naming bootstraps future-frame prediction in VLMs","Teach VLMs to predict frames by naming the action first","Inverse dynamics as teacher and judge for forward dynamics","From action captions to next-frame prediction in VLMs","VLM future vision: bootstrapped by inverse dynamics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the inverse model's automatically produced action labels for the unlabeled video pairs are correct enough to serve as training targets for the forward model; if the inverse model systematically names the wrong action, the forward model learns wrong (source, action, target) mappings and the reported gains would be an artifact of label noise rather than world knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Action-naming bootstraps future-frame prediction in VLMs","Teach VLMs to predict frames by naming the action first","Inverse dynamics as teacher and judge for forward dynamics","From action captions to next-frame prediction in VLMs","VLM future vision: bootstrapped by inverse dynamics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1428,"prompt_tokens":1042,"completion_tokens":386,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":658,"tokens_out":386,"duration_ms":3948,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T06:02:05.551975+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same IDM on data with a random subset of action labels shuffled, use it to generate the synthetic triplets, then train the FDM on those triplets and compare GPT-4o scores on AURORA-BENCH; if the gains persist with shuffled labels, the reported bootstrap effect is not due to accurate inverse-dynamics knowledge.","supporting_citations":[{"cited_title":"Learning Action and Reasoning-Centric Image Editing from Videos and Simulations","cited_arxiv_id":null,"evidence_quote":"Supplies the AURORA ground-truth trajectory triplets that train both the IDM and FDM, and the AURORA-BENCH subsets used for all evaluation."},{"cited_title":"Scaling egocentric vision: The EPIC-KITCHENS dataset","cited_arxiv_id":null,"evidence_quote":"Adds egocentric action clips from EPIC-Kitchen as additional IDM training data."},{"cited_title":"Moments in time dataset: one million videos for event understanding","cited_arxiv_id":null,"evidence_quote":"Source of unlabeled human-action video clips from which IDM-annotated synthetic trajectory triplets are extracted."},{"cited_title":"Video PreTraining (VPT) : Learning to act by watching unlabeled online videos","cited_arxiv_id":null,"evidence_quote":"Establishes the prior of training an inverse dynamics model on unlabeled video to synthesize trajectories, which this work ports to real-world frames with language actions."},{"cited_title":"InstructPix2Pix : Learning to follow image editing instructions","cited_arxiv_id":null,"evidence_quote":"PixInstruct serves as a specialized image-editing baseline the bootstrapped FDM is compared against."}],"review_version":1}