{"id":"1aaff55a-4c01-43a0-85ca-e4eac14e4141","arxiv_id":"2412.11785","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"InterDyn fine-tunes Stable Video Diffusion with a ControlNet-style branch so that a hand-mask control signal drives plausible, temporally consistent videos of object interactions.","lead":"This paper presents InterDyn, a video generation framework that takes one still image and a moving hand mask as a control signal, then produces a short video of the resulting object dynamics. It tests whether large video diffusion models can act as implicit physics simulators that propagate forces and generate counterfactual futures without explicit 3D reconstruction.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The quantitative Motion Fidelity metric does not actually test physical interaction, and the CLEVRER probes are only qualitative; the implicit-physics claim needs an identity-aware counterfactual check.","rationale":"The paper makes two kinds of contributions: a concrete control mechanism (a ControlNet-style branch on frozen SVD) and an interpretive claim that the model is an implicit physics simulator. The method part is well specified, the DexYCB ablations are informative, and the reported SSV2 gains over CosHand are large. I am not disputing that InterDyn produces smoother, more temporally consistent videos than the baselines. The load-bearing issue is whether the evidence supports the physics interpretation. The CLEVRER section is explicitly qualitative; it shows selected videos in which collisions appear to propagate. The quantitative SSV2 section uses Motion Fidelity as the only dynamics-related number, but Eqs. (1)-(2) define a bidirectional best-match cosine similarity between velocity tracklets. That construction is exactly what a metric would look like if the goal were to reward 'some motion in roughly the right direction,' not 'the same object undergoing the same causal event.' Combined with the fact that evaluation is on the same SSV2 distribution used for fine-tuning, the results are consistent with a weaker hypothesis: InterDyn learns action-class-specific motion priors, and the control signal selects which class-typical motion to produce. The paper's own limitation section supports this: complex non-translational interactions and underrepresented classes have notably lower motion fidelity (Table S3). This is not an ad hominem or a consensus dispute; it is a request to make the physics claim falsifiable. An identity-aware, counterfactual CLEVRER evaluation would settle the question. If that test fails, the correct claim would be the weaker one: InterDyn improves controllable video generation on interaction-heavy datasets. If it passes, the implicit-simulator claim is strengthened. I therefore do not move the reader's verdict; CONDITIONAL remains appropriate.","tokens_in":1060,"tokens_out":992,"duration_ms":81237,"concrete_test":"On the CLEVRER test set, use the dataset's collision-event annotations to select videos with one controlled collider and at least one uncontrolled object. For each video, generate two InterDyn videos: one with the collider's control signal as in the ground truth, and one with that collider removed. Then compute, per uncontrolled object and with object identities matched (no bidirectional max-matching over tracklets), the mean post-collision endpoint error and velocity cosine between the generated trajectory and the ground-truth trajectory of that same object. Also measure whether the generated trajectory difference between the two conditions matches the ground-truth counterfactual difference. Compare InterDyn against a no-interaction baseline that keeps uncontrolled objects at their initial motion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"InterDyn's central claim is that a large video diffusion model acts as an implicit physics simulator, evidenced by force propagation and counterfactual dynamics among uncontrolled objects. The only quantitative dynamics-related metric, Motion Fidelity (Eqs. 1-2), cannot support this claim: it uses bidirectional maximum matching over tracklets, so a generated tracklet can be matched to the GT tracklet of any object, not necessarily the same object. The per-frame correlation is cosine similarity of 2D displacements, ignoring speed and absolute position. Thus a video with the hand moving correctly and unrelated objects drifting in the same broad direction as the GT object can score highly without any causal collision or force propagation. On SSV2, the model is both fine-tuned and evaluated on the same action-class distribution, so high scores may simply reflect action-class motion priors rather than an internal causal model. The CLEVRER results (Fig. 4) are qualitative, with no quantitative accuracy, no error bars, no identity-aware tracking, and no comparison against a no-interaction baseline or a control model without the frozen SVD prior. The paper's own limitation section (Tab. S3) shows low motion fidelity precisely on complex interactive classes like spinning or poking, which is consistent with the weaker hypothesis that the model reproduces typical per-class kinematics rather than physics. This is an internal evaluation-validity concern, not a field-consensus disagreement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InterDyn, a framework for controllable interactive video generation. It extends Stable Video Diffusion (SVD) with a trainable ControlNet-like branch that takes an input image and a control signal (e.g., a sequence of binary hand masks) and generates a video of consequent object dynamics. The authors fine-tune the model on CLEVRER and Something-Something-v2 (SSV2), compare against CosHand, Seer, and DynamiCrafter, and argue that large video diffusion models can act as implicit physics simulators by demonstrating force propagation and counterfactual futures on CLEVRER and realistic human-object interactions on SSV2. The paper reports quantitative improvements in image quality, FVD, and a Motion Fidelity metric.","tokens_in":19555,"tokens_out":5813,"duration_ms":49662,"significance":"If the central claim holds, this work is a step toward using video diffusion models as generalizable dynamics engines without explicit physics simulation or 3D reconstruction. The architecture is simple and elegant: the SVD prior is frozen, the control branch is lightweight, and the method does not require fitted physical parameters or an explicit simulator. The paper also provides useful ablations on control signal type and robustness to noisy masks. However, the significance is currently limited by the evaluation: the only quantitative dynamics-related metric is not identity-aware, and the physics probes on CLEVRER are purely qualitative. With stronger evaluation, the contribution could be valuable to the video generation and intuitive physics communities.","major_comments":[{"comment":"The Motion Fidelity metric defined in Eqs. (1)-(2) does not measure physical interaction fidelity. The bidirectional maximum matching in Eq. (1) allows a generated tracklet to be matched to a ground-truth tracklet of any object rather than the same physical object, and the correlation in Eq. (2) is a cosine similarity of per-frame 2D displacements, which is invariant to speed and absolute position. A video with a correctly moving hand and uncontrolled objects drifting in a similar broad direction as the ground-truth object can therefore score highly without any causal collision or force propagation. Since this is the only quantitative dynamics-related metric in the paper, it cannot support the implicit-physics claim. Please adopt an identity-aware matching (e.g., by initial mask overlap) and report absolute trajectory errors and confidence intervals.","section":"Section 4.2"},{"comment":"The CLEVRER experiments are purely qualitative. There is no quantitative evaluation of collision outcomes, no identity-aware tracking of uncontrolled objects, no error bars, and no baseline comparing against a no-interaction condition or a model without the frozen SVD prior. The single counterfactual example (Fig. 4b) is illustrative but does not establish distribution-level counterfactual consistency. Because these probes are the core evidence for 'force propagation' and 'counterfactual dynamics,' the paper should include quantitative metrics (e.g., object-wise trajectory divergence between with-control and without-control generations, collision accuracy) and a control condition with no driving signal.","section":"Section 4.3"},{"comment":"The quantitative comparisons do not report error bars, confidence intervals, or repeated-seed statistics, and the claimed improvements are relative to the best baseline per metric; additionally, DynamiCrafter was not trained on SSV2, making cross-method comparison less direct. The class-wise breakdown in Table S3 shows that Motion Fidelity is lowest precisely for complex interactive classes such as spinning, poking, and burying, which is consistent with the weaker interpretation that the model reproduces per-class kinematic priors rather than causal physics. Please report variance across seeds and discuss the class-wise pattern in the main text.","section":"Section 4.4, Table 1"},{"comment":"The claim that InterDyn 'generalizes to unseen objects' is not systematically evaluated. All quantitative results are on the SSV2 validation split, which comes from the same action-class distribution used for fine-tuning; the few zero-shot examples are qualitative. Please add a held-out action-class or held-out object-category evaluation, or explicitly scope the generalization claim in the abstract.","section":"Abstract and Section 4.4"}],"minor_comments":[{"comment":"The caption contains a typo: '/searcZoom in for details.' should read 'Zoom in for details.'","section":"Figure 4 caption"},{"comment":"The reported KVD values for InterDyn are negative (-0.131 and -0.450); since KVD is an unbiased kernel distance estimate that can be negative in small samples, please clarify the estimator or provide standard errors.","section":"Table S4"},{"comment":"The introduction calls CosHand 'the previous SOTA' without noting that CosHand solves a different task (single-frame state transition); clarify the task relationship to avoid overstating the comparison.","section":"Introduction"},{"comment":"The control signal is introduced as c ∈ R^{N×H×W×3} but the experiments use binary masks; please clarify how the three channels are used for mask inputs.","section":"Section 3"},{"comment":"The paper would benefit from a brief discussion of the relationship between the 'motion ID' conditioning in SVD and the learned control branch; currently the choice of motion ID 40 is only justified by alignment with the frozen prior.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a clean architectural contribution with strong qualitative results, but the evaluation does not yet substantiate the 'implicit physics simulator' claim. The central quantitative metric is not identity-aware and the CLEVRER probes are anecdotal. I recommend a major revision focused on evaluation validity rather than broad changes to the method. The paper is within the journal's scope and likely to be of interest to the video generation community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take this seriously, but not as a proven physics engine. InterDyn is a clean, well-engineered extension of SVD with a frozen backbone and a trainable ControlNet-style temporal branch, fine-tuned on interaction data. The move from single-frame state transitions (CosHand) to temporally continuous video generation with a driving motion is real and useful. The SSV2 numbers are strong—big FVD/LPIPS gains over the CosHand variants—and the robust handling of noisy SAM2 masks plus the qualitative counterfactual demo on CLEVRER are nice. The paper is also admirably candid: the limitations section reports per-class motion fidelity and lists the failure modes.\n\nThe soft spots are exactly where the stress-test note lands. The Motion Fidelity metric (Eqs. 1-2) uses bidirectional maximum matching over tracklets without identity. A generated tracklet can be matched to any ground-truth tracklet, so a video that drifts in the right direction can score well even without object-specific causal interaction. Small objects are excluded from the metric when they have fewer than 100 trackable points, which removes many of the cases where real physics would be demonstrated. The CLEVRER force-propagation and counterfactual results are qualitative only; there is no no-interaction baseline, no identity-aware tracking, and no control model without the frozen SVD prior. And the SSV2 evaluation is in-distribution: the model is fine-tuned and evaluated on the same action classes, so high fidelity could reflect class motion priors rather than a learned physical model. The limitation table reinforces this concern—the bottom classes are spinning, poking, folding, collapsing, exactly the interactive cases where physics understanding would be visible.\n\nNone of this is fatal to the paper's core contribution. As a controllable video generation method, it works and improves on prior baselines. What is not supported is the abstract's stronger claim about acting as an implicit physics simulator. To nail that, the authors should add: an identity-aware counterfactual test (e.g., object X moves if and only if hit), a no-interaction control, and a metric that actually checks object-to-object causal influence. They should also report error bars, since all quantitative comparisons are point estimates.\n\nWho is this for? Anyone working on controllable video synthesis, action-conditioned generation, or using video diffusion as a world model. It deserves a serious referee, but the referee should push for the physics claim to be sharpened or made more modest. I would recommend conditional accept with requests for the counterfactual evaluation and a tighter framing.","headline":"Solid controllable-video paper whose implicit-physics claim is ahead of the evidence, but the gap is fixable with identity-aware counterfactual checks.","tokens_in":20054,"tokens_out":2321,"would_cite":true,"duration_ms":23582,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that large video generation models can act as implicit physics simulators, predicting full videos of interactive object dynamics from one image and a control signal that specifies only the driving motion.","keywords":["interactive dynamics","video diffusion models","implicit physics simulation","controllable video generation","human-object interaction","counterfactual futures","force propagation","motion fidelity"],"falsifier":"A quantitative failure test would be: on CLEVRER-style scenes with annotated collision events, measure whether the generated post-collision trajectories of uncontrolled objects match the momentum-conservation predictions, and whether counterfactual futures diverge at the moment of the controlled intervention. A model expressing dataset statistics rather than physical understanding should fail under varied masses, shapes, and collision angles.","tokens_in":19120,"feed_emoji":"🎬","tokens_out":6404,"duration_ms":52765,"temperature":0.7,"pith_summary":"This paper claims that large video generation models carry enough learned knowledge of everyday physics to act as implicit simulators: given a single starting frame and a control signal that specifies the motion of a driving hand or object, a model can generate a full video of the resulting interaction, including motion of objects that are never controlled. InterDyn is the proposed framework: a frozen Stable Video Diffusion backbone with a trainable temporal ControlNet-style branch that ingests a sequence of binary masks as the driving signal. The authors argue this beats the state-transition paradigm, in which a model only predicts one future image, because continuous dynamics such as force propagation through multiple objects, counterfactual futures, and post-contact rolling or pouring require temporal coherence that single-frame methods lack. On collision probes with CLEVRER and on the Something-Something-v2 hand-object dataset, InterDyn produces temporally consistent videos and outperforms the baselines on video metrics and motion fidelity. If the claim holds, interactive video generation becomes a way to probe and exploit the physical knowledge latent in generative video models.","feed_headline":"One image plus hand masks generates realistic interaction videos","feed_subtitle":"InterDyn plays out collisions, post-contact motion, and counterfactual futures with no 3D reconstruction or physics simulator.","key_machinery":"The central mechanism is a temporal ControlNet-style conditioning branch attached to a frozen Stable Video Diffusion (SVD) U-Net. The control signal is a per-frame, pixel-aligned map, usually a sequence of binary hand masks, encoded by a small CNN, processed by the branch's interleaved convolution, spatial, and temporal blocks, and injected into the frozen decoder through zero-initialized skip connections. This design lets InterDyn interpret noisy or coarse control sequences, such as masks from SAM2 with motion blur, while keeping the dynamics prior intact. The key design choice is that only the driving entity is conditioned; uncontrolled objects' trajectories are entirely generated, which is what turns video generation into a physics probe.","core_discovery":"InterDyn's central claim is that a large video diffusion model pre-trained on web-scale video already possesses a workable implicit model of interactive dynamics, and that this capacity can be unlocked by conditioning the generation process on the motion of a driving entity. The paper extends Stable Video Diffusion with a trainable control branch in the style of ControlNet, fed by a pixel-wise control signal (in most experiments a sequence of binary hand masks), while the main U-Net stays frozen to preserve the learned dynamics prior. The model conditions only the driving agent; all other objects are unconditioned, so the video generator must produce their consequential motion on its own. The CLEVRER experiments show generated force propagation across uncontrolled objects and counterfactually different futures for the same input frame under different control signals, while the Something-Something-v2 experiments show realistic hand-object interactions such as pouring, squeezing, dropping, and stacking. The authors' conclusion is that the video generator acts as both neural renderer and implicit physics engine, with no 3D reconstruction and no explicit simulation anywhere in the loop.","pith_inferences":["Beyond the paper: the force-propagation and counterfactual results suggest the model could be used as a proposal generator for physical prediction tasks, but those results should be stress-tested with out-of-distribution objects, lighting, and camera angles to separate genuine physics from dataset priors.","Beyond the paper: because InterDyn generates entire videos, its output could be used to bootstrap training of video-based world models or manipulation policies, treating generated trajectories as cheap synthetic experience, an application the paper does not pursue.","Beyond the paper: a natural testable extension is to quantify, rather than visualize, the implicit physics by measuring whether generated trajectories conserve momentum or energy; the current evidence is qualitative, and a trajectory-level metric would settle the question."],"forward_implications":["Because the control signal is a generic pixel-aligned map, the same framework can extend beyond hand masks to skeletons, meshes, or depth maps, letting a user steer scene dynamics without retraining the backbone.","Continuous, post-contact dynamics, such as objects rolling, water levels rising, and deformable objects compressing, become generated as video rather than as discrete future states, changing how interactive prediction can be evaluated and used.","The frozen-backbone design preserves the dynamics prior of the base video model, so fine-tuning on a new interaction dataset mostly teaches the control branch, reducing the data needed to adapt to new scenarios.","Counterfactual generation on a single input image offers a way to sample diverse physically plausible futures, which could serve as training data or test cases for reasoning about interventions."],"supporting_citations":[{"why":"Backbone video diffusion model whose learned dynamics prior InterDyn preserves by freezing; supplies the generation base.","marker":"[5]"},{"why":"State-transition baseline that InterDyn outdoes; cannot generate post-contact dynamics.","marker":"[70]"},{"why":"Collision dataset with annotated masks and events; used to probe force propagation and counterfactual futures.","marker":"[97]"},{"why":"Something-Something-v2: real-world hand-object interaction dataset used for fine-tuning and evaluation.","marker":"[21]"},{"why":"Source of the ControlNet conditioning-branch design adapted for temporal control signals.","marker":"[100]"},{"why":"Segment Anything 2: produces the binary hand masks used as InterDyn's control signal on SSV2.","marker":"[65]"},{"why":"Defines the motion fidelity metric based on point-tracking trajectories used to evaluate generated motion.","marker":"[93]"},{"why":"CoTracker3 point tracker computes the trajectories on which the motion fidelity metric relies.","marker":"[41]"},{"why":"Language-instructed video prediction baseline compared in the SSV2 experiments.","marker":"[23]"},{"why":"Language-instructed video generation baseline compared in the SSV2 experiments.","marker":"[89]"}],"fun_headline_variants":["Diffusion model gets hand masks, invents realistic physics","Control signal unlocks hidden physics in video diffusion","Interactive dynamics from one image, no simulator needed","Video generator plays out collisions with no explicit physics","Causal video generation: control one entity, predict the rest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the generated motion reflects learned physical understanding rather than memorized action-class patterns, since the model is fine-tuned and evaluated on the same hand-action video collection and its physics probes are qualitative.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion model gets hand masks, invents realistic physics","Control signal unlocks hidden physics in video diffusion","Interactive dynamics from one image, no simulator needed","Video generator plays out collisions with no explicit physics","Causal video generation: control one entity, predict the rest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1541,"prompt_tokens":972,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":494}},"tokens_in":588,"tokens_out":569,"duration_ms":5688,"temperature":1.0,"reasoning_tokens":494,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:35:03.861553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A quantitative failure test would be: on CLEVRER-style scenes with annotated collision events, measure whether the generated post-collision trajectories of uncontrolled objects match the momentum-conservation predictions, and whether counterfactual futures diverge at the moment of the controlled intervention. A model expressing dataset statistics rather than physical understanding should fail under varied masses, shapes, and collision angles.","supporting_citations":[{"cited_title":"Co- Tracker3: Simpler and better point tracking by pseudo- labelling real videos","cited_arxiv_id":null,"evidence_quote":"CoTracker3 point tracker computes the trajectories on which the motion fidelity metric relies."},{"cited_title":"DynamiCrafter: Animating open-domain images with video diffusion priors","cited_arxiv_id":null,"evidence_quote":"Language-instructed video generation baseline compared in the SSV2 experiments."},{"cited_title":"Controlling the world by sleight of hand","cited_arxiv_id":null,"evidence_quote":"State-transition baseline that InterDyn outdoes; cannot generate post-contact dynamics."},{"cited_title":"Tenenbaum","cited_arxiv_id":null,"evidence_quote":"Collision dataset with annotated masks and events; used to probe force propagation and counterfactual futures."},{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Source of the ControlNet conditioning-branch design adapted for temporal control signals."},{"cited_title":"Space-time diffusion features for zero-shot text-driven motion transfer","cited_arxiv_id":null,"evidence_quote":"Defines the motion fidelity metric based on point-tracking trajectories used to evaluate generated motion."}],"review_version":1}