{"id":"ce082a65-978b-4fd4-ae80-2b0abeea72c6","arxiv_id":"2411.10836","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-stage video generation system that converts camera, drag, and reference-video controls into unified optical flows, and adds a frequency-domain stabilizer to reduce flicker.","lead":"AnimateAnything turns camera paths, text prompts, and user drags into a single set of motion flows that guide a video diffusion model, producing stable animations from one reference image. It could make video generation easier to control for filmmakers and animators by unifying several control methods in one system.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unified optical-flow premise is untested for large-parallax/dynamic scenes; training data gap and lack of depth/disocclusion modeling make the central control claim conditional.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: optical flow is a sufficient intermediate representation for both global camera motion and local object motion without explicit depth, occlusion, or 3D structure modeling. The paper itself labels this a 'speculation' in Sec. 1, and the subsequent validation does not directly test it. Table 1 evaluates camera-trajectory control on Real10K, which is overwhelmingly static and indoor, and the training strategy in Sec. 3.5 acknowledges a gap: the camera-aware Stage 1 is trained on static-scene datasets plus fixed-camera dynamic videos, not on videos with simultaneous large camera motion and dynamic objects. This is an internal limitation, not merely a disagreement with an external consensus. If the FGM cannot produce physically correct flows in disoccluded or large-parallax regions, the unified-flow guidance will mislead Stage 2, invalidating the 'precise and consistent' claim for the very large-motion regime highlighted in the abstract. The proposed concrete test isolates Stage 1 and uses known depth and camera poses to compute ground-truth flow, directly measuring whether the learned conversion is geometrically sound. A failure there would not disprove the method for all inputs, but it would force a strict qualification of the central claim. Since the reader already gave a CONDITIONAL verdict, my analysis reinforces that position without moving it to reject or accept.","tokens_in":13435,"tokens_out":6667,"duration_ms":68406,"concrete_test":"Select a set of synthetic scenes (e.g., Kubric) or real scenes with known camera poses and depth maps, containing large parallax and moving objects. For each, run Stage 1 with the camera trajectory only, obtaining predicted dense optical flow F_pred. Compute the geometrically correct flow F_gt by projecting the depth under the given camera motion, treating pixels that become visible after occlusion as invalid (no correspondence). Report the average endpoint error (EPE) on valid pixels and the fraction of predicted flow vectors that fall in invalid (disoccluded) pixels with no counterpart. If EPE increases sharply with parallax magnitude or if the model assigns confident flow vectors in disoccluded regions, the unified-flow representation is not reliable for camera control in such scenes, and the central claim must be restricted to small-parallax or static settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that all control signals, including camera trajectories, can be reduced to a frame-by-frame optical flow that guides consistent generation. The load-bearing step is the learned conversion in Sec. 3.2: the Camera Reference Model (CRM) injects camera trajectory into the Flow Generation Model (FGM), which outputs dense flow. For a moving camera in a 3D scene, the true 2D motion field is depth-dependent and is undefined in disocclusion regions. The FGM must therefore infer scene depth and hallucinate newly visible content. The paper provides no direct evaluation of this inference. Table 1 reports camera-trajectory accuracy of final videos on Real10K, an indoor static-scene dataset (Sec. 3.5), and the qualitative figures do not isolate large-parallax cases. Moreover, Sec. 3.5 admits that the camera-control training data are 'primarily indoor or static scene datasets' and that dynamic video datasets lack camera poses; the authors therefore train on OpenVid videos with fixed cameras. Consequently, the FGM has never been trained on the combination of large camera motion and dynamic objects, which is precisely the regime the abstract promises ('large-scale motion'). The unified-flow representation could still work if the model learns implicit depth priors, but the paper presents no evidence for this, and its own speculation language in Sec. 1 signals that the premise is unverified. If the premise fails for large-parallax or heavily occluded scenes, the flow guidance will be unreliable in disoccluded regions, and the claimed 'precise and consistent' control will not hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AnimateAnything, a two-stage image-to-video generation framework that unifies multiple control signals (camera trajectories, user drag annotations, reference videos) into a common frame-by-frame optical flow representation. In Stage 1, a Flow Generation Model (FGM) converts explicit signals (e.g., arrow-based sparse flows) into dense optical flow, while a Camera Reference Model (CRM) implicitly injects camera trajectory information via Plücker embeddings and reference attention. In Stage 2, the generated optical flow is compressed by a 3D VAE and injected into a CogVideoX-based diffusion transformer through a ViT cross-attention block. A frequency stabilization (FS) module modifies attention weights via FFT/IFFT to reduce flicker. The authors report quantitative gains over prior methods on camera trajectory alignment (Table 1), video quality (Table 2), and consistency (Table 3), with qualitative results for drag and reference-video controls, plus an ablation study (Table 4).","tokens_in":13749,"tokens_out":4001,"duration_ms":47083,"significance":"If the central claim holds, the paper offers a genuinely unified control interface for I2V generation, which would be practically useful for film and VR applications. The main strengths are the architectural idea of converting heterogeneous control signals into a single optical-flow prior, the two-stage independent training scheme that decouples flow generation from video generation, and the explicit attempt to address large-motion flicker through frequency-domain feature modulation. The paper also provides fairly extensive comparisons with recent baselines on camera trajectory metrics and standard video-quality metrics. However, the evaluation is uneven across the three claimed control modalities, the training data for camera control do not cover the dynamic-camera-plus-dynamic-object regime promised in the abstract, and the FS module is described too imprecisely to be reproduced. The quantitative claims also lack error bars or significance tests, and the Stage 2 evaluation appears to use the same datasets that were used for training. These issues make the current evidence conditional rather than conclusive.","major_comments":[{"comment":"The training/evaluation gap undermines the large-scale-motion claim. The paper itself states (Sec. 3.5) that the camera-pose datasets Real10K and DL3DV10K are 'primarily indoor or static scene datasets' and that dynamic OpenVid videos are selected only when the camera is roughly fixed. Table 1 evaluates camera trajectory accuracy only on Real10K. Therefore, the FGM has never been trained on the combination of substantial camera motion and independent object motion, which is precisely the setting the abstract promises ('large-scale motion'). Please add an evaluation on dynamic scenes with known camera poses (e.g., synthetic benchmarks with ground-truth pose and object motion, or real datasets with SfM on rigid background plus moving foreground), or at a minimum a set of qualitative large-parallax examples with quantitative flow accuracy, and state the failure modes explicitly.","section":"§3.5 / Table 1"},{"comment":"The user-drag and reference-video control modalities, which are half of the claimed control types, are supported only by qualitative figures. No quantitative metric is reported for arrow annotations (e.g., point-trajectory accuracy, endpoint error, drag success rate) or for reference-video motion transfer (e.g., temporal flow alignment, identity preservation, motion fidelity). Since the paper's headline claim is 'precise and consistent' control across all conditions, these load-bearing modalities need quantitative evaluation against the same baselines that are used in the qualitative comparisons.","section":"§4.2 / Figs. 6–8"},{"comment":"The frequency stabilization module is underspecified. The text says an FFT is applied to 'each weight matrix' and that the spectral features are multiplied by a learnable weight before inverse FFT, but it does not specify along which axis the FFT is taken (e.g., the temporal dimension of the feature map, the embedding dimension, or the weight-matrix rows). Without this information, the method cannot be reproduced, and the claimed connection between modifying attention weight spectra and temporal coherence is not evident. Please provide the exact tensor shapes and operations, and add an ablation that isolates the effect of FS on flicker-specific metrics (e.g., temporal consistency, frame-wise LPIPS variance) rather than only on FVD.","section":"§3.4 / Fig. 4"},{"comment":"The Stage 2 evaluation appears to use the training distribution. The paper trains on WebVid10M and OpenVid (Sec. 3.5) and then reports video-quality and consistency metrics on WebVid and OpenVid (Tables 2 and 3). If the train and test splits are not explicitly separated, the FID/FVD gains may largely reflect overfitting to the training set rather than genuine generation quality. Please clarify the exact split procedure, or evaluate on a held-out set of videos not used in training, and report the overlap between training and evaluation data.","section":"§3.5 / §4.1"},{"comment":"The quantitative tables report single-run results without error bars, confidence intervals, or significance tests. Given the variability of video generation models across seeds and the small margins in some metrics (e.g., Table 2 LPIPS 0.135 vs 0.152 for Pyramid-Flow on WebVid, and Table 3 MoS values all near 0.99), the claimed improvements could be within run-to-run noise. Please report results over at least three random seeds with mean and standard deviation, or perform a paired significance test where appropriate.","section":"Tables 1–4"}],"minor_comments":[{"comment":"The text refers to 'Tab. 5' for the camera trajectory results, but the table is numbered Table 1 in the manuscript; please fix all cross-references.","section":"§4.2"},{"comment":"Equation (4) uses a hat symbol over d without defining it; the following line defines it only implicitly. Please clarify the notation for the normalized direction vector, and fix the typo 'image latens' in the same section.","section":"§3.2"},{"comment":"Equation (6) omits the standard softmax scaling factor 1/sqrt(d) for dot-product attention; if the implementation uses scaling, indicate it explicitly.","section":"§3.3"},{"comment":"The first two rows of the ablation are labeled 'Camera embedding' and 'ControlNet-Like' but these are alternative camera-conditioning architectures, not ablations of the optical-flow unification itself. Clarify that they are alternative designs and specify that the training data and schedule are identical across all rows.","section":"§4.3 / Table 4"},{"comment":"References [62] and [63] are duplicates of the same paper (Stereo Magnification); merge them and update the citation numbering.","section":"References"},{"comment":"The pipeline figure is very dense; enlarge the subfigure labels and consider separating Stage 1 and Stage 2 into two panels for readability.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid architectural core, but the current evidence is not yet sufficient for acceptance. The most serious issue is the data/evaluation alignment: the camera-control model is trained on static or fixed-camera scenes and evaluated on Real10K, while Stage 2 is trained and evaluated on the same WebVid/OpenVid collections. A reader cannot currently tell whether the reported improvements reflect genuine generalization. In addition, the two non-camera control modalities (drags and reference videos) are evaluated only qualitatively, and the FS module description lacks the precision needed for reproduction. These are fixable with additional experiments and rewriting, so I recommend major revision rather than rejection. I would also suggest that the authors make their code or a demo available to strengthen reproducibility claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know upfront. First, the paper does something genuinely new: it converts camera trajectories, drag annotations, and reference videos all into a single dense optical flow and then guides video generation with that flow. That is a clean, useful unification, and the camera-trajectory numbers on Real10K are better than CameraCtrl and MotionCtrl. Second, the central premise is not actually tested in the regime the abstract promises. The flow generation model is trained on indoor/static scenes and fixed-camera dynamic videos, so it has never seen large camera motion together with moving objects. The quantitative camera eval is also on Real10K, which is static. So \"precise control under large-scale motion\" is a claim about a regime the training data and evaluation never enter.\n\nWhat the paper does well: The two-stage architecture is thoughtfully designed, with explicit injection for pixel-level signals and implicit injection for camera trajectories through a reference-network mechanism. The ablation in Table 4 shows the unified-flow representation beats both a camera-embedding baseline and a ControlNet-style baseline, and the frequency stabilization module appears to help. The authors also honestly describe the data limitation in Sec. 3.5, which is to their credit. The writing is clear, and the qualitative figures look plausible.\n\nThe soft spots, in order of importance. The stress-test note is correct: camera-induced optical flow is depth-dependent and undefined in disocclusion regions, so the FGM must infer depth and hallucinate occluded content. There is no direct evaluation of that inference, and no large-parallax dynamic test cases. This is the load-bearing assumption, and it is unverified. Second, the drag and reference-video claims are supported only by qualitative comparison; no user study or quantitative metric. Third, there are no error bars or significance tests anywhere. Fourth, the frequency stabilization module is described at a high level (FFT, multiply by a parameterized weight, inverse FFT) but the implementation details matter and are not spelled out. Fifth, no code, so the work is not currently reproducible.\n\nThis is a solid systems paper with a promising idea, and it deserves peer review. But the reviewers should ask for code, error bars, and a direct stress test on dynamic scenes with large camera motion. As it stands, the paper is conditionally acceptable at best.","headline":"A genuinely unified flow-based controller, but the central large-motion premise is untested; worth serious review with requested revisions.","tokens_in":14283,"tokens_out":2570,"would_cite":true,"duration_ms":29246,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AnimateAnything turns camera trajectories, drags, and reference-video motion into one optical-flow representation that drives consistent video generation.","keywords":["controllable video generation","image-to-video","optical flow","camera trajectory control","drag-based animation","frequency stabilization","diffusion model","motion transfer"],"falsifier":"Run the model on shots where the camera moves sideways past objects at different depths while a foreground object moves in the opposite direction, and inspect the edges where background becomes newly visible. If the flat flow map cannot represent those newly revealed regions, the generated video should show ghosting, flicker, or wrong motion at those boundaries; that would refute the claim that a single frame-by-frame flow field is enough.","tokens_in":13236,"feed_emoji":"🎬","tokens_out":7195,"duration_ms":71033,"temperature":0.7,"pith_summary":"The paper argues that a video generator does not need separate controllers for each user input. Instead, it claims, every visual control—camera path, arrow-based drags, reference-video motion—can be converted into frame-by-frame optical flow, and that flow can then be the single motion condition for synthesis. A two-stage pipeline first produces the unified dense flow and then feeds it to a diffusion video generator, with a frequency-domain stabilization module to suppress flicker under large motion. If the claim holds, one framework can combine camera and object control in the same shot without conflict or careful tuning.","feed_headline":"Camera paths, drags, and reference videos all become one optical flow","feed_subtitle":"Every control signal becomes frame-by-frame flow, then a frequency-domain stabilizer cuts flicker.","key_machinery":"The load-bearing object is optical flow as a common motion 'language' for all control signals. Explicit signals (arrows, reference videos) become sparse or dense flows directly; camera trajectories, which cannot be written as pixel flows by hand, enter through the Camera Reference Model using Plücker embeddings, so the Flow Generation Model can synthesize a dense flow consistent with both signal types. Stage two then uses a flow VAE/encoder to supply key–value features to the generation transformer, and the frequency stabilization module applies adaptive FFT-based filtering inside attention to align features in time.","core_discovery":"AnimateAnything's central claim is that local object motion and global camera motion can be unified into a single frame-by-frame optical-flow representation, and that this representation—not the raw control signal—should condition video synthesis. Stage one explicitly converts drag annotations into sparse flows and camera trajectories into Plücker embeddings, which a reference network uses to guide the Flow Generation Model in synthesizing a dense unified flow. Stage two encodes that flow and injects it as key–value features into a transformer-based video diffusion model, while an FFT-based attention adaptor stabilizes large motions by modifying spectral features. The paper reports that this design yields better camera-trajectory alignment and video quality than camera-only, drag-only, and flow-based baselines.","pith_inferences":["If 2D optical flow is genuinely sufficient, new control types need only a converter into flow, making the framework extensible; but the paper's training data limitation (pose-rich datasets are mostly indoor or static) leaves dynamic outdoor scenes with strong parallax largely untested.","The frequency-stabilized attention could likely be reused in other transformer-based video generators as a drop-in module, independent of this pipeline's flow conditioning.","A natural stress test: generate flows for conflicting signals—object moving opposite to the camera—and check whether the learned model resolves the conflict or requires explicit occlusion reasoning."],"forward_implications":["A user can combine camera motion and object drags in one shot without the two signals fighting, because both share the same flow space.","The same second stage can transfer motion from a reference video, including face and lip motion, without skeleton or keypoint extraction.","Any new control modality that can be converted to sparse optical flow—audio, object landmarks, extracted video—can be plugged into the flow-generation stage.","Large camera or object motion is less likely to produce flicker because the frequency stabilization module enforces temporal consistency in the spectral domain.","The two stages can be trained on different datasets (static-scene pose data for flow, dynamic video for generation), reducing the need for large pose-annotated dynamic video."],"supporting_citations":[{"why":"Closest two-stage method that also uses optical flow as intermediate motion representation; this paper extends it to camera and drag conditions.","marker":"[38]"},{"why":"Provides the sparse-optical-flow generation scheme and flow-warping guidance that the explicit-injection path builds on.","marker":"[31]"},{"why":"Supplies the Plücker-embedding representation for camera trajectories and the trajectory encoder design.","marker":"[17]"},{"why":"Camera-control baseline whose trajectory-conditioning approach is compared and extended.","marker":"[45]"},{"why":"Contributes the reference-attention and progressive condition-injection design used by the Camera Reference Model.","marker":"[22]"},{"why":"Video diffusion backbone (CogVideoX) whose DiT blocks and expert adaptive layernorm the second stage inherits.","marker":"[51]"},{"why":"Unimatch is used to extract high-quality optical flow from training videos as ground truth for flow-generation training.","marker":"[50]"},{"why":"Classical latent diffusion model serving as the base architecture of the Flow Generation Model.","marker":"[34]"}],"fun_headline_variants":["One optical flow to rule all controls","Camera, drags, prompts: all become frame-by-frame flow","Every control becomes optical flow, then flicker is fixed","Drag, camera, text: unified into one motion flow","All control signals feed a single flow, then a stabilizer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline stands on the premise that a flat, frame-by-frame map of where each pixel moves can capture both camera motion and object motion without separately modeling occlusions, newly revealed background, parallax, or three-dimensional scene structure.","fun_headline_variants_meta":{"raw":{"variants":["One optical flow to rule all controls","Camera, drags, prompts: all become frame-by-frame flow","Every control becomes optical flow, then flicker is fixed","Drag, camera, text: unified into one motion flow","All control signals feed a single flow, then a stabilizer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001186,"raw_usage":{"total_tokens":4834,"prompt_tokens":821,"completion_tokens":4013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":3933}},"tokens_in":437,"tokens_out":4013,"duration_ms":24675,"temperature":1.0,"reasoning_tokens":3933,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:14:20.832213+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the model on shots where the camera moves sideways past objects at different depths while a foreground object moves in the opposite direction, and inspect the edges where background becomes newly visible. If the flat flow map cannot represent those newly revealed regions, the generated video should show ghosting, flicker, or wrong motion at those boundaries; that would refute the claim that a single frame-by-frame flow field is enough.","supporting_citations":[{"cited_title":"Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling","cited_arxiv_id":null,"evidence_quote":"Closest two-stage method that also uses optical flow as intermediate motion representation; this paper extends it to camera and drag conditions."},{"cited_title":"Motionctrl: A unified and flexible motion controller for video generation","cited_arxiv_id":null,"evidence_quote":"Camera-control baseline whose trajectory-conditioning approach is compared and extended."},{"cited_title":"Unifying flow, stereo and depth estimation","cited_arxiv_id":null,"evidence_quote":"Unimatch is used to extract high-quality optical flow from training videos as ground truth for flow-generation training."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Classical latent diffusion model serving as the base architecture of the Flow Generation Model."}],"review_version":1}