{"id":"eaa5c811-664a-4731-9117-d7688e8074a6","arxiv_id":"2504.19834","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AnimateAnywhere learns to animate the background of human videos directly from human pose sequences, using an epipolar-constrained 3D attention mechanism, and reports state-of-the-art results without camera trajectories.","lead":"A new method, AnimateAnywhere, makes the background move naturally in human animation videos by inferring background motion from the person's pose sequence, without needing camera trajectory input. It adds a learned module and an epipolar-geometry training loss to a diffusion model, and reports better quality than prior methods on two benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never demonstrates that the generated background motion is actually driven by the pose sequence; ablations report only global metrics, and Section V concedes that pose-to-background mapping is underdetermined. A controlled pose-scrambling test is needed to support the central claim.","rationale":"The reader's weakest_assumption identifies the underdetermination of pose-to-background mapping, which is a real concern and is acknowledged by the paper. My stress-test focuses on a related but distinct gap: even for pose sequences that are not ambiguous, the paper does not provide controlled evidence that the generated background motion is causally attributed to the pose sequence. The reported ablations and quantitative comparisons use global metrics on entire frames, so a model that produces pleasant but pose-independent background motion could still show the improvements reported in Tables I and III. This missing causal check is more directly load-bearing than the ambiguity itself, because without it the central mechanism (BML predicting background motion from pose) is supported only by qualitative examples and global scores. The proposed test is feasible with the authors' stated setup: it requires no new training, only inference-time manipulation of the input pose sequence and background motion estimation on the outputs. If the test shows pose-dependence, the reader's conditional accept can proceed; if it does not, the paper's headline claim should be downgraded. I therefore keep the reader's CONDITIONAL verdict unchanged rather than escalating to REJECT, because the concern is addressable by a single controlled experiment and the paper's other evidence (ablations, qualitative results, cross-human/background transfer) remains promising.","tokens_in":14554,"tokens_out":5980,"duration_ms":74428,"concrete_test":"Fix a reference human image and a reference background image from one test video. Run AnimateAnywhere under three conditions: (i) the original pose sequence; (ii) a temporally shuffled version of that same pose sequence (randomly permuting frame order); (iii) a pose sequence taken from a different video. For each generation, estimate background motion in the generated frames using DROID-SLAM or dense point trajectories restricted to background regions, and measure the resulting camera trajectory (translation and rotation over time). If the estimated background motion is statistically indistinguishable across conditions (e.g., similar translation/rotation magnitude and direction distributions), then BML is not actually using pose content, and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the Background Motion Learner (BML) 'learns background motions from human pose sequences' and that this produces backgrounds synchronized with human motion without camera trajectories at inference. For this claim to hold, the pose sequence must be causally informative about background motion in the generated video. The paper provides no direct evidence of this causality. The ablations in Table III compare 'Ours w/o BML' with 'Ours' and show global PSNR/FID/FVD improvements, but these metrics aggregate over the entire frame and do not isolate whether background motion tracks the pose. The improvement could come from extra model capacity, from the reference background conditioning, or from learning a generic camera-motion prior that is only weakly related to the specific pose. The Limitations section (Section V) explicitly acknowledges that 'a single human pose sequence may correspond to multiple plausible background motions' (skateboard zoom-in versus forward motion), which is precisely a case where any pose-driven prediction is not grounded. Without a controlled experiment that varies only the pose sequence and checks whether background motion changes accordingly, the core mechanism asserted in the abstract and Section III-B remains unverified. If BML ignores pose content and emits a generic background motion, the method reproduces a prior over camera moves rather than the claimed pose-conditioned synchronization, and the central contribution of the paper is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AnimateAnywhere, a human image animation framework built on CogVideoX that generates videos with dynamic backgrounds while being controlled only by a human pose sequence and separate reference human and background images. The two main technical contributions are a Background Motion Learner (BML), a LoRA-based module that injects pose-derived features into the denoising DiT to predict background motion, and an adaptive epipolar constraint applied to the 3D attention map, which suppresses low-confidence attention that is geometrically unreasonable according to the camera motion available during training. The method is evaluated on the Humanvid test set and a new Bilibili200 (BL200) dataset, compared against MagicAnimate, AnimateAnyone, Champ, MusePose, DynamiCtrl, and Humanvid, and ablated with respect to the BML, the epipolar loss, the VGG loss, and the attention threshold. The authors report state-of-the-art quantitative results and qualitative examples showing plausible dynamic backgrounds.","tokens_in":14810,"tokens_out":6658,"duration_ms":64770,"significance":"If the central claim is validated, the paper makes a practical contribution by removing the need for camera trajectories or reference videos at inference, which broadens the applicability of human image animation to ordinary users. The architecture is clearly described, and the ablations do isolate the contributions of the BML and the epipolar loss, which is a genuine strength. The idea of learning background motion from pose features and of regularizing 3D attention with an adaptive epipolar mask is reasonable and worthy of publication. However, the paper's headline claim that the BML 'learns background motions from human pose sequences' is not directly supported by the reported evidence, and the paper's own Limitations section concedes a fundamental ambiguity in the pose-to-background mapping. The provided metrics are also based on single runs and a hyperparameter selected on the test set, so the strength of the empirical claims is currently not fully established.","major_comments":[{"comment":"The central claim that the BML learns background motion from pose sequences is not directly supported by the reported experiments. Table III and Table I report only global frame-quality metrics (PSNR, SSIM, LPIPS, FID, FVD) aggregated over the entire video, so the gain from adding BML could come from extra model capacity, from the reference-background conditioning path, or from learning a generic camera-motion prior that is only weakly related to the specific pose content. A controlled experiment is needed: fix the reference human and reference background, vary only the pose sequence (for example, left-to-right walking versus in-place zooming, or a temporally scrambled pose sequence), and measure whether the generated background motion changes accordingly, for instance via optical-flow correlation between the background and the pose direction or via a human study. Section V's skateboard example shows that the pose-to-background mapping is ambiguous even in simple cases. Without such a test, the abstract's assertion that the method 'effectively learns the background motion from human pose sequences' remains unverified.","section":"Section III-B, Table III"},{"comment":"Equation (4) sums the epipolar loss over all pixels (u,v) without an explicit background/human mask, while the surrounding text says the constraint is applied to background pixels. Human body motion is not rigid under a global camera transformation, so applying the epipolar penalty to human-region queries would be geometrically incorrect and could distort the generated figure. The authors should state explicitly how the background region is identified in the loss, and either include the mask in Eq. (4) or define Aij(u,v) and Omega_ij(u,v) as restricted to background tokens. As written, the loss is ambiguous and cannot be reproduced from the paper.","section":"Section III-C, Eq. (4)"},{"comment":"The threshold delta is tuned on the Humanvid test set: Table IV reports the metrics for different percentiles and the authors then adopt the value that gives the best FVD and PSNR on that same set, and the final Table I numbers are produced with this selected value. This makes the headline comparison optimistic; a validation split or a sensitivity analysis that does not reuse the test set for model selection should be reported. Relatedly, all quantitative results are single runs without error bars or significance tests, and several differences from the nearest competitor are small (for example, PSNR 18.07 versus 17.91 in Table I), so run-to-run variance could change the ranking of methods.","section":"Section IV-A, Table IV"}],"minor_comments":[{"comment":"The caption of Table II does not state that the numbers are computed on the subset of Humanvid videos with significant background motion, even though the text introduces such a subset; the table should be labeled accordingly to avoid confusion with Table I.","section":"Table II"},{"comment":"In Table II, the row 'Ours with Lepipolar((1-M) cap (A<delta))' reports SSIM = 0.5831, which is identical to the 'Ours' row in Table I while all other metrics differ; the authors should confirm whether this is a copying error and clarify the evaluation subset.","section":"Table II"},{"comment":"The notation around the attention map is inconsistent: Aij(u,v) is sometimes a scalar and sometimes a map over the j-th frame, and the dimensions of Omega_ij(u,v) are not defined. The figures would be much easier to read if the masks and maps were defined with explicit tensor shapes.","section":"Section III-C and Fig. 2"},{"comment":"The VGG perceptual loss in Eq. (6) uses a hand-region mask m, but the range and resolution of m and the choice of VGG features are not specified, which makes this term non-reproducible.","section":"Section III-D, Eq. (6)"},{"comment":"DynaScene [18], which is discussed in Section II-A as a method for realistic dynamic backgrounds, is not included in the quantitative or qualitative comparisons; if it requires camera poses at test time, this should be stated explicitly.","section":"Related Work and Experiments"},{"comment":"There are minor typos and formatting issues, including 'V AE' for 'VAE', inconsistent capitalization of 'AnimateAnyone', and the use of 'eq. (4)' without the equation being labeled consistently; a careful proofread would improve the presentation.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main unresolved issue is evidentiary rather than architectural: the authors should add the controlled pose-variation experiment that directly tests whether background motion is driven by the pose sequence, and they should address the test-set tuning and lack of error bars in the quantitative evaluation. I would be willing to support acceptance after those points are addressed. The paper is within the scope of IEEE Transactions on Multimedia and the proposed framework is a useful step in human image animation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a legitimate step forward for human image animation. It identifies a real user-facing gap—dynamic backgrounds without camera trajectories—and builds a clean pipeline around it. The Background Motion Learner is a sensible idea, and the adaptive epipolar constraint is a reasonable way to impose geometric consistency without over-constraining the attention.\n\nWhat's actually new: the task formulation. Prior work either keeps backgrounds static or requires camera trajectories/tracking points from a reference video. AnimateAnywhere instead conditions background motion on the pose sequence alone, and it supports separate reference human and background images, which is convenient for customization. The architecture is standard diffusion stuff—CogVideoX, ControlNet, ReferenceNet, LoRA—but the combination is coherent and the ablations isolate the contributions of BML and the epipolar loss. The paper also has an honest Limitations section that admits pose-to-background mapping is underdetermined (the skateboard example).\n\nSoft spots: evaluation is single-run without error bars, and the threshold δ is selected on the evaluation set. That's a minor concern for a systems paper but worth noting. More substantive: the paper does not directly demonstrate that the generated background motion is actually driven by the pose sequence. A controlled pose-scrambling test—same reference images, different pose sequences, checking if background motion changes—would nail the central claim. The current ablations show global metric improvements when BML is added, but those could in principle come from a generic camera-motion prior. Given that the training loss uses ground-truth videos and the pose is the only varying signal about camera motion, I suspect the model is genuinely using pose, but the evidence is circumstantial. The limitations section's admission of ambiguity makes this even more relevant.\n\nAlso, code and data are not yet available, and the datasets are partly self-collected, so reproducing the numbers is not possible right now.\n\nOverall: this is a solid paper with an addressable evaluation gap. It deserves a serious referee; I'd suggest the reviewers ask for the pose-scrambling analysis and error bars. I would cite it if I worked in this area, and it's a good candidate for a reading group, especially for the task formulation.","headline":"A solid, well-engineered human animation paper that tackles a real gap—dynamic backgrounds without camera trajectories—but its central causality claim needs a direct test.","tokens_in":15364,"tokens_out":2622,"would_cite":true,"duration_ms":26831,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that in human image animation, the background should move in harmony with the person, and that the human pose sequence alone carries enough information to infer plausible camera and background motion.","keywords":["human image animation","background motion","video diffusion model","pose-driven animation","epipolar constraint","camera-free animation","LoRA","3D attention"],"falsifier":"Take a fixed pose sequence of a person who is stationary in world space while the camera zooms in, and the same pose sequence with the person moving forward while the camera zooms out so the on-screen pose is identical; run AnimateAnywhere on both. If the generated background motion is the same or chosen arbitrarily, or if on a held-out set with known camera trajectories the predicted background motion disagrees with the recorded camera path for these ambiguous cases, the central claim that pose sequences determine background motion fails in exactly the regime the paper concedes.","tokens_in":1671,"feed_emoji":"🎬","tokens_out":2204,"duration_ms":67380,"temperature":0.7,"pith_summary":"This paper argues that in human image animation, the background should not stay frozen, and that the human pose sequence alone carries enough information to infer plausible camera and background motion. It introduces AnimateAnywhere, built on a pretrained image-to-video diffusion model, which animates a reference human and a reference background from a pose sequence without needing any camera trajectory at inference. The central bet is that relative changes in a person's size and position across frames reveal zoom, pan, and other camera motion, and a learned Background Motion Learner turns pose features into background motion. If this bet holds, ordinary users could generate videos with lively, coherent backgrounds from a single image pair plus pose data, matching what previously required camera estimation or manual trajectories.","feed_headline":"Human pose alone predicts background motion in animated video","feed_subtitle":"AnimateAnywhere learns zoom, pan, and camera shifts from pose data alone, with no camera trajectory needed at inference.","key_machinery":"The two load-bearing components are the Background Motion Learner (BML) and the adaptive epipolar loss. BML is a LoRA module added to each DiT block of the base video diffusion model; it receives the sum of the denoising features and the human pose features extracted by a ControlNet, so the network can shift the background according to the implied camera motion. The epipolar loss uses the fundamental matrix from labeled camera poses during training to build, for each pixel, an epipolar line in every other frame; the model suppresses attention outside that line only when the vanilla attention score is below a 30th-percentile threshold ($(1-M_{ij}(u,v)) \\cap (A_{ij}(u,v) < \\delta)$), penalizing geometrically unreasonable cross-frame attention without destroying learned correlations. The combined effect is that geometry is injected as a soft training signal rather than as a control input at inference.","core_discovery":"The paper claims that background motion can be learned directly from human pose sequences. Concretely, AnimateAnywhere takes a reference human image, a reference background image, and a pose sequence, and produces a video in which the background moves coherently with the human. The Background Motion Learner, a LoRA-style adapter inserted into each DiT block of the base model, reads combined pose and denoising features and predicts background motion end-to-end, avoiding the error accumulation of explicitly estimating camera trajectories. To keep that motion geometrically plausible, training applies an adaptive epipolar constraint to the 3D attention map: attention from a background pixel to another frame is suppressed only where it falls outside the epipolar line and is already low-confidence, leaving the model's own learned correlations intact. At inference no epipolar masks or camera poses are required. The paper reports that this outperforms prior human-animation methods on the Humanvid test set and a wild-collected BL200 set, including methods that use ground-truth camera poses at test time.","pith_inferences":["An implication the paper leaves implicit is that the BML's reliability should degrade precisely on pose sequences whose camera motion is underdetermined, such as a stationary person with a zoom; a text condition or a learned distribution over camera hypotheses would be a natural extension.","The same 'pose implies camera' principle could transfer to non-human subjects, where object size and position changes relative to the frame are equally informative about camera motion.","The low-confidence-gated epipolar mask is a general trick: any video-diffusion model with 3D attention could use it during training whenever camera labels are available, even for tasks outside human animation.","A testable question is whether BML actually learns geometry or merely copies common background motions from the training distribution; ablating with a pose sequence whose implied camera motion contradicts the training prior would settle this."],"forward_implications":["Users can animate a character with a dynamic background using only a reference human, a reference background, and a pose sequence; no camera trajectory or reference video is needed at inference.","Because the reference human and reference background are injected separately, the same pose sequence can be reused with arbitrary human and background combinations, enabling cross-character and cross-scene customization.","Reported quantitative gains over static-background baselines and over a camera-pose-based method that needs test-time camera poses suggest the pose-to-background shortcut is learnable from camera-annotated video.","The adaptive epipolar masking avoids the artifacts of hard suppression, making geometric consistency compatible with the free-form 3D attention of a diffusion transformer.","When a pose sequence is ambiguous, the method still commits to a plausible background, and the paper names text descriptions as the next step for disambiguation."],"supporting_citations":[{"why":"Supplies the training and test videos with labeled camera pose sequences from which the epipolar masks are derived and on which the method is benchmarked.","marker":"[4]"},{"why":"The pretrained image-to-video diffusion model that AnimateAnywhere builds on; its frozen DiT and 3D full attention provide the backbone and the attention maps being constrained.","marker":"[9]"},{"why":"Defines the LoRA adaptation pattern used to implement the Background Motion Learner without tuning the base model.","marker":"[10]"},{"why":"Introduces epipolar constraints across all frames for camera-controlled image-to-video generation, the formulation the adaptive epipolar mask builds on.","marker":"[39]"},{"why":"ControlNet architecture used to inject the human pose sequence into the denoising network.","marker":"[41]"},{"why":"Represents the alternative paradigm of extracting background motion from reference videos, which the paper argues is impractical compared with predicting motion from pose alone.","marker":"[5]"}],"fun_headline_variants":["Pose alone drives background motion in human videos","No camera trajectory: pose predicts background moves","Background animation from human pose, no camera needed","Animate backgrounds with pose, skip camera planning"],"cache_read_input_tokens":17536,"weakest_assumption_plain":"The paper assumes that a human pose sequence alone reveals the background or camera motion, so that pose changes like apparent size and horizontal position translate into zoom and pan; this mapping is genuinely ambiguous in cases such as a skateboarder whose forward motion is indistinguishable from a camera zoom, and if such ambiguities dominate real usage, the learned background motion will be arbitrary or implausible.","fun_headline_variants_meta":{"raw":{"variants":["Pose alone drives background motion in human videos","No camera trajectory: pose predicts background moves","Background animation from human pose, no camera needed","Animate backgrounds with pose, skip camera planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1525,"prompt_tokens":985,"completion_tokens":540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":482}},"tokens_in":601,"tokens_out":540,"duration_ms":5049,"temperature":1.0,"reasoning_tokens":482,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:41:39.092798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed pose sequence of a person who is stationary in world space while the camera zooms in, and the same pose sequence with the person moving forward while the camera zooms out so the on-screen pose is identical; run AnimateAnywhere on both. If the generated background motion is the same or chosen arbitrarily, or if on a held-out set with known camera trajectories the predicted background motion disagrees with the recorded camera path for these ambiguous cases, the central claim that pose sequences determine background motion fails in exactly the regime the paper concedes.","supporting_citations":[],"review_version":1}