{"id":"0d8e631c-3477-4be6-9c4e-d00b070981ba","arxiv_id":"2507.15064","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"StableAnimator++ combines learnable SVD-guided pose alignment, a distribution-aware ID Adapter, and an HJB-based inference-time face optimizer to preserve identity in human image animation under severe pose misalignment.","lead":"StableAnimator++ is a video diffusion system that animates a still photo to follow a pose sequence while keeping the person's identity, even when the pose is scaled, rotated, or shifted far from the reference. It combines a learnable pose-aligner with identity-preserving face modules and an inference-time face optimizer, and reports strong results on a new high-misalignment benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MisAlign100 applies the same random pose transformations used to train the alignment module, so the reported CSIM/FVD margins may reflect train/test distribution match rather than robustness to natural misalignment.","rationale":"The paper is a coherent systems paper: the ablations are internally consistent, and the HJB/ID-adapter components plausibly contribute. However, the strongest empirical claim is about robustness to pose misalignment. That claim is currently evaluated on a benchmark whose construction mirrors the training-time augmentation of the very module intended to handle misalignment. This is a concrete, checkable threat to external validity, not a disagreement with consensus. The reader's 'weakest assumption' pointed in this direction; I agree and sharpen it to the exact overlap between Eq. 12's augmentation and Sec. IV-B's test construction. A conditional acceptance requiring an independent natural-misalignment evaluation or a held-out augmentation split is appropriate; no rejection is warranted because the ablations and user study support internal consistency. The HJB-derived face optimization is also not rigorously justified, but its contribution is ablated and is secondary to the benchmark-generalization concern.","tokens_in":20598,"tokens_out":5191,"duration_ms":61198,"concrete_test":"Construct an independent natural-misalignment test set (100-200 real videos with inherent body-size/position mismatch and no synthetic transforms) or split MisAlign100 by augmentation range disjoint from training (e.g., evaluate only on scale 1.2-1.8 if training used 0.6-1.4). Run StableAnimator++ and the same retrained competitors with common evaluation code and compute CSIM/FVD. If the margin over Animate-X drops substantially or becomes insignificant on natural/disjoint transforms, the reported superiority is an artifact of train/test distribution overlap, and the paper should be revised to claim gains only under simulated misalignment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that StableAnimator++ outperforms all SOTA under pose misalignment rests on the MisAlign100 benchmark. In Sec. III-A the learnable alignment block is trained by applying 'random scaling, rotation, and translation matrices' to driven poses and minimizing L_align (Eq. 12). In Sec. IV-B MisAlign100 is constructed by taking unseen videos and, again, 'randomly rotated, scaled, and translated' each driven pose. Thus the test-time misalignment distribution is the same family of synthetic transformations used to train the proposed alignment module. Only StableAnimator++ receives a learnable alignment trained on this exact task; competitors use static alignments (or alignment-free pipelines) and are retrained on the authors' private 5K-video dataset, which is not released. This does not make the method internally inconsistent, but it means the headline CSIM margins (0.831 vs. 0.475, 0.802 vs. 0.391 on MisAlign100) may be inflated by matching the training augmentation, not by genuine robustness to natural body-size/position mismatches. FVD and LPIPS gains could partly follow from better-aligned poses reducing motion/appearance conflict. Without code/checkpoints or an independently curated natural-misalignment test set, the generalization claim is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StableAnimator++, a video diffusion framework for human image animation that targets identity preservation under pose misalignment. The system adds a learnable SVD-guided pose-alignment module, a global content-aware face encoder, a distribution-aware ID adapter, and an inference-time face-optimization procedure presented as an approximate Hamilton-Jacobi-Bellman (HJB) solver integrated with the denoising process. The authors report state-of-the-art results on the TikTok dataset and on a self-constructed MisAlign100 benchmark, claiming improved CSIM and FVD without face-swapping post-processing.","tokens_in":20881,"tokens_out":4739,"duration_ms":57096,"significance":"If the empirical claims hold, the work is significant: it addresses a practical gap in human image animation, replaces brittle hand-crafted pose alignment with a learnable module, and eliminates face-swapping post-processing, which often degrades video quality. The modular design is tested across multiple backbones (Stable Video Diffusion, MagicAnimate, CogVideoX-I2V), and the ablations are reasonably thorough. However, the significance is currently constrained by three load-bearing weaknesses: the theoretical link between the HJB derivation and the actual algorithm is not established, the evaluation benchmark is private and its misalignment distribution matches the training augmentation, and no error bars or statistical tests accompany the headline numbers.","major_comments":[{"comment":"The derivation does not produce Algorithm 1. The chain from the HJB equation (15) through Eqs. (16)-(24) concludes with a score-based SDE that is structurally identical to a known reverse-time diffusion SDE, but Algorithm 1 performs an Adam-based optimization of a face-cosine loss on the predicted sample, with no explicit control variable, no denominator (1-t), no Brownian motion term, and no gradient log p(X_t) step. The claimed identification between solving the HJB equation and the diffusion denoising process is therefore not demonstrated by the given equations. Either provide a precise formulation in which the Algorithm 1 objective and Adam updates are the value function and control dynamics of the HJB problem, or reframe the method as a heuristic latent optimization; as written, the theoretical justification does not support the algorithm.","section":"§III-C, Algorithm 1, Eq. (15)-(24)"},{"comment":"The MisAlign100 benchmark is constructed by applying random scaling, rotation, and translation to driven poses, which is the exact family of transformations used to train the alignment module in Eq. (12). Since only StableAnimator++ is trained to predict these synthetic transformations, the large CSIM margins on MisAlign100 (0.802 vs. 0.391 for Animate-X) may be inflated by train/test distribution match rather than by genuine robustness to natural misalignment. The paper should report performance on a test set with naturally occurring body-size/position mismatches (not synthetically augmented), and should compare with a version of the competitors that also receives the same alignment augmentation during training, or at least with their released models on the same natural-misalignment videos.","section":"§III-A (Eq. 12) and §IV-B (MisAlign100)"},{"comment":"The headline claims are presented without error bars, number of runs, or statistical significance. All quantitative comparisons are single point estimates, including FVD (122.47 on TikTok, 384.27 on MisAlign100), and the evaluation protocol is not fully specified (number of frames, resolution, start-frame selection, number of seeds). For a state-of-the-art claim, the authors should report mean±std over multiple assessment runs or per-video confidence intervals, and should make the evaluation code available so the protocol can be reproduced independently.","section":"Table I"},{"comment":"The comparison protocol is not equally controlled across settings. TikTok results for competitors are cited from their original papers, while MisAlign100 results are obtained after retraining all competitors on the authors' private 5K-video dataset. No training code, checkpoints, or hyperparameters for the retrained competitors are released, so readers cannot verify that each competitor received an equal amount of tuning effort. This asymmetry makes the reported margins on MisAlign100 difficult to interpret as a fair comparison, and the absence of released code/checkpoints for the proposed method further limits reproducibility.","section":"§IV-A, §IV-D, and Table I"}],"minor_comments":[{"comment":"The denominator in the scale factor is ambiguous: it should be written as \\sum_i \\|X_d^i\\|^2 (the squared norm of each centered keypoint vector), and the summation index and variable types should be defined consistently with Eq. (2).","section":"Eq. (6)"},{"comment":"The pseudocode calls the Adam optimizer on xop before setting xop.requires_grad = True; in standard frameworks, the requires_grad flag must be set before constructing the optimizer. Also, the loop variable γ_i is set to 0 at the top of the loop and then conditionally overwritten; the initial assignment is misleading even if functionally harmless.","section":"Algorithm 1"},{"comment":"There is a verb-tense error in the conclusion: 'StableAnimator++ then used off-the-shelf models to gain image and face embeddings' should be present tense, and 'StableAnimator introduced a Face Encoder' should refer to StableAnimator++.","section":"Conclusion"},{"comment":"There is inconsistent spelling of ControlNeXt (also written 'ControlNext' in Table IX), and reference [4] lists the venue as 'EECV' instead of 'ECCV'. These should be corrected before publication.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's main selling point is the empirical margin on a private benchmark, but the benchmark construction and the missing theoretical link to HJB both need substantial work. I would strongly encourage the editors to request code/checkpoints release, since the training data and MisAlign100 are both private; without that, the reported margins are difficult to verify. The authors should also be asked to clearly delineate the novelty over their own StableAnimator [36], since the paper currently frames [36] as both the baseline and a preliminary version while claiming 'first' for the new alignment contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid systems paper that extends the authors' own StableAnimator with three genuinely new pieces — a learnable SVD-guided pose alignment that predicts similarity transforms, a distribution-aware ID adapter, and an HJB-flavored inference-time face optimization. The ablations are the strongest part: each component can be removed and performance drops, and the comparisons against ControlNeXt's alignment, FaceFusion, and IP-Adapter are informative. The headline numbers on TikTok (CSIM 0.831) and MisAlign100 (0.802) are plausible for a tuned system with competitors retrained on their data.\n\nConcerns in proportion. First, the HJB section does not actually derive the optimization in Algorithm 1. The text shows a known equivalence between a particular stochastic control problem and the diffusion SDE, then jumps to 'solve HJB to update latents' with Adam. The derivation is decoration; the algorithm is an empirical face-matching gradient step. That is fine as a system trick, but the paper overclaims the theory. Second, MisAlign100 is constructed by applying random scaling, rotation, and translation to poses — exactly the same augmentation family used to train the alignment block. Only this model gets a learnable alignment trained on that distribution; its competitors use static alignment or are retrained on the authors' private 5K-video dataset. The CSIM gap on MisAlign100 (0.802 vs 0.391 for Animate-X) may largely reflect train/test distribution match, not robustness to natural misalignment. That is not fatal — the TikTok results are on a public benchmark — but the 'dramatic misalignment' claim needs an independent test set. Third, no error bars, private training data, and no code or checkpoints are released (project page only). That makes the key numbers unverifiable right now.\n\nThe paper deserves peer review. It is a real systems contribution with honest ablations; the weak spots are fixable (clarify the HJB claim, add error bars, evaluate on a natural-misalignment set, release code). I would send it to a strong venue and require at least the first two fixes. The stress-test concern is on point; I would weight it heavily but not reject on it alone.","headline":"A genuine systems advance in pose alignment for human animation, with honest ablations, but the headline misalignment benchmark matches its own training augmentation and the HJB theory is mostly decoration.","tokens_in":21407,"tokens_out":1843,"would_cite":true,"duration_ms":20974,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StableAnimator++ preserves identity in human image animation even when poses are badly misaligned, and does it without face-swapping post-processing.","keywords":["human image animation","identity preservation","pose misalignment","video diffusion","SVD-guided pose alignment","distribution-aware ID adapter","Hamilton-Jacobi-Bellman face optimization","test-time diffusion guidance"],"falsifier":"A reader can check the central claim by rerunning StableAnimator++ and the strongest competitor on 100 naturally misaligned videos, measuring CSIM and FVD under identical conditions without synthetic transforms. If the CSIM gap over the best competitor collapses or the learnable aligner's contribution disappears when replaced by an oracle alignment that uses ground-truth poses, then the claim that learned SVD-guided alignment drives robustness would be falsified. A second check: the paper reports average Euclidean alignment error 0.105 for the full aligner versus 0.345 for SVD-only on MisAlign100; if forcing the predicted alignment to match an oracle rigid transform does not further improve CSIM, alignment error is not the limiting factor.","tokens_in":20409,"feed_emoji":"🎭","tokens_out":7450,"duration_ms":72909,"temperature":0.7,"pith_summary":"The paper sets out to show that identity-preserving human image animation can be made robust to the routine mismatch between a reference photo and a driving pose sequence—different body size, camera distance, or subject position—without falling back on face-swapping post-processing. Its proposal, StableAnimator++, is a video diffusion model with three training-time interventions and one inference-time intervention: a learnable SVD-guided aligner that predicts rotation, scale, and translation between reference and driving skeletons; a global content-aware Face Encoder; a distribution-aware ID Adapter that reconciles face and image cross-attention features before temporal layers; and an HJB-equation-based face optimization embedded in denoising. The paper reports that this system outperforms all compared methods in identity consistency (CSIM 0.831 on TikTok, 0.802 on MisAlign100) and video fidelity (FVD 122.47 and 384.27) while remaining a single end-to-end model. A sympathetic reader would care because the work targets the exact failure mode that makes current animation tools unusable outside studio-aligned inputs: the face drifts or the body distorts as soon as the reference and the driving video are not already framed alike.","feed_headline":"StableAnimator++ holds identity when poses misalign","feed_subtitle":"Learned SVD alignment and a distribution-aware ID adapter beat all tested animation models on TikTok and MisAlign100.","key_machinery":"The main engine is the learnable pose aligner: a Transformer encoder receives the repeated reference keypoints concatenated with driving keypoints, while a parallel SVD branch computes an intermediate rigid alignment from centered keypoint covariance; a Pose Fusion Block injects the SVD-aligned features via cross-attention, and an MLP predicts $R'$, $S'$, $t'$ so the driven pose becomes $P_d^{\\text{align}} = S'_d (R'_d \\cdot P_d) + t'_d$, trained against ground-truth keypoint distance. The second mechanism is the distribution-aware ID Adapter: after separate cross-attention with image and face embeddings, it enforces $\\frac{z^{\\text{img}}_i - \\mu_{\\text{img}}}{\\sigma_{\\text{img}}} = \\frac{z^{\\text{face}}_i - \\mu_{\\text{face}}}{\\sigma_{\\text{face}}}$ and adds the renormalized face feature to the image feature before temporal layers. The third is the HJB inference module: treating the predicted sample $x_{\\text{pred}}$ as the terminal state of an optimal control problem, the paper derives $c^*_t = \\frac{r(x_1 - X_t)}{1 + r(1 - t)}$, shows the resulting controlled SDE coincides with the diffusion score SDE, and implements the optimal direction by minimizing ArcFace cosine distance between the decoded latent and the reference with Adam over ten steps inside the EDM sampler. The SVD-guided aligner carries the misalignment argument; the ID Adapter carries the temporal-interference argument; the HJB step carries the post-processing-free face argument.","core_discovery":"The paper's central claim is that pose misalignment and face distortion in human image animation are better solved inside the diffusion pipeline than by post-processing. Concretely, it claims that a similarity transformation between reference keypoints and driving keypoints can be learned, with a single SVD solution used as cross-attention guidance to make learning tractable, and that this learned alignment reduces the Euclidean distance between aligned and ground-truth poses from 0.345 (SVD-only) or 0.430 (ControlNeXt's alignment) to 0.105 on the paper's MisAlign100 set. It further claims that identity is preserved by refining ArcFace face embeddings with a content-aware Face Encoder and by enforcing that the cross-attention output distributions from face and image embeddings match before every temporal layer, so that temporal modeling no longer erases identity. At inference, the paper claims that solving the HJB equation yields an optimal control direction for the denoising trajectory, and that ten Adam steps on the predicted sample against ArcFace cosine similarity progressively sharpens the face; this replaces FaceFusion-style post-processing and avoids its domain mismatch artifacts. The quantitative case rests on comparisons where all competitors were retrained on the authors' collected 5K-video dataset and evaluated on both TikTok and the self-built MisAlign100.","pith_inferences":["A direct extension would be to evaluate on naturally misaligned footage rather than randomly transformed poses; the MisAlign100 construction leaves open whether the learned aligner generalizes to real-world scale and position gaps that are not rigid transforms.","The distribution-alignment step is not specific to faces: the same mean-variance matching between two cross-attention streams could protect clothing, object, or style conditioning from temporal-layer interference in other video generators.","The HJB optimizer is a per-sample test-time adaptation on the decoded face; one could ablate step count and learning rate per video to trade sharpness against latency, or apply the same control law to other identity-bearing regions such as hands.","Because every competitor was retrained on the authors' private 5K-video dataset, an open benchmark with released checkpoints would be needed to know whether the reported margins are due to the architecture or to the training data distribution."],"forward_implications":["Animators can animate from a single reference photo even when the driving video is shot from a different distance or framing, without manually rescaling skeletons or running face-swap cleanup.","Removing face-swapping post-processing should eliminate the domain-mismatch artifacts, such as blurry or over-smoothed faces and degraded backgrounds, that the paper attributes to tools like FaceFusion.","The component design transfers: the paper shows that adding the Face Encoder, ID Adapter, and HJB optimization to a different video diffusion backbone preserves identity for text-to-video portrait generation.","Long animations of 500 or more frames remain stable under strong misalignment, which is the operating regime of dance and vlog animation products."],"supporting_citations":[{"why":"Baseline and predecessor StableAnimator; ablating pose alignment and face modules from it produces the w/o Pose Align setting.","marker":"[36]"},{"why":"Backbone Stable Video Diffusion whose pre-trained U-Net weights and latent space the model builds on.","marker":"[52]"},{"why":"EDM sampler whose denoising loop hosts the HJB-based face optimization.","marker":"[65]"},{"why":"DWPose supplies the reference and driving keypoint sequences that the learnable aligner transforms.","marker":"[62]"},{"why":"ArcFace provides the face embeddings used by the Face Encoder and the CSIM similarity metric.","marker":"[32]"},{"why":"ControlNeXt supplies a common pose-alignment baseline and its alignment is used as the w/ControlNeXt ablation.","marker":"[7]"},{"why":"FaceFusion is the face-swapping post-processing tool the paper replaces, and a comparison point in the face-enhancement ablation.","marker":"[28]"},{"why":"IP-Adapter is the face-enhancement baseline in the ID Adapter ablation showing that direct injection degrades video fidelity.","marker":"[24]"}],"fun_headline_variants":["Learnable pose alignment in diffusion preserves identity","SVD-guided poses and HJB face fix beat post-processing","In-pipeline pose alignment and face optimization for video","StableAnimator++: SVD alignment plus distribution-aware ID","Human animation without face distortion via learned alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation rests on a self-built test set (MisAlign100) in which misalignment is simulated by randomly scaling, rotating, and translating poses, and on retraining all competing methods on the authors' private 5K-video dataset; if those simulated transforms or that retraining protocol advantage the proposed modules, the reported margins may not transfer to naturally misaligned real-world footage.","fun_headline_variants_meta":{"raw":{"variants":["Learnable pose alignment in diffusion preserves identity","SVD-guided poses and HJB face fix beat post-processing","In-pipeline pose alignment and face optimization for video","StableAnimator++: SVD alignment plus distribution-aware ID","Human animation without face distortion via learned alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1492,"prompt_tokens":1053,"completion_tokens":439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":669,"tokens_out":439,"duration_ms":4775,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:41:42.078053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader can check the central claim by rerunning StableAnimator++ and the strongest competitor on 100 naturally misaligned videos, measuring CSIM and FVD under identical conditions without synthetic transforms. If the CSIM gap over the best competitor collapses or the learnable aligner's contribution disappears when replaced by an oracle alignment that uses ground-truth poses, then the claim that learned SVD-guided alignment drives robustness would be falsified. A second check: the paper reports average Euclidean alignment error 0.105 for the full aligner versus 0.345 for SVD-only on MisAlign100; if forcing the predicted alignment to match an oracle rigid transform does not further improve CSIM, alignment error is not the limiting factor.","supporting_citations":[{"cited_title":"Elucidating the design space of diffusion-based generative models,","cited_arxiv_id":null,"evidence_quote":"EDM sampler whose denoising loop hosts the HJB-based face optimization."},{"cited_title":"Effective whole-body pose estimation with two-stages distillation,","cited_arxiv_id":null,"evidence_quote":"DWPose supplies the reference and driving keypoint sequences that the learnable aligner transforms."},{"cited_title":"Facefusion,","cited_arxiv_id":null,"evidence_quote":"FaceFusion is the face-swapping post-processing tool the paper replaces, and a comparison point in the face-enhancement ablation."}],"review_version":1}