{"id":"0c7b9414-c194-439d-8c7a-c5344af9c770","arxiv_id":"2411.17697","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"StableAnimator uses a video diffusion model with face-embedding adapters and per-step latent optimization to generate pose-driven videos that preserve the reference person's identity end-to-end.","lead":"StableAnimator turns a single photo of a person into a video that follows a given pose sequence, while keeping the same face and identity throughout. It combines a video diffusion model with face-embedding modules and an inference-time optimization step, avoiding the blurry or face-swapped results of previous methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The HJB derivation in Sec. 3.2 does not rigorously connect the inner optimization loop to an optimal-control solution; Eq. 12's drift coefficient (1−t)∇log p does not match the reverse-SDE noise term, so the claimed HJB foundation is unsupported.","rationale":"The reader identified the HJB integration claim as the weakest assumption, and I agree. The central claim has two parts: an empirical SOTA result and a novel HJB-based inference mechanism. The empirical result could in principle survive even if the HJB theory is wrong, but the paper's abstract and contribution list explicitly present the HJB method as a key novelty. If Eq. 12 is not structurally equivalent to the score-based reverse SDE, the paper's claim of integrating HJB optimal control into denoising is unsupported; the method reduces to a heuristic test-time face-similarity gradient step. This is a correctness risk in the paper's own terms, not a disagreement with community consensus. The concrete re-derivation test would settle the issue analytically, without needing code or data. I therefore recommend keeping the reader's CONDITIONAL verdict: the empirical contributions may be valid, but the theoretical foundation requires verification or revision.","tokens_in":18110,"tokens_out":11495,"duration_ms":100377,"concrete_test":"Independently re-derive Eq. 12 from Eq. 4 by writing the Pontryagin necessary conditions for the stochastic control problem with drift control and additive Brownian noise, solving for c*, and comparing the resulting SDE with the reverse-time SDE of EDM under the time change t=1−t'. Specifically, check whether the coefficient of ∇log p equals the squared noise coefficient g^2(t) in the same SDE. If the two coefficients are not equal (e.g., (1−t) vs 1), then Eq. 12 is not a valid score-based reverse SDE and the HJB-integration claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's abstract and introduction present the Hamilton-Jacobi-Bellman (HJB) equation-based face optimization as a core novel contribution. Sec. 3.2 attempts to justify this by deriving, from the control problem in Eq. 4, the SDE dX_t = (1−t)∇log p(X_t)dt + dw_t (Eq. 12) and asserting it is 'structurally the same' as the score-based reverse SDE of Song et al. [42]. This equivalence is not established. In the standard reverse-time SDE, the drift coefficient of ∇log p equals g^2(t), the squared diffusion coefficient. If the Brownian term in Eq. 12 has coefficient 1, then g^2(t)=1, so the drift of ∇log p should be 1·∇log p, not (1−t)∇log p; alternatively, choosing g^2(t)=(1−t) would require the noise term to be √(1−t)dw_t, not dw_t. Thus Eq. 12 is internally inconsistent with the reverse SDE it claims to match. The derivation also relies on an asserted 'certainty equivalence' step (adding dw_t to the deterministic optimal control solution) and on treating the Tweedie estimate of X_1 as equal to x_pred, the output of the inner Adam loop on an ArcFace loss. Algorithm 2 never solves Eq. 4; it performs 10 gradient steps minimising a face-similarity loss. The HJB theory therefore appears to be a post-hoc wrapper around a heuristic latent-space optimisation. This matters because the paper's stated novelty includes 'a novel HJB equation-based face optimization method'; if the theoretical foundation is invalid, that central contribution is unsupported, even if the empirical procedure still improves CSIM.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"StableAnimator proposes an end-to-end identity-preserving human image animation framework built on Stable Video Diffusion, with three main contributions: a global content-aware Face Encoder that refines face embeddings using reference-image context, a distribution-aware ID Adapter that aligns face and image cross-attention features before temporal modeling, and an inference-time face optimization that the authors formulate as a Hamilton-Jacobi-Bellman (HJB) equation-based control problem. The model is trained with a face-mask-weighted reconstruction loss and evaluated on the TikTok dataset and an additional Unseen100 benchmark, reporting state-of-the-art CSIM and FVD values. The paper includes extensive ablations showing that each component contributes positively and that the inference-time optimization also improves other backbones.","tokens_in":18538,"tokens_out":6653,"duration_ms":60223,"significance":"If the empirical claims hold, StableAnimator addresses a practical problem—identity consistency in pose-driven human animation—without relying on external face-swapping post-processing, and its distribution-aware adapter is a plausible mechanism for injecting identity features into video diffusion models. The paper provides a detailed algorithm description, comprehensive ablations, and an honest discussion of limitations and ethical concerns. However, the central theoretical contribution, the claimed integration of HJB-based optimal control with diffusion denoising, is not rigorously established; the derivation in Sec. 3.2 is internally inconsistent with the reverse SDE it invokes, and Algorithm 2 is a heuristic latent-space gradient descent rather than a solution of the stated control problem. The empirical results are valuable regardless of the HJB framing, but the current presentation overstates the theoretical novelty.","major_comments":[{"comment":"The claimed equivalence between the derived SDE and the score-based reverse SDE is not established and is internally inconsistent. In Eq. (12), the Brownian term is dw_t, so the diffusion coefficient g^2 would be 1, in which case the drift coefficient of ∇log p in the standard reverse SDE (Eq. 16 in Sec. A.2) should be g^2 (up to sign), not (1−t). If one instead intends g^2=(1−t), the noise term must be √(1−t) dw_t, not dw_t. The step labeled 'certainty equivalence' (adding Brownian motion to the deterministic optimal-control solution) is asserted without a proof or a precise citation, so the derivation does not demonstrate that solving Eq. (4) corresponds to diffusion denoising. This undermines the claim in Sec. 3.2 that 'solving the HJB equation can be integrated with the diffusion denoising process.'","section":"Sec. 3.2, Eqs. (9)-(12)"},{"comment":"Algorithm 2 does not solve the optimal-control problem in Eq. (4). The control variable c_t is never defined in the algorithm, and the optimization performed is 10 Adam steps on an ArcFace cosine loss applied to a trainable copy of the diffusion model's prediction xpred. The statement 'xpred in Algorithm 2 is equivalent to x1' is unjustified: x1 in Eq. (4) is the terminal target of the control problem, whereas xpred after optimization is a face-similarity-optimized latent that is neither the reference latent nor a Tweedie estimate of the data. No Hamiltonian, terminal condition, or trajectory constraint appears in the algorithm. Therefore, the paper does not prove that the proposed optimization is HJB-based; at best it is an inference-time face-similarity guidance heuristic.","section":"Sec. 3.2, Algorithm 1 (and Algorithm 2 in Sec. A.6)"},{"comment":"The reported comparisons on the TikTok dataset are based on numbers taken from the competitors' papers, and no error bars, confidence intervals, or repeated-seed statistics are provided for any metric. Since the headline claim is a large improvement (e.g., CSIM 0.831 vs. 0.360 for ControlNeXt), the lack of uncertainty quantification and the inconsistent evaluation protocols across cited results make the exact magnitude of the improvement difficult to verify. The authors should report variance across multiple runs or at least state the number of evaluation videos and run the main competitors under a unified protocol, as they did for Unseen100.","section":"Table 1 and Sec. 4.2"}],"minor_comments":[{"comment":"The time variable t is overloaded: it denotes both the normalized time in the control problem (running from 0 to 1) and the EDM noise level (with σ(t)=t). This makes equations such as Eq. (11) confusing; please use distinct symbols for the control-time and the diffusion time.","section":"Sec. 3.2, notation"},{"comment":"In Hamilton's equations, dγ/dt should equal −∂H*/∂X, not ∂H*/∂X as written; the result is the same here because ∂H*/∂X=0, but the sign error should be corrected.","section":"Eq. (7)"},{"comment":"The cross-backbone experiments are informative, but the hyperparameters of the proposed optimization (Adam learning rate η and the number of inner steps k=10) are never varied or analyzed. A short sensitivity study would clarify how robust the reported gains are to these choices.","section":"Sec. 4.3, Table 5"},{"comment":"Two consecutive sentences are duplicated verbatim ('The reason is that applying the ArcFace ID loss requires...'). This should be fixed editorially.","section":"Supplementary A.8"}],"recommendation":"major_revision","confidential_remarks":"The empirical contributions—the Face Encoder, the distribution-aware ID Adapter, and the inference-time face-similarity optimization—appear solid and are supported by careful ablations. The main obstacle is the HJB derivation in Sec. 3.2, which is mathematically unsound and currently forms the basis of a stated contribution. I believe this is fixable: the authors could either provide a rigorous optimal-control derivation (matching coefficients with the reverse SDE and justifying the certainty-equivalence step) or, more realistically, reframe the inference-time method as a heuristic latent-space face-guidance procedure and remove the HJB claims from the contribution list. Given the strength of the empirical results, I recommend major revision rather than rejection, provided the authors are willing to correct or remove the unsupported theoretical framing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take: the empirical system is the real contribution; the HJB wrapper is not. Treat it as that.\n\nWhat's actually new: end-to-end identity-preserving human animation without post-processing. The design—global content-aware face encoder, distribution-aware ID adapter with mean/variance alignment, and inference-time face-similarity latent optimization—is coherent, and each component is ablated. The CSIM jump (0.831 vs 0.360 on TikTok) is large enough to matter, and the cross-backbone ablation shows the adapter and optimizer transfer. The note in A.8 about why they avoid ArcFace ID loss in training shows good judgment about compute/memory trade-offs.\n\nSoft spots, in order. The HJB derivation in Sec 3.2 does not work as a derivation. Eq. 12 has the wrong coefficient on the score term: with unit Brownian motion, the drift should be 1·∇log p, not (1−t)·∇log p. The 'certainty equivalence' step is asserted, and Algorithm 2 is ten Adam steps on a face-similarity loss, not a solution of the control problem in Eq. 4. A referee should ask them to either correct the derivation or honestly call it a heuristic latent-space refinement. This does not invalidate the empirical method, but it invalidates the claim that solving the HJB equation is being integrated into the denoising process.\n\nEmpirical concerns: no error bars or significance tests; Unseen100 is self-collected and not public; baselines on TikTok are taken from their papers, which is standard but fragile. CSIM is sensitive to face-crop details, and there is no code or data release. These are typical for the subfield, but they matter more when the headline is a 47% CSIM gain.\n\nBottom line: the paper is a competent systems contribution with a substantial empirical result. The HJB section should be reworked or reframed; the adapter and the inference-time optimization are strong enough to publish even as heuristics.","headline":"A strong, honest empirical systems contribution whose HJB theory section does not survive scrutiny; treat it as a heuristic latent optimization and judge it on the numbers.","tokens_in":19060,"tokens_out":1668,"would_cite":false,"duration_ms":16109,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StableAnimator is an end-to-end video diffusion framework that animates a reference person through pose sequences while keeping the face recognizable, and reports beating post-processing-based systems on identity and video-fidelity metrics.","keywords":["human image animation","identity preservation","video diffusion model","pose-guided animation","Hamilton-Jacobi-Bellman optimization","distribution-aware ID adapter","face embedding alignment","end-to-end animation"],"falsifier":"Re-derive the claimed equality with the diffusion noise coefficient $g^2$ retained: Eq. (12) as written omits $g^2$ and flips a sign relative to the standard reverse SDE, so a direct symbolic check of whether the optimizer's update is a solution of Eq. (4) would settle the theoretical claim. Empirically, one could run Algorithm 2 with the ArcFace gradient direction reversed or randomly perturbed; if identity scores stay high, the face-optimization loop is not what carries the reported identity preservation.","tokens_in":17903,"feed_emoji":"🎬","tokens_out":8563,"duration_ms":73157,"temperature":0.7,"pith_summary":"The paper asks whether identity consistency in human image animation can be achieved inside the diffusion model itself, without the face-swapping tools that current systems bolt on after generation. It answers yes with StableAnimator, a video diffusion pipeline that injects reference face embeddings through a content-aware face encoder and a distribution-aware ID adapter during training, then steers each denoising step with a Hamilton-Jacobi-Bellman (HJB) style optimization during inference. The reported payoff is that facial identity, measured as ArcFace cosine similarity, jumps to 0.831 on the TikTok benchmark versus 0.360 for the strongest open-source baseline, while video fidelity (FVD 140.62) also improves. If the claims hold, reliable pose-driven character animation no longer needs external face-swap or restoration post-processing.","feed_headline":"Face identity score hits 0.831 in pose-driven video, no post-processing","feed_subtitle":"Animates a reference person through any pose sequence while the face stays recognizable: CSIM 0.831.","key_machinery":"Three components carry the argument. The global content-aware Face Encoder is a stack of cross-attention blocks that refines ArcFace embeddings by attending to CLIP image embeddings, giving identity features awareness of the reference layout. The distribution-aware ID Adapter aligns the face cross-attention output to the image cross-attention output by matching per-feature mean and standard deviation before feeding the sum to temporal layers, which prevents the temporal layers from shifting the identity features out of the diffusion latents' domain. The inference-time HJB equation-based optimization treats the predicted clean sample at each denoising step as the control variable of an optimal-control problem whose terminal cost is face similarity to the reference; solving that problem yields a drift $(1-t)\\nabla\\log p(X_t)\\,dt + dw_t$, which the paper argues is structurally the score-based reverse SDE, so the optimized prediction can be folded into the EDM sampler as a steering gradient.","core_discovery":"In the paper's own framing, the discovery is that identity preservation and video fidelity can be treated as a distribution-matching problem rather than a post-processing problem. A global content-aware Face Encoder first lets face embeddings attend to the full-reference image embedding, so the identity signal carries context. A distribution-aware ID Adapter then computes separate cross-attention outputs for face and image embeddings and normalizes the face-attention features into the image-attention feature distribution before temporal layers act, so temporal modeling no longer corrupts the injected identity. At inference, the paper performs ten Adam steps on each denoised latent prediction, minimizing the ArcFace cosine distance to the reference, and derives this update from an HJB optimal-control problem whose solution it shows has the same structure as the score-based diffusion reverse SDE. The result, as reported, is end-to-end animation with no third-party face-swapping and the highest identity and video-fidelity scores among the compared methods.","pith_inferences":["A reasonable reading is that the empirical power of the inference step comes from the direct ArcFace-similarity gradient itself; the HJB derivation gives a principled framing, but the update would likely still improve identity even if the optimal-control equivalence fails.","The distribution-alignment idea is not specific to faces: any video-diffusion task that injects per-frame conditions into a U-Net with temporal layers, for example audio-driven expression or text-driven editing, could use the same mean-variance alignment before temporal processing.","A testable extension: measure identity preservation with a face embedder different from ArcFace, since the inference objective optimizes ArcFace similarity directly and part of the reported gain could be embedder-specific.","The face-inpainting extension the paper mentions in its limitation could be combined with the same HJB steering, turning a stated failure case into a targeted adapter rather than a general post-processing step."],"forward_implications":["If the central claim is right, identity-consistent human animation can be produced in one pass through a video diffusion model, eliminating the quality loss and domain mismatch that face-swapping post-processing introduces.","The distribution-alignment recipe gives a direct way to port image-domain identity adapters into video diffusion: align injected feature distributions with spatial latents before temporal layers.","Because the HJB optimization is training-free and applied at inference, it can be attached to other diffusion backbones, as the paper demonstrates with its cross-backbone ablations.","The reported gains, with CSIM rising from 0.360 to 0.831 on TikTok, indicate that identity-consistency headroom is large and that face-similarity metrics are not saturated by existing baselines.","Long sequences, including animations of over 300 frames and multi-person scenes, are claimed to remain identity-consistent, extending the method beyond short single-person clips."],"supporting_citations":[{"why":"Supplies the pre-trained Stable Video Diffusion backbone and frozen VAE whose latents and weights all components build on.","marker":"[3]"},{"why":"Provides the ArcFace face embeddings and face masks used by the Face Encoder, the training loss, and the inference-time similarity objective.","marker":"[7]"},{"why":"Defines the EDM sampling schedule and the Algorithm 2 structure into which the HJB optimization is inserted.","marker":"[26]"},{"why":"Gives the score-based reverse-SDE form that the HJB solution is claimed to match, grounding the inference-time theoretical justification.","marker":"[42]"},{"why":"Contributes the stochastic optimal-control formulation and certainty-equivalence step used to derive the HJB update.","marker":"[5]"},{"why":"The strongest open-source animation baseline against which identity and video-fidelity results are compared and which relies on post-processing.","marker":"[34]"},{"why":"The third-party face-swapping post-processing tool the paper avoids and uses as an ablation baseline.","marker":"[15]"},{"why":"The TikTok benchmark dataset used for the main quantitative comparison.","marker":"[25]"}],"fun_headline_variants":["End-to-end identity-preserving video animation, done","Face stays you: HJB-optimized video animation","StableAnimator: first ID-preserving video diffusion","No face-swap: video animation that keeps your face","Distribution-aware face identity for video animation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-step latent optimization genuinely solves the Hamilton-Jacobi-Bellman optimal-control problem and that the resulting drift is structurally the same as the diffusion reverse SDE; this equivalence is asserted rather than rigorously proven, and if it fails the method's theoretical foundation loses its basis even if the empirical face-similarity updates still help.","fun_headline_variants_meta":{"raw":{"variants":["End-to-end identity-preserving video animation, done","Face stays you: HJB-optimized video animation","StableAnimator: first ID-preserving video diffusion","No face-swap: video animation that keeps your face","Distribution-aware face identity for video animation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1695,"prompt_tokens":939,"completion_tokens":756,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":680}},"tokens_in":555,"tokens_out":756,"duration_ms":6575,"temperature":1.0,"reasoning_tokens":680,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:49:39.975950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-derive the claimed equality with the diffusion noise coefficient $g^2$ retained: Eq. (12) as written omits $g^2$ and flips a sign relative to the standard reverse SDE, so a direct symbolic check of whether the optimizer's update is a solution of Eq. (4) would settle the theoretical claim. Empirically, one could run Algorithm 2 with the ArcFace gradient direction reversed or randomly perturbed; if identity scores stay high, the face-optimization loop is not what carries the reported identity preservation.","supporting_citations":[{"cited_title":"Arcface: Additive angular margin loss for deep face recognition","cited_arxiv_id":null,"evidence_quote":"Provides the ArcFace face embeddings and face masks used by the Face Encoder, the training loss, and the inference-time similarity objective."},{"cited_title":"Elucidating the design space of diffusion-based generative models","cited_arxiv_id":null,"evidence_quote":"Defines the EDM sampling schedule and the Algorithm 2 structure into which the HJB optimization is inserted."},{"cited_title":"Score-based generative modeling through stochastic differential equa- tions","cited_arxiv_id":null,"evidence_quote":"Gives the score-based reverse-SDE form that the HJB solution is claimed to match, grounding the inference-time theoretical justification."},{"cited_title":"Genera- tive modeling with phase stochastic bridges","cited_arxiv_id":null,"evidence_quote":"Contributes the stochastic optimal-control formulation and certainty-equivalence step used to derive the HJB update."},{"cited_title":"Facefusion","cited_arxiv_id":null,"evidence_quote":"The third-party face-swapping post-processing tool the paper avoids and uses as an ablation baseline."},{"cited_title":"Learning high fidelity depths of dressed humans by watching social media dance videos","cited_arxiv_id":null,"evidence_quote":"The TikTok benchmark dataset used for the main quantitative comparison."}],"review_version":1}