{"id":"408471fd-2153-456c-aa26-741e26e0866f","arxiv_id":"2412.09296","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An audio-driven portrait animation framework that adds gaze control, prosody-aware head poses, and two-stage lip versus eye motion distillation to a latent-navigation animator.","lead":"GoHD generates talking-head videos from a single portrait and an audio clip, using separate modules for head pose, lip motion, eye motion, and gaze direction. It combines latent-space image animation with a diffusion model and a two-stage expression predictor, aiming to work on arbitrary identities from limited training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lip-sync evidence is partly self-confirming: Stage 1 distills from Wav2Lip, and Table 1 grades with the same SyncNet, so independent validation of mouth-shape correctness is the key missing test.","rationale":"The paper is a serious engineering effort: three modules are clearly specified, losses and ablations are detailed, and a user study is reported. I read the central claim as the combination of modules yielding accurate lip sync, rhythmic poses, natural gaze, and realistic expressions on unseen identities. The single assumption that most determines whether the central claim is supported is the quality of the Wav2Lip-resynced training target. It is load-bearing because Stage 1 of expression prediction is explicitly built on distillation from that expert; if the expert's mouth shapes are only synchronized but not articulatorily correct, the model inherits those errors. The evaluation cannot catch this because Table 1 uses the same SyncNet family that defines the expert. I agree with the reader's weakest_assumption. The check I propose is cost-effective because the 'w/o Distillation' ablation already exists; augmenting it with independent metrics would test the assumption directly. I do not think the concern warrants rejection: the paper acknowledges the SyncNet caveat, provides several losses (including a lip-reading loss) and shows qualitative results. However, it does warrant keeping the verdict CONDITIONAL until the independent check is done. Hence verdict_should_be = UNCHANGED relative to the reader's CONDITIONAL verdict.","tokens_in":18875,"tokens_out":8632,"duration_ms":89919,"concrete_test":"Run the Table 3 'w/o Distillation' variant (Stage 1 trained on original HDTF expression targets instead of Wav2Lip-resynced targets) and the Full model through the complete pipeline on the 220-clip HDTF test set. Evaluate both with (a) the Table 1 SyncNet LSE-C/LSE-D, (b) word error rate from an independently pretrained lip-reading model such as AV-HuBERT applied to the rendered videos, (c) mouth-landmark distance to the original, non-resynchronized HDTF frames, and (d) a forced-choice human rating of mouth-shape naturalness against GT and Wav2Lip references. Report per-clip means with 95% confidence intervals. If Full beats w/o Distillation on (b)-(d) as well as (a), the circularity concern is resolved; if the advantage appears only on (a), the reported lip-sync evidence is an artifact of training and evaluating on the same Wav2Lip/SyncNet model family.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Stage 1 of the expression predictor (Sec. 3.2; Ldistill in Supp. Eq. 18) trains on videos resynchronized by Wav2Lip (Prajwal et al. 2020), and the lip-sync scores in Table 1 (LSE-C/LSE-D) are computed with the same pre-trained SyncNet from that paper. Wav2Lip is itself trained to optimize SyncNet, as the authors state in Sec. 4.2 when explaining Wav2Lip's above-GT scores. The central lip-sync claim is therefore measured with the same model used to create the Stage 1 training targets. The authors' 'closest to ground truth' interpretation does not break the circularity: the relevant question is not whether the model beats Wav2Lip on the metric Wav2Lip was trained on, but whether the distilled mouth shapes are correct outside the SyncNet objective. The concern is not merely formal. The Table 3 ablation is consistent with it: 'w/o Distillation,' trained on the original non-resynchronized HDTF lip motions, reports worse MLD (2.012 vs 1.792) than the Wav2Lip-distilled Full model; if the comparison target is the Wav2Lip-resynced video rather than the original GT, this ordering is expected even if Wav2Lip introduces systematic mouth-shape artifacts. The user study asks about perceived lip synchronization, not phonetic correctness, so it does not independently settle the issue. If Wav2Lip's outputs contain systematic errors, those errors are baked into the model through Ldistill and are invisible to Table 1; the claim of accurate lip sync and realistic expressions on unseen identities then lacks independent support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GoHD proposes a modular audio-driven portrait animation framework. The system predicts 3DMM pose and expression coefficients from audio and animates a single reference image via a latent navigable face animator derived from LIA, with gaze direction as an explicit conditioning signal. Head poses are generated by a conformer-based conditional DDPM with classifier-free guidance on the initial pose. Expressions are produced by a two-stage predictor: Stage 1 distills per-frame lip motions from Wav2Lip-resynchronized videos, and Stage 2 uses an LSTM generator with discriminators to synthesize eye and brow motion features. The paper reports evaluations on HDTF and VoxCeleb, comparing with MakeItTalk, Wav2Lip, Audio2Head, EAMM, and SadTalker, including SyncNet-based lip-sync scores, FID, pose/eye SSIM and variance, identity preservation, ablations, and a user study.","tokens_in":19254,"tokens_out":7091,"duration_ms":67392,"significance":"The paper addresses relevant limitations in audio-driven portrait animation: identity-motion disentanglement, gaze control, prosody-aware head poses, and separation of lip vs. non-lip expressions. The modular design with intermediate 3DMM coefficients is sensible and supports multi-modal driving and motion interpolation. Strengths of the manuscript include the comprehensive ablation of loss terms (Supp. Tables 4-7), the inclusion of a lip-reading loss in Stage 1 training, a user study, and the public code release. If the quantitative claims hold up under independent evaluation, the framework would be a useful contribution. However, the evidence for the headline claims is weakened by evaluative circularity in the lip-sync metric and by the absence of statistical uncertainty estimates.","major_comments":[{"comment":"The lip-sync evaluation is partly self-confirming. Stage 1 of the expression predictor trains on videos resynchronized by Wav2Lip, and the primary lip-sync metrics LSE-C/LSE-D in Table 1 are computed with the same SyncNet model that Wav2Lip itself is trained to optimize. The paper's 'closest to ground truth' interpretation is reasonable as a relative ranking, but it does not establish that the distilled mouth shapes are phonetically correct. The MLD metric in Table 3 is more independent if computed against original ground-truth landmarks, and this should be stated explicitly. To break the circularity, please add an independent evaluation, e.g., word error rate from a lip-reading model not used in training, or a human phonetic-intelligibility task. If the MLD in Table 3 is indeed computed against original HDTF landmarks, that is valuable evidence and should be highlighted as such.","section":"Sec. 4.2, Table 1; Sec. 3.2 Stage 1; Supp. Eq. (18)"},{"comment":"All quantitative results are reported as point estimates without confidence intervals, error bars, or significance tests. Several differences between methods are small (e.g., Table 1 LSE-D: Ours 7.78 vs. SadTalker 7.70 on HDTF; Table 2 SSIMe: Ours 0.915 vs. SadTalker 0.904), so it is unclear if the claimed improvements are statistically reliable. Please report per-video score distributions and paired tests (e.g., bootstrap or Wilcoxon signed-rank) for the main metrics, and add uncertainty measures to the user-study percentages in Fig. 7.","section":"Tables 1-3; Fig. 7"},{"comment":"Gaze orientation is a title-level contribution, but the evaluation is only qualitative. The face animator is trained with a gaze loss using a pre-trained gaze estimator, and gaze manipulation is demonstrated in Fig. 1, yet no quantitative gaze accuracy is reported on the generated videos, nor is gaze control compared with any baseline. Please include a quantitative gaze-angle error evaluation under varying driving gaze directions (pitch/yaw) and on unseen identities to support the claimed gaze-oriented capability.","section":"Sec. 3.3; Sec. 4.3, Fig. 1; Supp. Eq. (14)"}],"minor_comments":[{"comment":"The text says 'The numerical results are reported in Table 6' but the two-stage ablation table in the main paper is Table 3; this cross-reference should be corrected.","section":"Sec. 4.4, Table 3"},{"comment":"Please clarify how the ground-truth eye-motion features f_t used in Stage 1 training are replaced at inference time: is the Stage 2 prediction always used, and is there a train/test mismatch or a scheduled-sampling procedure?","section":"Sec. 3.2, Eq. (5)"},{"comment":"The notation 'Ex0,n,ϵ' is nonstandard; please use the conventional expectation symbol with a subscript, e.g., \\mathbb{E}_{x_0,n,\\epsilon}.","section":"Sec. 3.1, Eq. (2)"},{"comment":"Please specify the noise schedule {β_n} and the total number of diffusion steps N, since these are important for reproducibility.","section":"Sec. 3.1"},{"comment":"Please state the bold/underline convention in the caption itself: currently bold means best and underline means closest to ground truth, but this is only explained in the body text.","section":"Table 1 caption"},{"comment":"The sentence 'our method exhibits better performance than SadTalker' is based primarily on LSE-C, but LSE-D is slightly worse on HDTF (7.78 vs. 7.70); a brief discussion of this trade-off would be helpful.","section":"Sec. 4.2"},{"comment":"The gaze loss from Supp. Eq. (14) is used in the animator training objective but is not referenced in the main text; please add a cross-reference when introducing the gaze loss.","section":"Sec. 3.3"},{"comment":"The lip-reading loss L_read is written with a raw dot product; please define the exact cosine-similarity formula and note whether the embeddings are normalized.","section":"Supp. Eq. (23)"}],"recommendation":"major_revision","confidential_remarks":"The central architecture is plausible and the paper is well-structured, with useful ablations and code release. The main concern is the circularity of the lip-sync evaluation and the lack of statistical rigor; these are fixable with additional experiments and clearer reporting. I would not recommend rejection, but the evidence as presented is not yet sufficient to support the strong claims in the title and abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing about this paper. First, it is a solid engineering contribution: GoHD combines the LIA latent-navigation animator with gaze-direction conditioning, a conformer-based diffusion model for audio-driven head poses, and a two-stage expression predictor that distills frame-wise lip motion from Wav2Lip while an LSTM generates eye and brow motion. The gaze control is genuinely new in this line of work, and the two-stage separation is a sensible response to limited training data. The paper includes ablations, a user study, and a code link, and the writing is straightforward. I would not desk-reject it.\n\nSecond, the central lip-sync claim is not independently supported. Stage 1 trains on videos resynchronized by Wav2Lip (Sec. 3.2, Supp. Eq. 18), and Table 1 evaluates lip sync with SyncNet, the same pre-trained model Wav2Lip was trained to optimize. The authors note that SyncNet scores may not be definitive and underline \"closest to ground truth\" values, but that does not break the circularity: a model trained to imitate Wav2Lip will naturally score well on SyncNet even if Wav2Lip's mouth shapes are wrong. The ablation \"w/o Distillation\" actually fits that story, since it scores worse on MLD when compared against the Wav2Lip-resynced target. The user study asks people whether lip movements look synchronized, not whether the phonemes are correct, so it cannot settle the question. The missing test is evaluation with a metric or human transcription that is not derived from Wav2Lip.\n\nThe other soft spots are milder. Quantitative tables report point estimates without error bars or significance tests, and some differences are small. The code link in the abstract is good, but reproducibility depends on several pre-trained experts (Wav2Lip, SyncNet, BiSeNet, gaze estimator), and the paper doesn't specify which external versions or data splits are used. The frame-by-frame animator also has known temporal jitter, which the authors acknowledge.\n\nOverall, the method is plausible and the new combination is worth publishing. The circular lip-sync evaluation is the load-bearing weakness, but it is fixable with an external eval. This paper deserves peer review and should be sent to a competent referee, not desk-rejected. I'd rather see it accepted after the lip-sync claim is re-tested.","headline":"Sound engineering combination for audio-driven talking heads with gaze control, but the lip-sync evidence is partly self-confirming because Stage 1 distills from Wav2Lip and Table 1 grades with SyncNet; an independent lip-accuracy check is needed before that claim lands.","tokens_in":19761,"tokens_out":2958,"would_cite":true,"duration_ms":28517,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GoHD claims that one audio clip plus one reference photo can drive an arbitrary portrait video with accurately synchronized lips, prosody-aware head poses, and controllable gaze, by splitting motion into three independently trained…","keywords":["audio-driven talking head generation","portrait animation","latent space navigation","diffusion model","head pose synthesis","expression prediction","gaze control","lip synchronization"],"falsifier":"A concrete check: take a held-out set of identities and audios, train GoHD without the distillation term using ground-truth lips only, and compare lip-shape errors against the distilled version using an independent lip-reading or optical-flow mouth measure not derived from SyncNet. If the non-distilled model matches or beats the distilled one on that external metric, the claimed benefit of expert distillation disappears; alternatively, if distilled videos show systematic mouth-shape artifacts against human-labeled ground truth, the load-bearing assumption fails.","tokens_in":18675,"feed_emoji":"🎭","tokens_out":7280,"duration_ms":66635,"temperature":0.7,"pith_summary":"The paper tries to establish that a portrait video can be driven entirely by audio plus a single source image while preserving identity and adding realistic spontaneous motion. Its central argument is to break the problem into three independent generators: a diffusion model that maps speech prosody to head-pose sequences, a two-stage predictor that first copies frame-accurate lip shapes from a pre-trained lip-sync expert and then generates blinks and brow movements with an LSTM, and a latent-space animator that moves the face along learned motion directions while keeping identity fixed. Gaze direction is added as an explicit control so eyes look where the animator intends. If the claims hold, the same system can animate stylized or exaggerated portraits that previous methods distort, and can do so with diversity: the same audio can produce multiple plausible takes. The paper reports comparisons with five prior methods across lip sync, pose naturalness, gaze, and identity preservation.","feed_headline":"One framework animates any portrait with audio-synced motion","feed_subtitle":"Lip, eye, and head motions split into independent generators, so the same audio gives realistic editable takes.","key_machinery":"The load-bearing objects are: 3DMM coefficients for pose $\\mathbf{p} \\in \\mathbb{R}^6$, expression $\\mathbf{e} \\in \\mathbb{R}^{64}$, and gaze $\\mathbf{g} \\in \\mathbb{R}^2$; a conditional denoising network whose residual blocks interleave dilated convolutions with conformer self-attention to predict residual pose sequences $\\Delta\\mathbf{p}_{1:T}$; a handcrafted 21-dimensional eye-motion feature $\\mathbf{f}_t$ combining blink ratio and brow displacements, used as input in the first stage and as the generation target of the second stage; a learnable motion codebook $\\mathbf{M}_\\theta$ whose orthogonal directions are weighted by an MLP to form the latent navigation vector $\\eta$; and adversarial discriminators for eye motion and temporal coherence. The mechanism that carries the argument is the separation of audio-to-lip alignment, treated as frame-wise and distillable, from audio-to-eye mapping, treated as temporal and generative, with gaze injected as an additional condition into the animator's driving branch.","core_discovery":"The central claim is that audio-driven talking-head generation generalizes to arbitrary identities when motion is represented by disentangled 3D morphable model coefficients and transferred through latent-space navigation rather than spatial warping. The authors identify three bottlenecks, gaze, prosody-aware head motion, and the different time scales of lips versus eyes, and assign one module to each: a conformer-based conditional diffusion model for poses, a two-stage expression predictor that distills frame-wise lip motion from a pre-trained lip-sync expert before learning temporal eye and brow dynamics, and a gaze-conditioned animator built on a learnable motion codebook. They assert that this decoupling, plus classifier-free guidance and handcrafted eye features, yields synchronized, natural, editable outputs on unseen identities and supports one-to-many generation.","pith_inferences":["A direct testable extension is to replace the Stage-1 lip-sync expert with a different expert or with a direct audio-to-landmark loss: if the two-stage decomposition itself is the source of the reported gains, the choice of expert should be less important than the separation of lips from eyes.","The explicit gaze parameter opens an application the paper only gestures at: gaze redirection for video conferencing and avatar interaction, where the same utterance can be rendered with the speaker looking at or away from the camera.","The authors' own limitation section implies that the frame-by-frame latent animator, not the motion generators, is the current bottleneck for resolution and temporal stability; combining it with a video-level refinement or a temporally smoothed decoder is the natural next step.","Because the motion descriptors are 3DMM coefficients, the pipeline could be ported to other animatable face models or game avatars, provided the decoder sees the corresponding identity space."],"forward_implications":["On unseen identities and out-of-distribution inputs, the latent-navigation animator should preserve identity while transferring predicted motions, instead of warping or distorting the face.","Head poses should track speech prosody and remain diverse, because pose generation is a stochastic diffusion process that allows the same audio to yield multiple natural sequences.","The two-stage predictor gives separate control over lips and eyes: lip shapes stay locked to the audio while blinks and frowns vary naturally and can be generated without a reference video.","Gaze direction becomes a user-set parameter, so outputs can be corrected toward natural eye directions or redirected for interactive avatars.","Because motion descriptors are independent of the audio source, the system also supports video-driven poses or expressions, multi-modal driving, and continuous motion interpolation."],"supporting_citations":[{"why":"Supplies the pre-trained lip-sync expert whose resynchronized videos serve as distillation targets in Stage 1 and whose SyncNet model is used for lip-sync evaluation.","marker":"(Prajwal et al. 2020)"},{"why":"Provides the latent image animator paradigm and the learned motion codebook that the gaze-oriented face animator adapts.","marker":"(Wang et al. 2022b)"},{"why":"Defines the DDPM diffusion process and noise-prediction loss used for head-pose generation.","marker":"(Ho, Jain, and Abbeel 2020)"},{"why":"Supplies the residual conditional denoising architecture that is extended with conformer attention for audio-conditioned pose synthesis.","marker":"(Kong et al. 2021)"},{"why":"Provides classifier-free guidance, used to balance audio conditioning against the initial pose.","marker":"(Ho and Salimans 2021)"},{"why":"Baseline SadTalker, which also uses 3DMM coefficients; the comparison establishes the reported gains in pose and expression naturalness.","marker":"(Zhang et al. 2023)"},{"why":"Baseline Audio2Head, used for head-motion comparisons and quantitative evaluation.","marker":"(Wang et al. 2021a)"},{"why":"Baseline MakeItTalk, used as a comparison for lip sync and identity generalization.","marker":"(Zhou et al. 2020)"},{"why":"Baseline EAMM, used as a comparison for emotional and expressive motion generalization.","marker":"(Ji et al. 2022)"},{"why":"Provides the VoxCeleb dataset used to train the face animator and as an unseen-identity test set.","marker":"(Nagrani, Chung, and Zisserman 2017)"}],"fun_headline_variants":["Disentangled audio-driven portrait animation with gaze and prosody","One model generates realistic talking heads with disentangled motion","GoHD: Portraits animate with audio, prosody, and gaze control","Audio-driven avatar animation with disentangled motion and eye gaze","From audio to realistic face: disentangled lips, eyes, and head"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pre-trained lip-sync expert's output videos are accurate enough to serve as teaching targets for mouth shapes, and that the same expert's scoring model is an honest yardstick for the result; if the expert distorts mouths systematically, GoHD inherits those distortions and its evaluation partly rewards matching them.","fun_headline_variants_meta":{"raw":{"variants":["Disentangled audio-driven portrait animation with gaze and prosody","One model generates realistic talking heads with disentangled motion","GoHD: Portraits animate with audio, prosody, and gaze control","Audio-driven avatar animation with disentangled motion and eye gaze","From audio to realistic face: disentangled lips, eyes, and head"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000899,"raw_usage":{"total_tokens":3850,"prompt_tokens":905,"completion_tokens":2945,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":2857}},"tokens_in":521,"tokens_out":2945,"duration_ms":19784,"temperature":1.0,"reasoning_tokens":2857,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:39.418209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: take a held-out set of identities and audios, train GoHD without the distillation term using ground-truth lips only, and compare lip-shape errors against the distilled version using an independent lip-reading or optical-flow mouth measure not derived from SyncNet. If the non-distilled model matches or beats the distilled one on that external metric, the claimed benefit of expert distillation disappears; alternatively, if distilled videos show systematic mouth-shape artifacts against human-labeled ground truth, the load-bearing assumption fails.","supporting_citations":[{"cited_title":"R.; Mukhopadhyay, R.; Namboodiri, V","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained lip-sync expert whose resynchronized videos serve as distillation targets in Stage 1 and whose SyncNet model is used for lip-sync evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the residual conditional denoising architecture that is extended with conformer attention for audio-conditioned pose synthesis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline SadTalker, which also uses 3DMM coefficients; the comparison establishes the reported gains in pose and expression naturalness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline MakeItTalk, used as a comparison for lip sync and identity generalization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VoxCeleb dataset used to train the face animator and as an unseen-identity test set."}],"review_version":1}