Pith. sign in

REVIEW 3 major objections 8 minor 55 references

GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression

T0 review · 3 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GoHD claims that one audio clip plus one reference photo can drive an arbitrary portrait video with accurately synchronized lips, prosody-aware head poses, and controllable gaze, by splitting motion into three independently trained…

desk verdict Sound engineering combination for audio-driven talking heads with gaze control, but the lip-sync evidence is partly self-confirming because Stage 1 distills from Wav2Lip and Table 1 grades with SyncNet; an independent lip-accuracy check is needed before that claim lands. read the letter →

arxiv 2412.09296 v2 pith:JOYJQQWH submitted 2024-12-12 cs.CV

Ziqi Zhou , Weize Quan , Hailin Shi , Wei Li , Lili Wang , Dong-Ming Yan This is my paper · ORCID
classification cs.CV
keywords audio-driventalkingheadgenerationportraitanimationlatentspacenavigationdiffusionmodelposesynthesisexpressionpredictiongazecontrollipsynchronization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a portrait video can be driven entirely by audio plus a single source image while preserving identity and adding realistic spontaneous motion. Its central argument is to break the problem into three independent generators: a diffusion model that maps speech prosody to head-pose sequences, a two-stage predictor that first copies frame-accurate lip shapes from a pre-trained lip-sync expert and then generates blinks and brow movements with an LSTM, and a latent-space animator that moves the face along learned motion directions while keeping identity fixed. Gaze direction is added as an explicit control so eyes look where the animator intends. If the claims hold, the same system can animate stylized or exaggerated portraits that previous methods distort, and can do so with diversity: the same audio can produce multiple plausible takes. The paper reports comparisons with five prior methods across lip sync, pose naturalness, gaze, and identity preservation.

What carries the argument

The load-bearing objects are: 3DMM coefficients for pose $\mathbf{p} \in \mathbb{R}^6$, expression $\mathbf{e} \in \mathbb{R}^{64}$, and gaze $\mathbf{g} \in \mathbb{R}^2$; a conditional denoising network whose residual blocks interleave dilated convolutions with conformer self-attention to predict residual pose sequences $\Delta\mathbf{p}_{1:T}$; a handcrafted 21-dimensional eye-motion feature $\mathbf{f}_t$ combining blink ratio and brow displacements, used as input in the first stage and as the generation target of the second stage; a learnable motion codebook $\mathbf{M}_\theta$ whose orthogonal directions are weighted by an MLP to form the latent navigation vector $\eta$; and adversarial discriminators for eye motion and temporal coherence. The mechanism that carries the argument is the separation of audio-to-lip alignment, treated as frame-wise and distillable, from audio-to-eye mapping, treated as temporal and generative, with gaze injected as an additional condition into the animator's driving branch.

What would settle it

A concrete check: take a held-out set of identities and audios, train GoHD without the distillation term using ground-truth lips only, and compare lip-shape errors against the distilled version using an independent lip-reading or optical-flow mouth measure not derived from SyncNet. If the non-distilled model matches or beats the distilled one on that external metric, the claimed benefit of expert distillation disappears; alternatively, if distilled videos show systematic mouth-shape artifacts against human-labeled ground truth, the load-bearing assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that audio-driven talking-head generation generalizes to arbitrary identities when motion is represented by disentangled 3D morphable model coefficients and transferred through latent-space navigation rather than spatial warping. The authors identify three bottlenecks, gaze, prosody-aware head motion, and the different time scales of lips versus eyes, and assign one module to each: a conformer-based conditional diffusion model for poses, a two-stage expression predictor that distills frame-wise lip motion from a pre-trained lip-sync expert before learning temporal eye and brow dynamics, and a gaze-conditioned animator built on a learnable motion codebook. They assert that this decoupling, plus classifier-free guidance and handcrafted eye features, yields synchronized, natural, editable outputs on unseen identities and supports one-to-many generation.

Load-bearing premise

The load-bearing premise is that the pre-trained lip-sync expert's output videos are accurate enough to serve as teaching targets for mouth shapes, and that the same expert's scoring model is an honest yardstick for the result; if the expert distorts mouths systematically, GoHD inherits those distortions and its evaluation partly rewards matching them.

Editorial extensions

If this is right

  • On unseen identities and out-of-distribution inputs, the latent-navigation animator should preserve identity while transferring predicted motions, instead of warping or distorting the face.
  • Head poses should track speech prosody and remain diverse, because pose generation is a stochastic diffusion process that allows the same audio to yield multiple natural sequences.
  • The two-stage predictor gives separate control over lips and eyes: lip shapes stay locked to the audio while blinks and frowns vary naturally and can be generated without a reference video.
  • Gaze direction becomes a user-set parameter, so outputs can be corrected toward natural eye directions or redirected for interactive avatars.
  • Because motion descriptors are independent of the audio source, the system also supports video-driven poses or expressions, multi-modal driving, and continuous motion interpolation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension is to replace the Stage-1 lip-sync expert with a different expert or with a direct audio-to-landmark loss: if the two-stage decomposition itself is the source of the reported gains, the choice of expert should be less important than the separation of lips from eyes.
  • The explicit gaze parameter opens an application the paper only gestures at: gaze redirection for video conferencing and avatar interaction, where the same utterance can be rendered with the speaker looking at or away from the camera.
  • The authors' own limitation section implies that the frame-by-frame latent animator, not the motion generators, is the current bottleneck for resolution and temporal stability; combining it with a video-level refinement or a temporally smoothed decoder is the natural next step.
  • Because the motion descriptors are 3DMM coefficients, the pipeline could be ported to other animatable face models or game avatars, provided the decoder sees the corresponding identity space.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. GoHD proposes a modular audio-driven portrait animation framework. The system predicts 3DMM pose and expression coefficients from audio and animates a single reference image via a latent navigable face animator derived from LIA, with gaze direction as an explicit conditioning signal. Head poses are generated by a conformer-based conditional DDPM with classifier-free guidance on the initial pose. Expressions are produced by a two-stage predictor: Stage 1 distills per-frame lip motions from Wav2Lip-resynchronized videos, and Stage 2 uses an LSTM generator with discriminators to synthesize eye and brow motion features. The paper reports evaluations on HDTF and VoxCeleb, comparing with MakeItTalk, Wav2Lip, Audio2Head, EAMM, and SadTalker, including SyncNet-based lip-sync scores, FID, pose/eye SSIM and variance, identity preservation, ablations, and a user study.

Significance. The paper addresses relevant limitations in audio-driven portrait animation: identity-motion disentanglement, gaze control, prosody-aware head poses, and separation of lip vs. non-lip expressions. The modular design with intermediate 3DMM coefficients is sensible and supports multi-modal driving and motion interpolation. Strengths of the manuscript include the comprehensive ablation of loss terms (Supp. Tables 4-7), the inclusion of a lip-reading loss in Stage 1 training, a user study, and the public code release. If the quantitative claims hold up under independent evaluation, the framework would be a useful contribution. However, the evidence for the headline claims is weakened by evaluative circularity in the lip-sync metric and by the absence of statistical uncertainty estimates.

major comments (3)
  1. [Sec. 4.2, Table 1; Sec. 3.2 Stage 1; Supp. Eq. (18)] The lip-sync evaluation is partly self-confirming. Stage 1 of the expression predictor trains on videos resynchronized by Wav2Lip, and the primary lip-sync metrics LSE-C/LSE-D in Table 1 are computed with the same SyncNet model that Wav2Lip itself is trained to optimize. The paper's 'closest to ground truth' interpretation is reasonable as a relative ranking, but it does not establish that the distilled mouth shapes are phonetically correct. The MLD metric in Table 3 is more independent if computed against original ground-truth landmarks, and this should be stated explicitly. To break the circularity, please add an independent evaluation, e.g., word error rate from a lip-reading model not used in training, or a human phonetic-intelligibility task. If the MLD in Table 3 is indeed computed against original HDTF landmarks, that is valuable evidence and should be highlighted as such.
  2. [Tables 1-3; Fig. 7] All quantitative results are reported as point estimates without confidence intervals, error bars, or significance tests. Several differences between methods are small (e.g., Table 1 LSE-D: Ours 7.78 vs. SadTalker 7.70 on HDTF; Table 2 SSIMe: Ours 0.915 vs. SadTalker 0.904), so it is unclear if the claimed improvements are statistically reliable. Please report per-video score distributions and paired tests (e.g., bootstrap or Wilcoxon signed-rank) for the main metrics, and add uncertainty measures to the user-study percentages in Fig. 7.
  3. [Sec. 3.3; Sec. 4.3, Fig. 1; Supp. Eq. (14)] Gaze orientation is a title-level contribution, but the evaluation is only qualitative. The face animator is trained with a gaze loss using a pre-trained gaze estimator, and gaze manipulation is demonstrated in Fig. 1, yet no quantitative gaze accuracy is reported on the generated videos, nor is gaze control compared with any baseline. Please include a quantitative gaze-angle error evaluation under varying driving gaze directions (pitch/yaw) and on unseen identities to support the claimed gaze-oriented capability.
minor comments (8)
  1. [Sec. 4.4, Table 3] The text says 'The numerical results are reported in Table 6' but the two-stage ablation table in the main paper is Table 3; this cross-reference should be corrected.
  2. [Sec. 3.2, Eq. (5)] Please clarify how the ground-truth eye-motion features f_t used in Stage 1 training are replaced at inference time: is the Stage 2 prediction always used, and is there a train/test mismatch or a scheduled-sampling procedure?
  3. [Sec. 3.1, Eq. (2)] The notation 'Ex0,n,ϵ' is nonstandard; please use the conventional expectation symbol with a subscript, e.g., \mathbb{E}_{x_0,n,\epsilon}.
  4. [Sec. 3.1] Please specify the noise schedule {β_n} and the total number of diffusion steps N, since these are important for reproducibility.
  5. [Table 1 caption] Please state the bold/underline convention in the caption itself: currently bold means best and underline means closest to ground truth, but this is only explained in the body text.
  6. [Sec. 4.2] The sentence 'our method exhibits better performance than SadTalker' is based primarily on LSE-C, but LSE-D is slightly worse on HDTF (7.78 vs. 7.70); a brief discussion of this trade-off would be helpful.
  7. [Sec. 3.3] The gaze loss from Supp. Eq. (14) is used in the animator training objective but is not referenced in the main text; please add a cross-reference when introducing the gaze loss.
  8. [Supp. Eq. (23)] The lip-reading loss L_read is written with a raw dot product; please define the exact cosine-similarity formula and note whether the embeddings are normalized.

Circularity Check

1 steps flagged · score 3.0 of 10

Lip-sync evidence is partly self-confirming: Stage 1 distills from Wav2Lip, and Table 1 grades with Wav2Lip's own SyncNet; independent mouth-shape validation is missing.

  1. fitted input called prediction [Sec. 3.2 Stage 1: Audio-to-lip Distillation; Sec. 4.1 Evaluation Metrics; Sec. 4.2 Table 1 discussion; Supp. Sec. C and Eq. 18-19.]
    "Notably, we distill the resynchronized results from a pre-trained lip expert (Prajwal et al. 2020) ( Ldistill) to inherit its lip-audio alignment capability learned on sufficient sample pairs ... we utilize the pre-trained SyncNet (Prajwal et al. 2020) for confidence (LSE-C) and distance (LSE-D) evaluations of lip motions ... Wav2Lip unquestionably achieves the best results, surpassing even the ground truth, because it directly trains with the SyncNet model used for evaluation ... we utilize the pre-trained lip-synchronization model (Prajwal et al."

    The Stage 1 training targets are videos resynchronized by Wav2Lip, and the Table 1 lip-sync metric is the SyncNet model from the same Wav2Lip paper. Since Wav2Lip is itself trained to optimize that SyncNet, a model distilled from Wav2Lip's outputs is partly optimized for the same scoring function used to evaluate it. The LSE-C/LSE-D values therefore cannot independently establish correct mouth shapes; they can partially reward faithful imitation of Wav2Lip's SyncNet-optimized artifacts. The paper's 'closest to ground truth' reinterpretation of the metric does not break this dependence, because no independent ground-truth mouth-shape check is provided.

full rationale

The core derivations of GoHD are not circular. The pose generator follows the standard DDPM formulation with classifier-free guidance (Eqs. 1-4), and the face animator builds on the external LIA latent-navigation framework (Eq. 9); neither reduces to the paper's own claims. The animator losses use external models (VGG, BiSeNet, ArcFace, L2CS-Net) that are not the paper's outputs. The main circularity burden is evaluative and localized to lip synchronization. Stage 1 of the expression predictor is trained on Wav2Lip-resynchronized videos, and Table 1 measures lip-sync with Wav2Lip's SyncNet. As the paper itself concedes, Wav2Lip beats ground truth because it is trained on that same SyncNet metric; therefore GoHD's lip-sync scores are partly a measure of how well the model imitates Wav2Lip's SyncNet-optimized outputs rather than of phonetic correctness. The MLD ablation in Table 3 does not state its reference target; if MLD were computed against the Wav2Lip-resynced targets, the 'w/o Distillation' degradation would also be expected from the experimental setup rather than from genuine lip-sync improvement. No load-bearing self-citation or uniqueness-importation pattern is present: the citation to Pang et al. 2023 is related work, not a premise. Overall, the pose and animation claims remain independent, but the central lip-sync evidence is partially self-confirming, warranting a score of 3 rather than 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest mainly on standard DDPM machinery, latent navigation from prior work, and pretrained external models. The main hand-set items are loss weights, the guidance coefficient, training sequence lengths, and a hand-designed eye motion feature. No new physical entities are introduced; the motion codebook is inherited from LIA (Wang et al. 2022b). The most fragile input is the assumption that Wav2Lip-resynced videos are reliable training targets, since the same model family is also used for evaluation.

free parameters (4)
  • Classifier-free guidance coefficient gamma = 0.4
    Chosen by hand to balance diversity and conditioning on the initial pose; used in Eq. (3) and reported in Supplementary Section C.
  • Loss weights lambda = lambda_gaze=100; lambda_lms=0.01; lambda_eye=2; lambda_shut=1; lambda_read=2; lambda_1=2; lambda_ssim=2; lambda_ad=1…
    Manually set in Eq. (17), Eq. (18), and Eq. (25); the reported results depend on these balances, and no sensitivity analysis is provided.
  • Gaze representation g in R^2 = pitch and yaw angles
    Hand-defined two-dimensional gaze condition used throughout the gaze-control experiments; the choice of representation affects all gaze-related results.
  • Training sequence lengths = 30 frames for Stage 1, 300 frames for Stage 2
    Hand-set in Supplementary Section C to separate frame-wise lip alignment from longer temporal eye motion dynamics.
assumptions (4)
  • domain assumption 3DMM coefficients (identity i in R^80, expression e in R^64, pose p = r concatenated with t) are a sufficient intermediate representation for facial motion.
    Equation (10) and Supplementary Section A define the motion descriptors. The whole pipeline generates p and e as intermediate outputs, and the authors acknowledge in Supplementary Section F that explicit descriptors lose subtle details such as wrinkles and teeth.
  • standard math DDPM forward and reverse processes and classifier-free guidance are valid for generating pose residual sequences.
    Equations (1) to (3) rely on the standard DDPM formulation of Ho et al. 2020 and classifier-free guidance of Ho and Salimans 2021; this is accepted formal background.
  • ad hoc to paper Wav2Lip lip-sync expert produces sufficiently accurate audio-aligned videos to serve as training targets.
    Stage 1 in Section 3.2 and the Ldistill loss in Supplementary Eq. (18) use Wav2Lip-resynced videos as targets to compensate for limited training data. If the expert outputs are biased, the learned lip shapes inherit the bias.
  • ad hoc to paper Handcrafted eye motion features, blink ratio and brow displacements, capture the perceptually relevant non-lip expression motion.
    Section 3.2 and Figure 4 define f_t in R^21. Stage 1 uses f_t as control and Stage 2 predicts it, so all eye and brow quality claims depend on this hand-designed feature being sufficient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression." pith.science (2026). https://pith.science/paper/JOYJQQWH

@misc{pith2026241209296,
  author       = {Pith},
  title        = {Pith review of: GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JOYJQQWH}},
  note         = {Machine review of arXiv:2412.09296}
}
read the original abstract

Audio-driven talking head generation necessitates seamless integration of audio and visual data amidst the challenges posed by diverse input portraits and intricate correlations between audio and facial motions. In response, we propose a robust framework GoHD designed to produce highly realistic, expressive, and controllable portrait videos from any reference identity with any motion. GoHD innovates with three key modules: Firstly, an animation module utilizing latent navigation is introduced to improve the generalization ability across unseen input styles. This module achieves high disentanglement of motion and identity, and it also incorporates gaze orientation to rectify unnatural eye movements that were previously overlooked. Secondly, a conformer-structured conditional diffusion model is designed to guarantee head poses that are aware of prosody. Thirdly, to estimate lip-synchronized and realistic expressions from the input audio within limited training data, a two-stage training strategy is devised to decouple frequent and frame-wise lip motion distillation from the generation of other more temporally dependent but less audio-related motions, e.g., blinks and frowns. Extensive experiments validate GoHD's advanced generalization capabilities, demonstrating its effectiveness in generating realistic talking face results on arbitrary subjects.

Figures

Figures reproduced from arXiv: 2412.09296 by the authors.

Figure 1
Figure 1. Illustration of gaze orientation experiments. The [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Demonstration of the residual denoising network [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Definition of the eye motion feature, where [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (8 more)
Figure 6
Figure 6. Figure 6: Qualitative comparison on the two datasets. Apart from accurate lip synchronization, our method presents the best [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The result of user study. randomly select 30 test examples and invite 37 volunteers to assess each example in terms of head pose naturalness, ex￾pression vividness (with a focus on eye motions like blinks and frowns), gaze naturalness, lip synchronization, and over￾all…
Figure 8
Figure 8. Figure 8: Visualization of motion interpolation. We can see a [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Demonstration of multi-modal driving results. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Illustration of semantic segmentation with a pre [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Landmarks indices visualization. where D is a discriminator for distinguishing reconstructed images from the real ones. In summary, the total loss of our face animator can be calculated as: Ltotal = λrecLrec + λpercLperc + λexpLexp +λgazeLgaze + λparsLpars + LGAN . (1…
Figure 12
Figure 12. Figure 12: Animation results for out-of-distribution images [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Diverse results for two example identities. [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 45 canonical work pages

  1. [1]

    A.; Hempel, T.; Khalifa, A.; and Al-Hamadi, A

    Abdelrahman, A. A.; Hempel, T.; Khalifa, A.; and Al-Hamadi, A. 2022. L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments. ArXiv, abs/2203.03339

    arXiv 2022

  2. [2]

    Alexanderson, S.; Nagy, R.; Beskow, J.; and Henter, G. E. 2023. Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models. ACM T rans. on G raphics (TOG) , 42(4): 44:1--44:20

    work page 2023

  3. [3]

    Algabri, R.; Shin, H.; and Lee, S. 2024. Real-time 6DoF full-range markerless head pose estimation. Expert Systems with Applications, 239: 122293

    work page 2024

  4. [4]

    K.; Duan, Z.; and Xu, C

    Chen, L.; Maddox, R. K.; Duan, Z.; and Xu, C. 2019. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR) , 7832--7841

    work page 2019

  5. [5]

    Chung, J.; Jamaludin, A.; Zisserman, A.; et al. 2017. You said that? In B ritish M achine V ision C onference (BMVC) . British Machine Vision Association and Society for Pattern Recognition

    work page 2017

  6. [6]

    Deng, J.; Guo, J.; Xue, N.; and Zafeiriou, S. 2019 a . Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4690--4699

    work page 2019

  7. [7]

    Deng, Y.; Yang, J.; Xu, S.; Chen, D.; Jia, Y.; and Tong, X. 2019 b . Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR)

    work page 2019

  8. [8]

    B.; Vougioukas, K.; Landgraf, Z.; Petridis, S.; and Pantic, M

    Drobyshev, N.; Casademunt, A. B.; Vougioukas, K.; Landgraf, Z.; Petridis, S.; and Pantic, M. 2024. EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars. arXiv:2404.19110

    arXiv 2024

Show all 55 references
  1. [9]

    J.; and Bolkart, T

    Feng, Y.; Feng, H.; Black, M. J.; and Bolkart, T. 2021. Learning an Animatable Detailed 3D Face Model from In-the-Wild Images. ACM Trans. Graph., 40(4)

    2021

  2. [10]

    Gao, G.; Xu, Z.; Li, J.; Yang, J.; Zeng, T.; and Qi, G.-J. 2023. Ctcnet: a cnn-transformer cooperation network for face image super-resolution. IEEE Transactions on Image Processing

    2023

  3. [11]

    He, T.; Guo, J.; Yu, R.; Wang, Y.; Zhu, J.; An, K.; Li, L.; Tan, X.; Wang, C.; Wu, H.; Zhao, S.; and Bian, J. 2023. GAIA: Zero-shot Talking Avatar Generation. In ICLR 2024

    2023

  4. [12]

    Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, 6629–6640....

    2017

  5. [13]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. 33: 6840--6851

    2020

  6. [14]

    Ho, J.; and Salimans, T. 2021. Classifier-Free Diffusion Guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications

    2021

  7. [15]

    Hong, F.-T.; Zhang, L.; Shen, L.; and Xu, D. 2022. Depth-Aware Generative Adversarial Network for Talking Head Video Generation

    2022

  8. [16]

    Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR

    2017

  9. [17]

    Ji, X.; Zhou, H.; Wang, K.; Wu, Q.; Wu, W.; Xu, F.; and Cao, X. 2022. EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model. In ACM SIGGRAPH 2022 Conference Proceedings, SIGGRAPH '22

    2022

  10. [18]

    Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; Lehtinen, J.; and Aila, T. 2020. Analyzing and improving the image quality of stylegan. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR) , 8110--8119

    2020

  11. [19]

    Kong, Z.; Ping, W.; Huang, J.; Zhao, K.; and Catanzaro, B. 2021. DiffWave: A Versatile Diffusion Model for Audio Synthesis. In I nternational C onference on L earning R epresentations (ICLR)

    2021

  12. [20]

    Liu, Y.; Lin, L.; Fei, Y.; Changyin, Z.; and Yu, L. 2023. MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentions. In Proceedings of the IEEE/CVF International Conference on Computer Vision

    2023

  13. [21]

    Ma, P.; Wang, Y.; Petridis, S.; Shen, J.; and Pantic, M. 2022. Training Strategies for Improved Lip-Reading. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 8472--8476

    2022

  14. [22]

    Ma, Y.; Wang, S.; Hu, Z.; Fan, C.; Lv, T.; Ding, Y.; Deng, Z.; and Yu, X. 2023 a . StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles. AAAI'23/IAAI'23/EAAI'23. AAAI Press

    2023

  15. [23]

    Ma, Y.; Zhang, S.; Wang, J.; Wang, X.; Zhang, Y.; and Deng, Z. 2023 b . DreamTalk: When Expressive Talking Head Generation Meets Diffusion Probabilistic Models. arXiv preprint arXiv:2312.09767

    2023 arXiv

  16. [24]

    Y.; Wang, Z.; and Paul Smolley, S

    Mao, X.; Li, Q.; Xie, H.; Lau, R. Y.; Wang, Z.; and Paul Smolley, S. 2017. Least squares generative adversarial networks. In P roceedings of the IEEE I nternational C onference on C omputer V ision (ICCV) , 2794--2802

    2017

  17. [25]

    Nagrani, A.; Chung, J.; and Zisserman, A. 2017. VoxCeleb: a large-scale speaker identification dataset. Interspeech

    2017

  18. [26]

    Pang, Y.; Zhang, Y.; Quan, W.; Fan, Y.; Cun, X.; Shan, Y.; and Yan, D.-M. 2023. DPE: Disentanglement of Pose and Expression for General Video Portrait Editing. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR) , 427--436

    2023

  19. [27]

    R.; Mukhopadhyay, R.; Namboodiri, V

    Prajwal, K. R.; Mukhopadhyay, R.; Namboodiri, V. P.; and Jawahar, C. 2020. A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild. MM '20, 484–492. Association for Computing Machinery

    2020

  20. [28]

    Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv e-prints, arXiv--2204

    2022

  21. [29]

    H.; and Liu, S

    Ren, Y.; Li, G.; Chen, Y.; Li, T. H.; and Liu, S. 2021. PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering. In P roceedings of the IEEE I nternational C onference on C omputer V ision (ICCV) , 13759--13768

    2021

  22. [30]

    L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al

    Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Syste...

    2022

  23. [31]

    Shen, S.; Zhao, W.; Meng, Z.; Li, W.; Zhu, Z.; Zhou, J.; and Lu, J. 2023. DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR)

    2023

  24. [32]

    Siarohin, A.; Lathuilière, S.; Tulyakov, S.; Ricci, E.; and Sebe, N. 2019. First Order Motion Model for Image Animation

    2019

  25. [33]

    Siarohin, A.; Woodford, O.; Ren, J.; Chai, M.; and Tulyakov, S. 2021. Motion Representations for Articulated Animation. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR)

    2021

  26. [34]

    Simonyan, K.; and Zisserman, A. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations

    2015

  27. [35]

    Song, Y.; Zhu, J.; Li, D.; Wang, A.; and Qi, H. 2019. Talking Face Generation by Conditional Recurrent Adversarial Network. 919--925. International Joint Conferences on Artificial Intelligence Organization

    2019

  28. [36]

    Tian, L.; Wang, Q.; Zhang, B.; and Bo, L. 2024. EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions. arXiv:2402.17485

    2024 arXiv

  29. [37]

    Vougioukas, K.; Petridis, S.; and Pantic, M. 2019. End-to-End Speech-Driven Realistic Facial Animation with Temporal GANs. In Proc. of the IEEE I nternational C onference on C omputer V ision W orkshops

    2019

  30. [38]

    Wang, S.; Li, L.; Ding, Y.; Fan, C.; and Yu, X. 2021 a . Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

    2021

  31. [39]

    Wang, S.; Li, L.; Ding, Y.; and Yu, X. 2022 a . One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning

    2022

  32. [40]

    Wang, T.-C.; Mallya, A.; and Liu, M.-Y. 2021. One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR)

    2021

  33. [41]

    Wang, X.; Li, Y.; Zhang, H.; and Shan, Y. 2021 b . Towards Real-World Blind Face Restoration with Generative Facial Prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2021

  34. [42]

    Wang, Y.; Yang, D.; Bremond, F.; and Dantcheva, A. 2022 b . Latent Image Animator: Learning to Animate Images via Latent Space Navigation. In I nternational C onference on L earning R epresentations (ICLR)

    2022

  35. [43]

    Wang, Z.; Bovik, A.; Sheikh, H.; and Simoncelli, E. 2004. Image quality assessment: from error visibility to structural similarity. IEEE T rans. on I mage P rocessing (TIP) , 13(4): 600--612

    2004

  36. [44]

    Xu, S.; Chen, G.; Guo, Y.-X.; Yang, J.; Li, C.; Zang, Z.; Zhang, Y.; Tong, X.; and Guo, B. 2024. VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time. arXiv:2404.10667

    2024 arXiv

  37. [45]

    Yin, F.; Zhang, Y.; Cun, X.; Cao, M.; Fan, Y.; Wang, X.; Bai, Q.; Wu, B.; Wang, J.; and Yang, Y. 2022. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In P roceedings of the E uropean C onference on C omputer V ision (ECCV) , 85--...

    2022

  38. [46]

    Yu, C.; Wang, J.; Peng, C.; Gao, C.; Yu, G.; and Sang, N. 2018. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), 325--341

    2018

  39. [47]

    Yu, Z.; Yin, Z.; Zhou, D.; Wang, D.; Wong, F.; and Wang, B. 2023. Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors. In P roceedings of the IEEE I nternational C onference on C omputer V ision (ICCV)

    2023

  40. [48]

    Zhang, W.; Cun, X.; Wang, X.; Zhang, Y.; Shen, X.; Guo, Y.; Shan, Y.; and Wang, F. 2023. SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecogniti...

    2023

  41. [49]

    Zhang, Z.; Li, L.; Ding, Y.; and Fan, C. 2021. Flow-Guided One-Shot Talking Face Generation With a High-Resolution Audio-Visual Dataset. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR) , 3661--3670

    2021

  42. [50]

    Zhao, J.; and Zhang, H. 2022. Thin-plate spline motion model for image animation. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR) , 3657--3666

    2022

  43. [51]

    Zhou, H.; Liu, Y.; Liu, Z.; Luo, P.; and Wang, X. 2019. Talking Face Generation by Adversarially Disentangled Audio-Visual Representation

    2019

  44. [52]

    C.; Wang, X.; and Liu, Z

    Zhou, H.; Sun, Y.; Wu, W.; Loy, C. C.; Wang, X.; and Liu, Z. 2021. Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation. In Proc. of the IEEE C onf. on C omputer V ision and P attern R ecognition (CVPR)

    2021

  45. [53]

    Zhou, Y.; Han, X.; Shechtman, E.; Echevarria, J.; Kalogerakis, E.; and Li, D. 2020. Makelttalk: speaker-aware talking-head animation. ACM T rans. on G raphics (TOG) , 39(6): 1--15

    2020

  46. [54]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.