Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

EDTalk++ decomposes a talking head into four orthogonal motion spaces — mouth, head pose, eyes, and expression — so each can be controlled separately and driven by either video or audio.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

EDTalk++ disentangles talking-head video into four orthogonal motion banks (mouth, pose, eyes, expression) and drives them from either video or audio inputs.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid incremental extension of EDTalk with strong numbers and a genuinely new eye/diffusion stage, but the 'full disentanglement' claim outruns the evidence. the 3 major comments →

arxiv 2508.13442 v1 pith:KFXX2NW2 submitted 2025-08-19 cs.CV

EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis

Shuai Tan , Bin Ji This is my paper
classification cs.CV
keywords talking head generationfacial disentanglementorthogonal latent spacesaudio-driven synthesisone-shot reenactmentemotional expressiondiffusion modellatent navigation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the full space of facial motion in a talking head can be split into four disjoint, independently controllable latent spaces: mouth shape, head pose, eye movement, and emotional expression. If this claim holds, a single one-shot generator can animate an identity image by copying the mouth from one video, the pose from another, the gaze from a third, and the emotion from a fourth — or by inferring all four from audio alone. The authors argue that earlier methods fall short because they rely on external priors, lack constraints between spaces, or retrain heavy networks; EDTalk++ instead learns an orthogonal set of basis directions per component and stores them in banks. The practical payoff is efficient training and fine-grained, composable control over talking-head video.

Core claim

The central claim is that facial dynamics can be fully separated into four component-aware latent spaces — mouth ($B^{m}$), pose ($B^{p}$), eye ($B^{ey}$), and expression ($B^{ex}$) — each spanned by learnable bases. Enforcing orthogonality both within each bank and across banks makes the spaces disjoint, so the final driving feature is the simple sum $f^{r\to d}=f^{r\to p}+f^{r\to m}+f^{r\to ey}+f^{r\to ex}$. A progressive training scheme assigns responsibility to each space in three stages: first head pose is separated from the face region via cross-reconstruction, then mouth and eye are separated by swapping eye regions, and finally expression is learned by self-reconstruction from what t

What carries the argument

The load-bearing object is the orthogonal base bank. Each of the four Component-aware Latent Navigation (CLN) modules maintains a bank of learnable vectors; a per-component MLP predicts weights $W^{*}$, and the component's motion feature is $f^{r\to *}=\sum_i w_i^{*} b_i^{*}$. Imposing $\langle b_i^{*}, b_j^{*}\rangle=0$ within a bank and $\langle b_i^{*}, b_j^{\diamond}\rangle=0$ across banks guarantees that the four components occupy orthogonal subspaces of the latent code, which is what makes additive recombination $f^{i\to d}=f^{i\to r}+f^{r\to p}+f^{r\to m}+f^{r\to ey}+f^{r\to ex}$ preserve independent control. The training strategy then uses cross-reconstruction and self-reconstruction

Load-bearing premise

The load-bearing premise is that the HDTF and VFHQ training videos are truly emotion-neutral, so the pose, mouth, and eye banks never absorb expression-related information during the first two training stages; if expression leaks into those banks, the later expression stage cannot make the four spaces fully disjoint.

What would settle it

Take frames from HDTF or VFHQ that the pipeline treats as neutral and run an independent expression classifier on them; if those frames carry systematic expression scores that correlate with the learned mouth or eye bank weights, the neutrality premise fails. More directly, build a paired test set in which only the mouth source changes while pose, eye, and expression sources are frozen, and check whether the generated head pose or expression changes: any measurable movement in the supposedly frozen components refutes complete disentanglement.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • With the four spaces disjoint, a user can combine any subset of facial motions — mouth from the audio, pose from one clip, expression from another — in one synchronized output.
  • The same banks trained from video transfer to audio-only input; only the weight predictors need to be trained, which the paper reports in hours rather than weeks.
  • Expression can be generated from the tone of voice and transcript, removing the need for a separate emotional reference video.
  • Because the bases are semantically meaningful, interpolating two expression weight vectors produces gradual expression morphing between two clips.
  • Freezing trained modules means adding a new component does not require retraining the whole generator, only the new lightweight module.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extension beyond the paper: one could test perceptual independence directly by sweeping one bank's weights while holding the other three fixed and measuring whether any change appears in the supposedly frozen components; latent orthogonality does not by itself guarantee perceptual independence after the nonlinear generator.
  • The additive bank structure suggests a reusable component library: the mouth, pose, eye, and expression banks could in principle be attached to a different renderer, since they are consumed as additive latent offsets — though the paper does not demonstrate this.
  • A concrete stress test: strip the expression from an emotionally expressive video via the neutral reconstruction and check whether the mouth bank still carries upper-face or brow information; if it does, the claimed disjointness is only approximate.
  • The paper's own limitation note points to a natural extension: conditioning the pose/eye diffusion on emotion labels once datasets with emotion-annotated head motion exist would close the loop between expression and pose.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces EDTalk++, a one-shot talking-head generator that aims to fully disentangle four facial-motion spaces—mouth shape, head pose, eye movement, and emotional expression—and to support both video-driven and audio-driven control. The method represents each motion space as a linear combination of learnable orthogonal bases stored in banks (Sec. III-A), and trains four lightweight navigation modules through a three-stage progressive scheme: Face-Pose Decouple, Mouth-Eye Decouple, and Expression Decouple (Sec. III-B). An Audio-to-Motion module then predicts bank weights from audio, optionally conditioned on transcripts, for audio-driven synthesis (Sec. III-C). Extensive experiments on MEAD, HDTF, VFHQ, Voxceleb2, and LRW report state-of-the-art or competitive scores on image quality, lip-sync, and emotion accuracy, together with user studies and efficiency comparisons.

Significance. If the central claim of complete decoupling is correct, EDTalk++ would be a practically valuable unified framework: it offers independent control of four facial components, additive composition of their features, and shared visual priors across video and audio modalities, while requiring only lightweight training of the disentanglement modules. The paper's strengths include consistently strong quantitative results across multiple datasets (Tables I, II, and IV), a clear and efficient training design, and a user study (Table III). However, the strongest contribution claim—'complete decoupling of these four spaces'—is currently supported mainly by qualitative manipulation examples rather than by quantitative disentanglement metrics. The manuscript also relies on an unverified dataset-neutrality assumption that is load-bearing for the training strategy. These issues need to be addressed before the central claim can be accepted.

major comments (3)
  1. [Sec. I, Sec. IV-C, Figs. 9-12] The central claim of 'complete decoupling' is not quantitatively tested. The evidence consists of selected qualitative manipulations (mouth/pose/eye/expression controls and base-wise edits) and ablation images (Fig. 14), but there is no metric measuring whether changing one component leaves the other three invariant. A concrete protocol should be added: e.g., fix identity and pose/eye/expression sources, drive mouth from target A, and report mouth landmark distance to A alongside pose/eye/expression distances to their fixed sources; repeat for each component. Without such non-interference measurements, the additive decomposition f_r→d = f_r→p + f_r→m + f_r→ey + f_r→ex (Sec. III-A) is not established.
  2. [Sec. IV-B, Sec. III-B] The statement that HDTF and VFHQ 'consistently exhibit a Neutral emotion' is used to justify training PLN, MLN, and EYLN exclusively on these datasets and then freezing them. No quantitative evidence for this neutrality is provided. If expression-correlated variation (smiles, eyebrow raises, squints) is present in these YouTube/interview videos, the cross-reconstruction and eye-swap losses will assign that variation to pose, mouth, or eye banks, because no emotion-neutrality or expression-invariance constraint is imposed in the first two stages. The later Expression Decouple stage (Sec. III-B.c) trains only on residuals and cannot remove expression information already absorbed into the frozen banks. Please verify neutrality quantitatively (e.g., emotion classifier/action-unit distribution on the actual training subsets) and, ideally, directly measure expression leakage by testing whether
  3. [Sec. III-C, Eq. (14), L_exp] The audio-to-motion modules are trained to regress weight vectors produced by the paper's own pretrained modules: W_m = MLN(E(I)) in Eq. (14), and similarly W_ex from pretrained EXLN for L_exp. This is self-distillation: it can teach the audio module to imitate the visual module, but it cannot by itself validate that the target spaces contain the intended semantic content. The external SyncNet and Emotion-Fan metrics partially mitigate this concern for lip-sync and expression accuracy, but the training loss for expression uses W_ex from the same model as ground truth. If EXLN has missed or leaked expression information, the audio expression module will inherit the defect. Please add an independent check, e.g., training with MEAD emotion labels or action units as auxiliary targets, or reporting agreement between audio-predicted weights and annotations on a held-out set.
minor comments (6)
  1. [Tables I and II] Several table entries are visually corrupted or hard to parse; for example, the EDTalk++ row in Table I shows '1.213/1.1960 15.6798.0368.0368.036 68.21'. Please regenerate the tables with unambiguous column separation and verify the numeric formatting.
  2. [Sec. III-B.b] Typo: 'bn training' should be 'by training'.
  3. [Sec. III-B.c] Typo: 'EXN L' should be 'EXLN'.
  4. [Sec. V-c] The limitation paragraph contains a duplicate 'Second, Second' and the cross-reference 'as depicted in Eq. (13)' appears to be incorrect; Eq. (13) is the mouth consistency loss, not the proposed extension with emotion label e.
  5. [Sec. I] The claim of disentanglement 'without relying on external knowledge' is overstated: the pipeline uses a pretrained LIA encoder/generator, VGG19, EMOCA, SyncNet, HuBERT, and EmoBERTa. The novelty lies in not using external labels for the bank decomposition, and the text should say so explicitly.
  6. [Sec. IV-C.c] The user study is based on 10 videos per method and 20 participants. Please report confidence intervals or significance tests for the differences in Table III, since the score gaps for some methods are small.

Circularity Check

0 steps flagged

No circularity: EDTalk++'s disentanglement and audio-to-motion pipelines are self-contained and externally validated.

full rationale

EDTalk++ does not exhibit circular reasoning. The four latent spaces are learned from video via cross-reconstruction and self-reconstruction losses (Eqs. 6-13) with the encoder/generator frozen; the expression bank is learned as the residual after pose/mouth/eye banks, which is a model design choice rather than a renaming of the target. The Audio-to-Motion modules (Eqs. 14-16) regress weight vectors produced by the paper's own CLN modules, but this is self-distillation rather than circularity: the training targets are intermediate representations, and the final claims are evaluated with external SyncNet, M-LMD, Emotion-Fan, and human studies. The self-citations ([1], [42], [96], [97]) are conference-version/baseline references, not load-bearing. The main weakness is the unverified 'Neutral' assumption for HDTF/VFHQ (Sec. IV-B: 'videos consistently exhibit a Neutral emotion'), which is a correctness/robustness risk for the full-disentanglement claim, not a circular step.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The framework rests on an additivity and orthogonality assumption about facial latent spaces, plus a strong dataset-neutrality assumption for the first two training stages. The free parameters (bank sizes, loss weights, orthogonality strength) are hand-chosen or unreported. The banks and canonical feature are internal constructs without external falsifiable handles.

free parameters (4)
  • Bank sizes (B_m=20, B_p=6, B_ey=10, B_ex=10) = 20, 6, 10, 10
    Hand-tuned; Table V explores 5/10/20/40 and picks 20 for mouth and 10 for expression; these capacities directly determine what each disentangled space can express.
  • Loss weight for L_mot = 10
    Set in Sec. IV-B; all other loss weights set to 1; not sensitivity-analyzed.
  • Latent and base dimension = 512
    Chosen in Sec. IV-B without justification; affects the capacity of the disentangled spaces.
  • Orthogonality regularization strength = unreported
    Sec. III-A states orthogonality is enforced but never states the loss term or weight, making the actual fitting unquantifiable.
axioms (4)
  • domain assumption Facial motion space can be decomposed as a linear sum of four mutually orthogonal bank subspaces, and the generator can combine them additively.
    Sec. III-A states f_r->d = f_r->p + f_r->m + f_r->ey + f_r->ex; no proof that real facial dynamics satisfy linearity or that the generator's latent space is additive.
  • domain assumption HDTF and VFHQ datasets are emotion-neutral, so training on them extracts pose/mouth/eye without expression contamination.
    Used in Sec. IV-B to train Face-Pose and Mouth-Eye decoupling; the entire expression decoupling stage depends on these banks being expression-free.
  • domain assumption Expression features change slowly, so averaging K frames yields a clean expression feature.
    Sec. III-B-c states 'we define a window of size K around I_d and average K extracted expression features'; K is never specified, and this is the core mechanism of the Expression Decouple stage.
  • domain assumption Pretrained EMOCA features (Sec. III-B-c, Eq. 12) provide reliable motion supervision for expression decoupling.
    L_mot uses the 3D face reconstruction and emotion networks of [33]; this external prior is loaded into the training but not questioned.
invented entities (2)
  • Orthogonal basis banks B_m, B_p, B_ey, B_ex no independent evidence
    purpose: Represent each facial motion component as a linear combination of learned orthogonal bases, enabling disjoint control and reuse for audio-driven synthesis.
    These are internal learned parameters with no external validation; the semantic meaning of each basis is demonstrated only through qualitative examples (Fig. 12).
  • Canonical feature f_r no independent evidence
    purpose: An intermediate latent feature such that f_i_to_* = f_i_to_r + f_r_to_* (Sec. III-A).
    Inherited from LIA [34] and assumed; not independently verified.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis." pith.science (2026). https://pith.science/paper/KFXX2NW2

@misc{pith2026250813442,
  author       = {Pith},
  title        = {Pith review of: EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KFXX2NW2}},
  note         = {Machine review of arXiv:2508.13442}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Achieving disentangled control over multiple facial motions and accommodating diverse input modalities greatly enhances the application and entertainment of the talking head generation. This necessitates a deep exploration of the decoupling space for facial features, ensuring that they a) operate independently without mutual interference and b) can be preserved to share with different modal inputs, both aspects often neglected in existing methods. To address this gap, this paper proposes EDTalk++, a novel full disentanglement framework for controllable talking head generation. Our framework enables individual manipulation of mouth shape, head pose, eye movement, and emotional expression, conditioned on video or audio inputs. Specifically, we employ four lightweight modules to decompose the facial dynamics into four distinct latent spaces representing mouth, pose, eye, and expression, respectively. Each space is characterized by a set of learnable bases whose linear combinations define specific motions. To ensure independence and accelerate training, we enforce orthogonality among bases and devise an efficient training strategy to allocate motion responsibilities to each space without relying on external knowledge. The learned bases are then stored in corresponding banks, enabling shared visual priors with audio input. Furthermore, considering the properties of each space, we propose an Audio-to-Motion module for audio-driven talking head synthesis. Experiments are conducted to demonstrate the effectiveness of EDTalk++.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PortraitDirector: A Hierarchical Disentanglement Framework for Controllable and Real-time Facial Reenactment

    cs.CV 2026-04 unverdicted novelty 5.0

    PortraitDirector uses hierarchical disentanglement of spatial physical motions and semantic emotions to deliver controllable, high-fidelity real-time facial reenactment at 20 FPS.

Reference graph

Works this paper leans on

111 extracted references · 49 canonical work pages · cited by 1 Pith paper · 5 internal anchors

  1. [1]

    Edtalk: Efficient disentanglement for emotional talking head synthesis,

    S. Tan, B. Ji, M. Bi, and Y . Pan, “Edtalk: Efficient disentanglement for emotional talking head synthesis,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 398–416

    2024

  2. [2]

    Style2talker: High-resolution talking head generation with emotion style and art style,

    S. Tan, B. Ji, and Y . Pan, “Style2talker: High-resolution talking head generation with emotion style and art style,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 5, 2024, pp. 5079–5087

    2024

  3. [3]

    Emotional voice puppetry,

    Y . Pan, R. Zhang, S. Cheng, S. Tan, Y . Ding, K. Mitchell, and X. Yang, “Emotional voice puppetry,”IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 5, pp. 2527–2535, 2023

    2023

  4. [4]

    Expressive talking avatars,

    Y . Pan, S. Tan, S. Cheng, Q. Lin, Z. Zeng, and K. Mitchell, “Expressive talking avatars,”IEEE Transactions on Visualization and Computer Graphics, 2024

    2024

  5. [5]

    Vasa-rig: Audio-driven 3d facial animation with ‘live’mood dynamics in virtual reality,

    Y . Pan, C. Liu, S. Xu, S. Tan, and J. Yang, “Vasa-rig: Audio-driven 3d facial animation with ‘live’mood dynamics in virtual reality,”IEEE Transactions on Visualization and Computer Graphics, 2025

    2025

  6. [6]

    Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images,

    Y . Guo, J. Cai, B. Jiang, J. Zhenget al., “Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images,” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 6, pp. 1294–1307, 2018

    2018

  7. [7]

    Styletalk++: A unified framework for controlling the speaking styles of talking heads,

    S. Wang, Y . Ma, Y . Ding, Z. Hu, C. Fan, T. Lv, Z. Deng, and X. Yu, “Styletalk++: A unified framework for controlling the speaking styles of talking heads,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 6, pp. 4331–4347, 2024

    2024

  8. [8]

    Dagan++: Depth-aware generative adversarial network for talking head video generation,

    F.-T. Hong, L. Shen, and D. Xu, “Dagan++: Depth-aware generative adversarial network for talking head video generation,”IEEE Transac- tions on Pattern Analysis and Machine Intelligence, vol. 46, no. 5, pp. 2997–3012, 2023

    2023

  9. [9]

    Metaearth: A generative foundation model for global-scale remote sensing image generation,

    Z. Yu, C. Liu, L. Liu, Z. Shi, and Z. Zou, “Metaearth: A generative foundation model for global-scale remote sensing image generation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  10. [10]

    Fixtalk: Taming identity leakage for high-quality talking head generation in extreme cases,

    S. Tan, B. Gong, B. Ji, and Y . Pan, “Fixtalk: Taming identity leakage for high-quality talking head generation in extreme cases,”arXiv preprint arXiv:2507.01390, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Pomp: Physics- consistent motion generative model through phase manifolds,

    B. Ji, Y . Pan, Z. Liu, S. Tan, X. Jin, and X. Yang, “Pomp: Physics- consistent motion generative model through phase manifolds,” inPro- ceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 22 690–22 701

    2025

  12. [12]

    Animate-x++: Universal character image animation with dynamic backgrounds,

    S. Tan, B. Gong, Z. Liu, Y . Wang, X. Chen, Y . Feng, and H. Zhao, “Animate-x++: Universal character image animation with dynamic backgrounds,”arXiv preprint arXiv:2508.09454, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation

    S. Tan, B. Gong, Y . Wei, S. Zhang, Z. Liu, D. Zheng, J. Chen, Y . Wang, H. Ouyang, K. Zhenget al., “Synmotion: Semantic-visual adaptation for motion customized video generation,”arXiv preprint arXiv:2506.23690, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    Dreamrelation: Relation-centric video customization,

    Y . Wei, S. Zhang, H. Yuan, B. Gong, L. Tang, X. Wang, H. Qiu, H. Li, S. Tan, Y . Zhanget al., “Dreamrelation: Relation-centric video customization,”arXiv preprint arXiv:2503.07602, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Vqtalker: Towards multilingual talking avatars through facial motion tokenization,

    T. Liu, Z. Ma, Q. Chen, F. Chen, S. Fan, X. Chen, and K. Yu, “Vqtalker: Towards multilingual talking avatars through facial motion tokenization,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 6, 2025, pp. 5586–5594

    2025

  16. [16]

    Ai-generated characters for supporting person- alized learning and well-being,

    P. Pataranutaporn, V . Danry, J. Leong, P. Punpongsanon, D. Novy, P. Maes, and M. Sra, “Ai-generated characters for supporting person- alized learning and well-being,”Nature Machine Intelligence, vol. 3, no. 12, pp. 1013–1022, 2021

    2021

  17. [17]

    Learning dynamic facial radiance fields for few-shot talking head synthesis,

    S. Shen, W. Li, Z. Zhu, Y . Duan, J. Zhou, and J. Lu, “Learning dynamic facial radiance fields for few-shot talking head synthesis,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 666–682

    2022

  18. [18]

    Real- istic one-shot mesh-based head avatars,

    T. Khakhulin, V . Sklyarova, V . Lempitsky, and E. Zakharov, “Real- istic one-shot mesh-based head avatars,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 345–362

    2022

  19. [19]

    Face2faceρ: Real-time high-resolution one-shot face reenactment,

    K. Yang, K. Chen, D. Guo, S.-H. Zhang, Y .-C. Guo, and W. Zhang, “Face2faceρ: Real-time high-resolution one-shot face reenactment,” in European conference on computer vision. Springer, 2022, pp. 55–71

    2022

  20. [20]

    Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan,

    F. Yin, Y . Zhang, X. Cun, M. Cao, Y . Fan, X. Wang, Q. Bai, B. Wu, J. Wang, and Y . Yang, “Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan,” inEuropean conference on computer vision. Springer, 2022, pp. 85–101

    2022

  21. [21]

    Progressive disentangled representation learning for fine-grained controllable talk- ing head synthesis,

    D. Wang, Y . Deng, Z. Yin, H.-Y . Shum, and B. Wang, “Progressive disentangled representation learning for fine-grained controllable talk- ing head synthesis,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 17 979–17 989

    2023

  22. [22]

    AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding

    T. Liu, F. Chen, S. Fan, C. Du, Q. Chen, X. Chen, and K. Yu, “Anitalker: Animate vivid and diverse talking faces through identity- decoupled facial motion encoding,”arXiv preprint arXiv:2405.03121, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Neural voice puppetry: Audio-driven facial reenactment,

    J. Thies, M. Elgharib, A. Tewari, C. Theobalt, and M. Nießner, “Neural voice puppetry: Audio-driven facial reenactment,” inComputer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16. Springer, 2020, pp. 716–731

    2020

  24. [24]

    First order motion model for image animation,

    A. Siarohin, S. Lathuili `ere, S. Tulyakov, E. Ricci, and N. Sebe, “First order motion model for image animation,”Advances in neural information processing systems, vol. 32, 2019

    2019

  25. [25]

    Depth-aware generative ad- versarial network for talking head video generation,

    F.-T. Hong, L. Zhang, L. Shen, and D. Xu, “Depth-aware generative ad- versarial network for talking head video generation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 3397–3406

    2022

  26. [26]

    Efficient emotional adaptation for audio-driven talking-head generation,

    Y . Gan, Z. Yang, X. Yue, L. Sun, and Y . Yang, “Efficient emotional adaptation for audio-driven talking-head generation,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 634–22 645

    2023

  27. [27]

    Facial action coding system,

    P. Ekman and W. V . Friesen, “Facial action coding system,”Environ- mental Psychology & Nonverbal Behavior, 1978

    1978

  28. [28]

    Talking head generation with probabilistic audio-to-visual diffusion priors,

    Z. Yu, Z. Yin, D. Zhou, D. Wang, F. Wong, and B. Wang, “Talking head generation with probabilistic audio-to-visual diffusion priors,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 7645–7655

    2023

  29. [29]

    Expressive talking head generation with granular audio-visual control,

    B. Liang, Y . Pan, Z. Guo, H. Zhou, Z. Hong, X. Han, J. Han, J. Liu, E. Ding, and J. Wang, “Expressive talking head generation with granular audio-visual control,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3387–3396

    2022

  30. [30]

    Dpe: Disentanglement of pose and expression for general video portrait editing,

    Y . Pang, Y . Zhang, W. Quan, Y . Fan, X. Cun, Y . Shan, and D.- m. Yan, “Dpe: Disentanglement of pose and expression for general video portrait editing,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 427–436

    2023

  31. [31]

    Pose- controllable talking face generation by implicitly modularized audio- visual representation,

    H. Zhou, Y . Sun, W. Wu, C. C. Loy, X. Wang, and Z. Liu, “Pose- controllable talking face generation by implicitly modularized audio- visual representation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4176–4186

    2021

  32. [32]

    Supervised contrastive learning,

    P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y . Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,”Advances in neural information processing systems, vol. 33, pp. 18 661–18 673, 2020

    2020

  33. [33]

    Emoca: Emotion driven monocular face capture and animation,

    R. Dan ˇeˇcek, M. J. Black, and T. Bolkart, “Emoca: Emotion driven monocular face capture and animation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20 311–20 322

    2022

  34. [34]

    Latent image animator: Learning to animate images via latent space navigation,

    Y . Wang, D. Yang, F. Bremond, and A. Dantcheva, “Latent image animator: Learning to animate images via latent space navigation,” in International Conference on Learning Representations, 2021

    2021

  35. [35]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation,

    W. Zhang, X. Cun, X. Wang, Y . Zhang, X. Shen, Y . Guo, Y . Shan, and F. Wang, “Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 8652–8661

    2023

  36. [36]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”NeurIPS, vol. 33, pp. 6840–6851, 2020

    2020

  37. [37]

    Denoising diffusion implicit mod- els,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit mod- els,” inICLR, 2021

    work page 2021

  38. [38]

    Animate-x: Universal character image animation with enhanced motion representation,

    S. Tan, B. Gong, X. Wang, S. Zhang, D. Zheng, R. Zheng, K. Zheng, J. Chen, and M. Yang, “Animate-x: Universal character image animation with enhanced motion representation,”arXiv preprint arXiv:2410.10306, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Mimir: Improving video diffusion models for precise text understanding,

    S. Tan, B. Gong, Y . Feng, K. Zheng, D. Zheng, S. Shi, Y . Shen, J. Chen, and M. Yang, “Mimir: Improving video diffusion models for precise text understanding,”arXiv preprint arXiv:2412.03085, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Sport: From zero- shot prompts to real-time motion generation,

    B. Ji, Y . Pan, Z. Liu, S. Tan, and X. Yang, “Sport: From zero- shot prompts to real-time motion generation,”IEEE Transactions on Visualization and Computer Graphics, 2025

    work page 2025

  41. [41]

    Audio- driven emotional video portraits,

    X. Ji, H. Zhou, K. Wang, W. Wu, C. C. Loy, X. Cao, and F. Xu, “Audio- driven emotional video portraits,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14 080–14 089

    work page 2021

  42. [42]

    Emmn: Emotional motion memory network for audio-driven emotional talking face generation,

    S. Tan, B. Ji, and Y . Pan, “Emmn: Emotional motion memory network for audio-driven emotional talking face generation,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 146–22 156. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 16

    work page 2023

  43. [43]

    AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis

    D. Li, K. Zhao, W. Wang, B. Peng, Y . Zhang, J. Dong, and T. Tan, “Ae-nerf: Audio enhanced neural radiance field for few shot talking head synthesis,”arXiv preprint arXiv:2312.10921, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  44. [44]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738

    2020

  45. [45]

    Video rewrite: Driving visual speech with audio,

    C. Bregler, M. Covell, and M. Slaney, “Video rewrite: Driving visual speech with audio,” inSeminal Graphics Papers: Pushing the Bound- aries, Volume 2, 2023, pp. 715–722

    work page 2023

  46. [46]

    Semantic-aware implicit neural audio-driven video portrait generation,

    X. Liu, Y . Xu, Q. Wu, H. Zhou, W. Wu, and B. Zhou, “Semantic-aware implicit neural audio-driven video portrait generation,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 106–125

    work page 2022

  47. [47]

    Emo: Emote portrait alive- generating expressive portrait videos with audio2video diffusion model under weak conditions,

    L. Tian, Q. Wang, B. Zhang, and L. Bo, “Emo: Emote portrait alive- generating expressive portrait videos with audio2video diffusion model under weak conditions,”arXiv preprint arXiv:2402.17485, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Vasa-1: Lifelike audio-driven talking faces generated in real time,

    S. Xu, G. Chen, Y .-X. Guo, J. Yang, C. Li, Z. Zang, Y . Zhang, X. Tong, and B. Guo, “Vasa-1: Lifelike audio-driven talking faces generated in real time,”arXiv preprint arXiv:2404.10667, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    Hallo2: Long-duration and high-resolution audio-driven portrait image animation,

    J. Cui, H. Li, Y . Yao, H. Zhu, H. Shang, K. Cheng, H. Zhou, S. Zhu, and J. Wang, “Hallo2: Long-duration and high-resolution audio-driven portrait image animation,”arXiv preprint arXiv:2410.07718, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Hallo3: Highly dynamic and realistic portrait image animation with diffusion transformer networks,

    J. Cui, H. Li, Y . Zhan, H. Shang, K. Cheng, Y . Ma, S. Mu, H. Zhou, J. Wang, and S. Zhu, “Hallo3: Highly dynamic and realistic portrait image animation with diffusion transformer networks,”arXiv preprint arXiv:2412.00733, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Echomimicv2: Towards striking, simplified, and semi-body human animation,

    R. Meng, X. Zhang, Y . Li, and C. Ma, “Echomimicv2: Towards striking, simplified, and semi-body human animation,”arXiv preprint arXiv:2411.10061, 2024

    arXiv 2024

  52. [52]

    Cyberhost: Taming audio-driven avatar diffusion model with region codebook attention,

    G. Lin, J. Jiang, C. Liang, T. Zhong, J. Yang, and Y . Zheng, “Cyberhost: Taming audio-driven avatar diffusion model with region codebook attention,”arXiv preprint arXiv:2409.01876, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Loopy: Taming audio-driven portrait avatar with long-term motion depen- dency,

    J. Jiang, C. Liang, J. Yang, G. Lin, T. Zhong, and Y . Zheng, “Loopy: Taming audio-driven portrait avatar with long-term motion depen- dency,”arXiv preprint arXiv:2409.02634, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning

    Y . Han, J. Zhu, Y . Feng, X. Ji, K. He, X. Li, Y . Liuet al., “Mimaface: Face animation via motion-identity modulated appearance feature learning,”arXiv preprint arXiv:2409.15179, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  55. [55]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation,

    H. Wei, Z. Yang, and Z. Wang, “Aniportrait: Audio-driven synthesis of photorealistic portrait animation,”arXiv preprint arXiv:2403.17694, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    Makelttalk: speaker-aware talking-head animation,

    Y . Zhou, X. Han, E. Shechtman, J. Echevarria, E. Kalogerakis, and D. Li, “Makelttalk: speaker-aware talking-head animation,”ACM Transactions On Graphics (TOG), vol. 39, no. 6, pp. 1–15, 2020

    work page 2020

  57. [57]

    Hierarchical cross-modal talking face generation with dynamic pixel-wise loss,

    L. Chen, R. K. Maddox, Z. Duan, and C. Xu, “Hierarchical cross-modal talking face generation with dynamic pixel-wise loss,” inProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2019, pp. 7832–7841

    work page 2019

  58. [58]

    Speech-driven facial animation using cascaded gans for learning of motion and texture,

    D. Das, S. Biswas, S. Sinha, and B. Bhowmick, “Speech-driven facial animation using cascaded gans for learning of motion and texture,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16. Springer, 2020, pp. 408–424

    2020

  59. [59]

    Few- shot adversarial learning of realistic neural talking head models,

    E. Zakharov, A. Shysheya, E. Burkov, and V . Lempitsky, “Few- shot adversarial learning of realistic neural talking head models,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 9459–9468

    work page 2019

  60. [60]

    Identity-preserving talking face generation with landmark and appear- ance priors,

    W. Zhong, C. Fang, Y . Cai, P. Wei, G. Zhao, L. Lin, and G. Li, “Identity-preserving talking face generation with landmark and appear- ance priors,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9729–9738

    work page 2023

  61. [61]

    Audio2head: Audio- driven one-shot talking-head generation with natural head motion,

    S. Wang, L. Li, Y . Ding, C. Fan, and X. Yu, “Audio2head: Audio- driven one-shot talking-head generation with natural head motion,” in International Joint Conference on Artificial Intelligence. IJCAI, 2021

    work page 2021

  62. [62]

    One-shot talking face generation from single-speaker audio-visual correlation learning,

    S. Wang, L. Li, Y . Ding, and X. Yu, “One-shot talking face generation from single-speaker audio-visual correlation learning,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 3, 2022, pp. 2531–2539

    work page 2022

  63. [63]

    Talking- head generation with rhythmic head motion,

    L. Chen, G. Cui, C. Liu, Z. Li, Z. Kou, Y . Xu, and C. Xu, “Talking- head generation with rhythmic head motion,” inEuropean Conference on Computer Vision. Springer, 2020, pp. 35–51

    work page 2020

  64. [64]

    A morphable model for the synthesis of 3d faces,

    V . Blanz and T. Vetter, “A morphable model for the synthesis of 3d faces,” inProceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999, pp. 187–194

    work page 1999

  65. [65]

    Lip movements generation at a glance,

    L. Chen, Z. Li, R. K. Maddox, Z. Duan, and C. Xu, “Lip movements generation at a glance,” inProceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 520–535

    work page 2018

  66. [66]

    Talking face generation by conditional recurrent adversarial network,

    Y . Song, J. Zhu, D. Li, A. Wang, and H. Qi, “Talking face generation by conditional recurrent adversarial network,” in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Aug 2019. [Online]. Available: http: //dx.doi.org/10.24963/ijcai.2019/129

    work page doi:10.24963/ijcai.2019/129 2019

  67. [67]

    Talking face generation by adversarially disentangled audio-visual representation,

    H. Zhou, Y . Liu, Z. Liu, P. Luo, and X. Wang, “Talking face generation by adversarially disentangled audio-visual representation,” inProceed- ings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 9299–9306

    2019

  68. [68]

    Vast: Vivify your talking avatar via zero-shot expressive facial style transfer,

    L. Chen, Z. Wu, R. Li, W. Bao, J. Ling, X. Tan, and S. Zhao, “Vast: Vivify your talking avatar via zero-shot expressive facial style transfer,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2977–2987

    work page 2023

  69. [69]

    Lipformer: High-fidelity and generalizable talking face generation with a pre-learned facial codebook,

    J. Wang, K. Zhao, S. Zhang, Y . Zhang, Y . Shen, D. Zhao, and J. Zhou, “Lipformer: High-fidelity and generalizable talking face generation with a pre-learned facial codebook,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 844–13 853

    work page 2023

  70. [70]

    Difftalk: Crafting diffusion models for generalized audio-driven portraits anima- tion,

    S. Shen, W. Zhao, Z. Meng, W. Li, Z. Zhu, J. Zhou, and J. Lu, “Difftalk: Crafting diffusion models for generalized audio-driven portraits anima- tion,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1982–1991

    work page 2023

  71. [71]

    A lip sync expert is all you need for speech to lip generation in the wild,

    K. Prajwal, R. Mukhopadhyay, V . P. Namboodiri, and C. Jawahar, “A lip sync expert is all you need for speech to lip generation in the wild,” inProceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 484–492

    work page 2020

  72. [72]

    Emotion- controllable generalized talking face generation,

    S. Sinha, S. Biswas, R. Yadav, and B. Bhowmick, “Emotion- controllable generalized talking face generation,” inInternational Joint Conference on Artificial Intelligence. IJCAI, 2021

    work page 2021

  73. [73]

    Mead: A large-scale audio-visual dataset for emotional talking-face generation,

    K. Wang, Q. Wu, L. Song, Z. Yang, W. Wu, C. Qian, R. He, Y . Qiao, and C. C. Loy, “Mead: A large-scale audio-visual dataset for emotional talking-face generation,” inComputer Vision–ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI. Springer, 2020, pp. 700–717

    work page 2020

  74. [74]

    Eamm: One-shot emotional talking face via audio-based emotion-aware motion model,

    X. Ji, H. Zhou, K. Wang, Q. Wu, W. Wu, F. Xu, and X. Cao, “Eamm: One-shot emotional talking face via audio-based emotion-aware motion model,” inACM SIGGRAPH 2022 Conference Proceedings, 2022, pp. 1–10

    work page 2022

  75. [75]

    StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles

    Y . Ma, S. Wang, Z. Hu, C. Fan, T. Lv, Y . Ding, Z. Deng, and X. Yu, “Styletalk: One-shot talking head generation with controllable speaking styles,”arXiv preprint arXiv:2301.01081, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  76. [76]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset,

    Z. Zhang, L. Li, Y . Ding, and C. Fan, “Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3661–3670

    2021

  77. [77]

    Perceptual losses for real-time style transfer and super-resolution,

    J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” inComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11- 14, 2016, Proceedings, Part II 14. Springer, 2016, pp. 694–711

    2016

  78. [78]

    The unreasonable effectiveness of deep features as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586–595

    2018

  79. [79]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”arXiv preprint arXiv:1409.1556, 2014

    Pith/arXiv arXiv 2014

  80. [80]

    Arbitrary style transfer in real-time with adaptive instance normalization,

    X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” inProceedings of the IEEE international conference on computer vision, 2017, pp. 1501–1510

    work page 2017

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.