Pith. sign in

REVIEW 5 major objections 5 minor 61 references

MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MEDTalk claims that separating lip-sync from facial expression—and recombining them while predicting emotion intensity per frame—produces more accurate, more controllable 3D talking-head animation.

desk verdict The lip-sync and reconstruction numbers are credible, but the dynamic-emotion evaluation is circular because the metric is a training loss on an unvalidated pseudo-intensity, so the headline SOTA claim outruns the evidence. read the letter →

arxiv 2507.06071 v4 pith:FTML6NEG submitted 2025-07-08 cs.CV cs.MM

Chang Liu , Ye Pan , Chenyang Ding , Susanto Rahardja , Xiaokang Yang This is my paper
classification cs.CVcs.MM
keywords 3Dfacialanimationaudio-driventalkingheademotionalexpressiondisentangledembeddingmultimodalguidanceemotionintensitypredictionMetaHumanrigcross-modalityfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the two things a talking-head animation must get right—lip-sync and emotional expression—can be trained as separate, swappable codes, and that doing so makes both easier to control and more accurate. MEDTalk learns a content embedding and an emotion embedding purely from motion sequences, using a self-supervised exchange-and-reconstruct scheme, so neither code has to come from error-prone speech emotion recognition. It then predicts how strongly the emotion should appear at each video frame from a fusion of audio and the spoken transcript, and lets the user choose the emotion by label, text description, or reference image. On its MetaHuman-rig benchmark, the paper reports that this design beats four recent speech-driven baselines on lip error, emotion error, and user-rated quality. If the method holds up, film and game production can generate emotionally dynamic 3D performances from just a voice clip, without manual keyframing.

What carries the argument

The load-bearing object is the pair of disentangled embedding spaces produced by a three-phase self-supervised training: self-reconstruction, overlap exchange, and cycle exchange. Self-reconstruction keeps each sequence faithful; overlap exchange swaps emotion codes between sequences with identical content (and vice versa), forcing the two codes to carry no redundant information; cycle exchange runs two rounds of swapping to prevent either code from silently dropping content. Around these codes, a Cross-Modality Fusion module uses cross-attention to combine audio emotion features with text features, predicting a scalar intensity per frame, which rescales the emotion code's norm; and two projection networks map text and image features into the same emotion space. All generation passes through one frozen decoder, which is why multimodal control is possible without retraining the motion model.

What would settle it

Take a clip where a speaker's emotion visibly rises from neutral to a peak within one sentence, obtain continuous human intensity ratings for every frame, and compare them with the paper's pseudo-intensity (the $\ell^1$ norm of its selected controller subset). If the pseudo-intensity does not track human ratings, then the EIE metric measures the wrong quantity and the claimed dynamic-emotion accuracy is not established.

Watch

Extended reading notes

Core claim

The central claim is that content (the articulations that produce speech) and emotion (the expressive deformation of the rest of the face) live in independent embedding spaces, and that training that separation from facial motion alone is sufficient to get better lip-sync and better emotion control than models that try to read emotion from audio. The evidence is the cross-reconstruction training: two sequences that share content but differ in emotion are swapped between encoders, and the decoder must reproduce both; if the two codes were entangled, the swap would fail. After the encoders and decoder are frozen, the model maps audio into the content space and maps either a label, a text embedding, or an image embedding into the emotion space, with the emotion embedding's norm rescaled frame by frame by a predicted intensity. Quantitative results on a 174-parameter MetaHuman rig show the claimed superiority in lip and emotion accuracy, with the user study rating MEDTalk above all four baselines.

Load-bearing premise

The dynamic-emotion mechanism and its evaluation metric both depend on treating the $\ell^1$ norm of a manually selected, never specified subset of MetaHuman controllers as ground-truth “emotional intensity,” with no perceptual validation that this pseudo-intensity matches what humans see.

Editorial extensions

If this is right

  • A production team could retarget an emotional performance onto any MetaHuman avatar without breaking lip-sync, because the content code and emotion code are independent.
  • Emotions within a single sentence are no longer frozen: the frame-wise intensity lets a performance move from calm to excited as the words require.
  • Users can specify emotion outside a fixed label set: a text description or a reference photo replaces the label embedding.
  • The whole training pipeline runs in under an hour on a single RTX 3090, so per-character or per-dataset retraining is practical in production.
  • On the paper's benchmark, MEDTalk improves lip error, emotion error, and user-rated vividness over FaceFormer, EmoTalk, EmoFace, and DiffPoseTalk.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether the same motion-only disentanglement transfers to other rig spaces, such as ARKit blendshapes or SMPL-X, where matched content-emotion pairs are harder to obtain; the method's reliance on paired data is its main transfer constraint.
  • Because the transcript helps predict intensity, the framework could be inverted for dubbing: replace or rewrite affective words in the transcript and re-run intensity prediction, guiding emotion without new audio.
  • The under-specified controller subset for pseudo-intensity should be published; without it, neither the EIE metric nor the intensity-supervision loss can be reproduced or audited.
  • The T-SNE and embedding-swap evidence for disentanglement is qualitative; a quantitative test would be to measure lip error under emotion-code swaps on a held-out set, which the paper only illustrates visually.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces MEDTalk, a framework for audio-driven 3D facial animation on MetaHuman rigs. It disentangles content and emotion embeddings through a self-supervised cross-reconstruction scheme, trains an audio-to-content mapping and a fusion intensity model that combines audio and text features to predict frame-wise intensity, and supports emotion labels, text descriptions, and reference images as guidance. Experiments compare MEDTalk against FaceFormer, EmoTalk, EmoFace, and DiffPoseTalk on an EmoFace-generated rig dataset using proposed metrics MLE, MEE, EIE, FRD plus a user study. The central claims are state-of-the-art accuracy, expressiveness, and diversity in facial animation generation.

Significance. The disentanglement-by-exchange idea is well motivated, and the lightweight training plus public code release are practical strengths. Multimodal guidance beyond fixed emotion labels is a timely direction. However, the dynamic-emotion evaluation is circular because the evaluation metric EIE is the L1 version of the training objective L_int, the ground-truth rig sequences are generated by a baseline method (EmoFace), and Table 1 does not fully support the 'state-of-the-art' claim. If these issues are resolved, the framework could be a useful contribution, but the current evidence is not sufficient to establish the headline claims.

major comments (5)
  1. [Sec. 3.3 and Sec. 4.2, Eqs. (8), (11), (17)] Eq. (17) defines EIE as the L1 error between Int(R_hat) and Int(R), where Int is the pseudo-intensity defined in Eq. (8) as the L1 norm of a hand-selected controller subset S_int. This is exactly the quantity that MEDTalk is trained against through L_int in Eq. (11), so the Table 1 comparison measures how well each method matches a pseudo-label that only MEDTalk was optimized to predict, favoring MEDTalk by construction. The paper never lists S_int or validates this pseudo-intensity against human ratings or an independent intensity measure, and the same definition is used to select reference frames in Sec. 3.4. The dynamic-emotion accuracy claim is therefore not established.
  2. [Sec. 4.2, Table 1] Table 1 shows MEDTalk is second on EIE behind FaceFormer (0.79055 vs. 0.69221) and third on FRD behind DiffPoseTalk and EmoTalk (0.00289 vs. 0.00075 and 0.00116), yet the text concludes that 'MEDTalk consistently outperforms competing methods' and 'achieves state-of-the-art performance in terms of accuracy, expressiveness, and diversity.' The authors acknowledge the EIE and FRD rankings in the same paragraph, so the overall SOTA claim is internally inconsistent. The conclusions should be restricted to MLE, MEE, and the user study, or additional evidence supporting the expressiveness and diversity claims should be provided.
  3. [Sec. 4.1, Datasets] The training and test rig sequences are generated by EmoFace [22], which is itself one of the four baseline methods in Table 1. Ground truth is therefore a synthetic output of a competing method, and all baselines are retrained on the same synthetic dataset. This limits the external validity of the quantitative comparison and makes 'state-of-the-art' claims relative to a self-created, EmoFace-like distribution rather than to real facial motion data. The paper should either evaluate on independently captured rig data or substantially temper the SOTA claim.
  4. [Sec. 4.5, Table 3] In the ablation, removing the disentanglement ('w/o disentangle') improves MEE (0.00861 vs. 0.00906), but the text states only that 'w/o disentangle' results in 'substantial increases in MLE, EIE, and FRD.' This selective reporting weakens the claim that the disentangled embedding is responsible for emotion accuracy, and the observed MEE improvement is left unexplained. The authors should report and discuss the MEE result.
  5. [Sec. 4.2, Eqs. (15)-(18)] The definitions of MLE, MEE, EIE, and FRD depend on controller subsets S_lip, S_emo, S_int, and S_up, but the paper never specifies which of the 174 MetaHuman controllers belong to these sets. Without this information, the quantitative results are not reproducible, and the semantic interpretation of MEE as an 'emotion error' cannot be independently checked. Full controller lists should be provided.
minor comments (5)
  1. [Sec. 3.3, Eq. (9)] Eq. (9) is typeset incorrectly: the formula for tilde f_label_t should be (hat_I_t / ||f_label||) * f_label, but the current rendering is ambiguous and should be fixed.
  2. [Sec. 4.2, Table 1] Table 1 reports no error bars or significance tests; several margins are small (e.g., MEE 0.00906 vs. 0.00952), so the numerical rankings may not be statistically meaningful without variance estimates.
  3. [Sec. 3.2] The assertion that mutual information minimization such as GRL and vCLUB 'was insufficient' is not accompanied by any experimental comparison in the paper; please add this ablation or remove the claim.
  4. [Sec. 3.3] The statement that 'emotional intensity is primarily reflected in the upper face and overall lip shape' is an unsupported assumption; please justify it with an anatomical or empirical reference or a sensitivity analysis over controller subsets.
  5. [Sec. 2.2] The sentence that DEITalk's predicted intensity 'lacked supervision and was heavily correlated with input audio' would benefit from a supporting citation or a precise explanation of how MEDTalk avoids the same issue.

Circularity Check

2 steps flagged · score 6.0 of 10

Dynamic-emotion evaluation reduces to the paper's own pseudo-intensity training target (EIE is the L1 twin of L_int), and the benchmark rigs come from the authors' own EmoFace model, making the central dynamic-emotion claim partially circular.

  1. self definitional [§3.3 (Eqs. 8, 11) and §4.2 (Eq. 17)]
    "Since manually annotating frame-wise emotional intensity is impractical, we derive intensity labels from the motion sequence. Specifically, emotional intensity is primarily reflected in the upper face and overall lip shape. We select specific controller rigs and compute their L1-norm as the pseudo-intensity. ... Additionally, to evaluate the dynamic characteristics of facial expressions, we introduce Emotion Intensity Error (EIE) ... EIE computes the ℓ1 error between the predicted and ground truth emotion intensity per frame (defined in Sec. 3.3)."

    Eq. 8 defines Int(R) as the L1 norm of a hand-selected, never-specified controller set S_int. Eq. 11 trains MEDTalk with L_int = ||Int(R_hat) - Int(R)||_2, and Eq. 17 evaluates dynamic emotion with EIE = ||Int(R_hat) - Int(R)||_1. EIE is therefore the L1 version of the training objective itself: the model is scored on how well it reproduces its own pseudo-labels. Baselines are not trained on L_int, so the EIE comparison is not a level playing field, and because S_int is never listed, no external check of the 'intensity' definition is possible. The dynamic-emotion claim thus reduces, by construction, to a self-defined target rather than to an independently validated quantity.

  2. self citation load bearing [§3.2 (data collection) and §4.1 (Datasets)]
    "To obtain data for cross-reconstruction training, we leverage EmoFace [22], a pre-trained model that takes audio and emotion labels as inputs and predicts facial animation. Specifically, we generate speech samples with seven emotions to create a large-scale dataset. ... Due to the limited availability of publicly accessible MetaHuman controller rig datasets, we utilize the dataset introduced by EmoFace [22] for training audio-driven expression generation (Sec. 3.3)."

    EmoFace [22] is the authors' own prior work (Liu, Lin, Zeng, and Pan). The training rigs and the ground-truth rigs used for quantitative evaluation are generated by this self-cited model, and EmoFace itself is also one of the baselines in Tables 1-2. The 'ground truth' is thus an output of the authors' own generator, with no externally validated MetaHuman motion-capture anchor. This makes the benchmark self-referential: MEDTalk is trained and evaluated on the distribution of the authors' own EmoFace model, so the comparison against EmoFace measures fidelity to the authors' own synthetic data rather than to an external standard.

full rationale

The lip-sync and user-study evidence is not circular: MLE/MEE compare predicted rig parameters to held-out rig sequences, and the 42-participant user study is an external human judgment; the disentanglement architecture is also not a tautology. However, the paper's central dynamic-emotion contribution is self-referential. Eq. 8 defines a pseudo-intensity from an unspecified controller set; Eq. 11 trains the model to match it (L2); Eq. 17 evaluates 'dynamic characteristics' with the L1 error of the same Int. The dynamic-emotion metric is thus the training objective in disguise, and no external validation of the intensity definition is supplied. This loop is compounded by the fact that the rigs themselves come from the authors' own EmoFace model [22], which is also a baseline, so the benchmark is self-generated. Table 1 independently undercuts the text: MEDTalk is second on EIE (behind FaceFormer) and third on FRD (behind DiffPoseTalk and EmoTalk), yet the text concludes 'state-of-the-art in terms of accuracy, expressiveness, and diversity.' The MLE/MEE results and user study support the accuracy subclaim, but the dynamic-emotion and diversity subclaims rest on self-defined/self-cited quantities, warranting a partial-circularity score of 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on unproven premises: EmoFace-generated rigs as ground truth, the L1 norm of selected controllers as emotional intensity, the insufficiency of mutual information minimization, and the adequacy of pretrained feature extractors. None are independently validated in the paper.

free parameters (3)
  • Intensity controller subset S_int = unspecified
    Hand-selected rig dimensions whose L1 norm defines pseudo-intensity (Eq. 8), used as supervision in Eq. 11 and metric in Eq. 17.
  • Loss weights lambda_sim, lambda_int = 0.1, 0.1
    Chosen by hand in Sec 4.1; no ablation on these weights is reported.
  • Controller subsets S_lip, S_emo, S_up = unspecified
    Manual selection of rig dimensions for metrics in Sec 4.2; exact indices not provided.
assumptions (4)
  • domain assumption EmoFace-generated synthetic rig sequences provide valid content-emotion ground truth
    Sec 3.2 uses EmoFace to generate the training dataset for cross-reconstruction; no real captured motion data is used.
  • ad hoc to paper L1 norm of S_int rigs captures emotional intensity
    Eq. 8 defines intensity as this norm; if false, intensity prediction and EIE are meaningless.
  • ad hoc to paper Mutual information minimization is insufficient for content-emotion separation
    Stated in Sec 3.2 without supporting experiment or citation.
  • domain assumption Pretrained feature extractors (wav2vec2.0, emotion2vec, Whisper, RoBERTa, CLIP) provide sufficient and compatible features
    Used throughout; standard practice but no analysis of failure modes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding." pith.science (2026). https://pith.science/paper/FTML6NEG

@misc{pith2026250706071,
  author       = {Pith},
  title        = {Pith review of: MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTML6NEG}},
  note         = {Machine review of arXiv:2507.06071}
}
read the original abstract

Audio-driven emotional 3D facial animation aims to generate synchronized lip movements and vivid facial expressions. However, most existing approaches focus on static and predefined emotion labels, limiting their diversity and naturalness. To address these challenges, we propose MEDTalk, a novel framework for fine-grained and dynamic emotional talking head generation. Our approach first disentangles content and emotion embedding spaces from motion sequences using a carefully designed cross-reconstruction process, enabling independent control over lip movements and facial expressions. Beyond conventional audio-driven lip synchronization, we integrate audio and speech text, predicting frame-wise intensity variations and dynamically adjusting static emotion features to generate realistic emotional expressions. Furthermore, to enhance control and personalization, we incorporate multimodal inputs-including text descriptions and reference expression images-to guide the generation of user-specified facial expressions. With MetaHuman as the priority, our generated results can be conveniently integrated into the industrial production pipeline. The code is available at: https://github.com/SJTU-Lucy/MEDTalk.

Figures

Figures reproduced from arXiv: 2507.06071 by the authors.

Figure 1
Figure 1. Results of MEDTalk. Given an input speech, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of MEDTalk. (a) The overall architecture of MEDTalk. It integrates audio, text, and multimodal guidance [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of our disentanglement strategy using [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Visualization of intensity fluctuation with audio, [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Disentangled Embedding Spaces. outperforms competing methods. MLE and MEE results demon￾strate superior accuracy in lip shape and upper-face expressions, indicating that our disentangled content-emotion embedding space enables high-quality expressive facial animation w…
Figure 6
Figure 6. Figure 6: Swapping emotion embedding alters the emotional [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Qualitative comparison of facial movements generated by different methods under the "angry" (left) and "disgusted" [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Results of multimodal guidance. Given reference [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 48 canonical work pages

  1. [22]

    Chang Liu, Qunfen Lin, Zijiao Zeng, and Ye Pan. 2024. EmoFace: Audio-driven emotional 3D face animation. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 387–397

    work page 2024

  2. [1]

    Ghaliya Al Farsi, Azmi bin Mohd Yusof, Awanis Romli, Ragad M Tawafak, So- hail Iqbal Malik, Jasiya Jabbar, and Mohd Ezanee Bin Rsuli. 2021. A Review of Virtual Reality Applications in an Educational Domain. International Journal of Interactive Mobile Technologies 15, 22 (2021)

    work page 2021

  3. [2]

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33 (2020), 12449–12460

    work page 2020

  4. [3]

    Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, et al. 2024. Emova: Empowering language models to see, hear and speak with vivid emotions.arXiv preprint arXiv:2409.18042 (2024)

    arXiv 2024

  5. [4]

    Peng Chen, Xiaobao Wei, Ming Lu, Yitong Zhu, Naiming Yao, Xingyu Xiao, and Hui Chen. 2023. Diffusiontalker: Personalization and acceleration for speech- driven 3d face diffuser. arXiv preprint arXiv:2311.16565 (2023)

    arXiv 2023

  6. [5]

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al . 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing 16, 6 (2022), 1505–1518

    work page 2022

  7. [6]

    Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. 2020. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning . PMLR, 1779–1788

    work page 2020

  8. [7]

    Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black. 2019. Capture, learning, and synthesis of 3D speaking styles. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10101– 10111

    work page 2019

Show all 61 references
  1. [8]

    Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu

  2. [9]

    Radek Daněček, Kiran Chhatre, Shashank Tripathi, Yandong Wen, Michael Black, and Timo Bolkart. 2023. Emotional speech-driven animation with content- emotion disentanglement. In SIGGRAPH Asia 2023 Conference Papers . 1–13

    2023

  3. [10]

    Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. 2016. Jali: an animator-centric viseme model for expressive lip synchronization. ACM Trans- actions on graphics (TOG) 35, 4 (2016), 1–11

    2016

  4. [11]

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. 2022. Faceformer: Speech-driven 3d facial animation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 18770– 18780

    2022

  5. [12]

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, et al. 2025. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957 (2025)

    2025 arXiv

  6. [13]

    Yuan Gan, Zongxin Yang, Xihang Yue, Lingyun Sun, and Yi Yang. 2023. Efficient emotional adaptation for audio-driven talking-head generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 22634–22645

    2023

  7. [14]

    Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In International conference on machine learning . PMLR, 1180– 1189

    2015

  8. [15]

    Yunhao Ge, Sami Abu-El-Haija, Gan Xin, and Laurent Itti. 2020. Zero-shot synthesis with group-supervised learning. arXiv preprint arXiv:2009.06586 (2020)

    2020 arXiv

  9. [16]

    Google. 2025. Gemini AI. https://gemini.google.com/app Accessed: 2025-03-19

    2025

  10. [17]

    Tianshun Han, Shengnan Gui, Yiqing Huang, Baihui Li, Lijian Liu, Benjia Zhou, Ning Jiang, Quan Lu, Ruicong Zhi, Yanyan Liang, et al. 2024. PMMTalk : Speech- Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features. IEEE Transactions on Multimedia (2024)

    2024

  11. [18]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840–6851

    2020

  12. [19]

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. 2021. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 14080–14089

    2021

  13. [20]

    Jisoo Kim, Jungbin Cho, Joonho Park, Soonmin Hwang, Da Eun Kim, Geon Kim, and Youngjae Yu. 2024. DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation. arXiv preprint arXiv:2408.06010 (2024)

    2024 arXiv

  14. [21]

    Matthew Korban and Xin Li. 2022. A survey on applications of digital human avatars toward virtual co-presence. arXiv preprint arXiv:2201.04168 (2022)

    2022 arXiv

  15. [23]

    Steven R Livingstone and Frank A Russo. 2018. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PloS one 13, 5 (2018), e0196391

    2018

  16. [24]

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. 2019. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172 (2019)

    2019 arXiv

  17. [25]

    Yifeng Ma, Suzhen Wang, Yu Ding, Bowen Ma, Tangjie Lv, Changjie Fan, Zhipeng Hu, Zhidong Deng, and Xin Yu. 2023. Talkclip: Talking head generation with text-guided expressive speaking styles. arXiv preprint arXiv:2304.00334 (2023)

    2023 arXiv

  18. [26]

    Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. 2023. emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185 (2023)

    2023 arXiv

  19. [27]

    Meinard Müller. 2007. Dynamic time warping. Information retrieval for music and motion (2007), 69–84

    2007

  20. [28]

    Federico Nocentini, Claudio Ferrari, and Stefano Berretti. 2024. EmoVOCA: Speech-Driven Emotional 3D Talking Heads. arXiv preprint arXiv:2403.12886 (2024)

    2024 arXiv

  21. [29]

    Ye Pan, Chang Liu, Sicheng Xu, Shuai Tan, and Jiaolong Yang. 2025. VASA-Rig: Audio-Driven 3D Facial Animation with ‘Live’Mood Dynamics in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics (2025)

    2025

  22. [30]

    Ye Pan, Ruisi Zhang, Shengran Cheng, Shuai Tan, Yu Ding, Kenny Mitchell, and Xubo Yang. 2023. Emotional voice puppetry. IEEE Transactions on Visualization and Computer Graphics 29, 5 (2023), 2527–2535

    2023

  23. [31]

    Ziqiao Peng, Yihao Luo, Yue Shi, Hao Xu, Xiangyu Zhu, Hongyan Liu, Jun He, and Zhaoxin Fan. 2023. Selftalk: A self-supervised commutative training diagram to comprehend 3d talking faces. In Proceedings of the 31st ACM International Conference on Multimedia. 5292–5301

    2023

  24. [32]

    Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. 2023. Emotalk: Speech-driven emotional disentanglement for 3d face animation. In Proceedings of the IEEE/CVF international conference on computer vision. 20687–20697

    2023

  25. [33]

    Hai X Pham, Samuel Cheung, and Vladimir Pavlovic. 2017. Speech-driven 3D facial animation with implicit emotional awareness: A deep learning approach. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 80–88

    2017

  26. [34]

    Hai Xuan Pham, Yuting Wang, and Vladimir Pavlovic. 2018. End-to-end learning for 3d facial animation from speech. In Proceedings of the 20th ACM International Conference on Multimodal Interaction . 361–365

    2018

  27. [35]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  28. [36]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning . PMLR, 28492–28518

    2023

  29. [37]

    Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh. 2021. Meshtalk: 3d face animation from speech using cross- modality disentanglement. In Proceedings of the IEEE/CVF international conference on computer vision. 1173–1182

    2021

  30. [38]

    Kang Shen, Haifeng Xia, Guangxing Geng, Guangyue Geng, Siyu Xia, and Zheng- ming Ding. 2024. DEITalk: Speech-Driven 3D Facial Animation with Dynamic Emotional Intensity Modeling. In Proceedings of the 32nd ACM International Conference on Multimedia. 10506–10514

    2024

  31. [39]

    Stefan Stan, Kazi Injamamul Haque, and Zerrin Yumak. 2023. Facediffuser: Speech-driven 3d facial animation synthesis using diffusion. In Proceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games . 1–11

    2023

  32. [40]

    Yasheng Sun, Wenqing Chu, Hang Zhou, Kaisiyuan Wang, and Hideki Koike

  33. [41]

    Zhiyao Sun, Tian Lv, Sheng Ye, Matthieu Lin, Jenny Sheng, Yu-Hui Wen, Min- jing Yu, and Yong-jin Liu. 2024. Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models. ACM Transactions on Graphics (TOG) 43, 4 (2024), 1–9

    2024

  34. [42]

    Shuai Tan, Bill Gong, Bin Ji, and Ye Pan. 2025. FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Cases. In Proceedings of the IEEE/CVF International Conference on Computer Vision

    2025

  35. [43]

    Shuai Tan, Bin Ji, Mengxiao Bi, and Ye Pan. 2025. Edtalk: Efficient disentanglement for emotional talking head synthesis. In European Conference on Computer Vision . Springer, 398–416

    2025

  36. [44]

    Shuai Tan, Bin Ji, Yu Ding, and Ye Pan. 2024. Say anything with any style. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 5088–5096

    2024

  37. [45]

    Shuai Tan, Bin Ji, and Ye Pan. 2023. Emmn: Emotional motion memory network for audio-driven emotional talking face generation. InProceedings of the IEEE/CVF International Conference on Computer Vision . 22146–22156. MM ’25, October 27–31, 2025, Dublin, Ireland Chang Liu, Ye Pa...

    2023

  38. [46]

    Shuai Tan, Bin Ji, and Ye Pan. 2024. FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26317–26327

    2024

  39. [47]

    Shuai Tan, Bin Ji, and Ye Pan. 2024. Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 5079–5087

    2024

  40. [48]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289 (2023)

    2023 arXiv

  41. [49]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)

    2024 arXiv

  42. [50]

    Jennifer Williams, Yi Zhao, Erica Cooper, and Junichi Yamagishi. 2021. Learning disentangled phone and speaker representations in a semi-supervised vq-vae paradigm. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 7053–7057

    2021

  43. [51]

    Sijing Wu, Yunhao Li, Yichao Yan, Huiyu Duan, Ziwei Liu, and Guangtao Zhai

  44. [52]

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien- Tsin Wong. 2023. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12780–12790

    2023

  45. [53]

    Chao Xu, Junwei Zhu, Jiangning Zhang, Yue Han, Wenqing Chu, Ying Tai, Chengjie Wang, Zhifeng Xie, and Yong Liu. 2023. High-fidelity generalized emotional talking face generation with multi-modal emotion space learning. In Proceedings of the IEEE/CVF conference on computer visi...

    2023

  46. [54]

    In Proceedings of the 32nd ACM International Conference on Multimedia

    MMHead: Towards Fine-grained Multi-modal 3D Facial Animation. In Proceedings of the 32nd ACM International Conference on Multimedia . 7966–7975

  47. [55]

    Qingcheng Zhao, Pengyu Long, Qixuan Zhang, Dafei Qin, Han Liang, Longwen Zhang, Yingliang Zhang, Jingyi Yu, and Lan Xu. 2024. Media2face: Co-speech facial animation generation with multi-modality guidance. In ACM SIGGRAPH 2024 conference papers. 1–13

    2024

  48. [56]

    Wei Zhao, Yijun Wang, Tianyu He, Lianying Yin, Jianxin Lin, and Xin Jin. 2023. Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shape. arXiv preprint arXiv:2310.20240 (2023)

    2023 arXiv

  49. [57]

    SiCheng Yang, Methawee Tantrawenith, Haolin Zhuang, Zhiyong Wu, Aolan Sun, Jianzong Wang, Ning Cheng, Huaizhen Tang, Xintao Zhao, Jie Wang, et al. 2022. Speech representation disentanglement with adversarial mutual information learning for one-shot voice conversion. arXiv prep...

    2022 arXiv

  50. [58]

    Yang Zhou, Zhan Xu, Chris Landreth, Evangelos Kalogerakis, Subhransu Maji, and Karan Singh. 2018. Visemenet: Audio-driven animator-centric speech animation. ACM Transactions on Graphics (ToG) 37, 4 (2018), 1–10

    2018

  51. [60]

    Yicheng Zhong, Huawei Wei, Peiji Yang, and Zhisheng Wang. 2024. Expclip: Bridging text and facial expressions via semantic alignment. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 7614–7622

    2024

  52. [2020]

    arXiv preprint arXiv:2004.13922 (2020)

    Revisiting pre-trained models for Chinese natural language processing. arXiv preprint arXiv:2004.13922 (2020)

    2020 arXiv

  53. [2024]

    IEEE Access (2024)

    AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation. IEEE Access (2024)

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.