Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Wav2Sem decouples audio semantics from phonetics so 3D lip animation stops averaging near-homophones.

desk verdict Useful plug-and-play module with consistent gains across six baselines, but the oracle BERT ablation undermines the claim that semantic decoupling is the cause. read the letter →

arxiv 2505.23290 v1 pith:PPDZKQZB submitted 2025-05-29 cs.SD cs.CVeess.AS

Hao Li , Ju Dai , Xin Zhao , Feng Zhou , Junjun Pan , Lei Li This is my paper · ORCID
classification cs.SDcs.CVeess.AS
keywords speech-drivenfacialanimationaudiosemanticdecouplingnear-homophonedisambiguationself-supervisedspeechrepresentationslipsyncBERTalignmenttemporalconvolutionalnetworktransformerencoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that pre-trained self-supervised audio encoders, which excel at phoneme-level features, entangle acoustically similar syllables that require different mouth shapes; the entanglement shows up as averaged, inaccurate lip motion in speech-driven 3D facial animation. To fix this without retraining the animation model, the authors propose Wav2Sem, a plug-and-play module that learns sentence-level semantic embeddings directly from audio and fuses them into the existing audio encoder's features. The added semantic context separates near-homophones in feature space, and the paper reports consistent reductions in lip vertex error and motion deviation across six different animation frameworks on two 3D face datasets, plus improved phoneme recognition on TIMIT. If correct, the module offers a drop-in upgrade for existing lip-sync pipelines that needs no text input at test time.

What carries the argument

The central object is Wav2Sem, a module trained once and then frozen, consisting of a 7-layer temporal convolutional network for local audio features and a 12-layer transformer encoder that aggregates global context. It maps an audio sequence to a sentence-level semantic vector $F_s$, trained with an L1 loss to match BERT's text embedding $F_{CLS}$ (or the mean of BERT's token embeddings, giving two variants, Wav2Semc and Wav2Semm). The fusion operation is $F_d = FC(FC(F_s) + F_p)$, where $F_p$ is the pre-trained self-supervised audio feature; this simple additive injection is what makes the module plug-and-play. The key mechanism is that a single global semantic vector, broadcast across the temporal dimension, acts as context that disambiguates phonetically similar syllables.

What would settle it

Take a curated set of minimal pairs such as 'sheep/ship' and 'backed/get' in matched prosodic contexts, compute the L2 distance between Wav2Sem-enhanced feature vectors for each pair in isolation versus in a disambiguating sentence, and check whether the increased distances correlate with reduced lip vertex error. A stronger test uses true homophones with identical phonemes but different meanings in identical sentential frames: since audio alone cannot distinguish them, their features should remain unchanged, and any claimed lip-shape separation there would indicate leakage of text information rather than genuine acoustic semantic recovery.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that the averaging effect in speech-driven facial animation is not a decoder problem but an encoder problem: near-homophonic syllables occupy overlapping regions in self-supervised audio feature spaces, so any downstream regressor maps them to a compromise lip shape. Wav2Sem addresses this by introducing a semantic space aligned to BERT sentence embeddings, learned from audio alone through a temporal convolutional network and transformer with an L1 loss against text-derived embeddings on LibriSpeech. Fusing a single global semantic vector into phoneme-level features via two linear layers and a sum produces decoupled features that preserve the acoustic distinction between pairs like /i:/ in 'sheep' and /I/ in 'ship'. The paper demonstrates the decoupling quantitatively, by showing larger L2 distances between near-homophone word embeddings after fusion, and qualitatively in generated mouth shapes, and shows the same semantic features also lower phoneme recognition error rates.

Load-bearing premise

The whole method rests on the premise that a single sentence-level semantic vector learned from audio alone can reliably recover enough of the meaning that separates near-homophone pronunciations; if the audio signal does not carry that distinguishing context, Wav2Sem cannot create it.

Editorial extensions

If this is right

  • Inserting frozen Wav2Sem into any of six representative speech-driven facial animation models—CNN-based, transformer-based, VQ-VAE, unified TCN, diffusion, and latent diffusion—reduces mean vertex error, lip vertex error, and upper-face dynamics deviation on both VOCASET and BIWI without changing the models' structures.
  • The same semantic feature increases the L2 distance between near-homophone word pairs in Wav2Vec 2.0 and HuBERT feature spaces, indicating that the decoupling happens at the feature level rather than only in the final rendered mouth shape.
  • Wav2Sem improves phoneme recognition, lowering phoneme error rates on TIMIT for both Wav2Vec 2.0 and HuBERT, which shows the semantic information transfers to a task outside facial animation.
  • Both semantic targets work, but the paper finds the mean-token version slightly better on the smaller VOCASET and the CLS-token version better on the larger BIWI, suggesting the choice depends on dataset scale.
  • No text is needed at test time: the module takes audio only, so it can be added to existing pipelines without changing their input requirements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The same decoupling idea may generalize beyond lips to other speech-to-motion tasks, such as co-speech gesture or full-body animation, wherever near-homophone ambiguity causes averaged poses.
  • Editorial inference: Because the semantic embedding is sentence-level and derived from audio, true homophones (identical pronunciation, different meaning) remain fundamentally underdetermined; the module can only separate pairs that differ acoustically, so its ceiling is set by how much semantic context the acoustic signal actually carries.
  • Editorial inference: A natural testable extension is to apply Wav2Sem to tonal languages such as Mandarin, where syllable meaning is carried by pitch contours; the current TCN and transformer may need explicit pitch features to separate such pairs.
  • Editorial inference: The paper's choice of BERT sentence embeddings as the semantic space could be probed by replacing BERT with other text encoders and checking whether lip-sync gains scale with the quality of the semantic representation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Wav2Sem, a plug-and-play module that learns to map an audio sequence to a sentence-level BERT semantic embedding (training on LibriSpeech with an L1 loss), and then integrates that embedding into the audio encoders of six existing 3D speech-driven facial animation models via a simple fusion layer. The authors report consistent improvements on LVE, MVE, and FDD on VOCASET and BIWI, along with a word-level feature-distance metric, t-SNE visualizations, a user study, and phoneme recognition results, and interpret these as evidence that semantic decoupling of near-homophone features improves lip-sync accuracy and animation quality.

Significance. The proposal is practically appealing: a frozen, audio-only semantic module that can be inserted into arbitrary pretrained audio encoders without requiring text at test time could benefit many speech-driven animation systems. The paper deserves credit for evaluating across six architecturally different baselines, two 3D face datasets, and a phoneme recognition task, and for making the source code publicly available. However, the current evidence does not isolate the semantic mechanism from the added learnable capacity of the TCN/transformer and fusion layers, and it lacks statistical validation; the oracle-BERT ablation in Table 3 actually complicates the causal story. If these gaps are filled, the contribution could be solid, but as it stands the central claim is not fully established.

major comments (4)
  1. [§4.6, Table 3] The oracle BERT ablation undermines the claim that semantic content is the active ingredient. Since BERT embeddings are the exact training targets of Wav2Sem in Eq. (8), injecting ground-truth BERT embeddings through the same fusion path should be at least as effective as Wav2Sem if semantics were the source of the improvement. Instead, the oracle is often worse: for UniTalker LVE, BERTm gives 3.5261 versus Wav2Semm's 3.1521; for LG-LDM MVE, BERTm gives 3.7557 versus the baseline 3.7162; for FaceDiffuse MVE, BERTc gives 4.3816 versus the baseline 4.3651. This pattern suggests that the improvements are carried by the learned TCN/transformer projection and the downstream-trained fusion layers, not by the semantic content per se. A capacity-matched control with non-semantic training targets (e.g., predicting a constant vector or a random but fixed sentence vector) is needed to support the paper's central claim.
  2. [§4.3, Tables 1 and 3] No error bars, confidence intervals, or significance tests are reported for any of the quantitative results. Many of the reported gains are small in absolute terms (e.g., FaceFormer LVE changes by roughly 0.02-0.04 × 10^-5; FDD values differ in the third decimal), so it is impossible to assess whether the differences are statistically reliable. Moreover, Table 1 contains a direct counterexample to the blanket statement in §4.3 that integration 'significantly enhances' every model: LG-LDM on BIWI shows MVE 7.7521 with Wav2Semm versus 7.7298 without, i.e., a clear degradation. The paper should report standard deviations over multiple runs and appropriate significance tests, and should revise claims of uniform improvement to acknowledge such exceptions.
  3. [§4.4, Table 2, Eq. (5), Eq. (7)] The word-level decoupling metric is not well-defined and does not measure the stated phenomenon. Fs in Eq. (5) is a single sentence-level vector that is broadcast to every frame via Eq. (7); therefore, adding Fs shifts the features of an entire utterance. The reported L2 distances between 'word-level' features likely reflect sentence-level differences in Fs rather than decoupling of near-homophone syllables within a sentence. The paper does not specify how word-level features are extracted from the audio, how word boundaries are determined, or why an increased L2 distance between near-homophone words is causally linked to lip-sync accuracy. A direct validation against per-word lip-vertex errors, or a carefully defined within-utterance metric, is needed before this evidence can be used to support the central claim.
  4. [§3.5, Eq. (7)] The fusion mechanism in Eq. (7) introduces additional learnable parameters (two linear layers) whose training protocol is unclear. The paper states that after Wav2Sem training, the module parameters are frozen and downstream networks are optimized with the original hyperparameters, but it does not clarify whether the two FC layers in Eq. (7) are part of the frozen Wav2Sem module or learned on the downstream facial animation datasets. If the FC layers are trained downstream, then part of the observed improvement may be due to added model capacity rather than semantic content. The paper should specify the training status of the fusion layers and include a control where the same downstream-trained linear layers are used with a non-semantic feature (e.g., a constant per-sentence vector) to isolate the effect of semantics.
minor comments (5)
  1. [Throughout] The dataset name 'LibriSpeech' is misspelled as 'Librispeech' and 'Librishpeech' in multiple places; please fix these typos.
  2. [Eq. (5)] The summation index runs from i=0 to N in the equation as written, which is likely a typo; it should be i=1 to N if there are N elements.
  3. [§4.4, Table 2] The numbers in Table 2 (e.g., 0.0397, 0.2689) are presented without units or a description of the exact feature extraction and distance computation; please add a caption or text explaining these values.
  4. [§4.6, Figure 5] The text refers to 'Figure 5' for the comparison with BERT, but the figure is not explained in detail; consider adding a clear description of what is being visualized.
  5. [§4.7, Figure 6(b)] The user study reports scores from 24 users but does not describe the rating procedure, the stimuli (e.g., number of videos shown), or any statistical test; please add these details.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Wav2Sem is trained against external BERT embeddings and evaluated on held-out facial datasets, so the central result is not derived from its own fitted inputs.

full rationale

Wav2Sem is trained (Eq. 8) to minimize ||FCLS − Fs||1 between a BERT sentence embedding of the transcript and a TCN/transformer audio representation, using LibriSpeech text–audio pairs. The downstream evaluations on VOCASET/BIWI lip metrics and TIMIT phoneme error are external to that training loss: none of the LVE/MVE/FDD numbers in Table 1 are optimized by Wav2Sem’s objective, and the module is frozen when inserted into the baseline pipelines. The word-level L2 comparison in §4.4 is not independent evidence of phoneme-level decoupling—because Fs is a single sentence-level vector (Eq. 5) broadcast through Eq. 7, part of the measured distance between near-homophone tokens from different sentences reflects the sentence-level BERT difference that Wav2Sem was trained to reproduce. That is a construct-validity limitation, not a circular reduction: the paper’s central claim rests on held-out facial-animation metrics and user studies, and Table 3’s ground-truth BERT injection—the exact training target—often performs worse than Wav2Sem, which runs counter to a self-fulfilling semantic explanation. No load-bearing argument reduces to the authors’ own prior work; self-citations in the reference list are not used to justify the method. Therefore no significant circularity is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No novel physical entities or hand-set scientific constants are introduced. The central claim rests on domain assumptions about the suitability of BERT as a semantic target, the L1 alignment loss, and the interpretation of feature-space distances as evidence of decoupling.

assumptions (4)
  • domain assumption BERT sentence embeddings define a complete and suitable semantic space for audio-text alignment (Section 3.1, Equation 1).
    The method relies on the assumption that BERT's CLS or mean token embedding captures sentence-level semantics that, when matched by an audio encoder, will disambiguate near-homophone phoneme features.
  • domain assumption An L1 loss between the mean audio feature and the BERT target is sufficient to distill semantic information (Section 3.4, Equation 8).
    No other losses are used; the quality of the semantic distillation rests entirely on this alignment objective.
  • domain assumption Near-homophone coupling in HuBERT and Wav2Vec 2.0 feature spaces is the main cause of lip-sync averaging errors (Section 1, Figure 1).
    The motivating observation is presented qualitatively; no causal experiment isolates feature coupling from other sources of error.
  • domain assumption Word-level L2 distance between feature vectors is a valid measure of useful decoupling (Section 4.4, Table 2).
    Larger distances are interpreted as better decoupling without showing they correlate with lip-sync quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation." pith.science (2026). https://pith.science/paper/PPDZKQZB

@misc{pith2026250523290,
  author       = {Pith},
  title        = {Pith review of: Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPDZKQZB}},
  note         = {Machine review of arXiv:2505.23290}
}
read the original abstract

In 3D speech-driven facial animation generation, existing methods commonly employ pre-trained self-supervised audio models as encoders. However, due to the prevalence of phonetically similar syllables with distinct lip shapes in language, these near-homophone syllables tend to exhibit significant coupling in self-supervised audio feature spaces, leading to the averaging effect in subsequent lip motion generation. To address this issue, this paper proposes a plug-and-play semantic decorrelation module-Wav2Sem. This module extracts semantic features corresponding to the entire audio sequence, leveraging the added semantic information to decorrelate audio encodings within the feature space, thereby achieving more expressive audio features. Extensive experiments across multiple Speech-driven models indicate that the Wav2Sem module effectively decouples audio features, significantly alleviating the averaging effect of phonetically similar syllables in lip shape generation, thereby enhancing the precision and naturalness of facial animations. Our source code is available at https://github.com/wslh852/Wav2Sem.git.

Figures

Figures reproduced from arXiv: 2505.23290 by the authors.

Figure 1
Figure 1. Our plug-and-play Wav2Sem module effectively allevi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of Wav2Sem, which is trained on a large dataset of text-audio pairs. Given input audio signals, Wav2Sem extracts se [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. T-SNE comparison for near-homophonic syllables. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Evaluations of facial motions with and without Wav2Sem for different methods on VOCASET (left) and BIWI (right). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visualizations of different audio encoding structures [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualizations of (a) T-SNE distributions of different [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 42 canonical work pages

  1. [1]

    Facetalk: Audio-driven motion diffusion for neu- ral parametric head models

    Shivangi Aneja, Justus Thies, Angela Dai, and Matthias Nießner. Facetalk: Audio-driven motion diffusion for neu- ral parametric head models. In CVPR, pages 21263–21273, 2024

    work page 2024

  2. [2]

    Gesturediffuclip: Gesture diffusion model with clip latents

    Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents. ACM TOG, 42: 1–18, 2023

    work page 2023

  3. [3]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In NeurIPS, 2020

    work page 2020

  4. [4]

    Expressive speech-driven facial animation

    Yong Cao, Wen C Tien, Petros Faloutsos, and Fr ´ed´eric Pighin. Expressive speech-driven facial animation. ACM TOG, 24(4):1283–1302, 2005

    work page 2005

  5. [5]

    Talking head generation with audio and speech related facial action units

    Sen Chen, Zhilei Liu, Jiaxing Liu, Zhengxiang Yan, and Longbiao Wang. Talking head generation with audio and speech related facial action units. In BMVC, page 353, 2021

    work page 2021

  6. [6]

    Peters, Michael J

    Kiran Chhatre, Radek Danecek, Nikos Athanasiou, Giorgio Becherini, Christopher E. Peters, Michael J. Black, and Timo Bolkart. Emotional speech-driven 3d body animation via disentangled latent diffusion. In CVPR, pages 1942–1953, 2024

    work page 1942

  7. [7]

    Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J. Black. Capture, learning, and syn- thesis of 3d speaking styles. In CVPR, pages 10101–10111, 2019

    work page 2019

  8. [8]

    Emotional speech- driven animation with content-emotion disentanglement

    Radek Dan ˇeˇcek, Kiran Chhatre, Shashank Tripathi, Yandong Wen, Michael Black, and Timo Bolkart. Emotional speech- driven animation with content-emotion disentanglement. In SIGGRAPH Asia, pages 1–13, 2023

    work page 2023

Show all 45 references
  1. [9]

    BERT: pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional trans- formers for language understanding. InNAACL, pages 4171– 4186, 2019

    2019

  2. [10]

    Cross modal audio search and retrieval with joint embeddings based on text and audio

    Benjamin Elizalde, Shuayb Zarar, and Bhiksha Raj. Cross modal audio search and retrieval with joint embeddings based on text and audio. InICASSP, pages 4095–4099, 2019

    2019

  3. [11]

    Unitalker: Scaling up audio-driven 3d facial animation through A unified model

    Xiangyu Fan, Jiaqi Li, Zhiqian Lin, Weiye Xiao, and Lei Yang. Unitalker: Scaling up audio-driven 3d facial animation through A unified model. In ECCV, pages 204–221, 2024

    2024

  4. [12]

    Faceformer: Speech-driven 3d facial anima- tion with transformers

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech-driven 3d facial anima- tion with transformers. In CVPR, pages 18749–18758, 2022

    2022

  5. [13]

    Joint audio-text model for expressive speech- driven 3d facial animation

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Joint audio-text model for expressive speech- driven 3d facial animation. CGIT, 5(1):16:1–16:15, 2022

    2022

  6. [14]

    Mimic: Speaking style disentanglement for speech-driven 3d facial animation

    Hui Fu, Zeqing Wang, Ke Gong, Keze Wang, Tianshui Chen, Haojie Li, Haifeng Zeng, and Wenxiong Kang. Mimic: Speaking style disentanglement for speech-driven 3d facial animation. In AAAI, pages 1770–1777, 2024

    2024

  7. [15]

    Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y

    Awni Y . Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y . Ng. Deep speech: Scaling up end-to-end speech recognition. CoRR, abs/1412.5567, 2014

    2014 arXiv

  8. [16]

    Facexhubert: Text-less speech-driven e (x) pressive 3d facial animation synthesis using self-supervised speech representation learn- ing

    Kazi Injamamul Haque and Zerrin Yumak. Facexhubert: Text-less speech-driven e (x) pressive 3d facial animation synthesis using self-supervised speech representation learn- ing. In ICML, pages 282–291, 2023

    2023

  9. [17]

    Phonemic similarity metrics to compare pronunciation methods

    Ben Hixon, Eric Schneider, and Susan L Epstein. Phonemic similarity metrics to compare pronunciation methods. In IN- TERSPEECH, pages 825–828, 2011

    2011

  10. [18]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. ACM TASLP, 29, 2021

    2021

  11. [19]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015

    2015

  12. [20]

    Mask-fpan: Semi-supervised face parsing in the wild with de-occlusion and uv gan

    Lei Li, Tianfang Zhang, Zhongfeng Kang, and Xikun Jiang. Mask-fpan: Semi-supervised face parsing in the wild with de-occlusion and uv gan. Computers & Graphics, 116:185– 193, 2023

    2023

  13. [21]

    Omg: Towards open-vocabulary motion generation via mixture of controllers

    Han Liang, Jiacheng Bao, Ruichi Zhang, Sihan Ren, Yuecheng Xu, Sibei Yang, Xin Chen, Jingyi Yu, and Lan Xu. Omg: Towards open-vocabulary motion generation via mixture of controllers. In CVPR, pages 482–493, 2024

    2024

  14. [22]

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J. Black. EMAGE: towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. In CVPR, pages 1144–1154, 2024

    2024

  15. [23]

    Convofusion: Multi-modal conversational diffu- sion for co-speech gesture synthesis

    Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, and Christian Theobalt. Convofusion: Multi-modal conversational diffu- sion for co-speech gesture synthesis. In CVPR, pages 1388– 1398, 2024

    2024

  16. [24]

    Librispeech: An ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In ICASSP, pages 5206–5210, 2015

    2015

  17. [25]

    Emotalk: Speech-driven emotional disentanglement for 3d face anima- tion

    Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech-driven emotional disentanglement for 3d face anima- tion. In ICCV, pages 20687–20697, 2023

    2023

  18. [26]

    K. R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Nam- boodiri, and C. V . Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In ACM MM, pages 484–492, 2020

    2020

  19. [27]

    Meshtalk: 3d face an- imation from speech using cross-modality disentanglement

    Alexander Richard, Michael Zollh ¨ofer, Yandong Wen, Fer- nando De la Torre, and Yaser Sheikh. Meshtalk: 3d face an- imation from speech using cross-modality disentanglement. In ICCV, pages 1153–1162

  20. [28]

    Expressive 3d facial animation generation based on local-to-global latent diffusion

    Wenfeng Song, Xuan Wang, Yiming Jiang, Shuai Li, Aimin Hao, Xia Hou, and Hong Qin. Expressive 3d facial animation generation based on local-to-global latent diffusion. TVCG, 2024

    2024

  21. [29]

    Talkingstyle: Personalized speech-driven 3d facial animation with style preservation

    Wenfeng Song, Xuan Wang, Shi Zheng, Shuai Li, Aimin Hao, and Xia Hou. Talkingstyle: Personalized speech-driven 3d facial animation with style preservation. TVCG, 2024

    2024

  22. [30]

    Facediffuser: Speech-driven 3d facial animation synthesis using diffusion

    Stefan Stan, Kazi Injamamul Haque, and Zerrin Yumak. Facediffuser: Speech-driven 3d facial animation synthesis using diffusion. In SIGGRAPH, pages 13:1–13:11, 2023

    2023

  23. [31]

    Sun and Li Deng

    Don X. Sun and Li Deng. Analysis of acoustic-phonetic vari- ations in fluent speech using TIMIT. In ICASSP, pages 201– 204, 1995

    1995

  24. [32]

    Diff- posetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models

    Zhiyao Sun, Tian Lv, Sheng Ye, Matthieu Gaetan Lin, Jenny Sheng, Yu-Hui Wen, Minjing Yu, and Yong-Jin Liu. Diff- posetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models. ACM TOG, 43: 46:1–46:9, 2024

    2024

  25. [33]

    Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica K

    Sarah L. Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica K. Hod- gins, and Iain A. Matthews. A deep learning approach for generalized speech animation. TOG, 36(4):93:1–93:11, 2017

    2017

  26. [34]

    3diface: Diffusion-based speech-driven 3d facial animation and editing

    Balamurugan Thambiraja, Sadegh Aliakbarian, Darren Cosker, and Justus Thies. 3diface: Diffusion-based speech-driven 3d facial animation and editing. CoRR, abs/2312.00870, 2023

    2023 arXiv

  27. [35]

    Imitator: Personalized speech-driven 3d facial animation

    Balamurugan Thambiraja, Ikhsanul Habibie, Sadegh Aliak- barian, Darren Cosker, Christian Theobalt, and Justus Thies. Imitator: Personalized speech-driven 3d facial animation. In ICCV, pages 20621–20631, 2023

    2023

  28. [36]

    Neural voice puppetry: Audio-driven facial reenactment

    Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. In ECCV, pages 716–731, 2020

    2020

  29. [37]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998– 6008, 2017

    2017

  30. [38]

    End-to-end speech-driven realistic facial animation with temporal gans

    Konstantinos V ougioukas, Stavros Petridis, and Maja Pan- tic. End-to-end speech-driven realistic facial animation with temporal gans. In CVPR, pages 37–40, 2019

    2019

  31. [39]

    One- shot talking face generation from single-speaker audio-visual correlation learning

    Suzhen Wang, Lincheng Li, Yu Ding, and Xin Yu. One- shot talking face generation from single-speaker audio-visual correlation learning. In AAAI, pages 2531–2539, 2022

    2022

  32. [40]

    Codetalker: Speech-driven 3d facial animation with discrete motion prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In CVPR, pages 12780–12790, 2023

    2023

  33. [41]

    Feng, Stacy Marsella, and Ari Shapiro

    Yuyu Xu, Andrew W. Feng, Stacy Marsella, and Ari Shapiro. A practical and configurable lip sync method for games. In MIG, pages 131–140, 2013

    2013

  34. [42]

    You only speak once to see

    Wenhao Yang, Jianguo Wei, Wenhuan Lu, and Lei Li. You only speak once to see. In ICASSP 2025-2025 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1–5. IEEE, 2025

    2025

  35. [43]

    Mining audio, text and visual information for talking face generation

    Lingyun Yu, Jun Yu, and Qiang Ling. Mining audio, text and visual information for talking face generation. InICDM, pages 787–795, 2019

    2019

  36. [44]

    Motiondif- fuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model. TPAMI, 46(6):4115–4128, 2024

    2024

  37. [45]

    Media2face: Co-speech facial animation gener- ation with multi-modality guidance

    Qingcheng Zhao, Pengyu Long, Qixuan Zhang, Dafei Qin, Han Liang, Longwen Zhang, Yingliang Zhang, Jingyi Yu, and Lan Xu. Media2face: Co-speech facial animation gener- ation with multi-modality guidance. InSIGGRAPH, page 18, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.