OT-Talk: Animating 3D Talking Head with Optimal Transportation

Heather Yu; Liang Peng; Xianfeng Gu; Xiang Gao; Xinmu Wang; Xiyun Song; Zongfang Lin

REVIEW 5 major objections 7 minor 55 references

OT-Talk: Animating 3D Talking Head with Optimal Transportation

T0 review · 5 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Optimal transport metric sharpens lip sync in 3D talking-head animation

desk verdict A clean OT-loss idea with a solid ablation, but the superiority claim is undercut by an undefined metric and fair-comparison gaps. read the letter →

arxiv 2505.01932 v2 pith:ZAADIOMW submitted 2025-05-03 cs.GR cs.CV

Xinmu Wang , Xiang Gao , Xiyun Song , Heather Yu , Zongfang Lin , Liang Peng , Xianfeng Gu This is my paper · ORCID

classification cs.GRcs.CV

keywords optimaltransportslicedWassersteindistancespeech-drivenanimation3DtalkingheadChebyshevgraphconvolutionmeshdeformationVOCASETMultiface

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces OT-Talk, a speech-driven 3D talking-head model whose training objective includes a sliced Wasserstein distance between predicted and ground-truth meshes represented as probability measures. It claims this optimal-transport loss, combined with Chebyshev graph convolutions on the mesh, produces smoother facial motion and better lip synchronization than existing models. Experiments on VOCASET and Multiface report the best scores on lip, face, and head reconstruction errors and on temporal alignment, and a 20-volunteer perception study favors it. The paper's central claim is stated directly: it achieves the best score on all metrics for both datasets. The claimed mechanism is that comparing surface distributions, not just per-vertex $\ell^2$ differences, captures whole-mouth and global-shape errors.

What carries the argument

The central object is the oriented varifold representation of a triangulated mesh: each face contributes a Dirac mass at its barycenter with weight equal to face area, embedded in a position-and-normal space. Projecting these measures onto random directions reduces comparison to one-dimensional Wasserstein distances, giving the sliced Wasserstein distance with $O(L m \log m)$ cost. ChebNet graph convolution supplies geometric features on the mesh graph, and the full loss combines per-vertex reconstruction, velocity, sliced Wasserstein, and weight regularization. The SWD term does the load-bearing work of making the network optimize surface-level agreement rather than isolated vertex positions.

What would settle it

Re-run VOCA, FaceFormer, and CodeTalker with the same training budget, evaluation script, and mask definitions as OT-Talk, report per-seed error bars, and compute DTW_lip from an explicit definition; if OT-Talk no longer wins on all metrics, or if the DTW advantage disappears under a different lip mask, the central claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a talking-head animation network can be trained to compare meshes not by vertex coordinates alone but as discrete probability measures supported on mesh faces, and that the sliced Wasserstein distance between those measures is a tractable training loss. With this loss plus a ChebNet mesh encoder, a HuBERT front-end, and a transformer, the model predicts vertex displacements that the paper reports as quantitatively closer to ground truth and more temporally aligned than VOCA, FaceFormer, and CodeTalker on two datasets. The claimed mechanism is that the optimal-transport distance measures the whole surface distribution, penalizing global shape and lip-location discrepancies in a way that per-vertex $\ell^2$ and velocity losses miss.

Load-bearing premise

The superiority claim rests on the comparison protocol: official pretrained baselines on VOCASET, official implementations on Multiface, no error bars or significance tests, and a temporal metric called DTW_lip that is never defined; if the baselines were not at their best or the metric was computed inconsistently, the claim of being best on all metrics would not be established.

Editorial extensions

If this is right

If the best-on-all-metrics claim holds, OT-Talk offers real-time audio-driven animation at about 0.007 seconds per frame.
The sliced Wasserstein loss, being mesh-agnostic in how it forms probability measures, transfers across datasets with different mesh sizes, as demonstrated on VOCASET and Multiface.
The ablation indicates ChebNet is the largest contributor to reconstruction accuracy, while the OT loss specifically improves lip-region errors and temporal alignment.
Because identity is encoded from mesh geometry rather than one-hot labels, the method can animate unseen identities without retraining for a fixed identity set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the sliced Wasserstein distance compares the distribution of face-area mass, it should be particularly sensitive to whole-mouth opening and closing; a direct test would compare OT-Talk's predicted lip apertures to ground truth frame by frame.
The paper never defines the DTW_lip temporal metric, so its temporal-alignment advantage cannot be independently reproduced without an explicit definition and evaluation code.
The baselines were evaluated with their official pretrained weights on VOCASET and official implementations on Multiface, without matched training budgets or significance tests; a fairer head-to-head with comparable tuning could narrow the reported gap.
The varifold-plus-sliced-Wasserstein loss could plausibly benefit other mesh-deformation tasks such as cloth or soft-body simulation, though the paper does not test that.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The short version: OT-Talk is a competently assembled talking-head pipeline whose genuinely new piece is a sliced Wasserstein distance loss computed on mesh-as-probability-measure representations, and the ablation argues that this loss buys real lip-sync and geometry accuracy. The engineering is solid, but the paper's central 'best on all metrics' claim rests on evaluation details that are not verifiable from the text, especially the undefined DTW_lip metric and the Multiface baseline setup.

What's good: The idea of treating meshes as varifold-type measures and using SWD as a training loss is clean and computationally sensible (O(L m log m)). The ablation study is helpful: removing ChebNet tanks performance, removing the OT loss hurts lip metrics on both datasets, and the transformer helps temporal alignment. That's an honest breakdown. The CoMA-style mesh resampling and PCA initialization are standard but credited, and the writing is straightforward.

Where it gets soft: DTW_lip appears in both tables but is never defined anywhere, which is a serious reproducibility gap. On Multiface, the baselines are 'official implementations' with no training details; if they weren't tuned to Multiface's 30 FPS or different mesh topology, the comparison could be unfair. The practice of evaluating baselines under the best of 8 identity conditions is unusual; if that identity is selected per metric, it's actually generous to the baselines, but the paper doesn't say whether the same identity is used in the user study. No error bars or significance tests; a margin like 0.87 vs 1.00 mm needs variance to mean anything. The mathematical transition from varifold measure (with normals) to SWD in Eqs. (4)-(10) is underspecified: are the measures on R^3 or R^3 x S^2? And the comparison omits ScanTalk, FaceDiffuser, DiffPoseTalk, and SelfTalk, all directly relevant and possibly affecting the 'first OT' and SOTA claims.

None of this is fatal to the core idea. The OT loss is novel and the positive ablation is the strongest evidence. But the superiority claim is not currently established.

I'd send this to a serious referee if I were an editor; it deserves constructive review, with a strong request for code, defined metrics, and convergence-aware baseline comparisons. For a reading group, it's a decent loss-design paper, though I'd only bring it up if the group cares about talking heads or optimal transport in graphics.

Recommendation: engage constructively; major revision before trusting the numbers.

Referee Report

5 major / 7 minor

Summary. This manuscript introduces OT-Talk, an audio-driven 3D talking-head animation method whose main components are a HuBERT audio front-end, a transformer temporal encoder, a ChebNet mesh encoder/decoder, and an auxiliary sliced Wasserstein distance (SWD) loss computed on varifold-encoded mesh measures. The authors claim that this is the first use of optimal transport for talking-head animation and that it yields smoother and better-synchronized motion. Experiments on VOCASET and Multiface compare with VOCA, FaceFormer, and CodeTalker, reporting per-frame lip/face/head errors, a temporal-alignment metric DTW_lip, ablations, qualitative side-by-sides, and a 20-participant user study. The central quantitative claim is stated in Section 4.1: 'Our method achieves the best score on all metrics for both datasets.'

Significance. If the evaluation is accepted, the contribution is a sensible and reasonably novel engineering combination: SWD as a training regularizer on mesh measures is computationally attractive, the ChebNet backbone is a natural fit for fixed-topology meshes, and the ablation study gives evidence that each component helps. The paper also provides enough implementation detail (loss terms, hyperparameters, architecture sizes) to reproduce the method itself. However, the strength of the claimed superiority is currently not established because several evaluation choices are underspecified or undefined, and no uncertainty quantification is provided. The theoretical novelty is moderate: the OT formulation is largely a loss-term swap, and the varifold-plus-SWD construction needs clarification. There is no circularity problem: the SWD is an additional training objective, and the reported metrics are computed independently of that loss.

major comments (5)

[Section 4.1, Tables 1-2] DTW_lip is reported in Tables 1 and 2 but is never defined anywhere in the manuscript. It is used to support the temporal-alignment part of the central claim, so the reader cannot check what is being measured, what the reference/query alignment is, or why the values are on the order of 600-900 (labeled mm). Please define the metric precisely (input trajectories, distance measure, warping constraints, normalization, units) and state the implementation used.
[Section 4.1, Baseline Methods] The fairness of the comparison is not fully documented. For Multiface, the paper only states that the authors adhere to the official implementations; no training epochs, hyperparameters, seeds, or validation-based checkpointing are given for VOCA, FaceFormer, or CodeTalker on this dataset. For VOCASET, pre-trained weights are used but the protocol selects the best-performing identity condition among all identities; the paper should state whether the same identity choice is used for every metric and in the qualitative/user-study comparisons, and should justify why this protocol is not biased. Without these details, the claimed margins (e.g., 0.87 mm versus 1.00 mm head error on VOCASET) cannot be attributed to the method rather than to comparison logistics.
[Tables 1-2] No error bars, repeated runs, or significance tests are reported. Several margins are small relative to plausible run-to-run variation (e.g., E_lip_mean 2.81 versus 2.93 mm and E_head_mean 0.87 versus 1.00 mm on VOCASET), and the ablation rows are single numbers. Please report standard deviations over test subjects or repeated training runs, and give a significance test or bootstrap confidence intervals for the main comparisons.
[Section 3.2, Eqs. (4)-(7)] The varifold construction places the mesh measure on R^3 x S^2 (position plus oriented normal), but the sliced Wasserstein distance is defined only for measures on R^d and the projection directions satisfy theta in S^{d-1}; the ambient dimension d is never specified. Clarify whether the SWD is computed on the six-dimensional (position, normal) vectors, and if so justify this choice; if only vertex positions are projected, explain what role the varifold/normal component plays. As written, the central OT step is ambiguous.
[Section 4, Baseline set] The abstract and Section 4.1 say the method outperforms state-of-the-art techniques, but the comparison set omits several recent methods discussed in Section 2, especially ScanTalk, which is designed for unregistered scans and multiple mesh topologies, as well as the diffusion-based FaceDiffuser and DiffPoseTalk. Either add at least one of these baselines or qualify the claim as applying to the three compared methods.

minor comments (7)

[Sections 3.3-3.4] The model name is written inconsistently as 'Hubert Large' and 'Huber model'; it should be 'HuBERT Large' throughout.
[Section 4.3, Figure 4 caption] The caption says the words are 'spoken by 2 subjects from VOCASET', but the figure and surrounding text describe Multiface; fix the caption.
[Section 3.4, Eq. (11)] 'toal loss' should be 'total loss'.
[Sections 3.1 and 3.4] W denotes the adjacency matrix in Section 3.1 but denotes the network weights in Eq. (11); rename one of them to avoid ambiguity.
[Table 2] On VOCASET, removing the transformer gives E_head_mean 0.86 mm, slightly lower than the 0.87 mm of OT-Talk; the claim that 'other components also help' should be qualified to acknowledge this exception.
[Section 4.4] Report the number of A/B responses and per-participant variance; with 20 volunteers, percentages such as 52.71% are based on a small sample and should be accompanied by confidence intervals.
[Abstract and Section 2.3] The claim of being 'the first approach to leverage optimal transportation' is stronger than Section 2.3 supports, since prior work already uses OT for mesh comparison; recommend saying 'first for audio-driven 3D talking-head animation'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the claimed superiority is an empirical comparison against external benchmarks, not a derivation from fitted inputs.

full rationale

The paper's central claim is an empirical performance comparison, not a derivation from fitted inputs or from the authors' own prior results. The training objective (Eq. 11) combines a per-vertex reconstruction loss L_r, a velocity-difference loss L_v, and a sliced-Wasserstein mesh-dissimilarity loss L_SW (Eq. 10). The reported metrics are independent test-set quantities: lip/face/head Euclidean vertex errors and a DTW_lip temporal-alignment score computed on synthesized meshes. None of these metrics is defined as the training loss itself, and the ablation 'w/o OT' in Table 2 shows that removing L_SW degrades DTW_lip and vertex errors, which is evidence of a causal contribution rather than a quantity forced by construction. The paper adopts ChebNet from external published work (Defferrard et al. 2016; CoMA), not from a self-citation chain, and it does not invoke any uniqueness theorem from the authors' prior work. The undefined DTW_lip metric and the absence of error bars concern comparability and reproducibility, not circularity. Therefore no circular step is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim depends on standard OT/varifold mathematics, dataset-synchronization assumptions, a deterministic audio-to-motion mapping, and a set of hand-chosen hyperparameters. No new physical or conceptual entities are introduced. The free parameters are the loss weights, the number of SWD projections, and the PCA-initialized 50-component motion subspace; the paper does not analyze sensitivity to these choices.

free parameters (3)

Loss weights beta1, beta2, beta3 = 10, 1, 0.01
Chosen in Section 3.4 without sensitivity analysis; the relative contribution of the SWD loss (beta2=1) vs reconstruction (implicit weight 1) and velocity (beta1=10) is load-bearing for the claimed improvement of OT.
Number of SWD projections L = 100
Set in Section 3.2; the Monte Carlo estimate of SWD depends on L, and no ablation over L is given.
PCA face motion components = 50 components
The motion decoder's linear layer is PCA-initialized on the training set (Section 3.3), providing a data-fitted linear subspace for face motions. This is a strong prior and may account for part of the performance.

assumptions (5)

standard math Sliced Wasserstein distance is a valid metric on probability measures and can be estimated by Monte Carlo projections (Eqs. 5-7).
Relied on throughout Section 3.2; cited from Bonneel et al. [4]. Forms the basis of the OT loss.
standard math A triangulated mesh can be represented as a discrete probability measure via an oriented varifold, approximated by face barycenters and face areas (Eq. 4).
Adopted from Charon and Trouve [6] and Kaltenmark et al. [26]; the approximation error for large or curved triangles is not analyzed.
domain assumption The audio-mesh pairs in VOCASET and Multiface are synchronized, and the facial motion is largely a deterministic function of speech plus identity/style.
The model learns a deterministic mapping from audio and template to displacements; Section 5 acknowledges non-deterministic motions such as blinks are not handled.
domain assumption HuBERT-large features, after linear interpolation, and a transformer encoder provide sufficient audio context for lip-sync.
This architectural choice is presented without comparison to other audio encoders or transformer variants in the main paper.
domain assumption The ChebNet mesh encoder, trained on each dataset's template mesh, provides an identity/style latent that generalizes to unseen subjects.
The model relies on the mesh encoder latent instead of one-hot identity codes; Section 5 notes the datasets contain few subjects and more data is needed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OT-Talk: Animating 3D Talking Head with Optimal Transportation." pith.science (2026). https://pith.science/paper/ZAADIOMW

@misc{pith2026250501932,
  author       = {Pith},
  title        = {Pith review of: OT-Talk: Animating 3D Talking Head with Optimal Transportation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZAADIOMW}},
  note         = {Machine review of arXiv:2505.01932}
}

read the original abstract

Animating 3D head meshes using audio inputs has significant applications in AR/VR, gaming, and entertainment through 3D avatars. However, bridging the modality gap between speech signals and facial dynamics remains a challenge, often resulting in incorrect lip syncing and unnatural facial movements. To address this, we propose OT-Talk, the first approach to leverage optimal transportation to optimize the learning model in talking head animation. Building on existing learning frameworks, we utilize a pre-trained Hubert model to extract audio features and a transformer model to process temporal sequences. Unlike previous methods that focus solely on vertex coordinates or displacements, we introduce Chebyshev Graph Convolution to extract geometric features from triangulated meshes. To measure mesh dissimilarities, we go beyond traditional mesh reconstruction errors and velocity differences between adjacent frames. Instead, we represent meshes as probability measures and approximate their surfaces. This allows us to leverage the sliced Wasserstein distance for modeling mesh variations. This approach facilitates the learning of smooth and accurate facial motions, resulting in coherent and natural facial animations. Our experiments on two public audio-mesh datasets demonstrate that our method outperforms state-of-the-art techniques both quantitatively and qualitatively in terms of mesh reconstruction accuracy and temporal alignment. In addition, we conducted a user perception study with 20 volunteers to further assess the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2505.01932 by the authors.

**Figure 1.** System Pipeline Hubert Large Linear Interpolation T r a n sfo r m e r E n c o d e r C h e b y s h e v G C N N C h e b y s h e v G C N N C h e b y s h e v G C N N P o olin g P o olin g Head Mesh Audio Signal F C L a y e r P o olin g F C L a y e r C o n c ate n atio n C h e b y s h e v G C N N C h e b y s h e v G C N N C h e b y s h e v G C N N F C L a y e r PCA Initialization P o olin g P o olin g P o olin g Face ? M… view at source ↗

**Figure 3.** VOCASET Qualitative results (face/lip motions to [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 6.** Face/lip motions at 6 time points for sentence "Did [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

Figures from the paper (2 more)

**Figure 7.** Figure 7: Scores on VOCASET Dataset [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Scores on Multiface Dataset 4.4 User Perception Study We predict meshes for audio files and mesh templates in the test data, render them into images, and generate the corresponding videos. We use the same identity conditioning for VOCA, FaceFormer, and CodeTalker in t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 37 canonical work pages

[1]

Mohammed M Alghamdi, He Wang, Andrew J Bulpitt, and David C Hogg. 2022. Talking head from speech audio using a pre-trained image generator. In Proceed- ings of the 30th ACM International Conference on Multimedia . 5228–5236

work page 2022
[2]

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33 (2020), 12449–12460

2020
[3]

Espen Bernton, Pierre E Jacob, Mathieu Gerber, and Christian P Robert. 2019. On parameter estimation with the Wasserstein distance. Information and Inference: A Journal of the IMA 8, 4 (2019), 657–676

work page 2019
[4]

Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. 2015. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision 51 (2015), 22–45

work page 2015
[5]

Giorgos Bouritsas, Sergiy Bokhnyak, Stylianos Ploumpis, Michael Bronstein, and Stefanos Zafeiriou. 2019. Neural 3d morphable models: Spiral convolutional networks for 3d shape representation learning and generation. In Proceedings of the IEEE/CVF international conference on computer vision . 7213–7222

work page 2019
[6]

Nicolas Charon and Alain Trouvé. 2013. The varifold representation of nonori- ented shapes for diffeomorphic registration. SIAM journal on Imaging Sciences 6, 4 (2013), 2547–2580

work page 2013
[7]

Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. 2020. Talking-head generation with rhythmic head motion. In European Conference on Computer Vision . Springer, 35–51

work page 2020
[8]

Shiyang Cheng, Michael Bronstein, Yuxiang Zhou, Irene Kotsia, Maja Pantic, and Stefanos Zafeiriou. 2019. Meshgan: Non-linear 3d morphable models of faces. arXiv preprint arXiv:1903.10384 (2019)

arXiv 2019

Show all 55 references

[9]

Shiyang Cheng, Georgios Tzimiropoulos, Jie Shen, and Maja Pantic. 2020. Faster, better and more detailed: 3d face reconstruction with graph convolutional net- works. In Proceedings of the Asian conference on computer vision

2020
[10]

Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia. 2008. MeshLab: an Open-Source Mesh Processing Tool. In Eurographics Italian Chapter Conference , Vittorio Scarano, Rosario De Chiara, and Ugo Erra (Eds.). The Eurograp...

2008
[11]

Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black. 2019. Capture, learning, and synthesis of 3D speaking styles. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10101– 10111

2019
[12]

Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems 26 (2013)

2013
[13]

Dipanjan Das, Sandika Biswas, Sanjana Sinha, and Brojeshwar Bhowmick. 2020. Speech-driven facial animation using cascaded gans for learning of motion and texture. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16 ...

2020
[14]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolu- tional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems 29 (2016)

2016
[15]

Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. 2016. Jali: an animator-centric viseme model for expressive lip synchronization. ACM Trans- actions on graphics (TOG) 35, 4 (2016), 1–11

2016
[16]

Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al. 2020. 3d morphable face models—past, present, and future. ACM Transactions on Graphics (ToG) 39, 5 (2020), 1–38

2020
[17]

Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. 2022. Faceformer: Speech-driven 3d facial animation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18770– 18780

2022
[18]

Gabriele Fanelli, Juergen Gall, Harald Romsdorfer, Thibaut Weise, and Luc Van Gool. 2010. A 3-d audio-visual corpus of affective communication. IEEE Transactions on Multimedia 12, 6 (2010), 591–598

2010
[19]

Michael Garland and Paul S Heckbert. 1997. Surface simplification using quadric error metrics. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. 209–216

1997
[20]

Joan Glaunes, Alain Trouvé, and Laurent Younes. 2004. Diffeomorphic matching of distributions: A new approach for unlabelled point-sets and sub-manifolds matching. In Proceedings of the 2004 IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition, 2004. C...

2004
[21]

Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, and Christian Theobalt. 2021. Learning speech-driven 3d conversational gestures from video. In Proceedings of the 21st ACM International Conference on Intelligent ...

2021
[22]

A Hannun. 2014. Deep Speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567 (2014)

2014 arXiv
[23]

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840–6851

2020
[24]

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Rus- lan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language process...

2021
[25]

Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. 2021. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 14080–14089

2021
[26]

Irene Kaltenmark, Benjamin Charlier, and Nicolas Charon. 2017. A general framework for curve and surface comparison and registration with oriented varifolds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3346–3355

2017
[27]

Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. 2017. Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics (ToG) 36, 4 (2017), 1–12

2017
[28]

Bo Li, Xiaolin Wei, Bin Liu, Zhifen He, Junjie Cao, and Yu-Kun Lai. 2024. Pose- aware 3D talking face synthesis using geometry-guided audio-vertices attention. IEEE Transactions on Visualization and Computer Graphics (2024)

2024
[29]

Kun Li, Jingying Liu, Yu-Kun Lai, and Jingyu Yang. 2019. Generating 3D Faces using Multi-column Graph Convolutional Networks. InComputer Graphics Forum, Vol. 38. Wiley Online Library, 215–224

2019
[30]

Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. 2017. Learn- ing a model of facial shape and expression from 4D scans. ACM Trans. Graph. 36, 6 (2017), 194–1

2017
[31]

Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia. 2018. De- formable shape completion with graph convolutional autoencoders. In Proceed- ings of the IEEE conference on computer vision and pattern recognition . 1886–1895

2018
[32]

Jingying Liu, Binyuan Hui, Kun Li, Yunke Liu, Yu-Kun Lai, Yuxiang Zhang, Yebin Liu, and Jingyu Yang. 2021. Geometry-guided dense perspective network for speech-driven facial animation. IEEE Transactions on Visualization and Computer Graphics 28, 12 (2021), 4873–4886

2021
[33]

Federico Nocentini, Thomas Besnier, Claudio Ferrari, Sylvain Arguillere, Ste- fano Berretti, and Mohamed Daoudi. 2025. Scantalk: 3d talking heads from unregistered scans. In European Conference on Computer Vision . Springer, 19–36

2025
[34]

Ziqiao Peng, Yihao Luo, Yue Shi, Hao Xu, Xiangyu Zhu, Hongyan Liu, Jun He, and Zhaoxin Fan. 2023. Selftalk: A self-supervised commutative training diagram to comprehend 3d talking faces. In Proceedings of the 31st ACM International Conference on Multimedia. 5292–5301

2023
[35]

Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. 2023. Emotalk: Speech-driven emotional disentanglement for 3d face animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 20687–20697

2023
[36]

Gabriel Peyré, Marco Cuturi, et al. 2019. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning 11, 5-6 (2019), 355–607

2019
[37]

Hai X Pham, Samuel Cheung, and Vladimir Pavlovic. 2017. Speech-driven 3D facial animation with implicit emotional awareness: A deep learning approach. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 80–88

2017
[38]

KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar
[39]

Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black. 2018. Gen- erating 3D faces using convolutional mesh autoencoders. In Proceedings of the European conference on computer vision (ECCV) . 704–720

2018
[40]

Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh. 2021. Meshtalk: 3d face animation from speech using cross- modality disentanglement. InProceedings of the IEEE/CVF International Conference on Computer Vision. 1173–1182

2021
[41]

Stefan Stan, Kazi Injamamul Haque, and Zerrin Yumak. 2023. Facediffuser: Speech-driven 3d facial animation synthesis using diffusion. In Proceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games . 1–11

2023
[42]

Zhiyao Sun, Tian Lv, Sheng Ye, Matthieu Lin, Jenny Sheng, Yu-Hui Wen, Min- jing Yu, and Yong-jin Liu. 2024. Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models. ACM Transactions on Graphics (TOG) 43, 4 (2024), 1–9

2024
[43]

Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2017. Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics (ToG) 36, 4 (2017), 1–13

2017
[44]

Sarah L Taylor, Moshe Mahler, Barry-John Theobald, and Iain Matthews
[45]

Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. 2020. Neural voice puppetry: Audio-driven facial reenactment. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16 . Springer, 716–731

2020
[46]

Cédric Villani. 2021. Topics in optimal transportation. Vol. 58. American Mathe- matical Soc

2021
[47]

Cédric Villani et al. 2009. Optimal transport: old and new . Vol. 338. Springer

2009
[48]

Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. 2020. Realistic speech-driven facial animation with gans. International Journal of Computer ICMR ’25, June 30-July 3, 2025, Chicago, IL, USA Xinmu Wang et al. Vision 128, 5 (2020), 1398–1413

2020
[49]

Haozhe Wu, Songtao Zhou, Jia Jia, Junliang Xing, Qi Wen, and Xiang Wen. 2023. Speech-driven 3d face animation with composite and regional facial movements. In Proceedings of the 31st ACM International Conference on Multimedia. 6822–6830

2023
[50]

Cheng-hsin Wuu, Ningyuan Zheng, Scott Ardisson, Rohan Bali, Danielle Belko, Eric Brockmeyer, Lucas Evans, Timothy Godisart, Hyowon Ha, Xuhua Huang, et al . 2022. Multiface: A dataset for neural face rendering. arXiv preprint arXiv:2207.11243 (2022)

2022 arXiv
[51]

Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien- Tsin Wong. 2023. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12780–12790

2023
[52]

Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. 2021. Pose-controllable talking face generation by implicitly modu- larized audio-visual representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 4176–4186

2021
[53]

Yang Zhou, Zhan Xu, Chris Landreth, Evangelos Kalogerakis, Subhransu Maji, and Karan Singh. 2018. Visemenet: Audio-driven animator-centric speech animation. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–10

2018
[2012]

In Proceedings of the 11th ACM SIG- GRAPH/Eurographics conference on Computer Animation

Dynamic units of visual speech. In Proceedings of the 11th ACM SIG- GRAPH/Eurographics conference on Computer Animation . 275–284
[2020]

In Proceedings of the 28th ACM international conference on multimedia

A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia . 484–492

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Mohammed M Alghamdi, He Wang, Andrew J Bulpitt, and David C Hogg. 2022. Talking head from speech audio using a pre-trained image generator. In Proceed- ings of the 30th ACM International Conference on Multimedia . 5228–5236

work page 2022

[2] [2]

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33 (2020), 12449–12460

2020

[3] [3]

Espen Bernton, Pierre E Jacob, Mathieu Gerber, and Christian P Robert. 2019. On parameter estimation with the Wasserstein distance. Information and Inference: A Journal of the IMA 8, 4 (2019), 657–676

work page 2019

[4] [4]

Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. 2015. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision 51 (2015), 22–45

work page 2015

[5] [5]

Giorgos Bouritsas, Sergiy Bokhnyak, Stylianos Ploumpis, Michael Bronstein, and Stefanos Zafeiriou. 2019. Neural 3d morphable models: Spiral convolutional networks for 3d shape representation learning and generation. In Proceedings of the IEEE/CVF international conference on computer vision . 7213–7222

work page 2019

[6] [6]

Nicolas Charon and Alain Trouvé. 2013. The varifold representation of nonori- ented shapes for diffeomorphic registration. SIAM journal on Imaging Sciences 6, 4 (2013), 2547–2580

work page 2013

[7] [7]

Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. 2020. Talking-head generation with rhythmic head motion. In European Conference on Computer Vision . Springer, 35–51

work page 2020

[8] [8]

Shiyang Cheng, Michael Bronstein, Yuxiang Zhou, Irene Kotsia, Maja Pantic, and Stefanos Zafeiriou. 2019. Meshgan: Non-linear 3d morphable models of faces. arXiv preprint arXiv:1903.10384 (2019)

arXiv 2019