MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation

Dan Xu; Yucheng Wang

REVIEW 4 major objections 4 minor 54 references

MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MoDiT achieves better same-identity lip sync, reduced jitter, and more natural blinks in talking-head videos by conditioning a diffusion transformer on 3DMM expression coefficients.

desk verdict A genuinely assembled diffusion-transformer system with a plausible blink module, but its headline lip-sync claim is contradicted by its own Table 1 because Wav2Lip — which is also inside the renderer — scores higher on both LSE metrics. read the letter →

arxiv 2507.05092 v1 pith:J3XRWPSO submitted 2025-07-07 cs.CV

Yucheng Wang , Dan Xu This is my paper

classification cs.CV

keywords talkingheadgeneration3Dmorphablemodeldiffusiontransformerlipsynchronizationtemporalconsistencyopticalflowpredictionblinkmodeling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a diffusion transformer conditioned on 3D Morphable Model (3DMM) expression coefficients can generate audio-driven talking-head videos that are more temporally consistent, identity-preserving, and natural in blinking than existing GAN- and UNet-based methods. The core idea is that explicit 3D coefficients give the renderer spatial constraints, while a hierarchical denoising strategy with biased attention and revised temporal attention reduces frame-to-frame jitter. The authors report the best same-identity lip-sync scores on the HDTF and VFHQ datasets and SOTA-comparable results for cross-identity and out-of-distribution inputs. If true, the work would strengthen the case for using explicit 3D representations inside diffusion models for face animation.

What carries the argument

The central object is the sequence of 3DMM expression coefficients $\beta$, which acts as the learned motion representation that the diffusion transformer directly denoises. The key mechanism is the bias injection strategy (Algorithm 1), which switches attention bias between a diagonal lip-focused mask at high noise and a dispersed expression mask at low noise, forcing the model to first lock onto lip motion and then refine full-face coherence. This, together with the revised temporal attention ($\tilde{s}_t = s_t + f(z_t)$), carries the temporal-consistency claim; the 3DMM coefficients carry the identity-preservation claim by providing explicit geometry to the optical-flow renderer.

What would settle it

Measure MoDiT's LSE-C/LSE-D on a fixed audio-visual test set while replacing the Wav2Lip reference with a deliberately mismatched or blanked lip region; if scores do not drop substantially, the renderer is not actually relying on the reference as described, undercutting the claim that the pipeline inherits Wav2Lip's accuracy.

Watch

Extended reading notes

Core claim

MoDiT represents the 3D face as a sequence of expression coefficients $\beta \in \mathbb{R}^{64}$ from the LSFM morphable model, and a diffusion transformer denoises this sequence conditioned on audio latents and the source image. A bias injection strategy applies a diagonal lip-focused attention bias during early high-noise denoising steps and a dispersed expression bias in later steps, while a revised temporal attention adds a learned latent variable to the attention scores. The denoised coefficients go through a mapping net that predicts head pose and eye blinks using a learnable blink sequence, and a renderer combines a Wav2Lip reference, a Reference UNet, and 3DMM-guided flow prediction to warp the source image. The paper claims this yields the best LSE-C/LSE-D lip-sync metrics on same-identity tests on HDTF and VFHQ and SOTA-comparable results otherwise.

Load-bearing premise

The load-bearing premise is that the Wav2Lip reference provides an accurate enough lip-sync start that the renderer's optical flow can refine it; if that reference is misaligned, the entire pipeline inherits the misalignment, so the claimed lip-sync improvement may in part come from Wav2Lip rather than from MoDiT's 3DMM motion prediction.

Editorial extensions

If this is right

If the claim holds, audio-driven talking heads can be generated with less visible jitter, since temporal attention adaptively weights sequence context rather than treating all frames alike.
Explicit 3DMM conditioning should make the framework controllable: practitioners could edit the expression coefficients directly to change emotion or emphasis without retraining.
Because the renderer predicts optical flow from a known 3D geometry, the approach may extend more gracefully to large head poses than latent-only diffusion methods.
The learnable blink sequence provides a template for modeling other subtle, fast facial motions that deterministic regression tends to smooth away.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper leaves implicit: feeding the renderer a deliberately corrupted Wav2Lip reference should degrade lip-sync metrics if the pipeline truly depends on that reference; the degree of degradation would disentangle MoDiT's contribution from Wav2Lip's.
The bias injection schedule suggests that attention masking could be time-tuned in other diffusion tasks where early denoising should focus on a small but semantically critical region, such as image inpainting of a face.
The revised temporal attention with a learned latent variable may transfer to non-face motion generation, where sequence-level context matters but full attention is too expensive.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a real system, not a toy, and the 3DMM-conditioned diffusion transformer with biased attention phases is a sensible design. But the paper's central sentence — "best lip-sync performance across the Same-Identity" — is false as printed. Table 1 lists Wav2Lip at LSE-C 9.652 / LSE-D 5.656 and MoDiT at 7.428 / 7.645 on HDTF; Wav2Lip wins on both. Same pattern on VFHQ. And Wav2Lip is not just a baseline: Section 3.4.5 feeds Wav2Lip's output into the Reference UNet. So the claim is internally contradicted, and the margin over Hallo/DreamTalk may be inherited from the Wav2Lip reference rather than produced by MoDiT's own motion prediction.

What's genuinely new: the hierarchical bias injection that switches from diagonal lip-focused attention to dispersed expression attention across diffusion phases, and the learned temporal attention offset. The idea of predicting 3DMM coefficients with a diffusion transformer and then using those to guide flow-based warping is a reasonable combination. The blink module with a learnable blink sequence is a small but sensible improvement over binary blink control. The ablations in Tables 3 and 4, inconsistent as they are, do show the components matter: removing 3DMM or temporal attention or the renderer reference degrades performance substantially. So there is real content here.

Soft spots, in order of severity. First, the Table 1 contradiction. Either Wav2Lip should be excluded from the "best" claim with an explicit caveat, or the claim should be reworded to "best among one-shot talking head methods." Second, the Wav2Lip-in-the-loop circularity: the evaluation uses SyncNet-family metrics (LSE-C/LSE-D) and the pipeline consumes Wav2Lip's output, so the reported lip-sync numbers partly reflect Wav2Lip's own optimization. The right ablation would toggle the Wav2Lip reference while keeping the Reference UNet; removing the whole unit doesn't separate the effects. Third, Table 3 and Table 4 report different values for "w/o 3DMM source β0" (5.365/9.431 vs 1.478/13.596), and the text says removing it gives "low LSE-D, high LSE-C" — the opposite of what the tables show. Fourth, no error bars, small training set (1k videos, 100 subjects), and FID is worse than several baselines, so the "state-of-the-art comparable" phrasing should be softened.

Who it's for: people building diffusion-based talking head systems will find the bias injection and blink ideas worth trying. The evaluation needs major rework before publication, and the authors should release code. It deserves a serious referee, but I would not accept the current claims as written. If I were handling it, I'd send to review with a request for corrected tables, the Wav2Lip-reference ablation, and reworded claims.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes MoDiT, a talking-head generation method that combines 3DMM expression coefficients with a diffusion transformer. It introduces a hierarchical denoising strategy with biased self/cross-attention, a revised temporal attention with a learned latent modifier, a blink module with a learnable blink sequence, and a renderer that uses Wav2Lip's output as a reference for a U-Net that predicts optical flow. Training uses an expression diffusion network with noise, velocity, lip-reading, and landmark losses. Evaluations on HDTF, VFHQ, and VoxCeleb-HQ report LSE-C/LSE-D, FID, LMD, AUE, and inference time, with claims of best same-identity lip-sync and SOTA-comparable consistency.

Significance. If the claims were supported, MoDiT would make a meaningful contribution: it injects explicit 3DMM spatial constraints into a diffusion-transformer backbone, introduces a curriculum-like bias strategy for lip focus, and proposes a learned temporal attention modification and blink modeling. The paper also provides an out-of-distribution evaluation and several external metrics (FID, LMD, AUE) that are not tied to Wav2Lip. However, the central lip-sync claim is undermined by the inclusion of Wav2Lip in Table 1 with better LSE scores, by the use of Wav2Lip as an input to the renderer, and by conflicting ablation values between Table 3 and Table 4. The contribution is therefore plausible but not demonstrated as claimed.

major comments (4)

[Table 1 / Section 4.1] The claim that MoDiT demonstrates the best lip-sync performance across Same-Identity on both datasets is contradicted by the table as printed: for HDTF Same-Identity, Wav2Lip achieves LSE-C 9.652 / LSE-D 5.656 versus MoDiT's 7.428 / 7.645, and for VFHQ Same-Identity Wav2Lip achieves 7.982 / 6.964 versus 6.205 / 7.437. Since Section 3.4.5 makes Wav2Lip's output an input to the Reference UNet of the renderer, the reported LSE margins over the other methods may reflect an injected lip-sync-optimized reference rather than the proposed 3DMM diffusion transformer. Please remove Wav2Lip from the comparison or provide a direct same-protocol comparison showing MoDiT outperforms it without relying on Wav2Lip as a reference.
[Section 4.2.2 / Table 4] This ablation removes the entire Reference UNet, but Section 3.4.5 says Wav2Lip's output is the input to the Reference UNet. The experiment therefore cannot separate the contribution of the proposed flow/3DMM pipeline from the contribution of the Wav2Lip reference itself. In addition, the same ablation row (w/o 3DMM source β0) reports LSE-C 5.365 / LSE-D 9.431 in Table 3 but LSE-C 1.478 / LSE-D 13.596 in Table 4; both cannot be correct for the same configuration. These discrepancies need to be reconciled before the renderer ablation can support the claim.
[Section 4.2.1 / Table 3] The sentence 'Removing the 3DMM coefficient constraints β0 significantly reduced spatial coherence and affected lip synchronization (low LSE-D, high LSE-C)' reverses the direction of the metrics: for the w/o 3DMM row, LSE-C drops from 7.428 to 5.365 and LSE-D rises from 7.645 to 9.431, i.e., the model is less confident and more distant, so the parenthetical should read 'low LSE-C, high LSE-D'. As written, the text reports the opposite of the tabulated result.
[Sections 3.5 and 4 / jitter claim] The paper's central motivation is reducing temporal jittering, but no temporal consistency metric (e.g., warping error, VTM, or frame-to-frame landmark stability) is reported; Table 1 reports only LSE-C/LSE-D and FID, and Table 3's temporal-attention ablation reports only LSE-C/LSE-D with the text saying it 'slightly reduced frame-to-frame consistency' without a measured quantity. Given that training is done on 12-frame clips and evaluation is on 8-second videos, the jitter-reduction claim is currently not directly supported. Please add a temporal metric or quantify the consistency gain.

minor comments (4)

[Table 2] The table has formatting errors and mismatched references: entries such as '4min27sDreamTalk' and '22.52.33' are merged, and the citations DreamTalk [27], SadTalker [51], and Hallo [47] do not match the reference list numbering used elsewhere.
[Section 3.4.1] The text refers to 'the bias injection strategy (stated in 3.3.2)', but the subsection describing bias injection is 3.4.2.
[Algorithm 1] Line 10 defines xt as a fresh sample from N(sqrt(1-beta_{t-1}) x_{t-1}, beta_{t-1} I) after x_{t-1} has already been computed in line 9; this is notationally confusing and should be clarified or removed.
[Throughout] The repository link in the abstract is missing, and there are inconsistent capitalization/spelling variants such as 'Aniportrait' vs. 'AniPortrait', 'Hubert-base-ls960' vs. 'Hubert-Base-LS960', and 'V oxCeleb-HQ' vs. 'VoxCeleb-HQ'.

Circularity Check

1 steps flagged · score 5.0 of 10

Lip-sync evaluation is self-referential: Wav2Lip is both a renderer input and the source of the LSE metric, while Table 1 contradicts the 'best lip-sync' claim.

self definitional [Section 3.4.5 (Renderer) and Section 4 (Evaluation Metrics)]
"Initially, the Wav2Lip [28] module aligns lip movements with the input audio, producing a synchronized reference. While Wav2Lip [28] serves as the input to the Reference UNet ... We evaluate the perceptual differences of the mouth shape from Wav2Lip [28], including the distance score (LSE-D) and confidence score (LSE-C)."

MoDiT's renderer is conditioned on Wav2Lip's lip-synced video (Section 3.4.5), so the generated mouth inherits Wav2Lip's synchronization. The reported lip-sync metrics, LSE-C and LSE-D, are Wav2Lip's own SyncNet scores. The 'best lip-sync performance across Same-Identity' claim is therefore a measurement of how closely the output follows the Wav2Lip reference, not an independent property of the 3DMM diffusion transformer. The comparison baselines (Hallo, SadTalker, AniPortrait, MakeItTalk) do not receive Wav2Lip as input, so the reported margin is confounded. Table 4's 'w/o Reference Unet' ablation removes the whole UNet, not just the Wav2Lip reference, so it cannot separate Wav2Lip's contribution.

full rationale

The derivation chain is mostly self-contained: 3DMM coefficients come from an external reconstruction network, the diffusion transformer denoises expression parameters with standard noise/velocity losses plus external lip-reading and landmark losses, and the renderer warps frames using predicted flow. The central circularity is confined to the lip-sync evaluation. Section 3.4.5 makes Wav2Lip's output an explicit input to the Reference UNet, and Section 4 defines the LSE-C/LSE-D metrics as coming from Wav2Lip's lip-sync expert. The headline claim of 'best lip-sync performance across the Same-Identity' is therefore not an independent measurement of MoDiT's 3DMM motion prediction: methods like Hallo, SadTalker, and AniPortrait do not receive Wav2Lip as input, while MoDiT does. The ablation in Table 4 removes the entire Reference UNet, not the Wav2Lip reference alone, so the improvement attributed to the renderer cannot separate Wav2Lip's contribution. Moreover, Table 1 itself lists Wav2Lip with higher LSE-C and lower LSE-D than Ours in Same-Identity on both HDTF and VFHQ, which contradicts the 'best lip-sync' sentence. These issues make the lip-sync claim partially self-referential, but the paper does contain independent evidence (FID, LMD, AUE, qualitative blink comparisons) and does not rely on self-citations for its core architecture, so the circularity is partial rather than total.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on standard diffusion theory, the LSFM 3DMM basis, several pre-trained models (Wav2Lip, HuBERT, 3D face reconstruction, lip-reading), and training on 12-frame clips from 100 subjects. The paper contributes a new architecture and training scheme on top of these, but does not independently validate the upstream models.

free parameters (5)

learned latent variable z_t in revised temporal attention = not reported
Introduced in Eq. 6 to adjust attention scores via an MLP; learned from the training data without a stated prior or constraint, and it is one of the claimed sources of temporal consistency improvement.
learnable blink sequence = not reported
The Blink Module (Section 3.4.4) employs a learnable blink sequence to determine eye closure per frame; it is pre-trained on a cropped eye dataset and then refined, so the final blink behavior is fitted to data and contributes to the reported blink realism.
loss weights lambda_t, lambda_read, lambda_lks, lambda_v = 10, 0.2, 0.1, 0.1
Hand-chosen weighting of noise prediction, lip-reading, landmark, and velocity losses in Eq. 8; no sensitivity analysis is provided, so the central results depend on these ad hoc values.
diagonal bias M_D and dispersed expression bias M_E = not reported
Bias matrices in Algorithm 1 are described qualitatively but their entries are not specified; the phase-dependent bias injection is a core contribution and its exact form is left open.
bias threshold t_T = not reported
Algorithm 1 switches from lip-focused diagonal bias to dispersed bias at threshold t_T; the value is not reported, so the hierarchical denoising schedule is under-specified.

assumptions (5)

standard math The DDPM/DDIM diffusion update and noise-prediction objective (Eqs. 2-3) correctly model the data distribution.
The method relies on standard diffusion model theory; no formal verification is provided, but this is a widely used framework.
domain assumption The LSFM 3DMM expression basis U_exp and identity basis U_id are orthonormal and span the relevant face variation (Eq. 1).
The intermediate representation is 3DMM expression coefficients; the quality of generated motion depends on the morphable model's coverage and the accuracy of the pre-trained reconstruction network [5].
domain assumption Wav2Lip provides a sufficiently reliable lip-sync reference for the renderer.
Section 3.4.5 uses Wav2Lip output as the reference input; if Wav2Lip's lip region is inaccurate, the final video inherits the error.
domain assumption The pre-trained lip-reading model [24] yields gradients that improve lip sync and correlates with perceptual quality.
The training loss L_read in Eq. 8 assumes the lip-reading model's outputs are a valid training signal for mouth shape; this is not independently validated in the paper.
ad hoc to paper Training on 12-frame clips from 100 subjects generalizes to long videos and out-of-distribution datasets.
The model is trained on continuous 12 frames (Section 4) but is evaluated on 8-second videos and an OOD dataset; the paper does not analyze cross-window consistency or training set sufficiency.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation." pith.science (2026). https://pith.science/paper/J3XRWPSO

@misc{pith2026250705092,
  author       = {Pith},
  title        = {Pith review of: MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J3XRWPSO}},
  note         = {Machine review of arXiv:2507.05092}
}

read the original abstract

Audio-driven talking head generation is critical for applications such as virtual assistants, video games, and films, where natural lip movements are essential. Despite progress in this field, challenges remain in producing both consistent and realistic facial animations. Existing methods, often based on GANs or UNet-based diffusion models, face three major limitations: (i) temporal jittering caused by weak temporal constraints, resulting in frame inconsistencies; (ii) identity drift due to insufficient 3D information extraction, leading to poor preservation of facial identity; and (iii) unnatural blinking behavior due to inadequate modeling of realistic blink dynamics. To address these issues, we propose MoDiT, a novel framework that combines the 3D Morphable Model (3DMM) with a Diffusion-based Transformer. Our contributions include: (i) A hierarchical denoising strategy with revised temporal attention and biased self/cross-attention mechanisms, enabling the model to refine lip synchronization and progressively enhance full-face coherence, effectively mitigating temporal jittering. (ii) The integration of 3DMM coefficients to provide explicit spatial constraints, ensuring accurate 3D-informed optical flow prediction and improved lip synchronization using Wav2Lip results, thereby preserving identity consistency. (iii) A refined blinking strategy to model natural eye movements, with smoother and more realistic blinking behaviors.

Figures

Figures reproduced from arXiv: 2507.05092 by the authors.

**Figure 2.** Overview of the Diffusion Transformer Pipeline, showcasing the denoising stages with temporal and spatial condition injection. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Illustration of structure details of the Transformer Block [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: Structure of the two-step renderer pipeline. We use [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Comparison of lip-syncing models: Visual results for Wav2Lip [ [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: This figure visualizes the attention heatmap shifts over [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 9.** Figure 9: Ablation study on Eye Blink in the source image, maxi [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

**Figure 8.** Figure 8: The figure is the ablation study on rendering, comparing [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 40 canonical work pages

[1]

A morphable model for the synthesis of 3d faces

V olker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In ACM SIGGRAPH, 1999. 1, 2, 3

work page 1999
[2]

Hierarchical cross-modal talking face generation with dynamic pixel-wise loss

Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In CVPR, 2019. 2

work page 2019
[3]

You said that?

Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? arXiv preprint arXiv:1705.02966, 2017. 2

work page Pith review arXiv 2017
[4]

Lip reading sentences in the wild

Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In CVPR,

work page
[5]

Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set

Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In CVPR Workshops, 2019. 7

work page 2019
[6]

Dae-talker: High fidelity speech-driven talking face generation with diffusion autoen- coder

Chenpeng Du, Qi Chen, Tianyu He, Xu Tan, Xie Chen, Kai Yu, Sheng Zhao, and Jiang Bian. Dae-talker: High fidelity speech-driven talking face generation with diffusion autoen- coder. In ACM MM, 2023. 1

work page 2023
[7]

Generative adversarial networks

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139–144, 2020. 2

2020
[8]

Ad-nerf: Audio driven neural radi- ance fields for talking head synthesis

Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radi- ance fields for talking head synthesis. In ICCV, 2021. 2

work page 2021

Show all 54 references

[9]

Facexhubert: Text-less speech-driven e (x) pressive 3d facial animation synthesis using self-supervised speech representation learn- ing

Kazi Injamamul Haque and Zerrin Yumak. Facexhubert: Text-less speech-driven e (x) pressive 3d facial animation synthesis using self-supervised speech representation learn- ing. In ICMI, 2023. 7

2023
[10]

Gaia: Zero-shot talking avatar generation

Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, and Han Hu. Gaia: Zero-shot talking avatar generation. arXiv preprint arXiv:2311.15230, 2023. 2

2023 arXiv
[11]

Gans trained by a two time-scale update rule converge to a local nash equilib- rium

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In NeurIPS, 2017. 7

2017
[12]

Implicit identity representation conditioned memory compensation network for talking head video generation

Fa-Ting Hong and Dan Xu. Implicit identity representation conditioned memory compensation network for talking head video generation. In ICCV, 2023. 2

2023
[13]

Dagan++: Depth- aware generative adversarial network for talking head video generation

Fa-Ting Hong, Li Shen, and Dan Xu. Dagan++: Depth- aware generative adversarial network for talking head video generation. arXiv preprint arXiv:2305.06225, 2023

2023 arXiv
[14]

Depth-aware generative adversarial network for talking head video generation

Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In CVPR, 2023. 2

2023
[15]

Audio-visual con- trolled video diffusion with masked selective state spaces modeling for natural talking head generation

Fa-Ting Hong, Zunnan Xu, Zixiang Zhou, Jun Zhou, Xiu Li, Qin Lin, Qinglin Lu, and Dan Xu. Audio-visual con- trolled video diffusion with masked selective state spaces modeling for natural talking head generation. arXiv preprint arXiv:2504.02542, 2025. 1, 3

2025 arXiv
[17]

Audio-driven emotional video portraits

Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio-driven emotional video portraits. In CVPR, 2021. 2

2021
[18]

Transformers in vision: A survey

Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. arXiv, 2021. 3, 4

2021
[19]

Deep video portraits

Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. Deep video portraits. TOG, 2018. 2

2018
[20]

Auto-encoding varia- tional bayes

Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. CoRR, abs/1312.6114, 2014. 2

2014 arXiv
[21]

Anitalker: Animate vivid and di- verse talking faces through identity-decoupled facial motion encoding

Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, and Kai Yu. Anitalker: Animate vivid and di- verse talking faces through identity-decoupled facial motion encoding. arXiv preprint arXiv:2405.03121, 2024. 2

2024 arXiv
[22]

Moda: Mapping-once audio-driven portrait animation with dual attentions

Yunfei Liu, Lijian Lin, Fei Yu, Changyin Zhou, and Yu Li. Moda: Mapping-once audio-driven portrait animation with dual attentions. In ICCV, 2023. 2

2023
[23]

Live speech por- traits: real-time photorealistic talking-head animation

Yuanxun Lu, Jinxiang Chai, and Xun Cao. Live speech por- traits: real-time photorealistic talking-head animation. TOG,
[24]

Training strategies for improved lip- reading

Pingchuan Ma, Yujiang Wang, Stavros Petridis, Jie Shen, and Maja Pantic. Training strategies for improved lip- reading. In ICASSP, 2022. 5

2022
[25]

Dreamtalk: When expressive talking head generation meets diffusion probabilistic models

Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767, 2023. 1, 2, 6

2023 arXiv
[26]

Diffspeaker: Speech-driven 3d facial animation with diffusion transformer

Zhiyuan Ma, Xiangyu Zhu, Guojun Qi, Chen Qian, Zhaox- iang Zhang, and Zhen Lei. Diffspeaker: Speech-driven 3d facial animation with diffusion transformer. arXiv preprint arXiv:2402.05712, 2024. 4

2024 arXiv
[27]

Librispeech: An asr corpus based on public do- main audio books

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public do- main audio books. In ICASSP, 2015. 7

2015
[28]

A lip sync expert is all you need for speech to lip generation in the wild

K R Prajwal, Rudrabha Mukhopadhyay, Vinay P.Namboodiri, and C.V .Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. InACM MM, 2020. 2, 5, 6, 7, 8

2020
[29]

High-resolution image syn- thesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 1

2022
[30]

pytorch-fid: FID Score for PyTorch

Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https://github.com/mseitzer/pytorch-fid ,
[31]

Difftalk: Crafting diffusion models for generalized audio-driven portraits animation

Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In CVPR, 2023. 1

2023
[32]

First order motion model for image animation

Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa ACM Transactions on Graphics, and Nicu Sebe. First order motion model for image animation. In NeurIPS, 2019. 2, 6, 7

2019
[33]

Denoising diffusion implicit models

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 3 9

2010 arXiv
[34]

Talking face generation by conditional recur- rent adversarial network

Yang Song, Jingwen Zhu, Dawei Li, Xiaolong Wang, and Hairong Qi. Talking face generation by conditional recur- rent adversarial network. arXiv preprint arXiv:1804.04786,

arXiv
[35]

Diffused heads: Diffusion models beat gans on talking-face genera- tion

Michał Stypułkowski, Konstantinos V ougioukas, Sen He, Maciej Zi˛ eba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face genera- tion. In WACV, 2024. 2

2024
[36]

Vividtalk: One-shot audio-driven talk- ing head generation based on 3d hybrid prior

Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Daiheng Gao, Liefeng Bo, and Xun Cao. Vividtalk: One-shot audio-driven talk- ing head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.01841, 2023. 2

2023 arXiv
[37]

Masked lip-sync prediction by audio-visual contextual exploitation in transformers

Yasheng Sun, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Zhibin Hong, Jingtuo Liu, Errui Ding, Jingdong Wang, Zi- wei Liu, and Koike Hideki. Masked lip-sync prediction by audio-visual contextual exploitation in transformers. In SIG- GRAPH Asia, 2022. 2

2022
[38]

Synthesizing obama: learn- ing lip sync from audio

Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learn- ing lip sync from audio. TOG, 2017. 2

2017
[39]

Human-centric founda- tion models: Perception, generation and agentic modeling

Shixiang Tang, Yizhou Wang, Lu Chen, Yuan Wang, Sida Peng, Dan Xu, and Wanli Ouyang. Human-centric founda- tion models: Perception, generation and agentic modeling. In IJCAI, 2025. 1

2025
[40]

Neural voice puppetry: Audio-driven facial reenactment

Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. In ECCV, 2020. 2

2020
[41]

Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485, 2024. 2

2024 arXiv
[42]

Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan

Liangbin Xie. Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution, 2022. 2

2022
[43]

Aniportrait: Audio-driven synthesis of photorealistic portrait animation

Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024. 2, 6, 7

2024 arXiv
[44]

Photorealistic audio-driven video portraits

Xin Wen, Miao Wang, Christian Richardt, Ze-Yin Chen, and Shi-Min Hu. Photorealistic audio-driven video portraits. TVCG, 26(12):3457–3466, 2020. 2

2020
[45]

Monocular depth estimation using multi-scale continuous crfs as sequential deep networks

Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, Nicu Sebe, et al. Monocular depth estimation using multi-scale continuous crfs as sequential deep networks. TPAMI, 41(6): 1426–1440, 2018. 2

2018
[46]

Hallo: Hierarchical audio-driven visual synthesis for portrait image animation

Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 1, 2, 6, 7, 8

2024 arXiv
[47]

Jointly attentive spatial-temporal pooling networks for video-based person re-identification

Shuangjie Xu, Yu Cheng, Kang Gu, Yang Yang, Shiyu Chang, and Pan Zhou. Jointly attentive spatial-temporal pooling networks for video-based person re-identification. In ICCV, 2017. 4

2017
[48]

Vasa-1: Lifelike audio-driven talking faces generated in real time

Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404.10667 ,

arXiv
[49]

Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In CVPR, 2023. 1, 2, 5, 6, 7, 8

2023
[50]

Flow-guided one-shot talking face generation with a high- resolution audio-visual dataset

Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high- resolution audio-visual dataset. In CVPR, 2021. 2

2021
[51]

Thin-plate spline motion model for image animation

Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In CVPR, 2022. 2

2022
[52]

Synergizing motion and appearance: Multi-scale com- pensatory codebooks for talking head video generation

Shuling Zhao, Fa-Ting Hong, Xiaoshui Huang, and Dan Xu. Synergizing motion and appearance: Multi-scale com- pensatory codebooks for talking head video generation. In CVPR, 2025. 2

2025
[53]

Talking face generation by adversarially disentangled audio-visual representation

Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially disentangled audio-visual representation. In AAAI, 2019. 2

2019
[54]

Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation

Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation. In CVPR, 2021. 1

2021
[55]

Makelttalk: speaker-aware talking-head animation

Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevar- ria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. TOG, 2020. 6, 7 10

2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

A morphable model for the synthesis of 3d faces

V olker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In ACM SIGGRAPH, 1999. 1, 2, 3

work page 1999

[2] [2]

Hierarchical cross-modal talking face generation with dynamic pixel-wise loss

Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In CVPR, 2019. 2

work page 2019

[3] [3]

You said that?

Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? arXiv preprint arXiv:1705.02966, 2017. 2

work page Pith review arXiv 2017

[4] [4]

Lip reading sentences in the wild

Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In CVPR,

work page

[5] [5]

Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set

Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In CVPR Workshops, 2019. 7

work page 2019

[6] [6]

Dae-talker: High fidelity speech-driven talking face generation with diffusion autoen- coder

Chenpeng Du, Qi Chen, Tianyu He, Xu Tan, Xie Chen, Kai Yu, Sheng Zhao, and Jiang Bian. Dae-talker: High fidelity speech-driven talking face generation with diffusion autoen- coder. In ACM MM, 2023. 1

work page 2023

[7] [7]

Generative adversarial networks

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139–144, 2020. 2

2020

[8] [8]

Ad-nerf: Audio driven neural radi- ance fields for talking head synthesis

Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radi- ance fields for talking head synthesis. In ICCV, 2021. 2

work page 2021