Pith. sign in

REVIEW 4 major objections 6 minor 53 references

GemTalk uses geometric blendshape priors to give audio-driven talking faces continuous, photorealistic emotion intensity control.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:58 UTC pith:S3ITURIT

load-bearing objection GEM is a genuinely new modulation idea and the ablations support it, but the D-GPG blendshape predictor is a black box with no validated ground truth, so the 'physical prior' and controllability claims rest on sand. the 4 major comments →

arxiv 2608.00663 v1 pith:S3ITURIT submitted 2026-08-01 cs.CV

Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation

Chenggong Hu , Shaoyin Ma , Yi Wang , Li Sun , Mingli Song , Jie Song This is my paper
classification cs.CV
keywords emotional talking face generationdiffusion modelblendshape coefficientsemotion intensity controlaudio-driven animationgeometric priorscross-attention magnitudeimplicit-explicit fusion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

GemTalk tries to resolve a trade-off in audio-driven emotional talking-face generation: implicit representations capture rich emotional semantics but cannot control how strong an expression is, while explicit geometric methods control expression but lose high-frequency texture. The paper's central move is to keep both—an implicit audio-to-visual emotion projection supplies emotional direction, and a diffusion-based generator predicts identity-aware blendshape coefficients that supply physical intensity. A Geometry-guided Emotion Modulation (GEM) module uses those coefficients to rescale the magnitude of the implicit features, leaving their semantic direction intact. If the approach works as claimed, a talking-face model can dial emotion intensity continuously—from a subtle smile to a broad laugh—while preserving photorealism, lip sync, and identity. On an emotional test set the system reports the highest emotion accuracy among compared methods and the lowest video distance.

Core claim

The central claim is that implicit emotion features in diffusion-based talking-face generation carry the right semantic direction (which emotion) but an ungrounded magnitude (how strong), because contrastive objectives normalize away vector length; GEM repairs this by injecting explicit geometric priors. Concretely, GEM L2-normalizes the implicit audio features onto a hypersphere, uses them as queries to retrieve muscle-activation patterns from predicted identity-aware blendshape coefficients, and applies a zero-initialized affine adapter that scales and shifts the normalized features while a directional-consistency loss keeps the emotion direction unchanged. During inference the blendshape

What carries the argument

The load-bearing object is the Geometry-guided Emotion Modulation (GEM) module, built on the geometric reading of cross-attention scores as ||Q||·||K||·cosθ. The direction term carries emotion category; the magnitude term carries physical intensity. GEM consists of two parts: a Geometry-aware Context Aggregation (GCA) that uses spherically normalized implicit features as queries over explicit blendshape tokens, and a Geometric Affine Adapter (GAA) that projects the retrieved context into per-token scale and shift parameters with zero initialization and a directional-consistency loss. The blendshape coefficients—45 retained muscle-activation dimensions generated from audio plus identity by a

Load-bearing premise

The framework's load-bearing assumption is that D-GPG's predicted identity-aware blendshape coefficients accurately track the true facial deformation of the target identity; if those predictions are imprecise, GEM rescales the implicit features by the wrong physical magnitudes and the claimed intensity-control mechanism loses its grounding.

What would settle it

Compute per-dimension error between D-GPG's predicted blendshape coefficients and ground-truth ARKit coefficients on held-out MEAD/RAVDESS clips; if the mean error is large (say per-dimension > 0.1) while generated emotion accuracy remains high, the physical-prior explanation is undermined. Alternatively, swap predicted blendshapes for ground-truth coefficients at inference: if emotion accuracy and intensity control do not improve, the geometric prior is not carrying the load.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Emotion intensity becomes a continuous, interpretable control: editing the top-10 blendshape coefficients tied to the target emotion produces smooth transitions from subtle to intense expressions without identity loss.
  • The decoupling of direction and magnitude suggests the same implicit feature can be re-anchored to physical deformation energy, reducing the 'average face' problem in emotion-driven generation.
  • Because the pipeline is pure audio-driven and needs only a reference image and audio, intensity control does not require driving video, discrete labels, or per-identity retraining.
  • Conflict-aware training with mismatched reference-audio emotion pairs suppresses reference emotion leakage, so the generated emotion follows the audio even when the reference face shows the opposite affect.
  • The reported gains in emotion accuracy and lower video distance on an aggregated emotional benchmark would extend to out-of-domain clips if the qualitative comparisons hold.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct error measurement of D-GPG's predicted blendshapes against ground-truth ARKit coefficients would settle whether the physical-prior story is doing the work; the paper doesn't report such a check, so the mechanism rests on an unvalidated premise.
  • The same magnitude/direction split could be applied to other conditional diffusion tasks needing intensity control—gesture amplitude, gaze strength, or body motion—where cosine-trained features would suffer the same radial invariance.
  • The intensity knob as described scales only blendshape magnitudes, not the emotion category; a testable extension is whether the same GEM adapter can interpolate between emotion clusters (e.g., happy-to-surprised) by moving direction continuously.
  • An ablation replacing predicted blendshapes with ground-truth coefficients (or with a simple low-rank conditioning vector) would reveal whether the geometric interpretability or merely the extra conditioning improves intensity control.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GemTalk, a diffusion-based framework for audio-driven emotional talking face generation that combines implicit emotional representations (V-AEP, LECM) with explicit geometric priors from ARKit-style blendshape coefficients (D-GPG). A Geometry-guided Emotion Modulation (GEM) module uses predicted blendshape coefficients to recalibrate the magnitude of implicit audio-visual features before cross-attention injection, aiming to enable continuous, interpretable control of emotion intensity without degrading visual quality. The method is evaluated on HDTF and MEAD+RAVDESS, reporting state-of-the-art emotion accuracy (Acc_emo=59.258%) and competitive FVD, plus ablations and a user study.

Significance. If the claims are substantiated, the work is a useful step toward controllable yet photorealistic emotional talking-face generation: it proposes a principled way to inject physically interpretable geometry into a diffusion model without sacrificing implicit-feature semantics. The use of explicit blendshape magnitude to modulate implicit features is a plausible and interesting design, and the three-stage training strategy, conflict-aware sampling, and directional-consistency loss are thoughtful engineering contributions. However, the load-bearing evidence for the geometric-prior story and the central intensity-control capability is currently missing, and the empirical claims lack uncertainty quantification. The paper is therefore of potential significance but requires substantial additional validation.

major comments (4)
  1. [§3.2, Eq. (3), and Appendix A] D-GPG is trained with a denoising objective on blendshape coefficients g, but the manuscript never states where ground-truth blendshape coefficients come from (e.g., which ARKit tracker or 3DMM fitter was used), nor does it report any quantitative validation of the predicted coefficients against ground truth. The Appendix, cited for 'Details about blendshape and D-GPG', only provides statistics and visualizations of the 45 retained dimensions; it does not describe D-GPG's architecture, training data, or accuracy. If D-GPG produces inaccurate or arbitrary coefficients, the 'physical prior' story in §3.3 is vacuous and the continuous-intensity-control mechanism is ungrounded. Please specify the generation of training targets, the network architecture, and report per-dimension coefficient error (e.g., MSE/MAE, correlation) on held-out identities.
  2. [§4.3, 'Fine-grained Controllability', and Fig. 4(a)] The paper's headline contribution is 'precise, continuous control over emotional expressions, especially emotion intensity' (Abstract §3.3). Yet the only evaluation is a set of qualitative frames showing a smile becoming broader. There is no quantitative metric for monotonicity, smoothness, or perceived intensity change as blendshape coefficients are scaled, and no comparison against simpler implicit-feature interpolation or against other controllable methods. Please add a quantitative evaluation (e.g., Acc_emo or Emotion-Fan score vs. scaling factor, with a monotonicity check; a user study asking raters to order output by intensity) to demonstrate that the geometric knob actually provides continuous control.
  3. [Tables 1 and 2] All reported metrics are point estimates with no error bars, no multiple seeds, and no statistical significance tests. Some gains are modest (e.g., GemTalk FVD 216.138 vs. Sonic 175.562 on HDTF is actually worse; on MEAD+RAVDESS the FVD improvement over Sonic is 28.604, while Acc_emo gains are large but variances are unknown). Given that single-run FVD/FID can fluctuate, please report mean±std over at least three runs and state the number of seeds. Also, Table 2 has incorrect arrows: FID should be ↓ and Sync-C should be ↑; as printed the direction contradicts standard practice and the surrounding text.
  4. [§4.3, Table 2 (M5- vs. M5+)] The ablation shows that removing GEM entirely reduces Acc_emo (M4 vs. M5+), but it does not establish that the specific predicted geometric priors are responsible. The GEM module adds parameters and a conditioning pathway, so the gain could come from extra capacity or from a generic affine modulation rather than from the physical validity of the blendshape predictions. A control experiment using random blendshape coefficients, or ground-truth coefficients (if available), in the same GEM pipeline would demonstrate that the geometric prior, not just the presence of a modulation module, is what drives the improvement. This is essential to support the 'geometry-guided' claim.
minor comments (6)
  1. [Table 1 caption] 'second raw' should be 'second row'.
  2. [§4.1, Metrics] The metric definitions for FID/FVD are standard, but the text says 'Fréchet Inception Distance (FID)' and 'Fréchet Video Distance (FVD)' - consider adding the standard references for FID [9] and FVD [34] explicitly, and clarify that lower is better for both.
  3. [Appendix A, Table 4] The threshold for discarding blendshape dimensions (max activation < 0.05) and the resulting K=45 are described, but the rationale for that particular threshold is not given. Please justify or remove the false-precision appearance.
  4. [Fig. 9 and Fig. 10 captions] 'MEDA' is a typo for 'MEAD'.
  5. [Appendix C, 'Lip Attention'; Eq. (11)] The mask M is downsampled to match intermediate feature maps; the text says positions outside the mouth region get M=-∞. This is clear, but the notation could be made more explicit (e.g., using M_lip rather than M) to avoid confusion with the mask in the blending-shape section.
  6. [References] References [23] and [24] are identical (same authors, title, venue, pages). Please deduplicate.

Circularity Check

0 steps flagged

No significant circularity: GEM edits the geometric input directly and headline metrics are external; the sole self-citation is non-load-bearing.

full rationale

I walked the derivation chain: V-AEP (Eqs. 1-2), D-GPG (Eq. 3), GEM (Eqs. 4-7), training/inference (Eq. 8), and evaluation. None of these equations assumes the headline outcomes. Acc_emo, FVD/FID, Sync, and E-FID are measured with external metrics (Emotion-Fan [26], FVD [34], etc.) on held-out identity splits of MEAD/RAVDESS and HDTF, and Table 2 ablates actual component variants, so the quantitative claims are not fitted inputs renamed as predictions. The continuous intensity demonstration in Sec. 4.3 is a direct manipulation of the D-GPG blendshape-conditioning input, i.e., an interface, not a prediction; it is true by construction but is not presented as independent evidence. The only self-citation, [10], supports the generic statement that diffusion models inject conditions via cross attention and is not load-bearing. The real weakness is evidential, not circular: D-GPG's blendshape training source and coefficient accuracy are never validated (Sec. 3.2 defers to an Appendix that only gives blendshape statistics), and Appendix F lists limitations but does not disclose this. Missing support of that kind affects correctness risk, not circularity per the review rules.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 0 invented entities

The headline mechanism adds a small number of hand-set hyperparameters and data-derived thresholds, and rests on an unverified physical premise: that blendshape coefficient norms are a valid intensity scale. No code or weights are released.

free parameters (9)
  • λ1=0.6, λ2=0.4 = 0.6 / 0.4
    Weights for visual contrastive losses; chosen by hand, no ablation reported (Appendix B, Eq. 9).
  • supervised contrastive temperature τ = not reported
    Temperature in Eq. (2)/(9); no value or sensitivity analysis is given.
  • hard-negative K = 16
    Number of hard negatives in supervised contrastive loss; fixed hyperparameter (Appendix B).
  • max visual anchors k = 4
    Maximum number of visual anchors for stochastic multi-target alignment (Appendix B).
  • conflict probability p_conf = 0.9
    Probability of mismatched reference-audio emotion pairs in conflict-aware training; no ablation (Appendix C).
  • blendshape activation threshold = 0.05
    Threshold for discarding 7 of 52 ARKit dimensions; data-derived from MEAD+RAVDESS max activations (Appendix A).
  • top-10 emotion blendshape selection = top-10 coefficients
    Coefficients selected by mean magnitude and high-activation frequency for intensity-control demo; dataset-dependent (Sec. 4.3).
  • safety clamp margin = 0.15
    Margin added to dataset coefficient extrema during intensity editing; chosen by hand (Sec. 4.3).
  • smoothing thresholds = data-driven
    Inter-frame discrepancy thresholds estimated from real video statistics; no explicit values or calibration reported (Appendix C).
axioms (6)
  • standard math Cross-attention score decomposes as ||Q||·||K||·cos θ (Eq. 12)
    Mathematical identity used as the basis for GEM; no additional proof needed.
  • domain assumption ARKit blendshape coefficient norm correlates with physical deformation energy
    Appendix D states 'a larger ||g||_2 directly corresponds to larger muscle displacements'; no calibration or evidence provided, and this premise underlies GEM.
  • domain assumption Contrastive training induces radial invariance (uninformative feature magnitude)
    Appendix D argues the cosine-similarity loss leaves magnitude free; this is a heuristic argument, not a proven property of the trained V-AEP encoder.
  • domain assumption Audio-only + identity image is sufficient to predict accurate blendshapes
    D-GPG regresses blendshape coefficients from audio and identity; no ground-truth evaluation of these predicted coefficients is reported (Sec. 3.2).
  • domain assumption Emotion labels of MEAD/RAVDESS and Emotion-Fan scores are reliable
    All training (V-AEP) and evaluation (Acc_emo) rely on emotion labels from MEAD/RAVDESS and the Emotion-Fan classifier; label noise is not analyzed.
  • ad hoc to paper The 45 retained blendshape dimensions form a complete and sufficient geometric basis
    Discarding dims with max activation <0.05 is data-specific and not justified theoretically; e.g., cheekPuff can be emotionally relevant outside the dataset.

pith-pipeline@v1.3.0-daily-deepseek · 21042 in / 15898 out tokens · 150334 ms · 2026-08-04T00:58:58.595550+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation." pith.science (2026). https://pith.science/paper/S3ITURIT

@misc{pith2026260800663,
  author       = {Pith},
  title        = {Pith review of: Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S3ITURIT}},
  note         = {Machine review of arXiv:2608.00663}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Audio-driven emotional talking face generation aims to synthesize realistic videos with expressive facial dynamics. However, existing methods struggle to balance controllability and visual fidelity. Although implicit representations capture rich semantics, they lack structural guidance, often resulting in averaged emotional expressions. In contrast, explicit geometric methods offer better control over facial expressions but tend to sacrifice high-frequency texture details. To address it, we propose GemTalk, a diffusion-based framework that combines the semantic richness of implicit representations with the structural precision of explicit geometric priors. We introduce a Vision-guided Audio Emotion Projection (V-AEP) module to extract implicit emotional lip and expression features. At the same time, a Diffusion-based Geometric Priors Generator (D-GPG) generates identity-aware blendshape coefficients as explicit structural priors. Crucially, our Geometry-guided Emotion Modulation (GEM) module leverages these geometric priors to recalibrate the magnitude of implicit features, enabling precise, continuous control over emotional expressions, especially emotion intensity, without sacrificing visual quality. Extensive experiments show GemTalk achieves superior performance in photo-realism, and facial emotional dynamics.

Figures

Figures reproduced from arXiv: 2608.00663 by Chenggong Hu, Jie Song, Li Sun, Mingli Song, Shaoyin Ma, Yi Wang.

Figure 1
Figure 1. Figure 1: Top: The explicit driving methods (left) yield unco [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Given a reference face and driving audio, our GemTalk generates high-fidelity and controllable emotional videos. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparisons with SOTA methods on out of domain data. Our method can obtain more accurate facial [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Visual results of ablation studies. (a) Visual comparisons of continuous intensity control by editing blendshape [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study of conflict-aware training strategy. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of four distinct ARKit blendshape bases [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The maximum and minimum value distribution of the remaining [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cosine similarity matrices of the projected audio features for a random batch of 16 samples. (a) shows the similarity [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative comparison with SOTA methods on MEDA with Angry expression. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative comparison with SOTA methods on HDTF with Happy expression. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Our GemTalk facial emotion generation performance on different datasets. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 10 linked inside Pith

  1. [1]

    Dan Bigioi, Shubhajit Basak, Michał Stypułkowski, Maciej Zieba, Hugh Jordan, Rachel McDonnell, and Peter Corcoran. 2024. Speech driven video editing via an audio-conditioned diffusion model.Image and Vision Computing142 (2024), 104911

    2024

  2. [2]

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. 2025. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 2403–2410

    2025

  3. [3]

    Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. 2022. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. InSIGGRAPH Asia 2022 Conference Papers. 1–9

    2022

  4. [4]

    Joon Son Chung and Andrew Zisserman. 2016. Out of time: automated lip sync in the wild. InAsian conference on computer vision. Springer, 251–263

    2016

  5. [5]

    Radek Daněček, Kiran Chhatre, Shashank Tripathi, Yandong Wen, Michael Black, and Timo Bolkart. 2023. Emotional speech-driven animation with content- emotion disentanglement. InSIGGRAPH Asia 2023 Conference Papers. 1–13

    2023

  6. [6]

    Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. 2019. Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops

    2019

  7. [7]

    Yuan Gan, Zongxin Yang, Xihang Yue, Lingyun Sun, and Yi Yang. 2023. Efficient emotional adaptation for audio-driven talking-head generation. InProceedings of the IEEE/CVF International Conference on Computer Vision. 22634–22645

    2023

  8. [8]

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. 2023. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725(2023)

    Pith/arXiv arXiv 2023

  9. [9]

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems30 (2017)

    2017

  10. [10]

    Chenggong Hu, Yi Wang, Mengqi Xue, Haofei Zhang, Jie Song, and Li Sun

  11. [11]

    Li Hu. 2024. Animate anyone: Consistent and controllable image-to-video syn- thesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8153–8163

    2024

  12. [12]

    Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. 2025. Sonic: Shifting focus to global audio perception in portrait animation. InProceedings of the Computer Vision and Pattern Recognition Conference. 193–203

    2025

  13. [13]

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. 2022. Eamm: One-shot emotional talking face via audio-based emotion- aware motion model. InACM SIGGRAPH 2022 conference proceedings. 1–10

    2022

  14. [14]

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. 2021. Audio-driven emotional video portraits. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 14080–14089

    2021

  15. [15]

    John P Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin, and Zhigang Deng. 2014. Practice and theory of blendshape facial models.Euro- graphics (State of the Art Reports)1, 8 (2014), 2

    2014

  16. [16]

    Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang, Zerong Zheng, and Yanbo Zheng. 2025. Cyberhost: A one-stage diffusion framework for audio- driven talking body generation. InThe Thirteenth International Conference on Learning Representations

    2025

  17. [17]

    Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, and Kai Yu. 2024. Anitalker: Animate vivid and diverse talking faces through identity- decoupled facial motion encoding. InProceedings of the 32nd ACM International Conference on Multimedia. 6696–6705

    2024

  18. [18]

    Yunfei Liu, Lijian Lin, Fei Yu, Changyin Zhou, and Yu Li. 2023. Moda: Mapping- once audio-driven portrait animation with dual attentions. InProceedings of the IEEE/CVF International Conference on Computer Vision. 23020–23029

    2023

  19. [19]

    Steven R Livingstone and Frank A Russo. 2018. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English.PloS one13, 5 (2018), e0196391

    2018

  20. [20]

    Yifeng Ma, Suzhen Wang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding, Zhi- dong Deng, and Xin Yu. 2023. Styletalk: One-shot talking head generation with controllable speaking styles. InProceedings of the AAAI conference on artificial intelligence, Vol. 37. 1896–1904

    2023

  21. [21]

    Soumik Mukhopadhyay, Saksham Suri, Ravi Teja Gadde, and Abhinav Shrivastava

  22. [22]

    Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. 2023. Emotalk: Speech-driven emotional disentanglement for 3d face animation. InProceedings of the IEEE/CVF international conference on computer vision. 20687–20697

    2023

  23. [24]

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar

  24. [25]

    Arnab Kumar Roy, Hemant Kumar Kathania, Adhitiya Sharma, Abhishek Dey, and Md Sarfaraj Alam Ansari. 2024. ResEmoteNet: bridging accuracy and loss reduction in facial emotion recognition.IEEE Signal Processing Letters(2024)

    2024

  25. [26]

    Elena Ryumina, Denis Dresvyanskiy, and Alexey Karpov. 2022. In search of a robust facial expressions recognition model: A large-scale visual cross-corpus study.Neurocomputing514 (2022), 435–450

    2022

  26. [27]

    Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition.arXiv preprint arXiv:1904.05862(2019)

    Pith/arXiv arXiv 2019

  27. [28]

    In Proceedings of the 28th ACM international conference on multimedia

    A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia. 484–492

  28. [29]

    Xuli Shen, Hua Cai, Dingding Yu, Weilin Shen, Qing Xu, and Xiangyang Xue

  29. [30]

    Michał Stypułkowski, Konstantinos Vougioukas, Sen He, Maciej Zięba, Stavros Petridis, and Maja Pantic. 2024. Diffused heads: Diffusion models beat gans on talking-face generation. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 5091–5100

    2024

  30. [31]

    Shuai Tan, Bin Ji, Mengxiao Bi, and Ye Pan. 2024. Edtalk: Efficient disentanglement for emotional talking head synthesis. InEuropean Conference on Computer Vision. Springer, 398–416

    2024

  31. [32]

    Fei Shen, Cong Wang, Junyao Gao, Qin Guo, Jisheng Dang, Jinhui Tang, and Tat- Seng Chua. 2025. Long-term talkingface generation via motion-prior conditional diffusion model.arXiv preprint arXiv:2502.09533(2025)

    Pith/arXiv arXiv 2025

  32. [33]

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. 2024. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. InEuropean Conference on Computer Vision. Springer, 244–260

    2024

  33. [34]

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. 2019. FVD: A new metric for video genera- tion. (2019)

    2019

  34. [35]

    Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. 2024. V-express: Conditional dropout for progressive training of portrait video generation.arXiv preprint arXiv:2406.02511 (2024)

    Pith/arXiv arXiv 2024

  35. [36]

    Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, and Baoyuan Wang. 2023. Progressive disentangled representation learning for fine-grained controllable talking head synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17979–17989

    2023

  36. [37]

    Weipeng Tan, Chuming Lin, Chengming Xu, FeiFan Xu, Xiaobin Hu, Xiaozhong Ji, Junwei Zhu, Chengjie Wang, and Yanwei Fu. 2025. Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generation. InProceedings of the 33rd ACM International Conference on Multimedia. 9987– 9995

    2025

  37. [38]

    Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. 2020. Mead: A large-scale audio- visual dataset for emotional talking-face generation. InEuropean conference on computer vision. Springer, 700–717

    2020

  38. [39]

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. 2021. One-shot free-view neu- ral talking-head synthesis for video conferencing. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10039–10049

    2021

  39. [40]

    Xu Wang, Yunpeng Jiao, Jiayi He, Yaxiong Wang, Lechao Cheng, and Shengeng Tang. 2025. Mamba-Lip: Revolutionizing Lip Reading Generation with Selective State Space Models. InInternational Conference on Artificial Intelligence, Human- Computer Interaction and Robotics (AIHCIR). IEEE, 1–6

    2025

  40. [41]

    Xu Wang, Shengeng Tang, Fei Wang, Lechao Cheng, Dan Guo, Feng Xue, and Richang Hong. 2025. Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering.arXiv preprint arXiv:2508.02362(2025). MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Chenggong Hu et al

    Pith/arXiv arXiv 2025

  41. [42]

    Haotian Wang, Yuzhe Weng, Yueyan Li, Zilu Guo, Jun Du, Shutong Niu, Jiefeng Ma, Shan He, Xiaoyan Wu, Qiming Hu, et al . 2025. Emotivetalk: Expressive talking head generation through audio information decoupling and emotional video diffusion. InProceedings of the Computer Vision and Pattern Recognition Conference. 26212–26221

    2025

  42. [43]

    You Xie, Hongyi Xu, Guoxian Song, Chao Wang, Yichun Shi, and Linjie Luo. 2024. X-portrait: Expressive portrait animation with hierarchical motion attention. In ACM SIGGRAPH 2024 conference papers. 1–11

    2024

  43. [44]

    Chao Xu, Junwei Zhu, Jiangning Zhang, Yue Han, Wenqing Chu, Ying Tai, Chengjie Wang, Zhifeng Xie, and Yong Liu. 2023. High-fidelity generalized emotional talking face generation with multi-modal emotion space learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6609–6619

    2023

  44. [45]

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. 2024. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801(2024)

    Pith/arXiv arXiv 2024

  45. [46]

    Bingyuan Zhang, Xulong Zhang, Ning Cheng, Jun Yu, Jing Xiao, and Jianzong Wang. 2024. Emotalker: Emotionally editable talking face generation via diffusion model. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 8276–8280

    2024

  46. [47]

    Huawei Wei, Zejun Yang, and Zhisheng Wang. 2024. Aniportrait: Audio-driven synthesis of photorealistic portrait animation.arXiv preprint arXiv:2403.17694 (2024)

    Pith/arXiv arXiv 2024

  47. [48]

    Yue Zhang, LIU Minhao, Zhaokang Chen, Bin Wu, Chao Zhan, Yingjie He, JUNXIN HUANG, Wenjiang Zhou, et al. 2024. Musetalk: Real-time high quality lip synchronization with latent space inpainting.arXiv preprint arXiv:2410.10122 (2024)

    Pith/arXiv arXiv 2024

  48. [49]

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. 2021. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3661–3670

    2021

  49. [50]

    Happy" vs

    Longtao Zheng, Yifan Zhang, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An, and Shuicheng Yan. 2024. Memo: Memory-guided diffusion for expressive talking video generation.arXiv preprint arXiv:2412.04448 (2024). Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generation MM ’26, Novemb...

    Pith/arXiv arXiv 2024

  50. [52]

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. 2023. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8652–8661

    2023

  51. [2024]

    In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Diff2lip: Audio conditioned diffusion models for lip-synchronization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 5292–5302

  52. [2025]

    EmoHead: Emotional Talking Head via Manipulating Semantic Expression Parameters.arXiv preprint arXiv:2503.19416(2025)

    Pith/arXiv arXiv 2025

  53. [2026]

    InProceedings of the AAAI Conference on Artificial Intelligence, Vol

    Semi-supervised latent disentangled diffusion model for textile pattern generation. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 4798–4806