Pith. sign in

REVIEW 3 major objections 6 minor 52 references

GGTalker: Talking Head Systhesis with Generalizable Gaussian Priors and Identity-Specific Adaptation

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GGTalker claims generalizable audio and texture priors, fine-tuned per identity in about 20 minutes, produce photorealistic talking heads that generalize to novel views and out-of-distribution audio.

desk verdict A strong engineering recipe for 3D talking heads with an unresolved question about what exactly is being measured in Table 1. read the letter →

arxiv 2506.21513 v2 pith:2GMQP7SA submitted 2025-06-26 cs.CV

Wentao Hu , Shunkai Li , Ziqiao Peng , Haoxian Zhang , Fan Shi , Xiaoqiang Liu , Pengfei Wan , Di Zhang
show 1 more author
Hui Tian
This is my paper
classification cs.CV
keywords talkingheadsynthesis3DGaussiansplattingaudio-drivenanimationgeneralizablepriorsFLAMEparametricmodellipsynchronizationdiffusiontransformeridentity-specificadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the failure of 3D talking head systems on large head rotations and out-of-distribution audio is a missing-priors problem, and that a two-stage Prior-Adaptation strategy fixes it. GGTalker first learns two generalizable priors from large datasets: an Audio-Expression prior, a diffusion transformer that maps speech to FLAME expression parameters, and an Expression-Visual prior, a generator that predicts a UV Gaussian texture map from a single reference image. For a new identity, it fine-tunes both priors together with the tracked FLAME parameters in about 20 minutes, then adds a color MLP and a Body Inpainter to sharpen and blend the rendered frames. If the claim holds, hours-long per-identity training becomes unnecessary and a single learned head prior can serve many speakers, including speakers of languages never seen in the target video.

What carries the argument

The load-bearing mechanism is the pairing of two priors bridged by a rigged-Gaussian head. The Audio-Expression prior is a conditional diffusion transformer that denoises sequences of FLAME expression parameters, conditioned on Wav2Vec 2.0 speech features, an identity embedding, and a diffusion timestep, with classifier-free guidance and temporal smoothness losses. The Expression-Visual prior is an Identity-Gaussian Generator that predicts a 14-channel UV Gaussian map from a single reference image; the map is sampled into 3D Gaussian primitives that are rigged to the triangles of the canonical FLAME mesh, and the whole head is trained self-supervised by rendering it from a second frame's pose and comparing with that frame. Adaptation then jointly optimizes the UV Gaussian map and the FLAME parameters on the target video, adds a color MLP that modulates Gaussian colors by pose and expression, and a lightweight U-Net Body Inpainter that blends the rendered head with the torso and background.

What would settle it

Render a single identity with a large head-rotation sequence and cross-language audio, then score the mouth with a lip-sync metric (LSE-D/LSE-C) and inspect novel-view frames for holes; if the tracker's expression bias is baked into the fine-tuned Audio-Expression prior, the mouth will systematically lag or under-articulate for that identity. A sharper test is to compare rendered novel views against true multi-view captures of the same subject and check whether the claimed 3D consistency holds outside the views seen in training.

Watch

Extended reading notes

Core claim

The central claim is that generalizable Gaussian head priors plus brief identity-specific adaptation are enough to synthesize photorealistic, 3D-consistent talking heads that survive large head rotations and out-of-distribution or cross-language audio. The paper argues that all human heads share universal patterns of lip movement and texture, so those patterns should be learned once from large-scale data and then fine-tuned per speaker instead of being retrained from scratch. Concretely, the system learns an audio-to-expression mapping with a diffusion transformer, and an expression-to-visual mapping that turns one reference image into a rigged 3D Gaussian head via a UV map; adaptation then refines both the Gaussian texture and the FLAME parameters on the target video. The reported outcome is state-of-the-art rendering quality, 3D consistency, and lip-sync accuracy, with roughly 0.3 hours of adaptation and 120 FPS rendering.

Load-bearing premise

The whole pipeline depends on the face-tracking parameters extracted from ordinary monocular video being accurate enough to serve as the teaching signal for lip motion and the driving signal for rendering; the paper itself admits that current tracking is imperfect and that mesh-based shading introduces errors, which is why it also optimizes those parameters during adaptation.

Editorial extensions

If this is right

  • A new identity can be brought up in about 20 minutes instead of hours, because only the priors are fine-tuned rather than retrained.
  • Large head rotations and novel viewpoints stay free of holes and distortion because the Gaussian head starts from a learned texture prior rather than being fit from scratch.
  • Lip-sync survives out-of-distribution audio, including unfamiliar languages, because the audio-to-expression mapping is learned from a large multi-speaker corpus.
  • The 120 FPS rendering and short adaptation make real-time, per-person digital avatars practical.
  • The one-time cost of learning universal head priors is separated from the per-identity cost of adaptation, so adding identities gets cheaper as the priors improve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the adaptation stage jointly optimizes FLAME parameters with the Gaussian texture, the method's accuracy ceiling may be set by the FLAME model's expressiveness rather than by the tracker; a version driven by a richer parametric model would test this.
  • The UV-map generation trick is not limited to faces: any avatar class with a canonical mesh and a UV layout could reuse the same single-image-to-Gaussian prior idea, for instance hands or full bodies.
  • A natural stress test the paper does not run is to withhold an entire language from the prior training set and measure how far lip-sync degrades, which would quantify how much of the cross-language gain comes from the prior versus from the audio encoder.
  • Training-time cost could be probed by shrinking the per-identity video below the reported 4 to 6 minutes to find the data floor at which adaptation still holds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes GGTalker, a two-stage Prior-Adaptation framework for speech-driven 3D talking-head synthesis. An Audio-Expression prior, implemented as a diffusion transformer conditioned on Wav2Vec 2.0 features and an identity embedding, is trained on large-scale audio-visual data to predict FLAME expression sequences. An Expression-Visual prior uses an Identity-Gaussian Generator to predict a UV-space Gaussian map from a single reference image, with Gaussians rigged to FLAME and trained self-supervised by re-rendering target frames. In Customized Adaptation, the UV map, a color MLP, and a body inpainter are fine-tuned per identity, FLAME parameters are jointly optimized, and the Audio-Expression model is fine-tuned on the target video. Experiments report state-of-the-art self-reenactment quality, 3D consistency, lip-sync accuracy, OOD audio generalization, 0.3h adaptation time, and 120 FPS rendering.

Significance. If the claims hold, GGTalker would be a meaningful step toward efficient, generalizable 3D talking heads: the two-stage prior-adaptation strategy is well motivated, the use of FLAME provides explicit control, and the large training corpora plus self-supervised objective reduce circularity. The efficiency gains (0.3h adaptation, 120 FPS) are substantial. However, the current manuscript does not unambiguously demonstrate that the reported self-reenactment improvements are achieved under audio-driven synthesis, which is the central claim.

major comments (3)
  1. [§4.2 / Table 1] The self-reenactment protocol is underspecified. Section 3.4 states that during Customized Adaptation the FLAME parameters are jointly optimized against the training video (Eq. 13 and the following paragraph). Table 1 reports PSNR, LPIPS, SSIM, FID, LMD, AUE, and LSE-C, but the paper does not state whether these results use expressions produced by the Audio-Expression model from the driving audio or the optimized FLAME parameters from the adaptation stage. If the latter, the metrics measure per-identity fitting and not the claimed generalizable audio-driven synthesis, and the comparison with baselines that do use audio-driven expressions is unfair. Please state the protocol explicitly and, if the optimized expressions are used, report all Table 1 metrics with Audio-Expression predicted expressions; also report image-quality metrics for the OOD experiments in Table 2 rather than only LSE-D/LSE-C.
  2. [§3.4 / Eq. 13] There is a concrete risk of mismatch between the expression space used to fine-tune the Audio-Expression model and the expression space used at render time. The Audio-Expression model is fine-tuned on the originally tracked FLAME expressions, while the renderer is later driven with jointly optimized FLAME parameters; if these two spaces diverge systematically, the learned audio-to-expression mapping is biased. The paper should either fine-tune the Audio-Expression model on the optimized expressions or provide evidence (for example, per-frame expression error between tracked and optimized sequences) that the two are consistent.
  3. [Table 1] The test set mixes public datasets and two self-recorded videos with larger head motion, and all results are pooled without per-subject breakdown or variance. Because the claimed margins over the second-best methods are small on several metrics (for example, LPIPS 0.0281 vs. 0.0334 for SyncTalk), subject-level variability could change the ranking. Report per-subject results and, if feasible, standard deviations or significance tests.
minor comments (6)
  1. [Title] The title contains a typo: 'Systhesis' should be 'Synthesis'.
  2. [§3.4] In the Body Inpainter paragraph, 'Mexpr, Fexp, Fexp, M_hat_id, MSH' lists 'Fexp' twice and uses the undefined symbol 'Mexpr'; please correct this list.
  3. [Eq. (5)] Equation (5) uses |n| for the sliding-window size, but n was previously used as the diffusion timestep; rename one of these to avoid ambiguity.
  4. [§3.4 / Eq. (16)] The color MLP is denoted MSH in Eq. (16) but described only as 'a lightweight MLP'; please clarify the notation and define MSH explicitly.
  5. [Table 1 footnote] The footnote about Wav2Lip's LSE-C being optimized by design is useful; consider adding an explicit comparison against ground-truth videos to help calibrate the LSE-C values reported for all methods.
  6. [References] Reference [45] is 'GaussianTalker: Speaker-specific talking head synthesis' while [7] is also 'GaussianTalker'; the similar names may confuse readers, so please consider distinguishing them in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the two-stage Prior-Adaptation pipeline is trained on external data and evaluated with external metrics, so the central claims are not forced by construction.

full rationale

The derivation chain is self-contained. GGTalker's Audio-Expression prior is trained on large external speech-video collections (HDTF, CN-CVS, and 100-hour internet videos) and the Expression-Visual prior on VFHQ and NeRSemble, so the two priors are not constructed from the evaluation identities or metrics. During Customized Adaptation the model is fine-tuned on each target video, but this is the paper's stated per-identity adaptation task, and all reported comparisons in Table 1 apply the same self-reenactment protocol to every baseline; the metrics (PSNR, LPIPS, SSIM, FID, LMD, AUE, LSE-D, LSE-C) are computed with external pre-trained models such as SyncNet and are not the training losses. The ablations in Table 3 show that removing each prior or fine-tuning stage degrades external metrics, which is an empirical claim rather than a definitional identity. The paper's own caveat in Section 3.4 that monocular FLAME tracking is imperfect and that FLAME parameters are jointly optimized is a robustness and correctness limitation, not a circularity: the audio-conditioned expression model is still trained on tracked pseudo-ground-truth and evaluated on held-out audio. Self-citations (SyncTalk, SyncTalk++, DualTalk, OmniSync) appear as baselines or related work, not as premises that define GGTalker's outputs. No predicted quantity in Tables 1-3 is equal by construction to an input fitted parameter, and no uniqueness or prior-result citation is used to force the design.

Assumptions & free parameters 8 free parameters · 7 assumptions · 0 invented entities

The central claim rests on the accuracy of FLAME tracking as pseudo ground truth and on the assumption that front-facing single images plus large-scale priors suffice for complete head reconstruction. The model also depends on several hand-chosen hyperparameters whose reported values are not always given. No new physical entities are introduced.

free parameters (8)
  • λtemp (temporal loss weight) = not reported
    Weights the temporal smoothness term in Audio-Expression training; chosen by hand.
  • λexp (expression loss weight) = not reported
    Weights the L2 expression reconstruction term; chosen by hand.
  • λL1, λSSIM, λvgg, λµ (visual loss weights) = not reported
    Weights for image, perceptual, and position losses in Expression-Visual training and adaptation.
  • Audio feature dimension d = 512
    Projection size for Wav2Vec 2.0 features; used throughout the audio condition encoder.
  • Transformer layers L = 8
    Depth of the diffusion transformer decoder for expression generation.
  • Sliding window frames n = 10
    Number of expression frames optimized jointly in the diffusion denoiser.
  • Classifier-free guidance probability p = 0.1
    Probability of dropping the condition during training.
  • Learning rates = 1e-4 (prior), 1e-5 (adaptation)
    Adam learning rates for the two stages; selected by hand.
assumptions (7)
  • standard math A differentiable 3D Gaussian rasterizer exists and is used for rendering.
    Invoked in Eqs. 1-2 and throughout rendering; standard from 3DGS [14].
  • domain assumption FLAME's parametric space covers the identity, expression, and pose variations needed for talking heads.
    Used as the intermediate representation and to rig Gaussians (Sections 3.1-3.3).
  • domain assumption Wav2Vec 2.0 audio features carry enough phonetic information to predict lip motion.
    The audio condition encoder projects Wav2Vec features to drive expressions (Section 3.2).
  • domain assumption VHAP monocular tracking gives FLAME parameters accurate enough to serve as pseudo ground truth.
    Tracked FLAME parameters supervise both the visual prior and the adaptation stage; Section 3.4 acknowledges tracking imperfections.
  • domain assumption A single front-facing reference image captures enough identity information to predict a complete UV Gaussian map.
    The Identity-Gaussian Generator uses one source frame to produce the coarse head texture (Section 3.3).
  • domain assumption All human heads share general texture and lip-movement patterns that are learnable from large data.
    Motivates training the two priors on HDTF, CN-CVS, VFHQ, NeRSemble, and private videos (Section 3.3).
  • domain assumption Standard quality metrics (PSNR, LPIPS, SSIM, FID, LMD, AUE, LSE-C/D) faithfully reflect the stated claims.
    Used for all quantitative comparisons; the LSE-C footnote shows one baseline directly optimizes that metric (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of GGTalker: Talking Head Systhesis with Generalizable Gaussian Priors and Identity-Specific Adaptation." pith.science (2026). https://pith.science/paper/2GMQP7SA

@misc{pith2026250621513,
  author       = {Pith},
  title        = {Pith review of: GGTalker: Talking Head Systhesis with Generalizable Gaussian Priors and Identity-Specific Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2GMQP7SA}},
  note         = {Machine review of arXiv:2506.21513}
}
read the original abstract

Creating high-quality, generalizable speech-driven 3D talking heads remains a persistent challenge. Previous methods achieve satisfactory results for fixed viewpoints and small-scale audio variations, but they struggle with large head rotations and out-of-distribution (OOD) audio. Moreover, they are constrained by the need for time-consuming, identity-specific training. We believe the core issue lies in the lack of sufficient 3D priors, which limits the extrapolation capabilities of synthesized talking heads. To address this, we propose GGTalker, which synthesizes talking heads through a combination of generalizable priors and identity-specific adaptation. We introduce a two-stage Prior-Adaptation training strategy to learn Gaussian head priors and adapt to individual characteristics. We train Audio-Expression and Expression-Visual priors to capture the universal patterns of lip movements and the general distribution of head textures. During the Customized Adaptation, individual speaking styles and texture details are precisely modeled. Additionally, we introduce a color MLP to generate fine-grained, motion-aligned textures and a Body Inpainter to blend rendered results with the background, producing indistinguishable, photorealistic video frames. Comprehensive experiments show that GGTalker achieves state-of-the-art performance in rendering quality, 3D consistency, lip-sync accuracy, and training efficiency.

Figures

Figures reproduced from arXiv: 2506.21513 by the authors.

Figure 1
Figure 1. The proposed GGTalker introduces generalizable Gaussian head priors from large-scale datasets to identity-specific talking head. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of GGTalker. We introduce a two-stage Prior-Adaptation approach for talking head synthesis. We use a conditional diffusion transformer to predict plausible facial movements from an audio sequence. For texture generation, we leverage the UV layout of the canonical mesh and use an Identity-Gaussian Generator to predict a UV Gaussian map from a single reference image, initializing a reasonable Gaussian distrib… view at source ↗
Figure 3
Figure 3. Visualization of the Gaussian color attributes on UV map M. We leverage the UV layout of the canonical mesh to map 2D identity features into 3D space. LE2V = λL1LL1 + λSSIMLSSIM + λvggLvgg + λµLµ (11) 3.4. Customized Adaptation Expression-Visual Fine-tuning During the Expression￾Visual Prior stage, our Identity-Gaussian Generator predicts a reasonable texture prior from a single image. However, in essence, a single … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of self-reenactment with previous methods. Compared to previous methods, our method maintains excellent identity consistency and rendering quality across various viewing angles. Our method also provides more accurate lip-sync and expression trans…
Figure 5
Figure 5. Figure 5: Qualitative ablations on Expression-Visual. We present a comparison of rendered results from novel views to demonstrate the contribution of our two-stage training. Remov￾ing either the prior or adaptation stage results in visual artifacts. supplementary video for bette…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 31 canonical work pages

  1. [1]

    Gaus- sianspeech: Audio-driven gaussian avatars

    Shivangi Aneja, Artem Sevastopolsky, Tobias Kirschstein, Justus Thies, Angela Dai, and Matthias Nießner. Gaus- sianspeech: Audio-driven gaussian avatars. arXiv preprint arXiv:2411.18675, 2024. 1, 2

    arXiv 2024

  2. [2]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural infor- mation processing systems, 33:12449–12460, 2020. 3

    work page 2020

  3. [3]

    Cross-dataset learning and person-specific normalisation for automatic action unit detection

    Tadas Baltru ˇsaitis, Marwa Mahmoud, and Peter Robinson. Cross-dataset learning and person-specific normalisation for automatic action unit detection. In 2015 11th IEEE Inter- national Conference and Workshops on Automatic Face and Gesture Recognition (FG), pages 1–6. IEEE, 2015. 7

    work page 2015

  4. [4]

    Cn-cvs: A mandarin audio-visual dataset for large vocabulary contin- uous visual to speech synthesis

    Chen Chen, Dong Wang, and Thomas Fang Zheng. Cn-cvs: A mandarin audio-visual dataset for large vocabulary contin- uous visual to speech synthesis. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 7

    work page 2023

  5. [5]

    Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136, 2024. 2

    arXiv 2024

  6. [6]

    Videoretalking: Audio-based lip synchronization for talking head video editing in the wild

    Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022. 1

    work page 2022

  7. [7]

    Gaus- siantalker: Real-time talking head synthesis with 3d gaussian splatting

    Kyusun Cho, Joungbin Lee, Heeji Yoon, Yeobin Hong, Jaehoon Ko, Sangjun Ahn, and Seungryong Kim. Gaus- siantalker: Real-time talking head synthesis with 3d gaussian splatting. In ACM Multimedia 2024, 2024. 1, 2, 3, 5, 6, 7, 8

    work page 2024

  8. [8]

    Generalizable and Animatable Gaussian Head Avatar

    Xuangeng Chu and Tatsuya Harada. Generalizable and animatable gaussian head avatar. arXiv preprint arXiv:2410.07971, 2024. 5

    work page Pith review arXiv 2024

Show all 52 references
  1. [9]

    Emoca: Emotion driven monocular face capture and animation

    Radek Dan ˇeˇcek, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 20311–20322, 2022. 5

    2022

  2. [10]

    Learning an animatable detailed 3d face model from in-the- wild images

    Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the- wild images. ACM Transactions on Graphics (ToG), 40(4): 1–13, 2021. 5

    2021

  3. [11]

    Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 5784–5794, 2021. 1, 2, 6, 7

    2021

  4. [12]

    Emotalk3d: high-fidelity free-view synthesis of emotional 3d talking head

    Qianyun He, Xinya Ji, Yicheng Gong, Yuanxun Lu, Zhengyu Diao, Linjia Huang, Yao Yao, Siyu Zhu, Zhan Ma, Song- cen Xu, et al. Emotalk3d: high-fidelity free-view synthesis of emotional 3d talking head. In European Conference on Computer Vision, pages 55–72. Springer, 2024. 2

    2024

  5. [13]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 7

    2017

  6. [14]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1,

  7. [15]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  8. [16]

    Nersemble: Multi-view ra- diance field reconstruction of human heads

    Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner. Nersemble: Multi-view ra- diance field reconstruction of human heads. ACM Trans. Graph., 42(4), 2023. 7

    2023

  9. [17]

    Gghead: Fast and generalizable 3d gaussian heads

    Tobias Kirschstein, Simon Giebenhain, Jiapeng Tang, Markos Georgopoulos, and Matthias Nießner. Gghead: Fast and generalizable 3d gaussian heads. arXiv preprint arXiv:2406.09377, 2024. 5

    2024 arXiv

  10. [18]

    Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7568– 7578, 2023. 1, 2, 6, 7, 8

    2023

  11. [19]

    Talkinggaussian: Structure-persistent 3d talking head synthesis via gaussian splatting

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Talkinggaussian: Structure-persistent 3d talking head synthesis via gaussian splatting. arXiv preprint arXiv:2404.15264, 2024. 2, 5, 6, 7, 8

    2024 arXiv

  12. [20]

    Er-nerf++: Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Jun Zhou, and Lin Gu. Er-nerf++: Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. Information Fusion, 110:102456, 2024. 2

    2024

  13. [21]

    Instag: Learning personalized 3d talking head from 9 few-second video

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Jun Zhou, and Lin Gu. Instag: Learning personalized 3d talking head from 9 few-second video. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 10690–10700,

  14. [22]

    Learning a model of facial shape and expression from 4d scans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194–1, 2017. 2

    2017

  15. [23]

    Learned spatial representations for few-shot talking-head synthesis

    Moustafa Meshry, Saksham Suri, Larry S Davis, and Abhi- nav Shrivastava. Learned spatial representations for few-shot talking-head synthesis. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 13829– 13838, 2021. 2

    2021

  16. [24]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM, 65(1):99–106, 2021. 1, 2

    2021

  17. [25]

    Real-time talking head driven by voice and its application to communication and entertainment

    Shigeo Morishima. Real-time talking head driven by voice and its application to communication and entertainment. In AVSP’98 International Conference on Auditory-Visual Speech Processing, 1998. 1

    1998

  18. [26]

    Synctalk: The devil is in the synchronization for talking head synthesis

    Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk: The devil is in the synchronization for talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 666–676,

  19. [27]

    Dualtalk: Dual-speaker in- teraction for 3d talking head conversations

    Ziqiao Peng, Yanbo Fan, Haoyu Wu, Xuan Wang, Hongyan Liu, Jun He, and Zhaoxin Fan. Dualtalk: Dual-speaker in- teraction for 3d talking head conversations. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 21055–21064, 2025. 2

    2025

  20. [28]

    Synctalk++: High-fidelity and efficient synchronized talking heads synthesis using gaussian splat- ting

    Ziqiao Peng, Wentao Hu, Junyuan Ma, Xiangyu Zhu, Xi- aomei Zhang, Hao Zhao, Hui Tian, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk++: High-fidelity and efficient synchronized talking heads synthesis using gaussian splat- ting. arXiv preprint arXiv:2506.14742, 2025. 1

    2025 arXiv

  21. [29]

    Omnisync: Towards universal lip synchronization via diffusion transformers.arXiv preprint arXiv:2505.21448,

    Ziqiao Peng, Jiwen Liu, Haoxian Zhang, Xiaoqiang Liu, Songlin Tang, Pengfei Wan, Di Zhang, Hongyan Liu, and Jun He. Omnisync: Towards universal lip synchronization via diffusion transformers.arXiv preprint arXiv:2505.21448,

  22. [30]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia , pages 484–492, 2020. 1, 2, 6, 7, 8

    2020

  23. [31]

    Vhap: Versatile head alignment with adaptive appearance priors, 2024

    Shenhan Qian. Vhap: Versatile head alignment with adaptive appearance priors, 2024. 5, 7

    2024

  24. [32]

    Gaus- sianavatars: Photorealistic head avatars with rigged 3d gaus- sians

    Shenhan Qian, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias Nießner. Gaus- sianavatars: Photorealistic head avatars with rigged 3d gaus- sians. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 2...

  25. [33]

    Difftalk: Crafting diffusion models for generalized audio-driven portraits animation

    Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 1982–1991, 2023. 1

    1982

  26. [34]

    Identity-preserving realistic talking face generation

    Sanjana Sinha, Sandika Biswas, and Brojeshwar Bhowmick. Identity-preserving realistic talking face generation. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–10. IEEE, 2020. 1

    2020

  27. [35]

    Everybody’s talkin’: Let me talk as you want

    Linsen Song, Wayne Wu, Chen Qian, Ran He, and Chen Change Loy. Everybody’s talkin’: Let me talk as you want. IEEE Transactions on Information Forensics and Se- curity, 17:585–598, 2022. 2

    2022

  28. [36]

    Real-time neural radiance talking portrait synthesis via audio-spatial decomposition

    Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211.12368, 2022. 2, 6, 7

    2022 arXiv

  29. [37]

    Neural voice puppetry: Audio-driven facial reenactment

    Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16 , pages 716–731. Spr...

    2020

  30. [38]

    Flashavatar: High-fidelity head avatar with efficient gaussian embedding

    Jun Xiang, Xuan Gao, Yudong Guo, and Juyong Zhang. Flashavatar: High-fidelity head avatar with efficient gaussian embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1802– 1812, 2024. 5

    2024

  31. [39]

    Vfhq: A high-quality dataset and bench- mark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and bench- mark for video face super-resolution. In The IEEE Con- ference on Computer Vision and Pattern Recognition Work- shops (CVPRW), 2022. 7

    2022

  32. [40]

    Hallo: Hierarchical audio-driven vi- sual synthesis for portrait image animation

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Li- wei Zhang, Ce Liu, Jingdong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven vi- sual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 2

    2024 arXiv

  33. [41]

    Geneface++: Generalized and stable real- time audio-driven 3d talking face generation

    Zhenhui Ye, Jinzheng He, Ziyue Jiang, Rongjie Huang, Ji- awei Huang, Jinglin Liu, Yi Ren, Xiang Yin, Zejun Ma, and Zhou Zhao. Geneface++: Generalized and stable real- time audio-driven 3d talking face generation. arXiv preprint arXiv:2305.00787, 2023. 6, 7, 8

    2023 arXiv

  34. [42]

    Geneface: Generalized and high- fidelity audio-driven 3d talking face synthesis.arXiv preprint arXiv:2301.13430, 2023

    Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, JinZheng He, and Zhou Zhao. Geneface: Generalized and high- fidelity audio-driven 3d talking face synthesis.arXiv preprint arXiv:2301.13430, 2023. 1, 2, 6, 7, 8

    2023 arXiv

  35. [43]

    Mimictalk: Mimicking a per- sonalized and expressive 3d talking face in minutes

    Zhenhui Ye, Tianyun Zhong, Yi Ren, Ziyue Jiang, Jiawei Huang, Rongjie Huang, Jinglin Liu, Jinzheng He, Chen Zhang, Zehan Wang, et al. Mimictalk: Mimicking a per- sonalized and expressive 3d talking face in minutes. Ad- vances in neural information processing systems , 37:1829–...

    2024

  36. [44]

    Real3d-portrait: One-shot realistic 3d talking portrait synthesis

    Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, et al. Real3d-portrait: One-shot realistic 3d talking portrait synthesis. arXiv preprint arXiv:2401.08503, 2024

    2024 arXiv

  37. [45]

    Gaussiantalker: Speaker-specific talking head synthesis via 3d gaussian splatting

    Hongyun Yu, Zhan Qu, Qihang Yu, Jianchuan Chen, Zhonghua Jiang, Zhiwen Chen, Shengyu Zhang, Jimin Xu, 10 Fei Wu, Chengfei Lv, et al. Gaussiantalker: Speaker-specific talking head synthesis via 3d gaussian splatting. arXiv preprint arXiv:2404.14037, 2024. 1, 2, 5

    2024 arXiv

  38. [46]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 7

    2018

  39. [47]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  40. [48]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021. 7

    2021

  41. [49]

    Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video

    Zhimeng Zhang, Zhipeng Hu, Wenjin Deng, Changjie Fan, Tangjie Lv, and Yu Ding. Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video. arXiv preprint arXiv:2303.03988, 2023. 1, 6, 7, 8

    2023 arXiv

  42. [50]

    Headgap: Few-shot 3d head avatar via generalizable gaussian priors

    Xiaozheng Zheng, Chao Wen, Zhaohu Li, Weiyi Zhang, Zhuo Su, Xu Chang, Yang Zhao, Zheng Lv, Xiaoyuan Zhang, Yongjie Zhang, et al. Headgap: Few-shot 3d head avatar via generalizable gaussian priors. arXiv preprint arXiv:2408.06019, 2024. 5

    2024 arXiv

  43. [51]

    Identity- preserving talking face generation with landmark and ap- pearance priors

    Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, and Guanbin Li. Identity- preserving talking face generation with landmark and ap- pearance priors. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 9729–...

    2023

  44. [52]

    Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation

    Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4176–4186,

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.