Pith. sign in

REVIEW 3 major objections 5 minor 68 references

Few-Shot Identity Adaptation for 3D Talking Heads via Global Gaussian Field

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single Global Gaussian Field shared across identities allows a 3D talking-head model to adapt to a new speaker from a few seconds of footage.

desk verdict A genuinely new shared Gaussian field idea for few-shot talking head adaptation, but the main evaluation only scores the adaptation clips themselves, so the core generalization claim is untested. read the letter →

arxiv 2506.22044 v1 pith:J3TTM5PO submitted 2025-06-27 cs.CV

Hong Nie , Fuyuan Cao , Lu Chen , Fengxin Chen , Yuefeng Zou , Jun Yu This is my paper
classification cs.CV
keywords talkingheadsynthesis3DGaussiansplattingfew-shotadaptationidentityembeddingaudio-drivenanimationuniversalmotionfieldpretrainandlipsynchronization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reconstruction-based talking-head avatars currently pay for identity preservation with per-person training: every new face needs its own neural field, taking hours and making multi-speaker systems expensive. This paper argues that the expensive part is unnecessary. Its FIAG framework builds one Global Gaussian Field—a single set of 3D Gaussian ellipsoids encoding the facial structure most people share—and lets an identity embedding network select and offset subsets of those ellipsoids for each speaker. A companion Universal Motion Field stores audio-to-motion mappings common to all identities, so adapting to a new person requires only seconds of that person's video and a small personalized motion network. The reported result is that this adapted model matches or beats per-identity and earlier pretrain-and-adapt methods on image quality and lip-sync metrics while cutting storage and training cost. If the central claim holds, the cost of adding a new speaker to a talking-head system drops from hours of training to a brief finetune.

What carries the argument

The mechanism that carries the method is the decomposition of the Gaussian field into a shared canonical field plus identity-conditioned offsets. Each ellipsoid is $g_i = \{\mu_i, r_i, s_i, SH_i, \alpha_i\}$, with center, quaternion, scale, spherical-harmonic coefficients, and opacity, and the shared collection $\theta_c$ encodes common facial geometry. The identity embedding module passes $\theta_c$ through a tri-plane hash encoder $\mathcal{H}$, fuses the result with an identity feature $F_{id}(id_i)$, and decodes with MLPs into the offset $\Delta\theta = MLP(\mathcal{H}(\theta_c) \oplus (\mathcal{H}(\theta_c) \odot F_{id}(id_i)))$; this both allocates ellipsoids to the correct identity and computes their parameter displacement. A triplet loss on offsets reinforces identity isolation, described by the paper as a non-physical way to manage identity conflict because no separate physical field is built. The motion side uses a Universal Motion Field for shared audio-to-lip and expression mappings, per-identity lightweight IM-Nets from an IM-Pool for personalized facial motion, and a Face-Mouth Hook with a Top-k selector to couple mouth motion to facial motion. Differentiable splatting renders the adapted ellipsoids into images. This design is what lets one pretrained checkpoint be reused across identities: adaptation initializes a new IM-Net and finetunes the offset mapping, without rebuilding the representation or motion fields.

What would settle it

Adapt an identity whose uniqueness lies in fine skin texture, such as deep wrinkles, scars, or heavy makeup, from five seconds of footage, render held-out frames with large head rotation, and compare identity-detail fidelity against an exclusive-field model trained on the same five seconds. If the adapted model cannot pass a same-different identity matching test on those frames, or if its perceptual fidelity is no better than the coarse shared-field reconstruction, the assumption that the adaptation stage recovers the attenuated detail is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that identity-exclusive Gaussian fields are not needed for high-quality talking-head synthesis, and it positions FIAG as the first pretrain-and-adapt method to share the representation field itself. It presents a shared Global Gaussian Field $\theta_c$ of canonical ellipsoids and an identity embedding module that, for each identity $i$, predicts a set of activated ellipsoids and offsets $\Delta\theta_i$ so that $\theta_i = \theta_c + \Delta\theta_i$; new ellipsoids are created only when large positional deviations would otherwise make optimization unstable. Because the shared field is deliberately built to represent common macro-level facial structure, the paper reports about 10,000 ellipsoids for ten identities (1.7 MB), compared with roughly 80,000 ellipsoids (8.4 MB) for five identities in the exclusive-field baseline InsTaG. Motion is shared in the same way: a Universal Motion Field learns common audio-to-mouth and audio-to-expression mappings, while a per-identity IM-Net from a shared pool personalizes them. Under a 5-second self-reconstruction setup, the paper reports a PSNR of 37.69, SSIM of 0.9716, LPIPS of 0.0296, LMD of 2.0954, and Sync-C of 1.909, ahead of all compared baselines, and shows that the adapted model retains lip-sync quality when the driving audio comes from a different speaker or a different language.

Load-bearing premise

The load-bearing premise is that a single set of shared Gaussian ellipsoids plus learned identity offsets can represent any face well enough that a few seconds of adaptation video recovers the identity-specific detail the global field deliberately leaves out.

Editorial extensions

If this is right

  • A new speaker can be added with seconds of footage and one new lightweight IM-Net, and the reported finetune cost is around 16 minutes on an RTX 3090 instead of hours of per-identity training.
  • Storage no longer scales linearly with the number of identities: one 1.7 MB global field replaces per-identity fields, and the paper reports 98.5% ellipsoid reuse compared with an exclusive-field variant.
  • Cross-domain driving works: lip-sync accuracy is retained when the audio comes from a different speaker or from a different language, with English and German tested.
  • Removing the identity embedding module makes the shared field worse than an exclusive field, so the identity-offset mechanism, not the shared geometry by itself, is what makes multi-identity pretraining viable.
  • The pipeline is robust to the choice of audio feature extractor, staying at or near the top of the compared metrics across four different extractors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The deliberate attenuation of fine wrinkles suggests a built-in ceiling: for identities whose recognizability depends on micro-texture, more of the burden moves to the adaptation stage, so the method's advantage over per-identity training should shrink as target-specific detail becomes finer in scale.
  • The same global-field-plus-embedding decoupling is a general recipe: any object class with shared topology, such as full bodies or hands, could replace per-instance fields with one shared field and instance embeddings, and the reuse rate of 98.5% here is the number to watch when the shared topology weakens.
  • Since the authors state that zero-shot performance remains unattainable in this framework, the sharpest next test is a data-scaling curve: adapt the same identity from 1, 2, 5, 10, and 20 seconds and record which quality metrics saturate first, revealing how close the pretrained shared field is to functioning as an identity prior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes FIAG, a few-shot identity adaptation framework for 3D talking heads built on 3D Gaussian Splatting. It introduces a Global Gaussian Field shared across identities, an identity embedding module that predicts per-identity Gaussian offsets, and a Universal Motion Field with identity-specific IM-Nets, following the pretrain-and-adapt (PAA) paradigm. The central claims are that the shared field supports rapid adaptation from a few seconds of footage, with better image quality and lip-sync accuracy than prior per-identity and PAA baselines. Experiments compare against zero-shot, one-shot, training-from-scratch, and PAA methods on self-reconstruction and cross-domain lip-sync, plus ablations and storage analyses.

Significance. If the claims hold, the paper makes a useful step: replacing identity-exclusive Gaussian fields with a shared field that reuses about 98.5% of ellipsoids, cuts per-identity storage (1.7 MB versus 8.4 MB for InsTaG), and adapts with seconds of footage. The ablation against exclusive-field variants supports the core contribution, and the code is promised. However, the evaluation protocol is self-reconstruction on the same clips used for adaptation, which cannot by itself establish identity generalization; the significance is therefore conditional on additional held-out evidence.

major comments (3)
  1. [IV-B, Tables I-III; Section VIII] All quantitative evidence for the central claim is self-reconstruction on the same few-second clips used for adaptation. With roughly 125 frames, a model can achieve high PSNR/SSIM/LMD by memorizing those frames without learning an identity-general representation. Section III-A states that the global field intentionally attenuates fine individual features, and Section VIII concedes that pretraining alone yields only coarse reconstruction, so the adaptation stage must be shown to recover identity; the current Tables I-III do not separate adaptation from overfitting. Please add held-out frames from the same identity (e.g., a temporal split or different utterances/poses), report an identity-preservation metric such as face-recognition similarity to the source identity versus other identities, and clarify whether the cross-domain test in Table III uses held-out frames or the training frames.
  2. [IV-B, Tables I-II; Section VI-A1 (supplementary)] All comparisons are single runs without variance or a per-identity breakdown, so the reported margins (e.g., FIAG PSNR 37.69 versus InsTaG 35.90, or Sync-C 1.909 versus 1.741) cannot be assessed for statistical reliability. Additionally, the supplementary says the test set comprises five videos but lists only four names. Please provide standard deviations across at least three seeds or a per-identity result table, and correct the test-set count.
  3. [IV-B, Table III] Cross-domain generalization is tested in only two settings (male/English and female/German) and only with lip-sync metrics; no image quality or identity-preservation metrics are reported, and the target identity's frames may still be the training frames. This is too thin to support the generalizability claim in the Abstract. Please expand the cross-domain evaluation to more language/gender/pose conditions and include held-out image metrics.
minor comments (5)
  1. [III-D, Eq. (10); III-B, Eq. (4); III-C] The values of the loss weights lambda1 and lambda2, the triplet margin tau, and the Top-k selector size k are never reported; please list them in the experimental setup, even if code is released.
  2. [Supplementary, Algorithm 2] The line 'theta_m_phi = theta_m_i + delta_m_phi' uses subscript i, which is undefined in the adaptation loop; it should be subscript phi.
  3. [III-A, Eq. (2)] The expression theta_i = theta_s + Delta_theta with theta_s subset of theta_c is informal; please clarify how offset addition acts on a subset of Gaussians and how new ellipsoids are instantiated when positional deviations are large.
  4. [IV-D, Table IV] The row-to-condition mapping for GGF/EGF and the IE-Net/IM-Net checkmarks is ambiguous; the text's comparisons (e.g., 'GGF in the absence of IM-Net outperforms EGF') do not map transparently to the table rows. Please restructure the table so each condition is explicit.
  5. [Supplementary, Sections VI-C and VI-D] There are typographical errors such as 'conducte' for 'conduct'; a careful proofread of the supplementary material is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical, the Global Gaussian Field is defined compositionally and tested by ablations against an exclusive-field baseline, and the reused motion components are attributed to external prior work.

full rationale

The paper does not derive any target quantity from a parameter that is defined in terms of it. Identity-specific fields are constructed as theta_i = theta_c + Delta_theta (Eq. 2), with Delta_theta produced by an identity embedding MLP from identity features and the shared field (Eq. 3); this is an architectural definition, not a circular prediction. The central quantitative evidence is a benchmark-style comparison and ablations (Tables I, II, IV, VI, VIII) against external baselines, and the GGF is tested against an Exclusive Gaussian Field variant, so the novel component is independently varied. The Face-Mouth Hook and universal motion structure are explicitly attributed to external works (Li et al. [45]; [41,42,53]), not to a self-citation chain, and no load-bearing uniqueness theorem is invoked. The self-reconstruction protocol (Table I) evaluates on the same few seconds used for adaptation (Algorithm 2), which limits generalization claims about unseen frames; that is a validity and overfitting concern, not a case where an alleged prediction equals its fitting input by construction, because the paper transparently labels the setting 'self-reconstruction' and does not present those frames as held out. No circular step meets the quoting-and-reduction bar.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method is empirical and does not introduce new physical entities. The central claim rests on learned representations, unstated hyperparameters, and domain assumptions about shared facial topology and the validity of self-reconstruction metrics.

free parameters (4)
  • Loss weights lambda1, lambda2 = not reported
    Weights in Eq. (10) for L1 and D-SSIM losses; chosen by hand and not ablated in the experiments.
  • Triplet margin tau = not reported
    Margin in Eq. (4) for identity isolation; its value is not disclosed.
  • Top-k selector size k = not reported
    Number of face-branch motion features propagated to the mouth branch via the Face-Mouth Hook in Section III-C2; the value is not disclosed.
  • Iteration count for pretraining and adaptation stages = 30K/40K in Table VIII
    The schedule inside Algorithm 1 is not specified; Table VIII shows iteration count changes the quality/sync trade-off, so reported metrics depend on this choice.
assumptions (5)
  • standard math 3D Gaussian splatting is differentiable under the L1 and D-SSIM losses.
    The optimization in Section III-D assumes gradients flow through ellipsoid parameters and the rasterizer.
  • domain assumption A single canonical Gaussian field plus identity offsets can represent the facial geometry of many identities.
    Core premise of the Global Gaussian Field in Section III-A; if false, shared ellipsoids cannot resolve identity conflicts.
  • domain assumption Pretraining on ten identities provides a sufficiently general facial topology and motion prior.
    The pretraining set is ten identities (Section VI-A); the method's generalization claim depends on this corpus being representative.
  • domain assumption Self-reconstruction on training footage is a valid proxy for identity preservation and adaptation quality.
    Headline results in Tables I and II evaluate on the same clips used for adaptation, so generalization beyond the training clips is assumed rather than measured.
  • domain assumption DeepSpeech audio features are sufficient to drive lip motion.
    All main experiments use DeepSpeech as the driving feature (Section VI-A); cross-domain tests use the same extractor.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Few-Shot Identity Adaptation for 3D Talking Heads via Global Gaussian Field." pith.science (2026). https://pith.science/paper/J3TTM5PO

@misc{pith2026250622044,
  author       = {Pith},
  title        = {Pith review of: Few-Shot Identity Adaptation for 3D Talking Heads via Global Gaussian Field},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J3TTM5PO}},
  note         = {Machine review of arXiv:2506.22044}
}
read the original abstract

Reconstruction and rendering-based talking head synthesis methods achieve high-quality results with strong identity preservation but are limited by their dependence on identity-specific models. Each new identity requires training from scratch, incurring high computational costs and reduced scalability compared to generative model-based approaches. To overcome this limitation, we propose FIAG, a novel 3D speaking head synthesis framework that enables efficient identity-specific adaptation using only a few training footage. FIAG incorporates Global Gaussian Field, which supports the representation of multiple identities within a shared field, and Universal Motion Field, which captures the common motion dynamics across diverse identities. Benefiting from the shared facial structure information encoded in the Global Gaussian Field and the general motion priors learned in the motion field, our framework enables rapid adaptation from canonical identity representations to specific ones with minimal data. Extensive comparative and ablation experiments demonstrate that our method outperforms existing state-of-the-art approaches, validating both the effectiveness and generalizability of the proposed framework. Code is available at: \textit{https://github.com/gme-hong/FIAG}.

Figures

Figures reproduced from arXiv: 2506.22044 by the authors.

Figure 1
Figure 1. Comparison of various frameworks for talking heads generation. (1) the zero-shot generative approach (top), which employs generative models with firsthand inference of new identities; (2) the RAR-based method (middle), utilizing identity-specific models for precise head reconstruction; and (3) the PAA paradigm (bottom), adopting multi-identity pretraining followed by targeted finetuning for novel identities. approac… view at source ↗
Figure 2
Figure 2. Overview of the proposed FIAG. In pretraining, FIAG constructs a Global Gaussian Field to encode canonical head features across multiple identities, with an identity embedding module resolving inter-identity conflicts, while learning generalized motion patterns stored in a Universal Motion Field. During finetuning on sparse footage of new idenities, the framework transforms neutral-to-specific representations via th… view at source ↗
Figure 3
Figure 3. The distinction between an Exclusive Gaussian Field and a Global Gaussian Field. The global Gaussian field models the majority of common head features (e.g., geometric contours and topological structures) while minimizing the emphasis on features specific to a particular identity. attenuation effect becomes more pronounced as the number of training identities increases. Consequently, the objective of the global Gaus… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Triplet loss for identity isolation. The triplet loss operates by constructing: (1) an anchor (identity i’s current offset), (2) a positive sample (same identity’s temporal offset), and (3) a negative sample (cross-identity offset). This formulation guides the network …
Figure 5
Figure 5. Figure 5: Visual comparison in lip image quality. The bottom right corner presents a magnified view of the mouth region to reveal finer details. The red dotted ellipse and arrow indicate some errors [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Visual comparison in lip synchronization accuracy. The white dashed rectangle highlights areas containing errors. D. Ablation Study To validate the effectiveness of our proposed method, we conducte ablation studies on the Global Gaussian Field (GGF) and comparative exp…
Figure 7
Figure 7. Figure 7: Visual comparison with and without identity embedding module. The absence of identity embedding will cause serious identity conflicts [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: The storage size of the Gaussian field for each identity. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Comparison of the number of Gaussian ellipsoids constituting the [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: When pretraining with 20 identities, each talking video having only a [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: High-definition comparison in the quality of rendered images and lip-sync accuracy. The red dashed rectangle highlights areas containing errors [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 55 canonical work pages

  1. [1]

    Few-shot adversarial learning of realistic neural talking head models,

    E. Zakharov, A. Shysheya, E. Burkov, and V . Lempitsky, “Few-shot adversarial learning of realistic neural talking head models,” in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , 2019, pp. 9459– 9468

    work page 2019

  2. [2]

    Audio- driven talking face video generation with learning-based personalized head pose,

    R. Yi, Z. Ye, J. Zhang, H. Bao, and Y .-J. Liu, “Audio- driven talking face video generation with learning-based personalized head pose,” arXiv:2002.10137, 2020

    arXiv 2002

  3. [3]

    What comprises a good talking-head video generation?

    L. Chen, G. Cui, Z. Kou, H. Zheng, and C. Xu, “What comprises a good talking-head video generation?” in IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020

    work page 2020

  4. [4]

    Makelttalk: speaker-aware talking-head animation,

    Y . Zhou, X. Han, E. Shechtman, J. Echevarria, E. Kalogerakis, and D. Li, “Makelttalk: speaker-aware talking-head animation,” ACM Transactions On Graphics (TOG), vol. 39, no. 6, pp. 1–15, 2020

    work page 2020

  5. [5]

    Text2video: Text-driven talking-head video synthesis with phonetic dictionary,

    S. Zhang, J. Yuan, M. Liao, and L. Zhang, “Text2video: Text-driven talking-head video synthesis with phonetic dictionary,” CoRR, 2021

    work page 2021

  6. [6]

    High-fidelity and freely controllable talking head video generation,

    Y . Gao, Y . Zhou, J. Wang, X. Li, X. Ming, and Y . Lu, “High-fidelity and freely controllable talking head video generation,” in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , 2023, pp. 5609–5619

    work page 2023

  7. [7]

    Difftalk: Crafting diffusion models for gener- alized audio-driven portraits animation,

    S. Shen, W. Zhao, Z. Meng, W. Li, Z. Zhu, J. Zhou, and J. Lu, “Difftalk: Crafting diffusion models for gener- alized audio-driven portraits animation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1982–1991

    work page 2023

  8. [8]

    Application of a 3d talking head as part of telecommunication ar, vr, mr system: Systematic review,

    N. Christoff, N. N. Neshov, K. Tonchev, and A. Manolova, “Application of a 3d talking head as part of telecommunication ar, vr, mr system: Systematic review,” Electronics, vol. 12, no. 23, p. 4788, 2023

    work page 2023

Show all 68 references
  1. [9]

    Toward user-aware interactive virtual agents: Generative multi-modal agent behaviors in vr,

    B. S. Gunawardhana, Y . Zhang, Q. Sun, and Z. Deng, “Toward user-aware interactive virtual agents: Generative multi-modal agent behaviors in vr,” in 2024 IEEE Inter- national Symposium on Mixed and Augmented Reality (ISMAR), 2024, pp. 1068–1077. 14

    2024

  2. [10]

    Language learning with interactive virtual agent scenar- ios and speech recognition: Lessons learned,

    J. N. Anderson, N. Davidson, H. Morton, and M. A. Jack, “Language learning with interactive virtual agent scenar- ios and speech recognition: Lessons learned,” Computer Animation and Virtual Worlds , vol. 19, no. 5, pp. 605– 619, 2008

    2008

  3. [11]

    A systematic literature review of language learning research based on teaching agents,

    X. Gu, Y . Huang, F. Wang, and X. Zheng, “A systematic literature review of language learning research based on teaching agents,” Educational Technology & Society , vol. 25, no. 2, pp. 1–14, 2022

    2022

  4. [12]

    Human-computer interaction system: A survey of talking-head generation,

    R. Zhen, W. Song, Q. He, J. Cao, L. Shi, and J. Luo, “Human-computer interaction system: A survey of talking-head generation,” Electronics, vol. 12, no. 1, p. 218, 2023

    2023

  5. [13]

    Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent,

    A. Delbosc, M. Ochs, N. Sabouret, B. Ravenet, and S. Ayache, “Towards the generation of synchronized and believable non-verbal facial behaviors of a talking virtual agent,” in Companion Publication of the 25th Interna- tional Conference on Multimodal Interaction , 2023, pp. 228–237

    2023

  6. [14]

    Unconstrained realtime facial performance capture,

    P.-L. Hsieh, C. Ma, J. Yu, and H. Li, “Unconstrained realtime facial performance capture,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1675–1683

    2015

  7. [15]

    Talking face generation by conditional recurrent adversarial network,

    Y . Song, J. Zhu, D. Li, X. Wang, and H. Qi, “Talking face generation by conditional recurrent adversarial network,” arXiv:1804.04786, 2018

    2018 arXiv

  8. [16]

    You said that?: Synthesising talking faces from audio,

    A. Jamaludin, J. S. Chung, and A. Zisserman, “You said that?: Synthesising talking faces from audio,” Interna- tional Journal of Computer Vision , vol. 127, pp. 1767– 1779, 2019

    2019

  9. [17]

    Hierar- chical cross-modal talking face generation with dynamic pixel-wise loss,

    L. Chen, R. K. Maddox, Z. Duan, and C. Xu, “Hierar- chical cross-modal talking face generation with dynamic pixel-wise loss,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition , 2019, pp. 7832–7841

    2019

  10. [18]

    Towards automatic face-to-face translation,

    P. KR, R. Mukhopadhyay, J. Philip, A. Jha, V . Nam- boodiri, and C. Jawahar, “Towards automatic face-to-face translation,” in Proceedings of the 27th ACM Interna- tional Conference on Multimedia , 2019, pp. 1428–1436

    2019

  11. [19]

    A lip sync expert is all you need for speech to lip generation in the wild,

    K. Prajwal, R. Mukhopadhyay, V . P. Namboodiri, and C. Jawahar, “A lip sync expert is all you need for speech to lip generation in the wild,” in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 484–492

    2020

  12. [20]

    Au- dio2head: Audio-driven one-shot talking-head generation with natural head motion,

    S. Wang, L. Li, Y . Ding, C. Fan, and X. Yu, “Au- dio2head: Audio-driven one-shot talking-head generation with natural head motion,” inProceedings of the Thirtieth International Joint Conference On Artificial Intelligence, Ijcai 2021, 2021, pp. 1098–1105

    2021

  13. [21]

    Facial: Synthesizing dynamic talking face with implicit attribute learning,

    C. Zhang, Y . Zhao, Y . Huang, M. Zeng, S. Ni, M. Buda- gavi, and X. Guo, “Facial: Synthesizing dynamic talking face with implicit attribute learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3867–3876

    2021

  14. [22]

    Audio-driven emotional video portraits,

    X. Ji, H. Zhou, K. Wang, W. Wu, C. C. Loy, X. Cao, and F. Xu, “Audio-driven emotional video portraits,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14 080–14 089

    2021

  15. [23]

    Make-a- video: Text-to-video generation without text-video data,

    U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni et al. , “Make-a- video: Text-to-video generation without text-video data,” arXiv:2209.14792, 2022

    2022 arXiv

  16. [24]

    Talking head generation with probabilistic audio-to- visual diffusion priors,

    Z. Yu, Z. Yin, D. Zhou, D. Wang, F. Wong, and B. Wang, “Talking head generation with probabilistic audio-to- visual diffusion priors,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 7645–7655

    2023

  17. [25]

    Diffused heads: Diffusion models beat gans on talking-face generation,

    M. Stypułkowski, K. V ougioukas, S. He, M. Zi˛ eba, S. Petridis, and M. Pantic, “Diffused heads: Diffusion models beat gans on talking-face generation,” inProceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2024, pp. 5091–5100

    2024

  18. [26]

    Nerf: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communi- cations of the ACM , vol. 65, no. 1, pp. 99–106, 2021

    2021

  19. [27]

    3d gaussian splatting for real-time radiance field render- ing

    B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field render- ing.” ACM Transactions on Graphics, vol. 42, no. 4, pp. 139–1, 2023

    2023

  20. [28]

    Dy- namic neural radiance fields for monocular 4d facial avatar reconstruction,

    G. Gafni, J. Thies, M. Zollhofer, and M. Nießner, “Dy- namic neural radiance fields for monocular 4d facial avatar reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8649–8658

    2021

  21. [29]

    Ad-nerf: Audio driven neural radiance fields for talking head synthesis,

    Y . Guo, K. Chen, S. Liang, Y .-J. Liu, H. Bao, and J. Zhang, “Ad-nerf: Audio driven neural radiance fields for talking head synthesis,” in Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, 2021, pp. 5784–5794

    2021

  22. [30]

    Semantic-aware implicit neural audio-driven video por- trait generation,

    X. Liu, Y . Xu, Q. Wu, H. Zhou, W. Wu, and B. Zhou, “Semantic-aware implicit neural audio-driven video por- trait generation,” in European Conference on Computer Vision, 2022, pp. 106–125

    2022

  23. [31]

    Head- nerf: A real-time nerf-based parametric head model,

    Y . Hong, B. Peng, H. Xiao, L. Liu, and J. Zhang, “Head- nerf: A real-time nerf-based parametric head model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20 374–20 384

    2022

  24. [32]

    Dfa- nerf: Personalized talking head generation via disentan- gled face attributes neural rendering,

    S. Yao, R. Zhong, Y . Yan, G. Zhai, and X. Yang, “Dfa- nerf: Personalized talking head generation via disentan- gled face attributes neural rendering,” arXiv:2201.00791, 2022

    2022 arXiv

  25. [33]

    Gaussianavatars: Pho- torealistic head avatars with rigged 3d gaussians,

    S. Qian, T. Kirschstein, L. Schoneveld, D. Davoli, S. Giebenhain, and M. Nießner, “Gaussianavatars: Pho- torealistic head avatars with rigged 3d gaussians,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 20 299–20 309

    2024

  26. [34]

    Headgas: Real-time animatable head avatars via 3d gaussian splatting,

    H. Dhamo, Y . Nie, A. Moreau, J. Song, R. Shaw, Y . Zhou, and E. Pérez-Pellitero, “Headgas: Real-time animatable head avatars via 3d gaussian splatting,” in European Conference on Computer Vision , 2024, pp. 459–476

    2024

  27. [35]

    Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians,

    Y . Xu, B. Chen, Z. Li, H. Zhang, L. Wang, Z. Zheng, and Y . Liu, “Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 1931–1941. 15

    2024

  28. [36]

    Learning dynamic facial radiance fields for few-shot talking head synthesis,

    S. Shen, W. Li, Z. Zhu, Y . Duan, J. Zhou, and J. Lu, “Learning dynamic facial radiance fields for few-shot talking head synthesis,” in European Conference on Computer Vision, 2022, pp. 666–682

    2022

  29. [37]

    Real-time neural radiance talk- ing portrait synthesis via audio-spatial decomposition,

    J. Tang, K. Wang, H. Zhou, X. Chen, D. He, T. Hu, J. Liu, G. Zeng, and J. Wang, “Real-time neural radiance talk- ing portrait synthesis via audio-spatial decomposition,” arXiv:2211.12368, 2022

    2022 arXiv

  30. [38]

    Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis,

    Z. Ye, Z. Jiang, Y . Ren, J. Liu, J. He, and Z. Zhao, “Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis,” arXiv:2301.13430, 2023

    2023 arXiv

  31. [39]

    Geneface++: Generalized and stable real-time audio-driven 3d talking face generation,

    Z. Ye, J. He, Z. Jiang, R. Huang, J. Huang, J. Liu, Y . Ren, X. Yin, Z. Ma, and Z. Zhao, “Geneface++: Generalized and stable real-time audio-driven 3d talking face generation,” arXiv:2305.00787, 2023

    2023 arXiv

  32. [40]

    Sd-nerf: Towards lifelike talking head animation via spatially-adaptive dual-driven nerfs,

    S. Shen, W. Li, X. Huang, Z. Zhu, J. Zhou, and J. Lu, “Sd-nerf: Towards lifelike talking head animation via spatially-adaptive dual-driven nerfs,” IEEE Transactions on Multimedia, vol. 26, pp. 3221–3234, 2023

    2023

  33. [41]

    Talkinggaussian: Structure-persistent 3d talking head synthesis via gaussian splatting,

    J. Li, J. Zhang, X. Bai, J. Zheng, X. Ning, J. Zhou, and L. Gu, “Talkinggaussian: Structure-persistent 3d talking head synthesis via gaussian splatting,” in European Con- ference on Computer Vision , 2024, pp. 127–145

    2024

  34. [42]

    Gaussiantalker: Real-time high-fidelity talking head synthesis with audio-driven 3d gaussian splatting,

    K. Cho, J. Lee, H. Yoon, Y . Hong, J. Ko, S. Ahn, and S. Kim, “Gaussiantalker: Real-time high-fidelity talking head synthesis with audio-driven 3d gaussian splatting,” arXiv:2404.16012, 2024

    2024 arXiv

  35. [43]

    Cstalk: Correlation supervised speech-driven 3d emotional facial animation generation,

    X. Liang, W. Zhuang, T. Wang, G. Geng, G. Geng, H. Xia, and S. Xia, “Cstalk: Correlation supervised speech-driven 3d emotional facial animation generation,” in 2024 IEEE 18th International Conference on Auto- matic Face and Gesture Recognition (FG) , 2024, pp. 1– 5

    2024

  36. [44]

    Mimictalk: Mimicking a personalized and expressive 3d talking face in minutes,

    Z. Ye, T. Zhong, Y . Ren, Z. Jiang, J. Huang, R. Huang, J. Liu, J. He, C. Zhang, Z. Wang et al. , “Mimictalk: Mimicking a personalized and expressive 3d talking face in minutes,” Advances in Neural Information Processing Systems, vol. 37, pp. 1829–1853, 2024

    2024

  37. [45]

    Instag: Learning personalized 3d talking head from few- second video,

    J. Li, J. Zhang, X. Bai, J. Zheng, J. Zhou, and L. Gu, “Instag: Learning personalized 3d talking head from few- second video,” arXiv:2502.20387, 2025

    2025 arXiv

  38. [46]

    Towards automatic face-to-face translation,

    P. K R, R. Mukhopadhyay, J. Philip, A. Jha, V . Nambood- iri, and C. V . Jawahar, “Towards automatic face-to-face translation,” in Proceedings of the 27th ACM Interna- tional Conference on Multimedia , 2019, p. 1428–1436

    2019

  39. [47]

    Hierar- chical cross-modal talking face generation with dynamic pixel-wise loss,

    L. Chen, R. K. Maddox, Z. Duan, and C. Xu, “Hierar- chical cross-modal talking face generation with dynamic pixel-wise loss,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition , 2019, pp. 7824–7833

    2019

  40. [48]

    Latentsync: Audio conditioned latent diffusion models for lip sync,

    C. Li, C. Zhang, W. Xu, J. Xie, W. Feng, B. Peng, and W. Xing, “Latentsync: Audio conditioned latent diffusion models for lip sync,” arXiv:2412.09262, 2024

    2024 arXiv

  41. [49]

    Float: Generative motion latent flow matching for audio-driven talking portrait,

    T. Ki, D. Min, and G. Chae, “Float: Generative motion latent flow matching for audio-driven talking portrait,” arXiv:2412.01064, 2024

    2024

  42. [50]

    Efficient region-aware neural radiance fields for high-fidelity talk- ing portrait synthesis,

    J. Li, J. Zhang, X. Bai, J. Zhou, and L. Gu, “Efficient region-aware neural radiance fields for high-fidelity talk- ing portrait synthesis,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 7568–7578

    2023

  43. [51]

    Synctalk: The devil is in the synchronization for talking head synthesis,

    Z. Peng, W. Hu, Y . Shi, X. Zhu, X. Zhang, H. Zhao, J. He, H. Liu, and Z. Fan, “Synctalk: The devil is in the synchronization for talking head synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 666–676

    2024

  44. [52]

    Gaussianspeech: Audio-driven gaussian avatars,

    S. Aneja, A. Sevastopolsky, T. Kirschstein, J. Thies, A. Dai, and M. Nießner, “Gaussianspeech: Audio-driven gaussian avatars,” arXiv:2411.18675, 2024

    2024 arXiv

  45. [53]

    Degstalk: Decomposed per-embedding gaussian fields for hair-preserving talking face synthesis,

    K. Deng, D. Zheng, J. Xie, J. Wang, W. Xie, L. Shen, and S. Song, “Degstalk: Decomposed per-embedding gaussian fields for hair-preserving talking face synthesis,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing , 2025, pp. 1–5

    2025

  46. [54]

    Emotalkinggaussian: Continuous emotion-conditioned talking head synthesis,

    J. Cha, S. Yoon, V . Strizhkova, F. Bremond, and S. Baek, “Emotalkinggaussian: Continuous emotion-conditioned talking head synthesis,” arXiv:2502.00654, 2025

    2025 arXiv

  47. [55]

    Facenet: A unified embedding for face recognition and clustering,

    F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2015, pp. 815–823

    2015

  48. [56]

    Facial action coding system,

    E. B. Prince, K. B. Martin, D. S. Messinger, and M. Allen, “Facial action coding system,” Environmental Psychology & Nonverbal Behavior , vol. 1, 2015

    2015

  49. [57]

    Differentiable surface splatting for point- based geometry processing,

    W. Yifan, F. Serena, S. Wu, C. Öztireli, and O. Sorkine- Hornung, “Differentiable surface splatting for point- based geometry processing,” ACM Transactions On Graphics, vol. 38, no. 6, pp. 1–14, 2019

    2019

  50. [58]

    Effi- cient emotional adaptation for audio-driven talking-head generation,

    Y . Gan, Z. Yang, X. Yue, L. Sun, and Y . Yang, “Effi- cient emotional adaptation for audio-driven talking-head generation,” in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 2023, pp. 22 634– 22 645

    2023

  51. [59]

    Real3d- portrait: One-shot realistic 3d talking portrait synthesis,

    Z. Ye, T. Zhong, Y . Ren, J. Yang, W. Li, J. Huang, Z. Jiang, J. He, R. Huang, J. Liu et al. , “Real3d- portrait: One-shot realistic 3d talking portrait synthesis,” in The Twelfth International Conference on Learning Representations

  52. [60]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset,

    Z. Zhang, L. Li, Y . Ding, and C. Fan, “Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3661–3670

    2021

  53. [61]

    The unreasonable effectiveness of deep fea- tures as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep fea- tures as a perceptual metric,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 586–595

    2018

  54. [62]

    Image quality assessment: from error visibility to structural similarity,

    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simon- celli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004

    2004

  55. [63]

    Lip reading in the wild,

    J. S. Chung and A. Zisserman, “Lip reading in the wild,” in Computer Vision–ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13 , 2017, pp. 87– 16 103

    2016

  56. [64]

    Out of time: automated lip sync in the wild,

    J. S. Chung and A. Zisserman, “Out of time: automated lip sync in the wild,” in Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, 2017, pp. 251–263

    2016

  57. [65]

    Lip movements generation at a glance,

    L. Chen, Z. Li, R. K. Maddox, Z. Duan, and C. Xu, “Lip movements generation at a glance,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 520–535

    2018

  58. [66]

    Deep speech: Scaling up end-to-end speech recognition,

    A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates et al. , “Deep speech: Scaling up end-to-end speech recognition,” arXiv:1412.5567, 2014

    2014 arXiv

  59. [67]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in Neural Infor- mation Processing Systems , vol. 33, pp. 12 449–12 460, 2020

    2020

  60. [68]

    Hubert: Self- supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 3451–3460, 2021

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.