Pith. sign in

REVIEW 3 major objections 6 minor 47 references

Comparative Analysis of Audio Feature Extraction for Real-Time Talking Portrait Synthesis

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Real-time talking portraits can use Whisper's compact encoder as their audio feature extractor, cutting latency by roughly 80 to 90 percent on longer clips while improving lip-sync confidence scores.

desk verdict The speed story is solid, the quality story isn't—one alignment recipe and three unreplicated clips can't carry it. read the letter →

arxiv 2411.13209 v1 pith:5HFCCF37 submitted 2024-11-20 cs.SD cs.AIcs.HCeess.AS

Pegah Salehi , Sajad Amouei Sheshkal , Vajira Thambawita , Sushant Gautam , Saeed S. Sabet , Dag Johansen , Michael A. Riegler , P{aa}l Halvorsen This is my paper · ORCID
classification cs.SDcs.AIcs.HCeess.AS
keywords talkingportraitsynthesisaudiofeatureextractionWhisperinteractiveavatarneuralradiancefieldslipsynchronizationreal-timeanimationinterviewertraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that swapping the audio feature extractor in a real-time talking-portrait pipeline from conventional speech-representation models to Whisper's compact encoder makes the avatar faster and, on lip-sync quality, better. It builds and tests a complete interactive avatar for interviewer training, where audio feature extraction is a known bottleneck. Across two real-time talking-head generators and three video datasets, Whisper is reported to cut feature-extraction time by 80 to 90 percent on longer audio and to raise mean SyncNet lip-sync confidence while keeping image-quality metrics roughly unchanged. The practical payoff is that an interactive training avatar can respond sooner and with more accurate mouth movements, without sacrificing visual fidelity.

What carries the argument

The load-bearing object is Whisper-Tiny's encoder: a compact transformer that converts log-Mel spectrograms into high-dimensional audio embeddings of width 384, replacing the dedicated speech-feature extractors used by talking-head networks. To feed these embeddings into NeRF-based portrait generators, the paper synchronizes them to 25 frames per second using a sliding window with $w=16$, stride $s=2$, and padding $p=7$, yielding a feature tensor of shape $(750, 16, 384)$ for 30 seconds of audio. This fixed alignment recipe is what allows the same extractor to be dropped into both RAD-NeRF and ER-NeRF, and it is also the premise on which the cross-model comparison rests.

What would settle it

Re-train both talking-head generators from scratch with each of the four extractors while varying the sliding-window width, stride, and padding per extractor, then compare wall-clock feature-extraction time and SyncNet confidence on the same three clips; if a non-Whisper extractor matches or beats Whisper under its own tuned alignment, the claimed advantage is an artifact of the shared recipe.

Watch

Extended reading notes

Core claim

The paper claims that Whisper-Tiny, a roughly 39-million-parameter transformer-based speech-recognition model, can be repurposed as the audio feature extractor in real-time talking-portrait systems. Evaluated inside RAD-NeRF and ER-NeRF on three datasets, it reports that Whisper reduces feature-extraction latency by 80 to 90 percent compared with DeepSpeech 2, Wav2Vec 2.0, and HuBERT, most clearly on longer clips. On rendering quality, Whisper matches or slightly improves PSNR, SSIM, LPIPS, LMD, FID, and AUE, and its clearest gains appear in SyncNet confidence: in the self-driven ER-NeRF setting the mean Sync score is 7.308 for Whisper versus 6.712 for DeepSpeech and 6.312 for Wav2Vec, and in the RAD-NeRF setting it is 6.966 versus 6.407 and 6.503. In cross-driven natural-speech tests Whisper also leads, while HuBERT produces near-zero Sync scores on natural speech; on a slow, pause-heavy synthetic TTS voice Whisper's advantage shrinks. The authors conclude that Whisper's encoder is a faster and, for lip-sync, better audio feature extractor for responsive interactive avatars.

Load-bearing premise

The paper's comparison rests on the assumption that one fixed way of slicing each audio feature stream into 25-frames-per-second pieces is equally fair to all four extractors; if that recipe favors Whisper or hurts HuBERT, the rankings would change.

Editorial extensions

If this is right

  • A talking avatar's response loop becomes faster because audio feature extraction no longer dominates, with Whisper roughly 80 to 90 percent faster on longer audio than DeepSpeech, Wav2Vec, and HuBERT.
  • Lip-sync quality, measured by SyncNet confidence, improves in both self-driven and cross-driven natural-speech settings, so avatars should appear better synchronized without degrading image fidelity.
  • HuBERT in its default configuration is a poor choice for lip-sync in these NeRF pipelines, with near-zero natural-speech Sync scores, so it can be ruled out for this application.
  • Frame rendering remains the largest latency component in the full system, so further responsiveness gains require faster renderers rather than only faster audio features.
  • Whisper's advantage shrinks with slow, pause-heavy synthetic TTS voices, meaning the best audio feature extractor depends on the voice source the avatar actually uses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because one alignment recipe is applied to all four extractors, a per-model sweep of window size, stride, and padding would test whether Whisper's SyncNet gains are intrinsic or partly an artifact of that shared recipe.
  • Beyond the paper: the 80 to 90 percent latency advantage is demonstrated on audio up to 30 seconds, and DeepSpeech's execution time grows with duration, so the speed gap should widen for longer avatar monologues and may shrink for very short utterances.
  • Beyond the paper: Whisper's training on multilingual and noisy speech could make it the preferred extractor in field deployments with background noise, a condition the three clean test clips do not cover.
  • Beyond the paper: the fixed tensor shape ties the comparison to 25 frames per second and 30-second clips, so talking-head systems with other frame rates or window lengths would need a new alignment recipe and the ranking could change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper integrates OpenAI Whisper-Tiny as the audio feature extractor (AFE) in two real-time NeRF-based talking-head frameworks, RAD-NeRF and ER-NeRF, and compares it against DeepSpeech, Wav2Vec 2.0, and HuBERT on three video clips. The authors report an 80-90% reduction in AFE latency for Whisper and, on the basis of PSNR/SSIM/LPIPS/LMD/FID/AUE and SyncNet confidence scores, claim that Whisper also improves lip synchronization and rendering realism. The manuscript further describes a complete interactive avatar pipeline (STT, TTS, GPT, AFE, frame rendering, audio overlay) and includes a component-level latency analysis.

Significance. If the speed result holds, swapping in Whisper-Tiny as an off-the-shelf AFE is a practically useful and nearly drop-in optimization for real-time talking-head systems, and the full pipeline integration is a useful engineering contribution. The paper ships code, compares four AFEs across two frameworks and three datasets, and reports both task-level and component-level timing. The work is not circular: the metrics, baselines, and pretrained models are external, and the comparisons are empirical measurements. However, the quality advantage is currently an unsupported load-bearing claim: the uniform alignment recipe, lack of repeated runs or significance tests, and arithmetic inconsistencies in Table 1 prevent the reader from distinguishing a genuine Whisper advantage from an artifact of the alignment interface or from run-to-run noise.

major comments (3)
  1. [Section 3.1.4, Tables 1-2] The claim that Whisper improves lip-sync quality rests on applying one hand-chosen sliding-window alignment (w=16, stride s=2, padding p=7) uniformly to all four AFE encoders, but the paper provides no per-model output frame-rate analysis, no layer-selection study, and no alignment ablation. The collapse of HuBERT Sync scores to 0.30-0.56 in Table 1 and to roughly 0.5 for natural audio in Table 2, while the synthetic-audio HuBERT scores remain around 5-6, is precisely the pattern one would expect from a systematic temporal misalignment rather than from intrinsic feature insufficiency. Without testing alternative window/stride combinations or alternative HuBERT encoder layers, the relative quality ordering among AFEs is not established.
  2. [Tables 1-2 and Section 4.3.2] The quality comparison is based on single runs on three clips with no variance estimates, repeated seeds, confidence intervals, or significance tests; the claimed "notable improvement" in Sync rests on mean differences of about 0.5-0.6 (e.g., RAD-NeRF Whisper 6.966 vs Wav2Vec 6.503) that are within plausible training noise for NeRF-based models. Moreover, several reported means are arithmetically inconsistent with the per-dataset rows, for example ER-NeRF Deep-Speech mean LPIPS 0.0689 versus a recalculation of 0.0499 from the listed rows, ER-NeRF Deep-Speech mean FID 20.92 versus 12.61, and ER-NeRF Whisper mean PSNR 28.12 versus 27.95. The authors should supply confidence intervals or repeated-seed results and correct these table entries before claiming a quality advantage.
  3. [Section 4.3.2 and Table 2] Whisper does not "generally outperform" the other AFEs in the cross-driven synthetic condition: in RAD-NeRF the synthetic mean Sync score for Whisper is 5.953 versus 6.093 for Wav2Vec, and in ER-NeRF it is 6.066 versus 6.323 for DeepSpeech and 6.159 for Wav2Vec. The paper's own caveat that the slower bot voice with more pauses may have benefited other models directly weakens the abstract's general claim that Whisper "improves specific aspects of rendering quality." The evidence supports only a qualified, natural-voice-specific advantage, and even that advantage needs statistical support.
minor comments (6)
  1. [Section 3.1.4, Eq. (1)] The equation presented as the Whisper log-Mel spectrogram is not a log-Mel spectrogram; it is a truncated discrete-time Fourier transform without the STFT window or mel filterbank, and it is dimensionally inconsistent. Please replace it with a correct STFT/mel-filterbank formulation or remove the equation.
  2. [Section 4.3.2, SyncNet metric] The Sync formula is described as yielding scores between 0 and 1, but Table 1 reports Sync confidence values as high as 8.575. Please clarify that the implementation uses the unbounded SyncNet confidence score or correct the formula/description.
  3. [Section 4.2 and Figure 2] The text says GeneFace++ is excluded because its AFE cannot be measured separately, but Figure 2 appears to include GeneFace++ dashed lines; please clarify the legend and caption so the reader knows which curves correspond to GeneFace++ and whether they include AFE.
  4. [Table 1, ER-NeRF Deep-Speech row] The mean LMD for ER-NeRF Deep-Speech is reported as 2.7932, but a recalculation from the listed per-dataset values 2.561, 2.647, and 2.900 gives approximately 2.7027; please verify all means in the table.
  5. [Throughout] The AFE model names are used inconsistently (e.g., "Deep-Speech," "DeepSpeech," "Wav2Vec," "wav2vec 2.0", "HuBERT"); please standardize the notation for a comparative paper.
  6. [Table 3] The column header "AA Tokens" is not defined in the text; please expand the definition (e.g., "avatar answer tokens") for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the AFE comparison is an external empirical measurement, not a derivation that reduces to its own inputs.

full rationale

The paper's central claims are that Whisper-Tiny reduces AFE latency by roughly 80-90% for longer clips and improves lip-sync (SyncNet confidence) while keeping PSNR/SSIM comparable. Both claims are supported by direct measurements using external baselines (DeepSpeech, Wav2Vec, HuBERT) and two external talking-head frameworks (RAD-NeRF, ER-NeRF) trained from scratch per AFE configuration. The metrics (PSNR, SSIM, LPIPS, LMD, FID, AUE, SyncNet) are computed against ground truth or against standard pretrained evaluators, and no equation in the paper defines the reported outcome in terms of the AFE choice by construction. The only hand-chosen element is the sliding-window alignment recipe (w=16, stride s=2, padding p=7) applied uniformly to all four AFE models in Section 3.1.4. This could bias the comparison if the recipe suits Whisper's 50 Hz feature grid better than HuBERT's or Wav2Vec's native rates, and the near-zero HuBERT Sync scores are a plausible symptom of temporal misalignment rather than intrinsic feature quality. However, that is an experimental validity concern, not circularity: the paper does not fit the recipe to Whisper's result, and the comparison remains an external, falsifiable measurement. Self-citations in the paper, such as [19] and [10], provide application context and prior system limitations rather than the load-bearing AFE comparison. No fitted parameter is renamed as a prediction, and no uniqueness theorem or prior author result is imported to force the conclusion. Therefore the derivation chain is self-contained with respect to circularity, and the honest finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper adds no new theory or fitted constants; its central claims depend on hand-set alignment parameters and on the transferability of ASR representations to lip-sync. The main free choice is the window/stride/padding configuration, which is not ablated or tuned per AFE. The quality conclusions additionally presuppose that standard image and sync metrics are a valid proxy for realism, a premise the authors themselves plan to test with subjective evaluations in future work.

free parameters (1)
  • AFE-to-frame sliding-window alignment (window w=16, stride s=2, padding p=7) = w=16, s=2, p=7
    Hand-chosen in Section 3.1.4 to map a 30-second Whisper encoder output to a 750 by 16 by 384 feature matrix synchronized to 25 FPS. The paper gives no ablation or per-AFE tuning, so the comparison may depend on this choice.
assumptions (4)
  • domain assumption Pre-trained ASR encoders, especially Whisper, provide audio representations suitable for talking-head lip animation without fine-tuning or per-encoder adaptation.
    Invoked in Sections 3.1.4 and 4.3; the paper does not fine-tune encoders for lip-sync or ablate which feature layer is used.
  • ad hoc to paper The fixed sliding-window mapping (w=16, stride s=2, padding p=7) gives precise temporal alignment for every AFE model.
    Stated in Section 3.1.4 as 'This setup ensures precise temporal alignment across 750 frames over 30 seconds'; no derivation, per-model tuning, or ablation is provided.
  • domain assumption Objective metrics (PSNR, SSIM, LPIPS, LMD, FID, AUE, SyncNet) capture the perceived realism and lip-sync quality relevant to interactive avatar training.
    Used as the sole quality evidence in Section 4.3.2; no human user study is performed, yet the abstract and discussion claim 'more realistic' interactions.
  • domain assumption The three selected clips, one privately sourced, each about 4.5 minutes long, are representative of real-time interviewer-training speech conditions.
    Dataset described in Section 4.1.1; no variation in languages, ages, accents, noise, or interview-specific dialogue is tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comparative Analysis of Audio Feature Extraction for Real-Time Talking Portrait Synthesis." pith.science (2026). https://pith.science/paper/5HFCCF37

@misc{pith2026241113209,
  author       = {Pith},
  title        = {Pith review of: Comparative Analysis of Audio Feature Extraction for Real-Time Talking Portrait Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5HFCCF37}},
  note         = {Machine review of arXiv:2411.13209}
}
read the original abstract

This paper examines the integration of real-time talking-head generation for interviewer training, focusing on overcoming challenges in Audio Feature Extraction (AFE), which often introduces latency and limits responsiveness in real-time applications. To address these issues, we propose and implement a fully integrated system that replaces conventional AFE models with Open AI's Whisper, leveraging its encoder to optimize processing and improve overall system efficiency. Our evaluation of two open-source real-time models across three different datasets shows that Whisper not only accelerates processing but also improves specific aspects of rendering quality, resulting in more realistic and responsive talking-head interactions. These advancements make the system a more effective tool for immersive, interactive training applications, expanding the potential of AI-driven avatars in interviewer training.

Figures

Figures reproduced from arXiv: 2411.13209 by the authors.

Figure 1
Figure 1. (a) System architecture of the interactive child avatar, detailing the integration of key modules: (1) Listening, (2) STT, (3) Language Processing, (4) TTS, (5) AFE, (6) Frames Rendering, and (7) Audio Overlay. This setup simulates natural conversation, allowing the user to interact with the avatar as if communicating with a real person. (b) User interaction with the child avatar system. AFE processes the incoming a… view at source ↗
Figure 2
Figure 2. Execution time comparison of open-source real-time talking-head generation models, including RAD [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Execution time comparison of different AFE models, including Deep-Speech [ [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Execution time comparison of RAD-NeRF [26] and ER-NeRF [28] across different AFE models. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Quality comparison: Examples of visualizations of RAD-NeRF [ [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Quality comparison: Examples of visualizations of ER-NeRF [ [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 30 canonical work pages

  1. [1]

    Simulation-based learning in higher education: A meta-analysis

    Olga Chernikova, Nicole Heitzmann, Matthias Stadler, Doris Holzberger, Tina Seidel, and Frank Fischer. Simulation-based learning in higher education: A meta-analysis. Review of educational research, 90(4):499–541, 2020

    work page 2020

  2. [2]

    Psychological foundations of emerging technologies for teaching and learning in higher education

    Helen Crompton, Matthew Bernacki, and Jeffrey A Greene. Psychological foundations of emerging technologies for teaching and learning in higher education. Current Opinion in Psychology, 36:101–105, 2020

    work page 2020

  3. [3]

    Michael E Lamb. Difficulties translating research on forensic interview practices to practitioners: Finding water, leading horses, but can we get them to drink? American psychologist, 71(8):710, 2016

    work page 2016

  4. [4]

    Designing effective training programs for investigative interviewers of children

    Martine B Powell. Designing effective training programs for investigative interviewers of children. Current issues in criminal justice, 20(2):189–208, 2008

    work page 2008

  5. [5]

    Michael E Lamb, Yael Orbach, Irit Hershkowitz, Phillip W Esplin, and Dvora Horowitz. A structured forensic interview protocol improves the quality and informativeness of investigative interviews with children: A review of research using the nichd investigative interview protocol. Child abuse & neglect, 31(11-12):1201–1231, 2007

    work page 2007

  6. [6]

    Interviewing children

    Thomas D Lyon. Interviewing children. Annual review of law and social science , 10(1):73–89, 2014

    work page 2014

  7. [7]

    Tell me what happened: Questioning children about abuse

    Michael E Lamb, Deirdre A Brown, Irit Hershkowitz, Yael Orbach, and Phillip W Esplin. Tell me what happened: Questioning children about abuse. John Wiley & Sons, 2018

    work page 2018

  8. [8]

    An overview of mock interviews as a training tool for interviewers of children

    Martine B Powell, Sonja P Brubacher, and Gunn Astrid Baugerud. An overview of mock interviews as a training tool for interviewers of children. Child Abuse & Neglect, 129:105685, 2022

    work page 2022

Show all 47 references
  1. [9]

    Towards an ai-driven talking avatar in virtual reality for investigative interviews of children

    Syed Zohaib Hassan, Pegah Salehi, Ragnhild Klingenberg Røed, Pål Halvorsen, Gunn Astrid Baugerud, Miriam Sinkerud Johnson, Pierre Lison, Michael Riegler, Michael E Lamb, Carsten Griwodz, et al. Towards an ai-driven talking avatar in virtual reality for investigative interviews...

    2022

  2. [10]

    Pegah Salehi, Syed Zohaib Hassan, Saeed Shafiee Sabet, Gunn Astrid Baugerud, Miriam Sinkerud Johnson, Pål Halvorsen, and Michael A Riegler. Is more realistic better? a comparison of game engine and gan-based avatars 1https://developer.nvidia.com/ace 14 A PREPRINT - NOVEMBER 21...

    2024

  3. [11]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99–106, 2021

    2021

  4. [12]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning , pages 28492–28518. PMLR, 2023

    2023

  5. [13]

    Whisper afe for talking heads generation

    Pegah Salehi. Whisper afe for talking heads generation. https://github.com/pegahs1993/ Whisper-AFE-TalkingHeadsGen, 2024. [Accessed 15-02-2024]

    2024

  6. [14]

    Technological acceptance of an avatar based interview training application: The development and technological acceptance study of the avbit application., 2021

    Kevin Charles Dalli. Technological acceptance of an avatar based interview training application: The development and technological acceptance study of the avbit application., 2021

    2021

  7. [15]

    A field assessment of child abuse investigators’ engagement with a child-avatar to develop interviewing skills.Child Abuse & Neglect, 143:106324, 2023

    Ragnhild Klingenberg Røed, Martine B Powell, Michael A Riegler, and Gunn Astrid Baugerud. A field assessment of child abuse investigators’ engagement with a child-avatar to develop interviewing skills.Child Abuse & Neglect, 143:106324, 2023

    2023

  8. [16]

    Evaluation of a comprehensive interactive training system for investigative interviewers of children

    Mairi S Benson and Martine B Powell. Evaluation of a comprehensive interactive training system for investigative interviewers of children. Psychology, public policy, and law, 21(3):309, 2015

    2015

  9. [17]

    Training in investigative interviews of children: Serious gaming paired with feedback improves interview quality

    F Pompedda. Training in investigative interviews of children: Serious gaming paired with feedback improves interview quality. doctoral dissertation. Turku: Åbo Akademi University , 2018

    2018

  10. [18]

    How to prepare for conversations with children about suspicions of sexual abuse? evaluation of an interactive virtual reality training for student teachers

    Niels Krause, Elsa Gewehr, Hermann Barbe, Marie Merschhemke, Frieda Mensing, Bruno Siegel, Jürgen L Müller, Renate V olbert, Peter Fromberger, Anett Tamm, et al. How to prepare for conversations with children about suspicions of sexual abuse? evaluation of an interactive virtu...

    2024

  11. [19]

    A theoretical and empirical analysis of 2d and 3d virtual environments in training for child interview skills

    Pegah Salehi, Syed Zohaib Hassan, Gunn Astrid Baugerud, Martine Powell, Miriam S Johnson, Dag Johansen, Saeed Shafiee Sabet, Michael A Riegler, and Pål Halvorsen. A theoretical and empirical analysis of 2d and 3d virtual environments in training for child interview skills. IEE...

    2024

  12. [20]

    Live speech portraits: real-time photorealistic talking-head animation

    Yuanxun Lu, Jinxiang Chai, and Xun Cao. Live speech portraits: real-time photorealistic talking-head animation. ACM Transactions on Graphics (ToG), 40(6):1–17, 2021

    2021

  13. [21]

    Generative pre-training for speech with autoregressive predictive coding

    Yu-An Chung and James Glass. Generative pre-training for speech with autoregressive predictive coding. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3497–3501. IEEE, 2020

    2020

  14. [22]

    Realtalk: Real-time and realistic audio-driven face generation with 3d facial prior-guided identity alignment network

    Xiaozhong Ji, Chuming Lin, Zhonggan Ding, Ying Tai, Jian Yang, Junwei Zhu, Xiaobin Hu, Jiangning Zhang, Donghao Luo, and Chengjie Wang. Realtalk: Real-time and realistic audio-driven face generation with 3d facial prior-guided identity alignment network. arXiv preprint arXiv:2...

    2024 arXiv

  15. [23]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1, 2023

    2023

  16. [24]

    Gstalker: Real-time audio-driven talking face generation via deformable gaussian splatting

    Bo Chen, Shoukang Hu, Qi Chen, Chenpeng Du, Ran Yi, Yanmin Qian, and Xie Chen. Gstalker: Real-time audio-driven talking face generation via deformable gaussian splatting. arXiv preprint arXiv:2404.19040, 2024

    2024 arXiv

  17. [25]

    Gaussiantalker: Real-time talking head synthesis with 3d gaussian splatting

    Kyusun Cho, Joungbin Lee, Heeji Yoon, Yeobin Hong, Jaehoon Ko, Sangjun Ahn, and Seungryong Kim. Gaussiantalker: Real-time talking head synthesis with 3d gaussian splatting. In ACM Multimedia 2024, 2024

    2024

  18. [26]

    Real-time neural radiance talking portrait synthesis via audio-spatial decomposition

    Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211.12368, 2022

    2022 arXiv

  19. [27]

    Ad-nerf: Audio driven neural radiance fields for talking head synthesis

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5784–5794, 2021

    2021

  20. [28]

    Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7568–7578, 2023

    2023

  21. [29]

    Geneface++: Generalized and stable real-time audio-driven 3d talking face generation

    Zhenhui Ye, Jinzheng He, Ziyue Jiang, Rongjie Huang, Jiawei Huang, Jinglin Liu, Yi Ren, Xiang Yin, Zejun Ma, and Zhou Zhao. Geneface++: Generalized and stable real-time audio-driven 3d talking face generation. arXiv preprint arXiv:2305.00787, 2023. 15 A PREPRINT - NOVEMBER 21, 2024

    2023 arXiv

  22. [30]

    R2-talker: Realistic real-time talking head synthesis with hash grid landmarks encoding and progressive multilayer conditioning

    Zhiling Ye, LiangGuo Zhang, Dingheng Zeng, Quan Lu, and Ning Jiang. R2-talker: Realistic real-time talking head synthesis with hash grid landmarks encoding and progressive multilayer conditioning. arXiv preprint arXiv:2312.05572, 2023

    2023 arXiv

  23. [31]

    Deep speech 2: End-to-end speech recognition in english and mandarin

    Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine l...

    2016

  24. [32]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems , 33:12449– 12460, 2020

    2020

  25. [33]

    Hubert: How much can a bad teacher benefit asr pre-training? In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6533–6537

    Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: How much can a bad teacher benefit asr pre-training? In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6533–65...

    2021

  26. [34]

    Bidirectional recurrent neural networks

    Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673–2681, 1997

    1997

  27. [35]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  28. [36]

    vq-wav2vec: Self-supervised learning of discrete speech representations

    Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910.05453, 2019

    1910 arXiv

  29. [37]

    Product quantization for nearest neighbor search

    Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence , 33(1):117–128, 2010

    2010

  30. [38]

    Librispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015

    2015

  31. [39]

    Libri-light: A benchmark for asr with limited or no supervision

    Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE Internati...

    2020

  32. [40]

    Spanbert: Improving pre-training by representing and predicting spans

    Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics , 8:64–77, 2020

    2020

  33. [41]

    Aws polly

    Amazon Web Services. Aws polly. In Amazon Web Services, page https://aws.amazon.com/polly/. Amazon, 2024. [Accessed 28-09-2024]

    2024

  34. [42]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018

    2018

  35. [43]

    Lip movements generation at a glance

    Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements generation at a glance. In Proceedings of the European conference on computer vision (ECCV) , pages 520–535, 2018

    2018

  36. [44]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  37. [45]

    Openface: an open source facial behavior analysis toolkit

    Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE winter conference on applications of computer vision (WACV) , pages 1–10. IEEE, 2016

    2016

  38. [46]

    Out of time: automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017

    2016

  39. [47]

    The uncanny valley [from the field]

    Masahiro Mori, Karl F MacDorman, and Norri Kageki. The uncanny valley [from the field]. IEEE Robotics & automation magazine, 19(2):98–100, 2012. 16

    2012

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.