Pith. sign in

REVIEW 3 major objections 5 minor 112 references

Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces MF-Talk, a mask-free three-stage pipeline that generates talking faces by first editing every frame to a closed-mouth neutral state and then adapting lips to the audio, requiring neither masked inputs nor identity…

desk verdict The mask-free pipeline is a genuine departure from Wav2Lip-style inpainting and the ablations are well done, but the cross-matching evaluation in Table 2 inflates the advantage by scoring against an input that masking baselines are deliberately asked to destroy. read the letter →

arxiv 2507.20953 v1 pith:JR5NN6Y7 submitted 2025-07-28 cs.CV

Dogucan Yaman , Fevziye Irem Eyiokur , Leonard Bärmann , Haz{i}m Kemal Ekenel , Alexander Waibel This is my paper
classification cs.CV
keywords talkingfacegenerationaudio-drivenlipreanimationmask-freeeditinglandmarkpredictionidentitypreservationsynchronizationclosed-mouthneutralizationGAN-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes MF-Talk, a three-stage pipeline for audio-driven talking face generation that abandons the standard inpainting recipe of masking the lower face and supplying an unmasked identity reference image. Instead it first predicts neutral closed-mouth landmarks for each frame, edits the input face to match them, and then adapts the lips to the audio on the fully unmasked image. The claim is that keeping all input pixels available simplifies the generation task and removes mask artifacts and reference-image interference, so visual quality and identity are preserved better while lip-sync stays competitive. On the LRS2 and HDTF benchmarks the method reports the best SSIM, PSNR, FID, and CSIM scores among the compared methods, and the masking ablation shows the mask-free version outperforms masked variants on every metric.

What carries the argument

The machinery is a cascade of three trained modules: a transformer-based landmark predictor (TL) that maps the current pose landmarks plus several previous frames' landmarks to a neutral, closed-mouth landmark set; a landmark-driven GAN face editor (GE) that modifies only the mouth region of the full-resolution input to match those predicted landmarks; and a lip-adaptation generator (GL) that takes the unmasked neutral-mouth image plus a mel-spectrogram and applies audio-conditioned SPADE/AdaIN modulation to generate synchronized speech lip shapes. The pipeline's key move is the intermediate neutral-mouth representation: it converts a video whose frames have arbitrary mouth states into a uniform closed-mouth state, so the final audio-conditioning stage always starts from the same clean, fully visible input. The authors also adapt a stabilized synchronization loss that compares generated lips against ground-truth lips directly, removing the reference-image term because no reference exists.

What would settle it

Feed a video frame that is already a closed neutral mouth into the pipeline, and compare the final output against the same frame processed through the full TL-to-GE-to-GL chain: if the pipeline degrades already-neutral frames materially, the loss is being injected by the editing stages, not the audio adaptation. Concretely, one can segment LRS2 test frames by mouth aperture and check whether videos whose mouths must be closed from a wide-open state show disproportionately lower SSIM and CSIM and more teeth artifacts than videos that start nearly closed; the paper's own limitation states teeth are suboptimal because the neutral input hides them.

Watch

Extended reading notes

Core claim

The central discovery is a reformulation of audio-driven lip reanimation: rather than regenerate a masked lower half from an identity reference, the pipeline converts every input frame into the same neutral state—lips closed—and then lets a single lip-adaptation network animate the unmasked mouth from audio. Because the input image is never masked, identity-bearing texture, pose, lighting, and background survive intact, and because no identity reference is used, the model cannot copy an unaligned lip shape from a reference frame. The authors demonstrate through quantitative and user studies that this mask-free route yields higher visual quality and identity preservation than the inpainting-based state of the art while remaining competitive in audio-lip synchronization, and they isolate the contribution of each stage through ablations.

Load-bearing premise

The closed-mouth editing stages must be accurate enough that their errors do not compound: if the landmarks or the edited neutral-mouth image are wrong, the final lip-adaptation stage receives corrupted inputs and the video inherits those errors.

Editorial extensions

If this is right

  • Inpainting-based talking face models can be replaced by mask-free editing without sacrificing lip-sync; the reported LSE-C and LSE-D scores are on par with or better than most masked baselines.
  • Identity preservation improves because no identity reference image is needed; CSIM is the best in both standard and cross-matched evaluations.
  • The pipeline needs only a single input image at inference, so it can animate a still face from audio alone.
  • The cross-matching experiment (random video-audio pairs) shows mask-free training largely avoids lip leakage, the copying of reference lip shapes that degrades masked models.
  • Removing the identity reference also simplifies the data pipeline, since there is no need to randomly sample reference frames during training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but the same neutral-mouth-then-animate recipe could be transplanted onto higher-resolution or diffusion-based backbones, where masking artifacts are even costlier than in the GAN implementation shown here.
  • A natural extension would be to feed ground-truth neutral-mouth frames into the lip-adaptation stage, separating errors introduced by the landmark predictor and face editor from errors of the lip-adaptation stage itself; the paper's ablations do not isolate this end-to-end error budget.
  • Because the method needs no identity reference, it may transfer more readily to one-shot and cross-identity dubbing settings where a clean reference frame is unavailable; this is an application path the paper does not explore.
  • The reported teeth limitation suggests the closed-mouth prior caps the model's ability to render tongue and teeth; an explicit teeth prior or masking of only the inner mouth could be a follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MF-Talk, a mask-free, three-stage pipeline for audio-driven talking face generation: a transformer-based landmark predictor (TL) that predicts neutral/closed-mouth lip and jaw landmarks, a GAN-based face editor (GE) that renders the input face with a closed mouth conditioned on those landmarks, and a lip adaptation model (GL) that injects audio features to synthesize synchronized lip movements. The central claim is that by avoiding masked inputs and identity-reference images, the method improves visual quality and identity preservation relative to inpainting-based baselines. Experiments on LRS2 and HDTF compare against eight prior methods with standard metrics, and several ablations, a user study, and a runtime analysis are reported.

Significance. If the central claim holds, MF-Talk is a useful alternative to the dominant masked-inpainting paradigm for lip reanimation, avoiding mask-related artifacts and identity-reference leakage while retaining competitive lip-sync. The paper's strengths include a coherent three-stage design, an extensive ablation program that includes the masking strategy itself, hyperparameters k and T, a user study, runtime measurements, and an explicit acknowledgment of the teeth-generation limitation. The main quantitative support, however, rests on an evaluation protocol that compares methods against the original unmasked input frames, which systematically favors the mask-free method when competitors receive masked inputs; this issue is load-bearing for the paper's headline claim and needs to be addressed before the conclusions can be fully accepted.

major comments (3)
  1. [Sec. 4.1, Table 2] The cross-matched evaluation on LRS2 computes SSIM, PSNR, and CSIM against the original input frame for all methods, as stated in the text: "we still use the input images to measure these metrics." Masking-based baselines (Wav2Lip, VideoReTalking, TalkLip, etc.) receive a lower-half-masked input, so these metrics penalize them for not reconstructing pixels the model never saw, while MF-Talk receives the full unmasked frame and can achieve high scores by editing only the mouth. The reported advantage in visual quality and identity preservation is therefore partly definitional in this table. Please re-evaluate all methods against the ground-truth frame of the same video (or, if that is not possible for the cross-paired setting, against a common masked reference so that all models are scored on the same visible information), and report the results separately from the standard paired evaluation.
  2. [Sec. 4.3.3, Table 6] The masking ablation in Table 6 appears to use the same input-frame reference for SSIM/PSNR/CSIM. In the 'Baseline' and 'Ours w/ masking' rows the lower half of the face is masked, while 'Ours (Mask-Free)' is not, so the comparison conflates access to input pixels with generation quality. The text does not explicitly state the reference for Table 6, but the caption-style claim that the mask-free approach "clearly demonstrates the best performance across all metrics" needs a clean protocol: compute the metrics for the masked variants against a masked ground-truth reference, or restrict the metrics to the unmasked upper-face region, and add variance or confidence intervals across runs. Without this, Table 6 does not establish that mask-free editing is superior for an honest reconstruction task.
  3. [Sec. 5, Limitations] The paper's own Limitations section concedes that the model generates suboptimal teeth because the lip-adaptation input has a closed mouth with no visible teeth, and it provides no end-to-end error analysis isolating how much of the final quality loss originates from the landmark predictor (TL) and face editor (GE) versus the lip adaptation model (GL). This is an honest limitation, but it directly touches the claim of "enhanced visual quality." Please add an experiment that feeds oracle neutral-mouth images (e.g., original closed-mouth frames from the training set, or the output of a perfect editor) into GL and compares the result with the full pipeline, to bound the compounding error from the first two stages.
minor comments (5)
  1. [Sec. 3.4 / Sec. 4] SyncNet is used both as the training objective (the adapted stabilized synchronization loss in Eq. 8) and as the evaluation metric (LSE-C/LSE-D). The user study provides some independent evidence, but the circularity should be explicitly acknowledged, and an additional sync metric not derived from the same network (e.g., a lip-reading based metric) would strengthen the claim of "comparable" lip-sync performance.
  2. [Tables 1, 2, 6] No table reports variance, confidence intervals, or significance tests. Several differences in Table 1 are small (e.g., LSE-C 7.76 for Ours vs. 7.87 for Diff2Lip; PSNR 33.96 vs. 32.64 for PLGAN), so the repeated use of "best" and "clearly outperforms" should be supported by repeated evaluation runs or paired statistical tests.
  3. [Table 6] The column header "Ep." is not defined in the text; please specify whether it denotes the number of training epochs.
  4. [Table 1 / Sec. 4.2] DINet is trained on HDTF while the other methods are not, as acknowledged in the qualitative discussion; this should be stated in the Table 1 caption so readers can interpret the HDTF numbers correctly.
  5. [Appendix F] There are typos in the supplementary text: "demonsttrate" should be "demonstrate" and "outout" should be "output."

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the pipeline is empirical and its central claim does not reduce to its inputs, though a few non-load-bearing self-citations appear.

full rationale

MF-Talk is an empirical three-stage pipeline; no equation in the paper derives an output from an input that is itself the claimed result. The landmark predictor, face editor, and lip adapter are trained with reconstruction, perceptual, adversarial, landmark, and synchronization losses, and their outputs are measured against held-out test data. The adapted stabilized synchronization loss (Eq. 8) is taken from the authors' prior work [94], and the evaluation metrics LSE-C/LSE-D share SyncNet with the training objective; these are self-referential elements, but the paper's central claim (mask-free editing preserves visual quality and identity better than masking) is not justified by those citations. The cross-matched and masking-ablation protocols compute SSIM/PSNR/CSIM against the original input image, as the paper states: 'we still use the input images to measure these metrics.' This disadvantages masked baselines, since they receive masked lower halves and thus cannot be as close to the original input by construction; however, this is an evaluation-fairness and correctness concern, not a circular derivation whose prediction is equivalent to its input. The masking ablation is the intended test of the information-loss hypothesis, and the mask-free model still must edit the mouth region without corrupting identity; the reported numbers are empirical, not entailed by the evaluation protocol alone. The Limitations section explicitly acknowledges suboptimal teeth due to closed-mouth inputs, which is an honest statement rather than a hidden circularity. Overall, the method is self-contained against external benchmarks and the self-citations are not load-bearing for the main result.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The paper's empirical results rest mainly on standard domain assumptions about landmark detectors, SyncNet as a sync measure, and perceptual losses as quality proxies; no new physical or conceptual entities are introduced. Its hand-tuned loss weights and hyperparameters are listed above.

free parameters (7)
  • Lip landmark weight lambda in TL loss (Eq. 5) = 10
    Chosen to emphasize lip landmark reconstruction over jaw landmark reconstruction.
  • Lambda weights for GE losses (Eq. 6) = (1, 0.1, 0.25)
    Weights for perceptual, feature matching, and landmark reconstruction losses, chosen empirically.
  • Lambda weights for GL losses (Eq. 7) = (4, 0.5, 10)
    Weights for perceptual, synchronization, and pixel reconstruction losses, chosen empirically.
  • Number of previous frames k for TL = 1
    Ablated in Table 12; k=1 gives the best or tied best landmark metrics.
  • Temporal window T = 1 for TL/GE, 5 for GL
    Follows Wav2Lip's T=5 convention for lip adaptation; T=1 for the neutral-mouth stages because inter-frame lip consistency is not needed there.
  • Face crop margin = 10% at the bottom
    Chosen empirically to avoid cutting the chin without adding too much background.
  • Input resolution = 128 x 128
    Constrained by the low resolution of LRS2 faces; all comparisons use this resolution.
assumptions (4)
  • domain assumption Mediapipe's 131-point landmark detection and FAN face detection with a 10% crop margin provide sufficiently accurate geometry for landmark-conditioned editing.
    Invoked in Section 3.1 for landmark extraction and Section 3.5 for cropping; if landmarks are noisy, TL and GE both fail.
  • domain assumption SyncNet cosine similarity is a valid proxy for audio-lip synchronization, in training loss and evaluation metrics.
    Section 3.4 uses SyncNet features for Lads and Appendix C uses LSE-C/LSE-D for evaluation; this is standard but unproven here.
  • domain assumption The distance between upper and lower lip landmarks reliably labels images as open or closed mouth for training the TL subset and mouth classifier.
    Sections 3.5, B.1, and B.3 define the closed-mouth subset by this distance; labeling errors propagate to both TL and GE training.
  • domain assumption Perceptual and feature-matching losses in unmasked editing preserve identity and avoid mask-related artifacts.
    Section 3.2 claims preservation from Lper and LF M without masking; this is a modeling assumption, not a theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation." pith.science (2026). https://pith.science/paper/JR5NN6Y7

@misc{pith2026250720953,
  author       = {Pith},
  title        = {Pith review of: Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JR5NN6Y7}},
  note         = {Machine review of arXiv:2507.20953}
}
read the original abstract

Audio-Driven Talking Face Generation aims at generating realistic videos of talking faces, focusing on accurate audio-lip synchronization without deteriorating any identity-related visual details. Recent state-of-the-art methods are based on inpainting, meaning that the lower half of the input face is masked, and the model fills the masked region by generating lips aligned with the given audio. Hence, to preserve identity-related visual details from the lower half, these approaches additionally require an unmasked identity reference image randomly selected from the same video. However, this common masking strategy suffers from (1) information loss in the input faces, significantly affecting the networks' ability to preserve visual quality and identity details, (2) variation between identity reference and input image degrading reconstruction performance, and (3) the identity reference negatively impacting the model, causing unintended copying of elements unaligned with the audio. To address these issues, we propose a mask-free talking face generation approach while maintaining the 2D-based face editing task. Instead of masking the lower half, we transform the input images to have closed mouths, using a two-step landmark-based approach trained in an unpaired manner. Subsequently, we provide these edited but unmasked faces to a lip adaptation model alongside the audio to generate appropriate lip movements. Thus, our approach needs neither masked input images nor identity reference images. We conduct experiments on the benchmark LRS2 and HDTF datasets and perform various ablation studies to validate our contributions.

Figures

Figures reproduced from arXiv: 2507.20953 by the authors.

Figure 1
Figure 1. Demonstration of the traditional talking face generation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Mask-related problems. Generated images are clearly [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Our mask-free talking face generation pipeline in inference. First, a landmark predictor ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of our model with SOTA methods. The samples are randomly selected from generated videos in the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Generated samples with neutral mouth by different [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Generated faces with baseline, our pipeline with mask [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Additional results from the HDTF dataset. We compare the performance of different models with our model. Each set of three [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Generated landmark samples. In each block, the first row shows the landmark map of an input video frame, generally a talking [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Output images demonstrating the performance of our face editing model ( [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Demonstration of the output of each submodules along with ground-truth (GT) samples. In each block, rows demonstrate the [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

112 extracted references · 67 canonical work pages

  1. [1]

    Deep audio-visual speech recognition

    Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 44(12):8717–8727, 2018. 4, 5, 6

    2018

  2. [2]

    Self-supervised learning of audio- visual objects from video

    Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio- visual objects from video. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16 , pages 208–224. Springer, 2020. 3

    2020

  3. [3]

    A morphable model for the synthesis of 3d faces

    V olker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2 , pages 157–164, 2023. 3

    2023

  4. [4]

    Large scale 3d mor- phable models

    James Booth, Anastasios Roussos, Allan Ponniah, David Dunaway, and Stefanos Zafeiriou. Large scale 3d mor- phable models. International Journal of Computer Vision, 126(2):233–254, 2018. 3

    2018

  5. [5]

    V oice puppetry

    Matthew Brand. V oice puppetry. InProceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 21–28, 1999. 2

    1999

  6. [6]

    How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)

    Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In International Conference on Computer Vision, 2017. 5, 2

    2017

  7. [7]

    JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation

    Sai Tanmay Reddy Chakkera, Aggelina Chatziagapi, and Dimitris Samaras. Jean: Joint expression and audio- guided nerf-based talking face generation. arXiv preprint arXiv:2409.12156, 2024. 3

    work page Pith review arXiv 2024

  8. [8]

    TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans

    Aggelina Chatziagapi, Bindita Chaudhuri, Amit Kumar, Rakesh Ranjan, Dimitris Samaras, and Nikolaos Sarafianos. Talkinnerf: Animatable neural fields for full-body talking humans. arXiv preprint arXiv:2409.16666, 2024. 3

    work page Pith review arXiv 2024

Show all 112 references
  1. [9]

    Implicit neural head synthesis via controllable lo- cal deformation fields

    Chuhan Chen, Matthew O’Toole, Gaurav Bharaj, and Pablo Garrido. Implicit neural head synthesis via controllable lo- cal deformation fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 416–426, 2023. 3

    2023

  2. [10]

    Audio-visual synchronisation in the wild

    Honglie Chen, Weidi Xie, Triantafyllos Afouras, Ar- sha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Audio-visual synchronisation in the wild. arXiv preprint arXiv:2112.04432, 2021. 3

    2021 arXiv

  3. [11]

    Hierarchical cross-modal talking face generation with dynamic pixel-wise loss

    Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7832–7841, 2019. 2, 3, 6

    2019

  4. [12]

    Videoretalking: Audio-based lip synchronization for talking head video editing in the wild

    Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022. 2, 3, 5, 6

    2022

  5. [13]

    Gpavatar: Generaliz- able and precise head avatar from image (s)

    Xuangeng Chu, Yu Li, Ailing Zeng, Tianyu Yang, Lijian Lin, Yunfei Liu, and Tatsuya Harada. Gpavatar: Generaliz- able and precise head avatar from image (s). arXiv preprint arXiv:2401.10215, 2024. 3

    2024 arXiv

  6. [14]

    Lip reading in the wild

    Joon Son Chung and Andrew Zisserman. Lip reading in the wild. In Computer Vision–ACCV 2016: 13th Asian Confer- ence on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13 , pages 87–103. Springer, 2017. 3

    2016

  7. [15]

    Out of time: auto- mated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: auto- mated lip sync in the wild. InComputer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017. 6, 2

    2016

  8. [16]

    Perfect match: Improved cross-modal embeddings for audio-visual synchronisation

    Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang. Perfect match: Improved cross-modal embeddings for audio-visual synchronisation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3965–3969. IEEE, 2019. 3

    2019

  9. [17]

    Speech-driven facial animation us- ing cascaded gans for learning of motion and texture

    Dipanjan Das, Sandika Biswas, Sanjana Sinha, and Bro- jeshwar Bhowmick. Speech-driven facial animation us- ing cascaded gans for learning of motion and texture. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, p...

    2020

  10. [18]

    Imagenet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2

    2009

  11. [19]

    Arcface: Additive angular margin loss for deep face recognition

    Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 4690–4699, 2019. 6, 3

    2019

  12. [20]

    End-to-end generation of talking faces from noisy speech

    Sefik Emre Eskimez, Ross K Maddox, Chenliang Xu, and Zhiyao Duan. End-to-end generation of talking faces from noisy speech. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1948–1952. IEEE, 2020. 3

    2020

  13. [21]

    Efficient emotional adaptation for audio- driven talking-head generation

    Yuan Gan, Zongxin Yang, Xihang Yue, Lingyun Sun, and Yi Yang. Efficient emotional adaptation for audio- driven talking-head generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22634–22645, 2023. 3

    2023

  14. [22]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. 4

    2014

  15. [23]

    Stylesync: High-fidelity generalized and personalized lip sync in style-based gener- ator

    Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Stylesync: High-fidelity generalized and personalized lip sync in style-based gener- ator. In Proceedings of the IEEE/CVF Conference on C...

  16. [24]

    Ad-nerf: Audio driven neural radiance fields for talking head synthesis

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hu- jun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 5784–5794, 2021. 3

    2021

  17. [25]

    Audio vision: Using 9 audio-visual synchrony to locate sounds

    John Hershey and Javier Movellan. Audio vision: Using 9 audio-visual synchrony to locate sounds. Advances in neu- ral information processing systems, 12, 1999. 3

    1999

  18. [26]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 6, 2

    2017

  19. [27]

    Diffted: One-shot audio-driven ted talk video generation with diffusion-based co-speech gestures

    Steven Hogue, Chenxu Zhang, Hamza Daruger, Yapeng Tian, and Xiaohu Guo. Diffted: One-shot audio-driven ted talk video generation with diffusion-based co-speech gestures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1922– 1931, 2024. 3

    1922

  20. [28]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Pro- ceedings of the IEEE international conference on computer vision, pages 1501–1510, 2017. 5

    2017

  21. [29]

    Discohead: audio-and- video-driven talking head generation by disentangled con- trol of head pose and facial expressions

    Geumbyeol Hwang, Sunwon Hong, Seunghyun Lee, Sung- woo Park, and Gyeongsu Chae. Discohead: audio-and- video-driven talking head generation by disentangled con- trol of head pose and facial expressions. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and...

    2023

  22. [30]

    Sparse in space and time: Audio-visual synchronisation with trainable selectors

    Vladimir Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Sparse in space and time: Audio-visual synchronisation with trainable selectors. arXiv preprint arXiv:2210.07055, 2022. 3

    2022 arXiv

  23. [31]

    Batch normalization: Accelerating deep network training by reducing internal co- variate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. In International conference on machine learn- ing, pages 448–456. pmlr, 2015. 5

    2015

  24. [32]

    You said that?: Synthesising talking faces from audio

    Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. You said that?: Synthesising talking faces from audio. In- ternational Journal of Computer Vision , 127:1767–1779,

  25. [33]

    Audio-driven emotional video portraits

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14080–14089, 2021. 3

    2021

  26. [34]

    Eamm: One-shot emotional talking face via audio-based emotion-aware motion model

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. Eamm: One-shot emotional talking face via audio-based emotion-aware motion model. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022. 3

    2022

  27. [35]

    Audio-driven facial an- imation with deep learning: A survey

    Diqiong Jiang, Jian Chang, Lihua You, Shaojun Bian, Robert Kosk, and Greg Maguire. Audio-driven facial an- imation with deep learning: A survey. Information, 15(11): 675, 2024. 1

    2024

  28. [36]

    Percep- tual losses for real-time style transfer and super-resolution

    Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Percep- tual losses for real-time style transfer and super-resolution. In Computer Vision–ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694–711. Springer, 2016. 4

    2016

  29. [37]

    V ocalist: An audio-visual synchronisation model for lips and voices

    Venkatesh S Kadandale, Juan F Montesinos, and Gloria Haro. V ocalist: An audio-visual synchronisation model for lips and voices. arXiv preprint arXiv:2204.02090, 2022. 3

    2022 arXiv

  30. [38]

    Nerffacespeech: One-shot audio-diven 3d talk- ing head synthesis via generative prior

    Gihoon Kim, Kwanggyoon Seo, Sihun Cha, and Juny- ong Noh. Nerffacespeech: One-shot audio-diven 3d talk- ing head synthesis via generative prior. arXiv preprint arXiv:2405.05749, 2024. 3

    2024 arXiv

  31. [39]

    End-to-end lip synchronisation based on pattern classification

    You Jin Kim, Hee Soo Heo, Soo-Whan Chung, and Bong- Jin Lee. End-to-end lip synchronisation based on pattern classification. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 598–605. IEEE, 2021. 3

    2021

  32. [40]

    Towards automatic face-to-face translation

    Prajwal KR, Rudrabha Mukhopadhyay, Jerin Philip, Ab- hishek Jha, Vinay Namboodiri, and CV Jawahar. Towards automatic face-to-face translation. In Proceedings of the 27th ACM international conference on multimedia , pages 1428–1436, 2019. 3

    2019

  33. [41]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems, 25, 2012. 5

    2012

  34. [42]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv e-prints , pages arXiv–1607,

  35. [43]

    Efficient region-aware neural radiance fields for high- fidelity talking portrait synthesis

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. Efficient region-aware neural radiance fields for high- fidelity talking portrait synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7568–7578, 2023. 3

    2023

  36. [44]

    One-shot high- fidelity talking-head synthesis with deformable neural ra- diance field

    Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Zhigang Wang, Mulin Chen, Bang Zhang, Zhongjian Wang, Liefeng Bo, and Xuelong Li. One-shot high- fidelity talking-head synthesis with deformable neural ra- diance field. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2023

  37. [45]

    Expressive talking head generation with granular audio-visual control

    Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. Expressive talking head generation with granular audio-visual control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2022

  38. [46]

    Font: Flow-guided one-shot talking head generation with natural head motions

    Jin Liu, Xi Wang, Xiaomeng Fu, Yesheng Chai, Cai Yu, Jiao Dai, and Jizhong Han. Font: Flow-guided one-shot talking head generation with natural head motions. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 2099–2104. IEEE, 2023. 3

    2023

  39. [47]

    Opt: One-shot pose- controllable talking head generation

    Jin Liu, Xi Wang, Xiaomeng Fu, Yesheng Chai, Cai Yu, Jiao Dai, and Jizhong Han. Opt: One-shot pose- controllable talking head generation. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 3

    2023

  40. [48]

    Semantic-aware implicit neural audio- driven video portrait generation

    Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. Semantic-aware implicit neural audio- driven video portrait generation. In European conference on computer vision, pages 106–125. Springer, 2022. 3

    2022

  41. [49]

    Moda: Mapping-once audio-driven portrait animation with dual attentions

    Yunfei Liu, Lijian Lin, Fei Yu, Changyin Zhou, and Yu Li. Moda: Mapping-once audio-driven portrait animation with dual attentions. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 23020– 23029, 2023. 3 10

    2023

  42. [50]

    Me- diapipe: A framework for building perception pipelines

    Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mc- Clanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo- Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Me- diapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019. 3

    1906 arXiv

  43. [51]

    Cvthead: One-shot controllable head avatar with vertex-feature transformer

    Haoyu Ma, Tong Zhang, Shanlin Sun, Xiangyi Yan, Kun Han, and Xiaohui Xie. Cvthead: One-shot controllable head avatar with vertex-feature transformer. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6131–6141, 2024. 3

    2024

  44. [52]

    Styletalk: One-shot talking head generation with controllable speak- ing styles

    Yifeng Ma, Suzhen Wang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding, Zhidong Deng, and Xin Yu. Styletalk: One-shot talking head generation with controllable speak- ing styles. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, pages 1896–1904, 2023. 3

    1904

  45. [53]

    Dreamtalk: When ex- pressive talking head generation meets diffusion probabilis- tic models

    Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When ex- pressive talking head generation meets diffusion probabilis- tic models. arXiv preprint arXiv:2312.09767, 2023. 3

    2023 arXiv

  46. [54]

    Otavatar: One-shot talking face avatar with control- lable tri-plane rendering

    Zhiyuan Ma, Xiangyu Zhu, Guo-Jun Qi, Zhen Lei, and Lei Zhang. Otavatar: One-shot talking face avatar with control- lable tri-plane rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16901–16910, 2023. 3

    2023

  47. [55]

    Sidgan: High-resolution dubbed video generation via shift-invariant learning

    Urwa Muaz, Wondong Jang, Rohun Tripathi, Santhosh Mani, Wenbin Ouyang, Ravi Teja Gadde, Baris Gecer, Ser- gio Elizondo, Reza Madad, and Naveen Nair. Sidgan: High-resolution dubbed video generation via shift-invariant learning. In Proceedings of the IEEE/CVF International Confe...

    2023

  48. [56]

    Diff2lip: Audio conditioned dif- fusion models for lip-synchronization

    Soumik Mukhopadhyay, Saksham Suri, Ravi Teja Gadde, and Abhinav Shrivastava. Diff2lip: Audio conditioned dif- fusion models for lip-synchronization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 5292–5302, 2024. 3, 6, 4

    2024

  49. [57]

    Rectified linear units improve restricted boltzmann machines

    Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010. 5

    2010

  50. [58]

    Audio-visual scene analysis with self-supervised multisensory features

    Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. InPro- ceedings of the European conference on computer vision (ECCV), pages 631–648, 2018. 3

    2018

  51. [59]

    in-the-wild

    Foivos Paraperas Papantoniou, Panagiotis P Filntisis, Pet- ros Maragos, and Anastasios Roussos. Neural emotion director: Speech-preserving semantic control of facial ex- pressions in" in-the-wild" videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2022

  52. [60]

    Synctalkface: Talking face generation with precise lip-syncing via audio-lip memory

    Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, and Yong Man Ro. Synctalkface: Talking face generation with precise lip-syncing via audio-lip memory. InProceedings of the AAAI Conference on Artificial Intelligence, pages 2062– 2070, 2022. 3

    2022

  53. [61]

    Semantic image synthesis with spatially-adaptive normalization

    Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2337– 2346, 2019. 4

    2019

  54. [62]

    Synctalk: The devil is in the synchronization for talking head synthesis

    Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk: The devil is in the synchronization for talking head synthesis. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages...

    2024

  55. [63]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia , pages 484–492, 2020. 1, 2, 3, 5, 6

    2020

  56. [64]

    U- net: Convolutional networks for biomedical image seg- mentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image seg- mentation. In Medical Image Computing and Computer- Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Pro- ceedings,...

    2015

  57. [65]

    Learning dynamic facial radiance fields for few-shot talking head synthesis

    Shuai Shen, Wanhua Li, Zheng Zhu, Yueqi Duan, Jie Zhou, and Jiwen Lu. Learning dynamic facial radiance fields for few-shot talking head synthesis. In European conference on computer vision, pages 666–682. Springer, 2022. 3

    2022

  58. [66]

    Difftalk: Crafting dif- fusion models for generalized audio-driven portraits anima- tion

    Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting dif- fusion models for generalized audio-driven portraits anima- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 1982–1991,

    1982

  59. [67]

    Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

    Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on compute...

    2016

  60. [68]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 4

    2014 arXiv

  61. [69]

    Facesync: A linear operator for measuring synchronization of video facial im- ages and audio tracks

    Malcolm Slaney and Michele Covell. Facesync: A linear operator for measuring synchronization of video facial im- ages and audio tracks. Advances in neural information pro- cessing systems, 13, 2000. 3

    2000

  62. [70]

    Audio-driven dubbing for user generated contents via style-aware semi-parametric synthesis

    Linsen Song, Wayne Wu, Chaoyou Fu, Chen Change Loy, and Ran He. Audio-driven dubbing for user generated contents via style-aware semi-parametric synthesis. IEEE Transactions on Circuits and Systems for Video Technology, 33(3):1247–1261, 2022. 3

    2022

  63. [71]

    Everybody’s talkin’: Let me talk as you want

    Linsen Song, Wayne Wu, Chen Qian, Ran He, and Chen Change Loy. Everybody’s talkin’: Let me talk as you want. IEEE Transactions on Information Forensics and Se- curity, 17:585–598, 2022. 3

    2022

  64. [72]

    Talking face generation by conditional recur- rent adversarial network

    Yang Song, Jingwen Zhu, Dawei Li, Xiaolong Wang, and Hairong Qi. Talking face generation by conditional recur- rent adversarial network. arXiv preprint arXiv:1804.04786,

    arXiv

  65. [73]

    Diffused 11 heads: Diffusion models beat gans on talking-face gen- eration

    Michał Stypułkowski, Konstantinos V ougioukas, Sen He, Maciej Zi˛ eba, Stavros Petridis, and Maja Pantic. Diffused 11 heads: Diffusion models beat gans on talking-face gen- eration. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision , pages 5...

    2024

  66. [74]

    Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior

    Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Daiheng Gao, Liefeng Bo, and Xun Cao. Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.01841, 2023. 3

    2023 arXiv

  67. [75]

    Masked lip-sync predic- tion by audio-visual contextual exploitation in transform- ers

    Yasheng Sun, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Zhibin Hong, Jingtuo Liu, Errui Ding, Jingdong Wang, Ziwei Liu, and Koike Hideki. Masked lip-sync predic- tion by audio-visual contextual exploitation in transform- ers. In SIGGRAPH Asia 2022 Conference Papers , pages 1–9, 2022. 3

    2022

  68. [76]

    Synthesizing obama: learning lip sync from audio

    Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics (ToG), 36(4):1–13, 2017. 2, 3

    2017

  69. [77]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2818–2826, 2016. 2

    2016

  70. [78]

    Emmn: Emotional mo- tion memory network for audio-driven emotional talking face generation

    Shuai Tan, Bin Ji, and Ye Pan. Emmn: Emotional mo- tion memory network for audio-driven emotional talking face generation. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision , pages 22146– 22156, 2023. 3

    2023

  71. [79]

    Real-time neural radiance talking portrait synthesis via audio-spatial decomposition

    Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211.12368, 2022. 3

    2022 arXiv

  72. [80]

    Neural voice puppetry: Audio-driven facial reenactment

    Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pages 716–

    2020

  73. [81]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485, 2024. 3

    2024 arXiv

  74. [82]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 3

    2017

  75. [83]

    Realistic speech-driven facial animation with gans

    Konstantinos V ougioukas, Stavros Petridis, and Maja Pan- tic. Realistic speech-driven facial animation with gans. International Journal of Computer Vision , 128(5):1398– 1413, 2020. 3

    2020

  76. [84]

    Progressive disentangled representa- tion learning for fine-grained controllable talking head syn- thesis

    Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, and Baoyuan Wang. Progressive disentangled representa- tion learning for fine-grained controllable talking head syn- thesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17979– 1...

    2023

  77. [85]

    Seeing what you said: Talking face gen- eration guided by a lip reading expert

    Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. Seeing what you said: Talking face gen- eration guided by a lip reading expert. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14653–14662, 2023. 2, 3, 5, 6

    2023

  78. [86]

    Lipformer: High- fidelity and generalizable talking face generation with a pre- learned facial codebook

    Jiayu Wang, Kang Zhao, Shiwei Zhang, Yingya Zhang, Yu- jun Shen, Deli Zhao, and Jingren Zhou. Lipformer: High- fidelity and generalizable talking face generation with a pre- learned facial codebook. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2023

  79. [87]

    Styletalk++: A unified framework for controlling the speaking styles of talking heads

    Suzhen Wang, Yifeng Ma, Yu Ding, Zhipeng Hu, Changjie Fan, Tangjie Lv, Zhidong Deng, and Xin Yu. Styletalk++: A unified framework for controlling the speaking styles of talking heads. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 3

    2024

  80. [88]

    High-resolution image synthesis and semantic manipulation with conditional gans

    Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018. 4

    2018

  81. [89]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image pro- cessing, 13(4):600–612, 2004. 6

    2004

  82. [90]

    Imitating arbitrary talking style for realistic audio-driven talking face synthesis

    Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng. Imitating arbitrary talking style for realistic audio-driven talking face synthesis. InProceedings of the 29th ACM International Conference on Multimedia , pages 1478–1486, 2021. 3

    2021

  83. [91]

    Ganhead: Towards generative animatable neural head avatars

    Sijing Wu, Yichao Yan, Yunhao Li, Yuhao Cheng, Wenhan Zhu, Ke Gao, Xiaobo Li, and Guangtao Zhai. Ganhead: Towards generative animatable neural head avatars. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 437–447, 2023. 3

    2023

  84. [92]

    High-fidelity generalized emotional talking face generation with multi-modal emotion space learning

    Chao Xu, Junwei Zhu, Jiangning Zhang, Yue Han, Wenqing Chu, Ying Tai, Chengjie Wang, Zhifeng Xie, and Yong Liu. High-fidelity generalized emotional talking face generation with multi-modal emotion space learning. In Proceedings of the IEEE/CVF conference on computer vision and...

    2023

  85. [93]

    Audio-visual speech representation expert for enhanced talking face video generation and evaluation

    Dogucan Yaman, Fevziye Irem Eyiokur, Leonard Bär- mann, Seymanur Akti, Hazım Kemal Ekenel, and Alexan- der Waibel. Audio-visual speech representation expert for enhanced talking face video generation and evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sio...

    2024

  86. [94]

    Audio-driven talking face generation with stabilized synchronization loss

    Dogucan Yaman, Fevziye Irem Eyiokur, Leonard Bärmann, Hazim Kemal Ekenel, and Alexander Waibel. Audio-driven talking face generation with stabilized synchronization loss. arXiv preprint arXiv:2307.09368, 2024. 2, 3, 5, 6

    2024 arXiv

  87. [95]

    Dfa-nerf: Personalized talking head generation via disentangled face attributes neural rendering

    Shunyu Yao, RuiZhe Zhong, Yichao Yan, Guangtao Zhai, and Xiaokang Yang. Dfa-nerf: Personalized talking head generation via disentangled face attributes neural rendering. arXiv preprint arXiv:2201.00791, 2022. 3

    2022 arXiv

  88. [96]

    Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis

    Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. arXiv preprint arXiv:2301.13430, 2023. 3 12

    2023 arXiv

  89. [97]

    Real3d-portrait: One- shot realistic 3d talking portrait synthesis

    Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, We- ichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, et al. Real3d-portrait: One- shot realistic 3d talking portrait synthesis. arXiv preprint arXiv:2401.08503, 2024. 3

    2024 arXiv

  90. [98]

    Quantitative association of vocal-tract and facial behavior

    Hani Yehia, Philip Rubin, and Eric Vatikiotis-Bateson. Quantitative association of vocal-tract and facial behavior. Speech Communication, 26(1-2):23–43, 1998. 2

    1998

  91. [99]

    Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan

    Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In European conference on computer vision , pages 85–101. Sp...

    2022

  92. [100]

    Multimodal image synthesis and editing: A survey and taxonomy

    Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, Lingjie Liu, Adam Kortylewski, Chris- tian Theobalt, and Eric Xing. Multimodal image synthesis and editing: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 1

    2023

  93. [101]

    Emotalker: Emotionally editable talking face generation via diffusion model

    Bingyuan Zhang, Xulong Zhang, Ning Cheng, Jun Yu, Jing Xiao, and Jianzong Wang. Emotalker: Emotionally editable talking face generation via diffusion model. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8276–8280. ...

    2024

  94. [102]

    Facial: Synthesizing dynamic talking face with implicit attribute learning

    Chenxu Zhang, Yifan Zhao, Yifei Huang, Ming Zeng, Saifeng Ni, Madhukar Budagavi, and Xiaohu Guo. Facial: Synthesizing dynamic talking face with implicit attribute learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3867–3876, 2021. 3

    2021

  95. [103]

    Emodiffhead: Continuously emotional control in talking head generation via diffusion

    Jian Zhang, Weijian Mai, and Zhijun Zhang. Emodiffhead: Continuously emotional control in talking head generation via diffusion. arXiv preprint arXiv:2409.07255, 2024. 3

    2024 arXiv

  96. [104]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  97. [105]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021. 3, 6

    2021

  98. [106]

    Dinet: Deformation inpainting network for realistic face visually dubbing on high reso- lution video

    Zhimeng Zhang, Zhipeng Hu, Wenjin Deng, Changjie Fan, Tangjie Lv, and Yu Ding. Dinet: Deformation inpainting network for realistic face visually dubbing on high reso- lution video. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3543–3551, 2023. 3, 6

    2023

  99. [107]

    Learning dynamic tetrahedra for high-quality talking head synthesis

    Zicheng Zhang, Ruobing Zheng, Bonan Li, Congying Han, Tianqi Li, Meng Wang, Tiande Guo, Jingdong Chen, Zi- wen Liu, and Ming Yang. Learning dynamic tetrahedra for high-quality talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  100. [108]

    Human-computer interaction system: A survey of talking-head generation

    Rui Zhen, Wenchao Song, Qiang He, Juan Cao, Lei Shi, and Jia Luo. Human-computer interaction system: A survey of talking-head generation. Electronics, 12(1):218, 2023. 1

    2023

  101. [109]

    Identity- preserving talking face generation with landmark and ap- pearance priors

    Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, and Guanbin Li. Identity- preserving talking face generation with landmark and ap- pearance priors. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 9729–...

    2023

  102. [110]

    Talking face generation by adversarially disentan- gled audio-visual representation

    Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially disentan- gled audio-visual representation. In Proceedings of the AAAI conference on artificial intelligence , pages 9299– 9306, 2019. 3

    2019

  103. [111]

    Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation

    Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4176–4186,

  104. [112]

    Makelttalk: speaker-aware talking-head animation

    Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevar- ria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions On Graphics (TOG), 39(6):1–15, 2020. 2, 3 13 Mask-Free Audio-driven Talking Face Generation for Enhanced Visual ...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.