Pith. sign in

REVIEW 4 major objections 7 minor 54 references

Audio-Driven Talking Face Video Generation with Joint Uncertainty Learning

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that audio-driven talking-face video generation improves when the model is trained to predict its own per-pixel visual uncertainty—an error map and a matching uncertainty map—and shows that JULNet, built on this joint…

desk verdict The uncertainty regularizer works for image quality, but the paper's lip-sync superiority claim is contradicted by its own tables. read the letter →

arxiv 2504.18810 v1 pith:NB24LJIY submitted 2025-04-26 cs.CV cs.AI

Yifan Xie , Fei Ma , Yi Bin , Ying He , Fei Yu This is my paper · ORCID
classification cs.CVcs.AI
keywords talkingfacevideogenerationuncertaintylearningjointaudio-lipsynchronizationhistogramKLmatchingJULNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a talking-face video generator can be made better and more reliable by teaching it to predict its own visual uncertainty. Its proposed network, JULNet, synthesizes a frame and then produces two maps: an error map showing the pixel-by-pixel difference from the ground truth, and an uncertainty map predicting where the model is likely to be wrong. A histogram-based KL divergence term forces the uncertainty distribution to match the error distribution, so that optimizing error and uncertainty together becomes a single training objective. The authors report that this joint uncertainty learning outperforms prior person-generic methods in image fidelity and audio-lip synchronization, and that the uncertainty module can be bolted onto an existing method such as Wav2Lip to improve it.

What carries the argument

The machinery is the joint uncertainty loss $L_{\mathrm{un}} = L_{\mathrm{un1}} + L_{\mathrm{un2}}$. $L_{\mathrm{un1}}$ is the standard heteroscedastic uncertainty-weighted regression loss: pixels with large predicted uncertainty $\sigma_i$ contribute less to the error term, while the $\tau_i$ sum prevents the model from inflating uncertainty everywhere. $L_{\mathrm{un2}}$ is the distinctive piece: the paper approximates the error distribution and the predicted uncertainty distribution with a differentiable histogram, using soft bin assignments $w_j(\epsilon_i) = \lambda_1 \exp(-(u_\epsilon + \alpha_j s_\epsilon - \epsilon_i)^2 / \lambda_2)$, where $u_\epsilon$ and $s_\epsilon$ are the mean and standard deviation of the error and $\alpha_m$ is a temperature-scale bin layout. It then computes the KL divergence between the two histograms, so that uncertainty is explicitly trained to have the same distribution as the error. This histogram tie makes the uncertainty representation directly related to visual error and is what carries the claimed improvement.

What would settle it

Rerun the ablation in Table 4 over multiple random seeds; if removing $L_{\mathrm{un2}}$ (the KL histogram term) yields no consistent degradation in PSNR, LPIPS, FID, or LSE metrics, or if replacing it with a simple moment-matching or variance penalty reproduces identical gains, the claim that histogram-based distribution matching is the active ingredient would be falsified.

Watch

Extended reading notes

Core claim

JULNet is a person-generic audio-driven talking-face generator whose training loss includes a joint uncertainty term. After the generator produces an image $I_{gen}$, an uncertainty module computes an error map $\epsilon_i = |\hat{p}_i - p_i|$ (the L1 difference between generated and ground-truth pixels) and predicts an uncertainty map $\sigma_i$, modeled as the scale of a Laplacian distribution. The uncertainty loss has two components: an uncertainty-weighted regression loss $L_{\mathrm{un1}} = \frac{1}{n}\sum_i (\epsilon_i / \exp(\tau_i) + \tau_i)$, where $\tau_i = \log \sigma_i$, and a KL matching loss $L_{\mathrm{un2}}$ computed between differentiable histograms of the error and uncertainty values. The paper's central discovery is that jointly optimizing these two terms—forcing the uncertainty map's histogram to mirror the actual error histogram—improves PSNR, LPIPS, FID, and lip-sync metrics over prior methods, and that the same module improves Wav2Lip when attached to it.

Load-bearing premise

The paper assumes that the pixel-by-pixel difference between generated and real video frames faithfully captures how bad the output looks, and that pushing the predicted uncertainty map to have the same distribution as this difference teaches the network something it would not otherwise learn.

Editorial extensions

If this is right

  • On HDTF and MEAD, JULNet reports the best PSNR, LPIPS, and FID among Wav2Lip, EAMM, VideoReTalking, DINet, and TalkLip, with the largest margins in FID (7.825 vs 8.437 on HDTF; 19.525 vs 23.350 on MEAD).
  • Adding the uncertainty module to Wav2Lip improves its PSNR from 31.836 to 32.108 and its FID from 9.137 to 6.996 on HDTF, indicating the module is a plug-and-play enhancement.
  • Ablations show each loss component matters: removing perception loss hurts realism, removing lip-sync loss hurts LSE metrics, and removing the uncertainty loss degrades both quality and sync (full model PSNR 32.266 vs 31.939 without the uncertainty loss).
  • The full uncertainty loss works best with 11 histogram bins, and both components $L_{\mathrm{un1}}$ and $L_{\mathrm{un2}}$ individually contribute, with $L_{\mathrm{un2}}$ providing a smaller but consistent gain (PSNR 32.214 without it vs 32.266 with it).
  • The method's current limitations are dynamic scenes, changing illumination, moving backgrounds, and non-frontal head poses, where it can produce visible artifacts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The differentiable-histogram KL scheme is not specific to talking faces; it could serve as a generic calibration regularizer for any generative model with access to pixel-level ground truth, such as inpainting or super-resolution.
  • Editorial extension: The predicted uncertainty map may have practical value as a failure detector: regions with high predicted uncertainty could be highlighted for human review or fed back into a second-stage refinement network.
  • Editorial extension: Since Wav2Lip still leads on LSE-C on HDTF, a direct further step would be to pre-train the sync expert on a larger dataset and then add the uncertainty module, testing whether the two improvements are complementary rather than overlapping.
  • Editorial extension: A testable hypothesis is that the KL histogram term mostly acts as a regularizer that smooths the uncertainty map; if true, a simpler variance-minimization or entropy penalty on the uncertainty map could reproduce the gains, which would simplify the method.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes JULNet, an audio-driven person-generic talking face generation method built on a DINet-style deformation-inpainting generator with an added joint uncertainty learning module. After generating a frame, a network predicts a per-pixel error map (defined as the L1 distance to the ground truth) and an uncertainty map, trained with a heteroscedastic regression loss L_un1 (Eq. 4) and a histogram-based KL divergence L_un2 (Eq. 8) that matches the uncertainty distribution to the error distribution. The total training loss combines uncertainty, adversarial, perceptual, and lip-sync losses. Experiments on HDTF and MEAD compare JULNet against Wav2Lip, EAMM, VideoReTalking, DINet, and TalkLip, reporting PSNR, LPIPS, FID, LSE-D, and LSE-C. The paper reports the best image-quality metrics among the compared methods, mixed lip-sync metrics (worse than Wav2Lip on both datasets), and ablations showing gains from the uncertainty losses. A plug-in experiment on Wav2Lip suggests that the uncertainty module can transfer to another baseline.

Significance. If the reported performance holds, joint uncertainty learning is a simple, architecture-agnostic auxiliary loss that could improve the image quality of talking-face generators and possibly their robustness. The paper's strengths are the clear quantitative comparisons on two datasets, the ablation studies that isolate the contributions of L_un1 and L_un2, and the plug-in experiment on Wav2Lip, which is a convincing way to demonstrate the module's generality. However, the headline claim of superior audio-lip synchronization is contradicted by the paper's own LSE-D/LSE-C numbers, and several load-bearing technical details (the inference-time use of the uncertainty module, the exact form of the histogram KL, and the plug-in integration protocol) need clarification before the claims can be fully assessed.

major comments (4)
  1. [Abstract; Conclusion; Tables 1 and 2; Section 4.1.2] The abstract and the conclusion state that JULNet achieves 'superior high-fidelity and audio-lip synchronization' compared to previous methods, but the quantitative results in Tables 1 and 2 contradict the lip-sync part. On HDTF, JULNet has LSE-D 7.024 and LSE-C 8.208 versus Wav2Lip's 6.792 and 8.877; on MEAD, JULNet has LSE-D 7.801 and LSE-C 7.147 versus Wav2Lip's 7.655 and 7.902. Section 4.1.2 explicitly concedes that 'our method falls short compared to Wav2Lip' on audio-visual synchronization. Since the abstract explicitly couples fidelity with lip-sync as the basis of the claimed advantage, this overclaim is load-bearing. The authors should either revise the claims to match the evidence (image quality improvements and sync improvements over the no-uncertainty ablation) or rerun the evaluation with a SyncNet setup that is directly comparable to Wav2Lip's, such as using the same pretrained sync expert.
  2. [Section 3.3, Eq. (5) and Eq. (8)] The KL divergence direction is inconsistent and likely incorrect. Equation (5) writes D(P_σ(p))∥Q_ε(p), placing the uncertainty distribution as the left argument, while the text says the error distribution is the reference. Equation (8), however, computes Σ_j H_ε(j) log(H_ε(j)/H_σ(j)), which is KL(H_ε || H_σ), the opposite direction. Because KL is asymmetric, this inconsistency changes the optimization objective and the interpretation of the regularizer. Please correct Eq. (5) and unambiguously state which distribution is the reference.
  3. [Section 3.2, Eq. (2); Fig. 2] The role of the uncertainty module at inference is not specified. The error map is defined using the ground truth (Eq. 2), and Eq. (3) conditions the Laplacian on both predicted and ground-truth pixels, so it is unclear whether the uncertainty module can be executed at test time when ground truth is absent. The paper should state explicitly whether the uncertainty map is a training-only auxiliary output or is also produced for new videos, and if the latter, what input is used at inference. Without this, the claim that the model 'provides reliable uncertainty estimates for the generated content' is not verifiable.
  4. [Section 4.3.3, Table 5] The plug-and-play generalization experiment on Wav2Lip is reported without any implementation details. The paper does not describe how the uncertainty module is attached to Wav2Lip, which parts of the network are trained, how the losses are combined, or which hyperparameters are used. The improvements in Table 5, including better LSE-D and LSE-C than the original Wav2Lip, are therefore impossible to reproduce or interpret. Please include the integration and training protocol, or explicitly frame the result as a preliminary proof-of-concept and list all required details.
minor comments (7)
  1. [Section 4.2.1; Section 4.0.2] There are citation errors: Section 4.2.1 says 'Wav2Lip [43] and TalkLip [38]' but Wav2Lip is reference [38] and TalkLip is reference [43]; Section 4.0.2 also cites TalkLip as [47] instead of [43].
  2. [Section 3.2, Eq. (2)] Equation (2) uses the same symbol p_i for both the predicted and ground-truth pixels; please use distinct notation, for example \ hat{p}_i and p_i, consistently throughout the paper.
  3. [Section 3.3, Eq. (7)] Equation (7) is garbled in the typeset version: the denominator appears as '˝m_j=0 exp(w_j(ε_i))', which is illegible. Please provide a clean formula, e.g., a softmax normalization over the histogram bin weights, so that the histogram construction is unambiguous.
  4. [Figure 2] The symbols ⊕ and … in Figure 2 are not explained in the caption; please label the concatenation operation and clarify which connections are omitted.
  5. [Section 3.1.1] The sentence 'The source image is identical to the ground truth image, with the exception that it masks the lower-half mouth region' is confusing; please rephrase to make clear that the source image is a masked version of the ground truth.
  6. [Section 4.1.2] The explanation for the lower lip-sync scores is that the syncnet is trained from scratch, but the paper does not specify the training data, architecture, or training protocol for this syncnet; please provide these details.
  7. [Keywords] The keyword 'Taking Face Video Generation' is a typo; it should be 'Talking Face Video Generation'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the uncertainty losses are supervised training objectives, and the paper's quantitative claims are empirically tested rather than derived from their own definitions.

full rationale

The proposed method's derivation chain is self-contained and does not reduce to its inputs. The error map (Eq. 2) is the per-pixel L1 difference between the generated and ground-truth images, and the uncertainty map is trained through Eq. 4, which is the standard heteroscedastic aleatoric uncertainty regression loss of Kendall and Gal (cited as [22]), together with the histogram-based KL matching term of Eq. 8. Both losses are directly optimized against the same supervised signal (the GT image), so there is no fitted parameter that is later renamed as a prediction. The central claims of improved image quality and robustness are supported by the ablation study in Tables 3 and 4 and the plug-in experiment in Table 5; these are empirical comparisons, not consequences of the definitions. The paper does include several self-citations ([11], [12], [28]-[30], [45]), but these appear only as contextual references for ASR, related work, and the authors' prior talking-head work; none is load-bearing for the uncertainty derivation or the evaluation. The manuscript's own admission in Section 4.1.2 that JULNet 'falls short compared to Wav2Lip' on audio-visual synchronization, observed in Tables 1 and 2, is a correctness/overclaim issue (the abstract claims superior synchronization) but it is not circularity: the sync metrics are external evaluations using a pretrained SyncNet, and the paper reports them honestly. No step in the derivation relies on an unverified self-citation, a uniqueness theorem, or an ansatz smuggled in by citation; hence the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or representational entities. It relies on standard assumptions about uncertainty-error correlation and on a generator borrowed from DINet. The main hidden degrees of freedom are the loss weights and histogram hyperparameters, one of which was tuned directly on the test set.

free parameters (4)
  • Global loss weights lambda_1..lambda_4 = 1.0, 1.0, 10.0, 0.1
    Set by hand in Section 3.4; no systematic tuning or sensitivity analysis is reported.
  • Number of histogram bins B = 11
    Selected as the best value in the ablation on the test set (Table 4), which is a form of test-set-based hyperparameter selection.
  • Temperature coefficient alpha_m in histogram bin range
    Appears in Section 3.3 as controlling the bin extent but its numerical value is never reported.
  • lambda_1 and lambda_2 in Eq. 6 (histogram weight function)
    Hyperparameters in the Gaussian weight function for histogram computation; values are unstated and the names collide with the global loss weights in Eq. 14.
assumptions (4)
  • domain assumption Uncertainty is highly correlated with prediction error in visual tasks.
    Invoked in Section 1 and Section 3.2, citing SEDNet [6]; this motivates training the uncertainty map to predict the L1 error.
  • domain assumption Masking the lower-half mouth region in the source image prevents the generator from copying lip movements directly from the source.
    Described in Section 3.1.1; a standard training setup inherited from prior talking-face works.
  • domain assumption Five reference images randomly sampled from the same video preserve identity and head pose.
    Assumed in Section 3.1.1; the mechanism by which the ID encoder fuses these images is not analyzed.
  • domain assumption The Laplacian distribution is appropriate for modeling L1 pixel error.
    Taken from Ilg et al. [18] in Section 3.2; no validation for this specific task is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Audio-Driven Talking Face Video Generation with Joint Uncertainty Learning." pith.science (2026). https://pith.science/paper/NB24LJIY

@misc{pith2026250418810,
  author       = {Pith},
  title        = {Pith review of: Audio-Driven Talking Face Video Generation with Joint Uncertainty Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NB24LJIY}},
  note         = {Machine review of arXiv:2504.18810}
}
read the original abstract

Talking face video generation with arbitrary speech audio is a significant challenge within the realm of digital human technology. The previous studies have emphasized the significance of audio-lip synchronization and visual quality. Currently, limited attention has been given to the learning of visual uncertainty, which creates several issues in existing systems, including inconsistent visual quality and unreliable performance across different input conditions. To address the problem, we propose a Joint Uncertainty Learning Network (JULNet) for high-quality talking face video generation, which incorporates a representation of uncertainty that is directly related to visual error. Specifically, we first design an uncertainty module to individually predict the error map and uncertainty map after obtaining the generated image. The error map represents the difference between the generated image and the ground truth image, while the uncertainty map is used to predict the probability of incorrect estimates. Furthermore, to match the uncertainty distribution with the error distribution through a KL divergence term, we introduce a histogram technique to approximate the distributions. By jointly optimizing error and uncertainty, the performance and robustness of our model can be enhanced. Extensive experiments demonstrate that our method achieves superior high-fidelity and audio-lip synchronization in talking face video generation compared to previous methods.

Figures

Figures reproduced from arXiv: 2504.18810 by the authors.

Figure 1
Figure 1. The process of audio-driven talking face video gen [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of the proposed JULNet. By utilizing audio signals, a masked source image, and a few reference [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The detailed structure of the generator. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of talking face video generation by different methods on the HDTF dataset. JULNet has the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison of talking face video generation by different methods on the MEAD dataset. JULNet has the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of the uncertainty maps generated by [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: User study results on three aspects, i.e. Image Quality, Video Realness, and Audio-Visual Synchronization. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 33 canonical work pages

  1. [1]

    Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In International Conference on Machine Learning . PMLR, 173–182

    work page 2016

  2. [2]

    Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE winter conference on applications of computer vision (W ACV). IEEE, 1–10

    2016

  3. [3]

    Volker Blanz and Thomas Vetter. 2023. A morphable model for the synthesis of 3D faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2 . 157–164

    work page 2023

  4. [4]

    Anh-Quan Cao, Angela Dai, and Raoul de Charette. 2024. Pasco: Urban 3d panoptic scene completion with uncertainty awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14554–14564

    work page 2024

  5. [5]

    Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. 2020. Talking-head generation with rhythmic head motion. In European Conference on Computer Vision . Springer, 35–51

    2020

  6. [6]

    Liyan Chen, Weihan Wang, and Philippos Mordohai. 2023. Learning the distribu- tion of errors in stereo matching for joint disparity and uncertainty estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. 17235–17244

    work page 2023

  7. [7]

    Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. 2022. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia 2022 Conference Papers. 1–9

    work page 2022

  8. [8]

    dissimilar

    Antonin Cheymol, Anatole Lécuyer, Jean-Marie Normand, Ferran Argelaguet, et al. 2023. Beyond my real body: Characterization, impacts, applications and perspectives of “dissimilar” avatars in virtual reality. IEEE Transactions on Visu- alization and Computer Graphics (2023)

    work page 2023

Show all 54 references
  1. [9]

    Joon Son Chung and Andrew Zisserman. 2017. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II

    2017

  2. [10]

    Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Kather- ine Heller, Balaji Lakshminarayanan, and Dustin Tran. 2020. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning. PMLR, 2782–2792

    2020

  3. [11]

    Tao Feng, Yifan Xie, Xun Guan, Jiyuan Song, Zhou Liu, Fei Ma, and Fei Yu. 2025. UniSync: A Unified Framework for Audio-Visual Synchronization. arXiv preprint arXiv:2503.16357 (2025)

    2025 arXiv

  4. [12]

    Tao Feng, Zhiyuan Zhao, Yifan Xie, Yuqi Ye, Xiangyang Luo, Xun Guan, and Yu Li. 2025. STFTCodec: High-Fidelity Audio Compression through Time-Frequency Domain Representation. arXiv preprint arXiv:2503.16989 (2025)

    2025 arXiv

  5. [13]

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang

  6. [14]

    Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al

  7. [15]

    Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew M Dai, and Dustin Tran. 2020. Training independent subnetworks for robust prediction. arXiv preprint arXiv:2010.06610 (2020)

    2020 arXiv

  8. [16]

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems 30 (2017)

    2017

  9. [17]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840–6851

    2020

  10. [18]

    Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox. 2018. Uncertainty estimates and multi-hypotheses networks for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV). 652–667

    2018

  11. [19]

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. 2022. Eamm: One-shot emotional talking face via audio-based emotion- aware motion model. In ACM SIGGRAPH 2022 Conference Proceedings . 1–10

    2022

  12. [20]

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. 2021. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14080–14089

    2021

  13. [21]

    Dima Kagan, Galit Fuhrmann Alpert, and Michael Fire. 2023. Zooming into video conferencing privacy. IEEE Transactions on Computational Social Systems 11, 1 (2023), 933–944

    2023

  14. [22]

    Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems 30 (2017)

    2017

  15. [23]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  16. [24]

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems 30 (2017)

    2017

  17. [25]

    Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-Marc Martinez, Andrei Bursuc, and Gianni Franchi. 2022. Packed-ensembles for effi- cient uncertainty estimation. arXiv preprint arXiv:2210.09184 (2022)

    2022

  18. [26]

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. 2023. Efficient region- aware neural radiance fields for high-fidelity talking portrait synthesis. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision . 7568–7578

    2023

  19. [27]

    Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mo- canu. 2021. Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. arXiv p...

    2021

  20. [28]

    Xiangyang Luo, Xin Zhang, Yifan Xie, Xinyi Tong, Weijiang Yu, Heng Chang, Fei Ma, and Fei Richard Yu. 2024. Codeswap: Symmetrically face swapping based on prior codebook. In Proceedings of the 32nd ACM International Conference on Multimedia. 6910–6919

    2024

  21. [29]

    Fei Ma, Yukan Li, Yifan Xie, Ying He, Yi Zhang, Hongwei Ren, Zhou Liu, Wei Yao, Fuji Ren, Fei Richard Yu, et al. 2024. A Review of Human Emotion Synthesis Based on Generative Technology. arXiv preprint arXiv:2412.07116 (2024)

    2024 arXiv

  22. [30]

    Fei Ma, Yucheng Yuan, Yifan Xie, Hongwei Ren, Ivan Liu, Ying He, Fuji Ren, Fei Richard Yu, and Shiguang Ni. 2024. Generative technology for human emotion recognition: A scoping review. Information Fusion (2024), 102753

    2024

  23. [31]

    Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. 2023. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767 (2023)

    2023 arXiv

  24. [32]

    David JC MacKay. 1992. A practical Bayesian framework for backpropagation networks. Neural computation 4, 3 (1992), 448–472

    1992

  25. [33]

    Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. 2017. Least squares generative adversarial networks. InProceedings of the IEEE international conference on computer vision . 2794–2802

    2017

  26. [34]

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 1 (2021), 99–106

    2021

  27. [35]

    Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. In- stant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) 41, 4 (2022), 1–15

    2022

  28. [36]

    Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Jun He, Hongyan Liu, and Zhaoxin Fan. 2024. SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  29. [37]

    Matteo Poggi, Filippo Aleotti, Fabio Tosi, and Stefano Mattoccia. 2020. On the uncertainty of self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 3227–3237

    2020

  30. [38]

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar

  31. [39]

    Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. 2023. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1982–1991

    2023

  32. [40]

    Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

    2014 arXiv

  33. [41]

    Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. 2022. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition.arXiv preprint arXiv:2211.12368 (2022)

    2022 arXiv

  34. [42]

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. 2024. EMO: Emote Portrait Alive-Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions. arXiv preprint arXiv:2402.17485 (2024)

    2024 arXiv

  35. [43]

    Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. 2023. Seeing what you said: Talking face generation guided by a lip reading expert. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14653–14662

    2023

  36. [44]

    Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. 2020. Mead: A large-scale audio- visual dataset for emotional talking-face generation. In European Conference on Computer Vision. Springer, 700–717

    2020

  37. [45]

    Yifan Xie, Tao Feng, Xin Zhang, Xiangyang Luo, Zixuan Guo, Weijiang Yu, Heng Chang, Fei Ma, and Fei Richard Yu. 2025. Pointtalk: Audio-driven dynamic lip point cloud for 3d gaussian-based talking head synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence ...

    2025

  38. [46]

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang

  39. [47]

    Zhimeng Zhang, Zhipeng Hu, Wenjin Deng, Changjie Fan, Tangjie Lv, and Yu Ding. 2023. Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 3543–3551

    2023

  40. [48]

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. 2021. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3661–3670

    2021

  41. [49]

    Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, and Guanbin Li. 2023. Identity-preserving talking face generation with landmark and appearance priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 9729–9738

    2023

  42. [50]

    Ke Zou, Xuedong Yuan, Xiaojing Shen, Meng Wang, and Huazhu Fu. 2022. Tbrats: Trusted brain tumor segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 503–513

    2022

  43. [2014]

    arXiv preprint arXiv:1412.5567 (2014)

    Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567 (2014)

    2014 arXiv

  44. [2018]

    In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 586–595. ICMR ’25, June 30–July 3, 2025, Chicago, IL, USA. Xie et al

    2025

  45. [2020]

    In Proceedings of the 28th ACM International Conference on Multimedia

    A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia . 484–492

  46. [2021]

    In Proceedings of the IEEE/CVF International Conference on Computer Vision

    Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 5784– 5794

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.