Pith. sign in

REVIEW 4 major objections 6 minor 36 references

LokiTalk: Learning Fine-Grained and Generalizable Correspondences to Enhance NeRF-based Talking Head Synthesis

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LokiTalk contends that the artifacts and high training costs of NeRF-based talking heads both come from missing correspondences, and that region-specific cascaded deformation fields plus cross-identity pretraining eliminate the artifacts…

desk verdict Solid talking-head recipe with an unexplained 3.26 dB mismatch between its main table and ablation table; the architecture and pretraining idea are worth engaging, but the headline numbers need a fix before they can be trusted. read the letter →

arxiv 2411.19525 v2 pith:2MHU2BW2 submitted 2024-11-29 cs.CV cs.LG

Tianqi Li , Ruobing Zheng , Bonan Li , Zicheng Zhang , Meng Wang , Jingdong Chen , Ming Yang This is my paper · ORCID
classification cs.CVcs.LG
keywords talkingheadsynthesisNeuralRadianceFieldsdeformationknowledgetransferaudio-drivenanimationlipsynceyeblinkingcanonicalspace
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LokiTalk argues that the two remaining obstacles to practical NeRF-based talking heads—visible artifacts and steep per-identity data and compute cost—have a common cause: the absence of precise, reusable correspondences between driving signals and the face and torso regions they control. To fix this, it introduces region-specific cascaded deformation fields that separate lip, eye, head-pose, and torso motion, and an identity-aware pretraining module that learns shared static and dynamic knowledge from a handful of other identities. If the claims hold, studios and avatar pipelines could keep the realism of NeRF rendering while training a new character with a quarter of the usual footage and reusing a single pretrained model across many identities. The paper also shows the transfer module can be grafted onto an existing strong baseline, suggesting the benefit is not tied to LokiTalk's own architecture.

What carries the argument

The load-bearing object is the pair of cascaded deformation fields defined in Section 3.2: the face field $\Phi_{\mathrm{face}}(PE(x); F_a, F_e) \to \Delta x_{\mathrm{face}}$ combines an audio-derived motion feature $F_a$ with a landmark eye-aspect ratio $F_e$, and the torso field $\Phi_{\mathrm{torso}}(PE(x); F_h, \Delta x_{\mathrm{face}}) \to \Delta x_{\mathrm{torso}}$ uses the 3DMM head pose $F_h$ together with the face deformation itself, so that jaw movement indirectly drives the torso. All coordinates are warped to one canonical space by $x' = x + \Delta x_{\mathrm{face}} + \Delta x_{\mathrm{torso}}$, which is what lets a single radiance field render the whole portrait. The second object is the ID-Aware Knowledge Transfer module: an ID-Encoder extracts per-identity dynamic features (a hyper-network replacing the last MLP layer) and static features (appearance and geometry features plus an ID-aware canonical offset), and during fine-tuning the encoder is discarded after initializing from the first frame, turning the ID-specific outputs into learnable parameters. Region-regularization and cross-attention losses ($L_\Delta$, $L_{\mathrm{att}}$) keep the deformation fields from distorting regions they should not affect.

What would settle it

Run the full LokiTalk pipeline on 20 new identities with 10 auxiliary videos and 25% target footage, and compare PSNR against ER-NeRF trained on 100%. If the average improvement over ER-NeRF does not reproduce the paper's margin (about 1.4 dB in PSNR at 25%)—or if the gain disappears for identities whose 3DMM fits are poorly aligned to the common canonical space—then the shared-canonical-space assumption and the claimed data-efficiency advantage would be disproved.

Watch

Extended reading notes

Core claim

The paper's central claim is that two families of artifacts in NeRF-based talking heads—imprecise lip sync, unnatural blinking, head–torso disconnection—and the high per-identity training cost are both symptoms of a single failure: the model does not learn fine-grained, generalizable correspondences between driving signals and the portrait regions they affect. LokiTalk therefore builds two components. Region-Specific Deformation Fields decompose motion into a face deformation $\Delta x_{\mathrm{face}}$ conditioned on audio features and an eye-aspect-ratio signal, and a torso deformation $\Delta x_{\mathrm{torso}}$ conditioned on the 3DMM head pose plus the face deformation; both warp observation-space coordinates into a unified canonical space $x' = x + \Delta x_{\mathrm{face}} + \Delta x_{\mathrm{torso}}$. ID-Aware Knowledge Transfer pretrains an ID-Encoder on a small set of multi-identity videos so that dynamic and static knowledge is shared, while per-identity hyper-networks and canonical offsets preserve individuality; at fine-tuning, only the first frame is passed through the encoder and the encoder is then discarded. The paper reports that with 10 auxiliary training videos and only 25% of a target identity's video, LokiTalk surpasses ER-NeRF's PSNR (29.832 vs 28.425), and that the transfer module is plug-and-play on ER-NeRF itself.

Load-bearing premise

The method assumes that different individuals' faces can be placed in a shared canonical space with only per-identity offsets and first-frame features, so pretrained knowledge actually transfers; if identities are too different in geometry, the shared space cannot absorb them and the data-efficiency gain disappears.

Editorial extensions

If this is right

  • Speech drives only lips and related facial motion, the eye-aspect ratio handles blinking, and the torso follows head pose jointly with jaw-driven face deformation, so artifacts like broken neck, unsynced lips, and erratic blinking are reduced.
  • With 10 auxiliary multi-identity videos, the target identity's video can be cut to 25% and still beat ER-NeRF in PSNR (29.832 vs 28.425), LMD, and LPIPS.
  • ID-Aware Knowledge Transfer can be attached to an existing NeRF-based method such as ER-NeRF and improve it, especially when the target video is short.
  • Replacing per-identity geometry learning with a shared canonical space plus per-ID offsets means a single radiance field renders the whole portrait, improving depth consistency between face and torso.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same pretrained ID-Encoder could be applied to an unseen identity at inference time with no fine-tuning, which would turn the data-efficiency result into a few-shot avatar system; the paper does not examine this.
  • Editorial inference: because blinking is controlled by an explicit eye-aspect-ratio signal, a natural extension is treating that ratio as a user-controllable input for gaze or blink-style transfer, a scenario the authors do not test.
  • Editorial inference: the near-zero transfer gain on ER-NeRF at 100% data (28.428 vs 28.425 PSNR) suggests the module's value is concentrated in low-data regimes; a stress test across identities with varied head shapes and hairstyles would reveal whether the shared-canonical-space assumption is the bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. LokiTalk is a NeRF-based talking-head synthesis method that augments a base NeRF model with two components: (i) Region-Specific Deformation Fields, which decompose portrait motion into face, eye, head-pose, and torso components through two cascaded deformation fields, and (ii) ID-Aware Knowledge Transfer, a pretraining module that learns shared static and dynamic representations from multi-identity videos while extracting per-identity features, with fine-tuning on a target identity. The paper reports quantitative comparisons against Wav2Lip, AD-NeRF, RAD-NeRF, and ER-NeRF, claiming superior fidelity and reduced data requirements, and provides ablations of the proposed modules and a transfer experiment on ER-NeRF.

Significance. If the reported results are reproducible, LokiTalk would be a useful contribution to NeRF-based talking heads: the region-specific decomposition is well motivated, the ID-aware pretraining addresses a real data-efficiency bottleneck, and the evaluation uses official implementations of several strong baselines. The paper also provides a clear set of ablations and a plug-and-play transfer experiment. However, the central empirical claims are currently undermined by a serious internal inconsistency between the main quantitative table and the ablation table, and the training-efficiency claim is contradicted by the reported training costs. The contribution cannot be fairly assessed until these issues are resolved.

major comments (4)
  1. [Table 1 vs Table 3] Table 1 reports 'Ours' on the portrait reconstruction benchmark as PSNR 33.744, LPIPS 0.029, LMD 2.732, Sync 5.736. Table 3, however, reports the configuration 'Pretrain IDs 10, Finetune Data 100%' — which, by the description in §3.3 and §4.3, is the same full LokiTalk model with ID-aware pretraining and full target data — as PSNR 30.482, LPIPS 0.050, LMD 3.034, Sync 5.873. The PSNR gap of 3.26 dB is larger than the advertised margin over ER-NeRF (1.24 dB in Table 1), and the Sync metric also conflicts (5.736 vs 5.873). The paper never explains this discrepancy. If Table 1 is correct, the ablation table is not measuring the full system; if Table 3 is correct, the headline results in Table 1 are not reproducible. Please reconcile the two tables, state exactly which training protocol each row uses, and re-run the comparison if needed.
  2. [Table 1, §4.3] The abstract and conclusion claim improved training efficiency, and §4.3 claims that with 10 auxiliary videos and 25% target data LokiTalk surpasses ER-NeRF. Yet Table 1's Cost column lists Ours as 3h versus ER-NeRF 2h, and this cost does not include the multi-ID pretraining stage (50 epochs over 10 videos) described in §3.3 and §4. The data-efficiency comparison in Table 3 is not accompanied by any training-time measurement for the 25%-data setting, so the paper does not substantiate either 'accelerating training' or a net efficiency advantage. Please report wall-clock time including pretraining for the configurations compared, or revise the efficiency claim to refer specifically to target-data requirements.
  3. [Tables 3 and 4] No error bars or significance tests are reported for any metric. This matters because the key transfer gains are numerically small: at 100% finetune data, adding 10 pretrain IDs changes PSNR from 30.464 to 30.482 (+0.018 dB) for LokiTalk, and for ER-NeRF in Table 4 it changes PSNR from 28.425 to 28.428 (+0.003 dB). These differences are within typical run-to-run variation for NeRF training. Please report multiple seeds with standard deviations (or per-identity variances across the evaluation set) and state whether the reported differences are statistically reliable.
  4. [Table 3] The claim 'With an increase in the number of auxiliary training videos, the performance continues to improve' is contradicted by Table 3: at 50% finetune data, 5 pretrain IDs give PSNR 30.387, identical to 10 pretrain IDs (30.387), and at 100% data the gain from 0 to 10 IDs is only 0.018 dB. Please soften or qualify this claim based on the actual numbers.
minor comments (6)
  1. [§1] In the first bullet of the introduction, 'Vsisual artifacts' is a typo for 'Visual artifacts'.
  2. [§3.2] The sentence 'Torso deformation filed is predicted...' uses 'filed' instead of 'field'.
  3. [Table 3] The table header layout is confusing: the ER-NeRF column is not labeled with a 'Pretrain IDs' value, and it is left to the reader to infer that its finetune data is 100%. Please make the column structure explicit.
  4. [§3.3] The phrase 'the hyper-network becomes regular network weights' is ambiguous: clarify whether the hyper-network is unrolled during fine-tuning or whether its output weights are frozen as initializations.
  5. [§3.4 and Implementation Details] Equation (12) lists four loss weights (λ_Δ, λ_att, λα, λ_lpips), but the implementation details specify only λα, λΔ, and λlpips; the value and schedule for λatt are missing.
  6. [Figure 5] The sentence 'The reason for the bright area close to the hair edges is due to the jitter in parsing results which mislead learning of the deformations' would read better as 'The bright area near the hair edges is caused by jitter in the parsing results, which misleads the learning of deformations.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: LokiTalk's claims are empirical reconstructions compared against held-out frames and official baselines; no stated prediction reduces to its input by construction.

full rationale

The core claims are evaluated by PSNR/LPIPS/LMD/Sync on validation frames against official implementations (AD-NeRF, RAD-NeRF, ER-NeRF, Wav2Lip), so the quantities called 'predictions' are rendered images compared with ground truth not used in fitting. The ID-Aware Knowledge Transfer module is a pretrain-then-finetune procedure (Section 3.3): multi-identity pretraining initializes the target model, then only target-identity data is used for fine-tuning and evaluation; improvement is measured against the no-pretrain base model in Tables 3-4, not defined by the module's own outputs. The deformation fields condition on audio, pose, and eye-ratio inputs (Eqs. 3-5), and losses (Eqs. 6-12) are standard reconstruction and regularization terms; no equation defines the reported metric in terms of the fitted signal. The one author-overlap reference ([22], Learning Dynamic Tetrahedra) is not cited in the body and is not load-bearing. The numerical mismatch between Table 1 ('Ours' PSNR 33.744) and the corresponding Table 3 row ('Pretrain IDs 10, Finetune Data 100%' PSNR 30.482) is an internal-consistency/reproducibility concern, not a circularity pattern: the two numbers are claimed for the same configuration but no reduction shows one is derived from the other. Under the circularity definitions, the derivation chain is self-contained.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method is a supervised learning system; the only free parameters are the loss weights and the hand-set training schedule. The domain assumptions are non-trivial because the claimed gains depend on the causal decomposition of motion and on the existence of a transferable canonical face space. No new physical entities are introduced; all components are network modules or learned offsets.

free parameters (4)
  • lambda_delta = 1e-5
    Weight for region regularization loss L_delta in Eq. 12; value chosen by hand, no sensitivity analysis reported.
  • lambda_alpha = 1e-4
    Weight for alpha entropy regularization in Eq. 12; fixed without ablation.
  • lambda_lpips = 5e-3 for final 20% of iterations
    LPIPS loss is disabled for the first 80% of iterations and activated at 5e-3 for the rest; schedule chosen empirically.
  • lambda_att = not reported
    Attention regularization weight in Eq. 12 is not given, so the exact loss balance is not reproducible.
assumptions (5)
  • standard math The NeRF volume rendering equations (Eq. 1 and Eq. 2) correctly model image formation for the talking head scenes.
    Adopted from Mildenhall et al. [15]; all pixel-space supervision in Eq. 6 relies on this model.
  • domain assumption Every observed point can be warped to a single unified canonical space by adding the two predicted deformation fields (Eq. 5).
    Section 3.2 assumes the cascaded face and torso deformation fields explain all motion; no test for topological changes such as occlusions or hair parsing errors.
  • domain assumption The causal decomposition is correct: audio drives lip/face motion, the eye aspect ratio drives blinking, and head pose plus face deformation drives torso motion.
    Section 3.2 asserts this hierarchy rather than learning or validating the direction of causality.
  • domain assumption A shared canonical space plus per-identity offsets can represent different identities during multi-ID pretraining.
    Section 3.3 relies on common geometry across identities; Table 4 shows weak transfer for ER-NeRF, so this premise is only weakly evidenced.
  • domain assumption Face/torso segmentation masks and 3DMM tracking used in Eq. 7 and head pose estimation are accurate.
    Figure 5 caption acknowledges that jitter in parsing results misleads deformation learning, so mask quality is a known sensitivity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LokiTalk: Learning Fine-Grained and Generalizable Correspondences to Enhance NeRF-based Talking Head Synthesis." pith.science (2026). https://pith.science/paper/2MHU2BW2

@misc{pith2026241119525,
  author       = {Pith},
  title        = {Pith review of: LokiTalk: Learning Fine-Grained and Generalizable Correspondences to Enhance NeRF-based Talking Head Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2MHU2BW2}},
  note         = {Machine review of arXiv:2411.19525}
}
read the original abstract

Despite significant progress in talking head synthesis since the introduction of Neural Radiance Fields (NeRF), visual artifacts and high training costs persist as major obstacles to large-scale commercial adoption. We propose that identifying and establishing fine-grained and generalizable correspondences between driving signals and generated results can simultaneously resolve both problems. Here we present LokiTalk, a novel framework designed to enhance NeRF-based talking heads with lifelike facial dynamics and improved training efficiency. To achieve fine-grained correspondences, we introduce Region-Specific Deformation Fields, which decompose the overall portrait motion into lip movements, eye blinking, head pose, and torso movements. By hierarchically modeling the driving signals and their associated regions through two cascaded deformation fields, we significantly improve dynamic accuracy and minimize synthetic artifacts. Furthermore, we propose ID-Aware Knowledge Transfer, a plug-and-play module that learns generalizable dynamic and static correspondences from multi-identity videos, while simultaneously extracting ID-specific dynamic and static features to refine the depiction of individual characters. Comprehensive evaluations demonstrate that LokiTalk delivers superior high-fidelity results and training efficiency compared to previous methods. The code will be released upon acceptance.

Figures

Figures reproduced from arXiv: 2411.19525 by the authors.

Figure 1
Figure 1. Overview of the proposed Region-Specific Deformation Fields. The driving signals (audio, pose, eye ratio) participate in the two-stage prediction of face and torso deformation fields, respectively. The mask subsequent to each driving signal represents the cross-attention loss between the driving signal and the corresponding region. A colored cubic grid is used to illustrate the predicted deformation fields, with the… view at source ↗
Figure 2
Figure 2. ID-Aware Knowledge Transfer. The blue modules are the common correspondences among multiple identities, comprising dynamic (light blue) and static (dark blue) corre￾spondences. The colored modules are dynamic (facial actions) and static information (geometry and appearance) of individual identities. During the pre-training (entire yellow panel), both upper and lower parts are trained simultaneously on multi-ID data,… view at source ↗
Figure 3
Figure 3. The comparison of the keyframes and details of generated portraits. We mark the un-sync and bad rendering quality results with red arrows, around which the generated eyes, mouths, neck broken or wrinkles are clearly not in line with the real ones. We also show the details of the eyes, teeth, forehead wrinkles and mouth area. Please zoom in for better visualization. present ID-Aware Knowledge Transfer, a plug-and-pla… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of the depth maps generated by our method and the baseline methods. Our depth map shows more details on the face area, especially the mouth and eye expressions (differences between open and closed eyes). The connection between the face and torso is more cons…
Figure 5
Figure 5. Figure 5: Heatmaps of the P [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 26 canonical work pages

  1. [1]

    Obamanet: Photo-realistic lip-sync from text

    Rithesh Kumar, Jose Sotelo, Kundan Kumar, Alexan- dre De Brebisson, and Yoshua Bengio. Obamanet: Photo-realistic lip-sync from text. arXiv preprint arXiv:1801.01442, 2017

    arXiv 2017

  2. [2]

    Audio2face: Generating speech/face animation from single audio with attention-based bidirectional lstm networks

    Guanzhong Tian, Yi Yuan, and Yong Liu. Audio2face: Generating speech/face animation from single audio with attention-based bidirectional lstm networks. In 2019 IEEE International Conference on Multimedia & Expo Work- shops (ICMEW), pages 366–371. IEEE, 2019

    work page 2019

  3. [3]

    A neural lip-sync framework for synthesizing photorealistic virtual news anchors

    Ruobing Zheng, Zhou Zhu, Bo Song, and Changjiang Ji. A neural lip-sync framework for synthesizing photorealistic virtual news anchors. In 2020 25th International Confer- ence on Pattern Recognition (ICPR) , pages 5286–5293. IEEE, 2021

    work page 2020

  4. [4]

    Depth-aware generative adversarial network for talking head video generation

    Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3397–3406, 2022

    work page 2022

  5. [5]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. InProceedings of the IEEE /CVF Conference on Computer Vision and Pattern Recognition, pages 8652–8661, 2023

    work page 2023

  6. [6]

    Videoretalking: Audio-based lip synchronization for talking head video editing in the wild

    Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. Videoretalking: Audio-based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022

    work page 2022

  7. [7]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485, 2024

    arXiv 2024

  8. [8]

    Talking-head generation with rhythmic head motion

    Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. Talking-head generation with rhythmic head motion. In European Conference on Computer Vision, pages 35–51. Springer, 2020

    work page 2020

Show all 36 references
  1. [9]

    Synthesizing obama: learn- ing lip sync from audio

    Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learn- ing lip sync from audio. ACM Transactions on Graphics (ToG), 36(4):1–13, 2017

    2017

  2. [10]

    Neural voice puppetry: Audio-driven facial reenactment

    Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, pages 716–

    2020

  3. [11]

    Learning pose-adaptive lip sync with cascaded temporal convolu- tional network

    Ruobing Zheng, Bo Song, and Changjiang Ji. Learning pose-adaptive lip sync with cascaded temporal convolu- tional network. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4255–4259. IEEE, 2021

    2021

  4. [12]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484–492, 2020

    2020

  5. [13]

    Talking face generation by adversarially disentan- gled audio-visual representation

    Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially disentan- gled audio-visual representation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 9299–9306, 2019. Preprint. Under review. 8

    2019

  6. [14]

    Synctalk: The devil is in the synchronization for talking head synthesis

    Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk: The devil is in the synchronization for talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 66...

    2024

  7. [15]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM, 65(1):99–106, 2021

    2021

  8. [16]

    Ad-nerf: Audio driven neural radiance fields for talking head synthesis

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5784–5794, 2021

    2021

  9. [17]

    Real-time neural radiance talking portrait synthesis via audio-spatial decomposition

    Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211.12368, 2022

    2022 arXiv

  10. [18]

    Efficient region-aware neural radiance fields for high- fidelity talking portrait synthesis

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, and Lin Gu. Efficient region-aware neural radiance fields for high- fidelity talking portrait synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7568–7578, 2023

    2023

  11. [19]

    Dfa-nerf: Personalized talking head generation via disentangled face attributes neural rendering

    Shunyu Yao, RuiZhe Zhong, Yichao Yan, Guangtao Zhai, and Xiaokang Yang. Dfa-nerf: Personalized talking head generation via disentangled face attributes neural rendering. arXiv preprint arXiv:2201.00791, 2022

    2022 arXiv

  12. [20]

    Semantic-aware implicit neural audio- driven video portrait generation

    Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. Semantic-aware implicit neural audio- driven video portrait generation. In European Conference on Computer Vision, pages 106–125. Springer, 2022

    2022

  13. [21]

    Learning dynamic facial radiance fields for few-shot talking head synthesis

    Shuai Shen, Wanhua Li, Zheng Zhu, Yueqi Duan, Jie Zhou, and Jiwen Lu. Learning dynamic facial radiance fields for few-shot talking head synthesis. In European Conference on Computer Vision, pages 666–682. Springer, 2022

    2022

  14. [22]

    Learning dynamic tetrahedra for high-quality talking head synthesis

    Zicheng Zhang, Ruobing Zheng, Bonan Li, Congying Han, Tianqi Li, Meng Wang, Tiande Guo, Jingdong Chen, Zi- wen Liu, and Ming Yang. Learning dynamic tetrahedra for high-quality talking head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  15. [23]

    Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans

    Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceed- ings of the IEEE/CVF Conference on Computer Vision a...

    2021

  16. [24]

    An- imatable neural radiance fields for modeling dynamic human bodies

    Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. An- imatable neural radiance fields for modeling dynamic human bodies. In Proceedings of the IEEE /CVF Inter- national Conference on Computer Vision, pages 14314– 14323, 2021

    2021

  17. [25]

    Neural scene flow fields for space-time view syn- thesis of dynamic scenes

    Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view syn- thesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498–6508, 2021

    2021

  18. [26]

    Dynamic neural radiance fields for monocular 4d facial avatar reconstruction

    Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8649–8658, 2021

    2021

  19. [27]

    Barron, Sofien Bouaziz, Dan B Goldman, Steven M

    Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021

    2021

  20. [28]

    Hypernerf: A higher- dimensional representation for topologically varying neu- ral radiance fields

    Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin- Brualla, and Steven M Seitz. Hypernerf: A higher- dimensional representation for topologically varying neu- ral radiance fields. arXiv preprint arXiv:2106.13228, 2021

    2021 arXiv

  21. [29]

    Dynamic neural radiance fields for monoc- ular 4d facial avatar reconstruction

    Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monoc- ular 4d facial avatar reconstruction. 2021 IEEE /CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8645–8654, 2020. URL https://api. semanticschola...

    2021

  22. [30]

    Fast bi-layer neural syn- thesis of one-shot realistic head avatars

    Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural syn- thesis of one-shot realistic head avatars. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pages 524–540....

    2020

  23. [31]

    Few-shot video-to-video synthesis

    Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video synthesis. arXiv preprint arXiv:1910.12713, 2019

    1910 arXiv

  24. [32]

    Live speech portraits: Real-time photorealistic talking-head anima- tion

    Yuanxun Lu, Jinxiang Chai, and Xun Cao. Live speech portraits: Real-time photorealistic talking-head anima- tion. ACM Trans. Graph., 40(6), dec 2021. ISSN 0730-

    2021

  25. [33]

    The unreasonable e ffectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable e ffectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018

    2018

  26. [34]

    Maddox, Zhiyao Duan, and Chenliang Xu

    Lele Chen, Zhiheng Li, Ross K. Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements generation at a glance. ArXiv, abs /1803.10404, 2018. URL https: //api.semanticscholar.org/CorpusID:4435268

    2018 arXiv

  27. [35]

    Out of time: Automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: Automated lip sync in the wild. InComputer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017

    2016

  28. [301]

    URL https: //doi.org/10.1145/3478513.3480484

    doi: 10.1145 /3478513.3480484. URL https: //doi.org/10.1145/3478513.3480484

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.