Pith. sign in

REVIEW 3 major objections 6 minor 56 references

M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A talking-head model decouples head, face, and mouth motion to beat prior 3D Gaussian renderers by 2.43 dB PSNR.

desk verdict A solid engineering contribution to 3DGS talking-head generation with real novelties, but the SOTA claim hinges on a narrow self-reconstruction protocol and one internal inconsistency in the ablation table. read the letter →

arxiv 2507.08307 v3 pith:CYGMM75Q submitted 2025-07-11 cs.CV

Kui Jiang , Shiyu Liu , Junjun Jiang , Hongxun Yao , Xiaopeng Fan This is my paper
classification cs.CV
keywords audio-driventalkingheadgeneration3DGaussianSplattingmotiondecouplingalternatingoptimizationposeestimationsemanticsegmentationlipsynchronizationfacialActionUnits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that talking-head video generation improves when you stop treating the face as one moving surface and instead split its motion into three parts: rigid head rotation, facial expression, and internal mouth articulation, each modeled by its own branch and supervised by clean, preprocessed masks and camera parameters. The authors claim that this multi-granular decoupling, together with a motion consistency constraint that links face to torso and an alternating optimization scheme that refines the face and mouth branches in turns, removes the motion blur, temporal jitter, and face-torso penetration artifacts that plague existing 3D Gaussian splatting methods. If the claim holds, it matters because the method runs at 150 FPS, fast enough for real-time film-production and avatar use, while reportedly beating the strongest prior system by 2.43 dB PSNR and scoring 0.64 higher in user-rated video realness. The reader should care because the bottleneck in talking-head avatars is no longer raw fidelity but stable, fine-grained motion, and this is a direct attack on that bottleneck.

What carries the argument

The central machinery is the three-way motion factorization realized in a dual-branch 3D Gaussian deformation field. A rigid head trajectory is estimated by optical-flow tracking of deformation-invariant keypoints (ears, hairlines) and encoded as camera rotation, scale, and focal length; a Face Branch deforms Gaussians for macro expressions using attention-modulated phoneme and Action-Unit features; an Inside Mouth Branch deforms Gaussians for tongue and teeth articulation using only phoneme-aware features. Region masks from the preprocessing pipeline gate both rendering and losses, and a full-portrait compositing loss couples facial deformation to torso motion, while alternating optimization cycles between the two branches to stabilize the facial-oral boundary.

What would settle it

Train M2DAO-Talker on six of the seven identities and reconstruct the held-out seventh, or adapt it to an entirely unseen speaker, and compare against TalkingGaussian under the same protocol; if the 2.43 dB PSNR and 0.64 user-study realness advantages shrink to statistical noise or reverse, the claim that the architecture itself improves talking-head generation is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a talking-head synthesis system built on 3D Gaussian Splatting reaches state-of-the-art reconstruction quality when motion is factored into rigid head rotation, non-rigid facial expression, and oral articulation, with each factor assigned its own deformation branch. The paper shows that precise 2D preprocessing—segmenting motion regions with positional-prompt-guided masks refined by a teeth-aware parser, and estimating camera parameters from deformation-invariant optical-flow keypoints instead of 3DMM fitting—gives the deformation branches clean inputs that previous methods lack. Two further mechanisms carry the argument: a motion consistency constraint that composites facial renderings over the torso background to enforce kinematic coherence, and an alternating optimization that first trains the face branch with the mouth branch frozen, then fine-tunes both, preventing lip-color leakage and tooth discoloration. On seven per-identity self-reconstruction clips the system reports PSNR 34.47 dB, LPIPS 0.0229, LMD 2.636, and sync confidence 7.756, outperforming the previous best 3DGS method TalkingGaussian by 2.43 dB PSNR and a user-study realness gain of 0.64, at 150 FPS.

Load-bearing premise

The load-bearing premise is that scores from per-identity self-reconstruction on seven short video clips—where the network trains and tests on the same speaker—measure the quality that matters for real talking-head generation; if those gains mostly reflect memorizing one speaker's appearance, the reported advantages will not carry over to unseen identities or production footage.

Editorial extensions

If this is right

  • Real-time use: at 150 FPS on an RTX 3090, the system is fast enough for interactive avatar and film-preview workflows without quality loss.
  • The motion consistency constraint should eliminate head-torso kinematic discontinuity and penetration artifacts that prior 3DGS talking-head systems exhibit.
  • The alternating optimization scheme should prevent lip-color leakage and tooth discoloration that arise from joint training of face and mouth branches.
  • The preprocessing pipeline (positional-prompt masks plus teeth-aware refinement, and flow-based HPE) transfers to other architectures; ablations show SyncTalk and TalkingGaussian also improve when fitted with MRS and HPE.
  • In cross-domain lip-sync tests, the method generalizes to unseen speakers and cross-gender audio, maintaining sync confidence above specialized 2D baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the per-identity self-reconstruction gains are real and not memorization, a natural next test is one-shot or few-shot adaptation to an unseen identity; the current design trains per identity, so the preprocessing pipeline, not the trained Gaussians, is what would need to transfer.
  • The dependence on optical-flow-based pose estimation is a stated limitation: edited videos with temporal cuts break the rigid-trajectory assumption; a testable extension is to fuse the flow-based trajectory with inertial or landmark-based priors on discontinuous footage.
  • The use of six Action Units (AU 1, 4, 5, 6, 7, 45) for expression control suggests an explicit, interpretable control space; a user-controllable extension would let animators drive expressions by AU intensities rather than only by audio.
  • Because MRS improved even architectures without explicit deformation modeling (SyncTalk), the segmentation pipeline may be a separately reusable asset for the broader talking-face field, with or without 3DGS.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript proposes M2DAO-Talker, a per-identity audio-driven talking-head synthesis method built on 3D Gaussian Splatting, organized as three stages: video preprocessing (hybrid motion-region segmentation and optical-flow-based head pose estimation), motion representation (decoupling rigid head motion from non-rigid facial and oral deformations via two Gaussian branches), and rendering reconstruction with a full-portrait motion consistency constraint and an alternating optimization schedule between the Face and Inside Mouth branches. On seven standard talking-head clips, the authors report state-of-the-art self-reconstruction results, including PSNR 34.47 dB versus 32.04 for TalkingGaussian, LPIPS 0.0229, LMD 2.636, Sync-C 7.756, and 150 FPS inference, together with a cross-domain lip-synchronization comparison (Table 3), component ablations (Tables 4-6 and appendix Tables 7-8), and a 20-participant user study.

Significance. If the results hold, the contribution is a well-executed systems-level synthesis: the claimed gains (2.43 dB PSNR and 0.64 user-rated realness over TalkingGaussian) combine three independently reasonable ideas - precise mask and pose preprocessing, multi-granular motion decoupling, and staged optimization - rather than resting on a single new theoretical mechanism. The manuscript is candid about borrowed components (SyncTalk's audio-visual encoder, SAM2, OpenFace, 3DGS) and about its own limitations: Appendix E concedes that the audio encoder can cause lip jitter, that binary segmentation masks produce jagged boundaries, and that optical-flow camera estimation fails on edited videos, while Appendix F describes the framework as identity-specific with generalization left to future work. These admissions narrow the scope of the abstract's film-production framing. The central weakness is evidential: all quantitative results are per-identity self-reconstruction on seven clips with no held-out identity, no error bars, and a thin user study, and the MRS ablation contains a table/text contradiction (Table 4).

major comments (3)
  1. [Ablation Study (Ablation Results; Table 4)] The text states that replacing the standard segmentation backbone with MRS yields 'consistent improvements' and that 'even SyncTalk ... benefits from MRS,' but Table 4 shows the opposite for two of the three tested backbones: for TalkingGaussian, MRS decreases PSNR from 33.260 to 31.519 and Sync-C from 6.868 to 5.971, while for SyncTalk, PSNR decreases from 34.355 to 33.955 and Sync-C from 7.819 to 7.663. Only for M2DAO-Talker itself does MRS improve PSNR (34.400 to 35.041). This contradiction is load-bearing because the ablation is the main evidence that MRS is a general contribution rather than a component that happens to help the proposed architecture. Please reconcile the text with the table, report per-clip results, and either soften the claim or explain the mechanism by which MRS helps M2DAO-Talker but harms the other methods.
  2. [Experimental Settings (Dataset, Comparison Settings); Tables 1-3; User Study] The headline claims (Abstract, Table 1: 2.43 dB over TalkingGaussian; 0.64 user-rated realness gain) rest entirely on per-identity self-reconstruction with a 10:1 temporal split on seven clips. All target identities are seen during training, no held-out-identity reconstruction is reported, and no per-clip breakdown, confidence interval, or significance test is given for Tables 1-3. The user study is too thin to carry the perceptual claim: 20 participants each rate only seven half-minute videos (49 videos total across six models and ground truth, so roughly three ratings per video on average), and the reported 87%-of-GT and 0.49 lip-sync-error figures are not accompanied by variance or a test. Because self-reconstruction on training identities cannot distinguish genuine motion-modeling ability from identity-specific memorization of static appearance, the abstract's SOTA claim is not yet supported. Please add per-identity results, error bars over identities/clips, a held-out-identity (or few-shot) experiment, and a proper statistical summary of the user study.
  3. [Ablation Study (Table 6; Figure 6)] The claim that 'disabling MCC results in poor quality' is only weakly supported by Table 6: removing MCC changes PSNR from 34.475 to 34.375 and Sync-C from 7.757 to 7.556, while LMD actually improves (2.636 to 2.620), and removing AOS similarly improves LMD (2.636 to 2.624). The case therefore rests mainly on the qualitative Figure 6 panels. Please report per-clip numbers and clarify how LMD can improve when MCC or AOS is removed; otherwise the component-wise benefit of MCC and AOS is not quantitatively established.
minor comments (6)
  1. [Table 4 header] Table 4's header introduces the abbreviation 'BN' for BiseNet, but the abbreviation is never defined in the text or caption; please write out the baseline backbone names and mark which row uses the proposed MRS.
  2. [Figure 1 caption; Video Preprocessing; Motion Region Segmentation] Figure 1's caption contains 'two separate motion branches bashed on 3D Gaussian primitives' (should be 'based'), and the Video Preprocessing and Motion Region Segmentation sections contain the typos 'imgae' and 'incoporating'; a copyedit pass is needed.
  3. [Method (Alternating Optimization Reconstruction) vs Implementation Details] The alternating optimization is described in the Method section as first freezing the Inside Mouth Branch and then jointly fine-tuning both branches, whereas Implementation Details state the opposite schedule (50K joint iterations followed by 20K alternating iterations); these two descriptions should be aligned.
  4. [Tables 1, 4, 5, 6] The same configuration (M2DAO-Talker with all components) is reported as 34.47 dB in Table 1, 35.041 dB in Table 4, 33.746 dB in Table 5, and 34.475 dB in Table 6; the Ablation Setting text names the different subsets, but each table caption should state its subset so readers can trace the numbers.
  5. [Table 3] Table 3 does not bold the best entry in each column, which makes the claim that M2DAO-Talker 'demonstrates stable and consistent performance' harder to verify; please mark best results and note where ER-NeRF is superior (e.g., Shaheen audio on Obama, Sync-D 8.240 vs 8.486).
  6. [Reproducibility] No code, trained models, or evaluation scripts are released, despite the project homepage being listed; releasing the preprocessing and evaluation pipeline would materially improve reproducibility for this systems paper.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the SOTA claims are empirical comparisons against external baselines, and the framework's components are ablative design choices rather than fitted inputs renamed as predictions.

full rationale

The paper's central claim is state-of-the-art performance measured against external baselines (Tables 1-3 and the user study), and no equation defines the headline improvement from an assumed value: PSNR 34.47 vs 32.04 is a reported comparison, not a derived identity. The losses (Eqs. 10, 11, 13) are standard reconstruction objectives; MRS, HPE, MCC, and AOS are architectural choices whose contributions are assessed by ablations, not fitted parameters renamed as predictions. The only author-overlap reference I can identify is Liu and Hao 2023, cited in Appendix B as an example of blinking-pattern regulation; it is a related-work aside and is not load-bearing. Load-bearing components (3DGS, SyncTalk's audio-visual encoder, TalkingGaussian, SAM2, OpenFace) are external. Appendix E honestly lists limitations of AVE, binary masks, and optical-flow camera estimation, but none of these is an admission of circularity. The self-reconstruction protocol with per-identity training is a generalizability limitation—possible identity memorization could inflate scores—but that is an evaluation-design risk, not a reduction of the result to its inputs. Accordingly, no significant circularity; score 1.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim depends on external pretrained components including 3DGS, SyncTalk's audio-visual encoder, OpenFace, SAM2, and EasyPortrait, as well as on a hand-tuned training recipe. The main unverified assumption is that per-identity self-reconstruction metrics on seven clips are a sufficient benchmark for the claimed state-of-the-art status.

free parameters (5)
  • Loss weight lambda = 0.2
    Set by hand in Equations (10), (11), and (13) to balance L1 and DSSIM; no sensitivity analysis is reported.
  • Loss weight gamma = 0.5
    Set by hand in Equations (11) and (13) to weight LPIPS; no sensitivity analysis is reported.
  • Alternating optimization schedule = 50K joint plus 20K alternating iterations
    Training phase durations are chosen by hand; no ablation on iteration counts or phase lengths is provided.
  • Flow-error filtering threshold = sequence-level statistics, unspecified
    Frames with high flow error are filtered based on sequence-level statistics, but the exact criterion is not reported, making the threshold a hidden hand-set parameter.
  • Facial Action Unit subset = {1,4,5,6,7,45}
    Six AUs are selected by hand to represent facial expressions; the selection criterion and the excluded AUs are not justified beyond anatomical intuition.
assumptions (5)
  • domain assumption 3D Gaussian Splatting, including Gaussian primitives, projection, and alpha blending, is a valid representation for talking-head scenes.
    Preliminaries (Equations 1-4) adopt Kerbl et al. 2023 without modification; the paper assumes this rendering model captures facial geometry and dynamics.
  • domain assumption The SyncTalk audio-visual encoder, pretrained on LRS2 with a lip-sync discriminator, provides phoneme-aligned features sufficient for talking-head synthesis.
    Appendix B adopts this pretrained encoder; no re-training or validation of this feature space is provided in this paper.
  • domain assumption OpenFace 2.0 AU estimates are accurate ground-truth facial expression signals for the six selected AUs.
    Appendix B extracts frame-wise AU features via OpenFace and treats them as control conditions for the Face Branch.
  • domain assumption SAM2-based segmentation with positional prompts produces masks accurate enough for region-specific Gaussian deformation.
    Equation (5) and the Motion Region Segmentation section depend on SAM2 mask quality; mask accuracy is a stated motivation for the whole pipeline.
  • domain assumption Self-reconstruction on training identities with a 10-to-1 split is a valid proxy for talking-head generation quality.
    Experimental Settings section; no held-out identity reconstruction or cross-dataset evaluation is used for the headline PSNR and LPIPS numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation." pith.science (2026). https://pith.science/paper/CYGMM75Q

@misc{pith2026250708307,
  author       = {Pith},
  title        = {Pith review of: M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYGMM75Q}},
  note         = {Machine review of arXiv:2507.08307}
}
read the original abstract

Audio-driven talking head generation holds significant potential for film production. While existing 3D methods have advanced motion modeling and content synthesis, they often produce rendering artifacts, such as motion blur, temporal jitter, and local penetration, due to limitations in representing stable, fine-grained motion fields. Through systematic analysis, we reformulate talking head generation into a unified framework comprising three steps: video preprocessing, motion representation, and rendering reconstruction. This framework underpins our proposed M2DAO-Talker, which addresses current limitations via multi-granular motion decoupling and alternating optimization. Specifically, we devise a novel 2D portrait preprocessing pipeline to extract frame-wise deformation control conditions (motion region segmentation masks, and camera parameters) to facilitate motion representation. To ameliorate motion modeling, we elaborate a multi-granular motion decoupling strategy, which independently models non-rigid (oral and facial) and rigid (head) motions for improved reconstruction accuracy. Meanwhile, a motion consistency constraint is developed to ensure head-torso kinematic consistency, thereby mitigating penetration artifacts caused by motion aliasing. In addition, an alternating optimization strategy is designed to iteratively refine facial and oral motion parameters, enabling more realistic video generation. Experiments across multiple datasets show that M2DAO-Talker achieves state-of-the-art performance, with the 2.43 dB PSNR improvement in generation quality and 0.64 gain in user-evaluated video realness versus TalkingGaussian while with 150 FPS inference speed. Our project homepage is https://m2dao-talker.github.io/M2DAO-Talk.github.io.

Figures

Figures reproduced from arXiv: 2507.08307 by the authors.

Figure 1
Figure 1. The M2DAO-Talker pipeline comprises three stages. i) Video Preprocessing: it extracts key features from input [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Iterative optimization of the facial 3DGS point [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Qualitative results of Image Quality Comparison. Our method effectively eliminates edge blur and corrects facial [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: User Study. The rating is in the range of 1-5, higher [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: High-definition Comparison. We compare the generation results of different methods across five complex scenarios. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 47 canonical work pages

  1. [1]

    S.; Senior, A.; Vinyals, O.; and Zisserman, A

    Afouras, T.; Chung, J. S.; Senior, A.; Vinyals, O.; and Zisserman, A. 2018. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12): 8717--8727

    work page 2018

  2. [2]

    C.; and Morency, L.-P

    Baltrusaitis, T.; Zadeh, A.; Lim, Y. C.; and Morency, L.-P. 2018. OpenFace 2.0: Facial Behavior Analysis Toolkit. In International Conference on Automatic Face and Gesture Recognition, 59--66. IEEE

    work page 2018

  3. [3]

    Brox, T.; Bruhn, A.; Papenberg, N.; and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In European Conference on Computer Vision, 25--36. Springer

    work page 2004

  4. [4]

    Cho, K.; Lee, J.; Yoon, H.; Hong, Y.; Ko, J.; Ahn, S.; and Kim, S. 2024. GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splatting. In ACM International Conference on Multimedia, 10985--10994

    work page 2024

  5. [5]

    S.; and Zisserman, A

    Chung, J. S.; and Zisserman, A. 2017. Lip reading in the wild. In Asian Conference on Computer Vision, 87--103. Springer

    work page 2017

  6. [6]

    Dosovitskiy, A.; Fischer, P.; Ilg, E.; Hausser, P.; Hazirbas, C.; Golkov, V.; Van Der Smagt, P.; Cremers, D.; and Brox, T. 2015. Flownet: Learning optical flow with convolutional networks. In IEEE/CVF International Conference on Computer Vision, 2758--2766

    work page 2015

  7. [7]

    Duan, Y.; Wei, F.; Dai, Q.; He, Y.; Chen, W.; and Chen, B. 2024. 4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes. In ACM SIGGRAPH 2024 Conference Papers, 1--11

    work page 2024

  8. [8]

    Ekman, P.; and Friesen, W. V. 1978. Facial action coding system. Environmental Psychology & Nonverbal Behavior

    work page 1978

Show all 56 references
  1. [9]

    Fei, B.; Xu, J.; Zhang, R.; Zhou, Q.; Yang, W.; and He, Y. 2024. 3D Gaussian Splatting as New Era: A Survey. IEEE Transactions on Visualization and Computer Graphics, (01): 1--20

    2024

  2. [10]

    Guan, J.; Zhang, Z.; Zhou, H.; Hu, T.; Wang, K.; He, D.; Feng, H.; Liu, J.; Ding, E.; Liu, Z.; et al. 2023. Stylesync: High-fidelity generalized and personalized lip sync in style-based generator. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1505--1515

    2023

  3. [11]

    Guo, M.-H.; Liu, Z.-N.; Mu, T.-J.; and Hu, S.-M. 2022. Beyond self-attention: External attention using two linear layers for visual tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5): 5436--5447

    2022

  4. [12]

    Guo, Y.; Chen, K.; Liang, S.; Liu, Y.-J.; Bao, H.; and Zhang, J. 2021. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In IEEE/CVF International Conference on Computer Vision, 5784--5794

    2021

  5. [13]

    K.; and Schunck, B

    Horn, B. K.; and Schunck, B. G. 1981. Determining optical flow. Artificial Intelligence, 17(1-3): 185--203

    1981

  6. [14]

    Hu, L.; Zhang, H.; Zhang, Y.; Zhou, B.; Liu, B.; Zhang, S.; and Nie, L. 2024. Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 634--644

    2024

  7. [15]

    Huang, N.; Wei, X.; Zheng, W.; An, P.; Lu, M.; Zhan, W.; Tomizuka, M.; Keutzer, K.; and Zhang, S. 2024. S ^3 Gaussian: Self-Supervised Street Gaussians for Autonomous Driving. arXiv preprint arXiv:2405.20323

    2024 arXiv

  8. [16]

    C.; Qin, H.; Dai, J.; and Li, H

    Huang, Z.; Shi, X.; Zhang, C.; Wang, Q.; Cheung, K. C.; Qin, H.; Dai, J.; and Li, H. 2022. Flowformer: A transformer architecture for optical flow. In European Conference on Computer Vision, 668--685. Springer

    2022

  9. [17]

    Kerbl, B.; Kopanas, G.; Leimk \"u hler, T.; and Drettakis, G. 2023. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4)

    2023

  10. [18]

    Kvanchiani, K.; Petrova, E.; Efremyan, K.; Sautin, A.; and Kapitanov, A. 2023. EasyPortrait--Face Parsing and Portrait Segmentation Dataset

    2023

  11. [19]

    Li, J.; Zhang, J.; Bai, X.; Zheng, J.; Ning, X.; Zhou, J.; and Gu, L. 2025. Talkinggaussian: Structure-persistent 3d talking head synthesis via gaussian splatting. In European Conference on Computer Vision, 127--145

    2025

  12. [20]

    Li, J.; Zhang, J.; Bai, X.; Zhou, J.; and Gu, L. 2023. Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. In IEEE/CVF International Conference on Computer Vision, 7568--7578

    2023

  13. [21]

    Liu, S.; and Hao, J. 2023. Generating Talking Face With Controllable Eye Movements by Disentangled Blinking Feature. IEEE Transactions on Visualization and Computer Graphics, 29(12): 5050--5061

    2023

  14. [22]

    Lu, Y.; Chai, J.; and Cao, X. 2021. Live speech portraits: real-time photorealistic talking-head animation. ACM Transactions on Graphics, 40(6): 1--17

    2021

  15. [23]

    M \'e min, E.; and P \'e rez, P. 1998. Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Transactions on Image Processing, 7(5): 703--719

    1998

  16. [24]

    P.; Tancik, M.; Barron, J

    Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; and Ng, R. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99--106

    2021

  17. [25]

    T.; Bouaziz, S.; Goldman, D

    Park, K.; Sinha, U.; Barron, J. T.; Bouaziz, S.; Goldman, D. B.; Seitz, S. M.; and Martin-Brualla, R. 2021 a . Nerfies: Deformable neural radiance fields. In IEEE/CVF International Conference on Computer Vision, 5865--5874

    2021

  18. [26]

    T.; Bouaziz, S.; Goldman, D

    Park, K.; Sinha, U.; Hedman, P.; Barron, J. T.; Bouaziz, S.; Goldman, D. B.; Martin-Brualla, R.; and Seitz, S. M. 2021 b . HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG), 40(6): 1--12

    2021

  19. [27]

    Paysan, P.; Knothe, R.; Amberg, B.; Romdhani, S.; and Vetter, T. 2009. A 3D face model for pose and illumination invariant face recognition. In IEEE International Conference on Advanced Video and Signal based Surveillance, 296--301. IEEE

    2009

  20. [28]

    Peng, Z.; Hu, W.; Shi, Y.; Zhu, X.; Zhang, X.; Zhao, H.; He, J.; Liu, H.; and Fan, Z. 2024. Synctalk: The devil is in the synchronization for talking head synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 666--676

    2024

  21. [29]

    P.; and Jawahar, C

    Prajwal, K.; Mukhopadhyay, R.; Namboodiri, V. P.; and Jawahar, C. 2020. A lip sync expert is all you need for speech to lip generation in the wild. In ACM International Conference on Multimedia, 484--492

    2020

  22. [30]

    Pumarola, A.; Corona, E.; Pons-Moll, G.; and Moreno-Noguer, F. 2021. D-nerf: Neural radiance fields for dynamic scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10318--10327

    2021

  23. [31]

    Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; R \"a dle, R.; Rolland, C.; Gustafson, L.; et al. 2024. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714

    2024 arXiv

  24. [32]

    Sun, S.; Chen, Y.; Zhu, Y.; Guo, G.; and Li, G. 2022 a . Skflow: Learning optical flow with super kernels. Advances in Neural Information Processing Systems, 35: 11313--11326

    2022

  25. [33]

    Sun, Y.; Zhou, H.; Wang, K.; Wu, Q.; Hong, Z.; Liu, J.; Ding, E.; Wang, J.; Liu, Z.; and Hideki, K. 2022 b . Masked lip-sync prediction by audio-visual contextual exploitation in transformers. In SIGGRAPH Asia 2022 Conference Papers, 1--9

    2022

  26. [34]

    Tang, J.; Wang, K.; Zhou, H.; Chen, X.; He, D.; Hu, T.; Liu, J.; Zeng, G.; and Wang, J. 2022. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211.12368

    2022 arXiv

  27. [35]

    Teed, Z.; and Deng, J. 2020. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision, 402--419. Springer

    2020

  28. [36]

    Tretschk, E.; Tewari, A.; Golyanik, V.; Zollh \"o fer, M.; Lassner, C.; and Theobalt, C. 2021. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In IEEE/CVF International Conference on Computer Vision, 12959--12970

    2021

  29. [37]

    T.; and Li, H

    Wang, J.; Qian, X.; Zhang, M.; Tan, R. T.; and Li, H. 2023. Seeing what you said: Talking face generation guided by a lip reading expert. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14653--14662

    2023

  30. [38]

    Wang, M.; Zhao, S.; Dong, X.; and Shen, J. 2024 a . High-Fidelity and High-Efficiency Talking Portrait Synthesis With Detail-Aware Neural Radiance Fields. IEEE Transactions on Visualization and Computer Graphics, (01): 1--14

    2024

  31. [39]

    Wang, Q.; Ye, V.; Gao, H.; Austin, J.; Li, Z.; and Kanazawa, A. 2024 b . Shape of motion: 4d reconstruction from a single video. arXiv preprint arXiv:2407.13764

    2024

  32. [40]

    Wang, S.; Li, L.; Ding, Y.; Fan, C.; and Yu, X. 2021. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. In International Joint Conference on Artificial Intelligence, 1098--1105

    2021

  33. [41]

    C.; Sheikh, H

    Wang, Z.; Bovik, A. C.; Sheikh, H. R.; and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600--612

    2004

  34. [42]

    Wedel, A.; Cremers, D.; Pock, T.; and Bischof, H. 2009. Structure-and motion-adaptive regularization for high accuracy optic flow. In IEEE/CVF International Conference on Computer Vision, 1663--1668. IEEE

    2009

  35. [43]

    Wu, G.; Yi, T.; Fang, J.; Xie, L.; Zhang, X.; Wei, W.; Liu, W.; Tian, Q.; and Wang, X. 2024. 4d gaussian splatting for real-time dynamic scene rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20310--20320

    2024

  36. [44]

    Yang, L.; Deng, B.; and Zhang, J. 2025. Scalable and high-quality neural implicit representation for 3D reconstruction. IEEE Transactions on Visualization & Computer Graphics

    2025

  37. [45]

    Yang, Z.; Gao, X.; Zhou, W.; Jiao, S.; Zhang, Y.; and Jin, X. 2024 a . Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20331--20341

    2024

  38. [46]

    Yang, Z.; Yang, H.; Pan, Z.; and Zhang, L. 2024 b . Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting. In International Conference on Learning Representations (ICLR)

    2024

  39. [47]

    Yao, S.; Zhong, R.; Yan, Y.; Zhai, G.; and Yang, X. 2022. Dfa-nerf: Personalized talking head generation via disentangled face attributes neural rendering. arXiv preprint arXiv:2201.00791

    2022 arXiv

  40. [48]

    Ye, Z.; Jiang, Z.; Ren, Y.; Liu, J.; He, J.; and Zhao, Z. 2023. GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis

    2023

  41. [49]

    Yu, C.; Wang, J.; Peng, C.; Gao, C.; Yu, G.; and Sang, N. 2018. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In European Conference on Computer Vision, 325--341

    2018

  42. [50]

    Zhang, C.; Ni, S.; Fan, Z.; Li, H.; Zeng, M.; Budagavi, M.; and Guo, X. 2023 a . 3D Talking Face With Personalized Pose Dynamics. IEEE Transactions on Visualization and Computer Graphics, 29(2): 1438--1449

    2023

  43. [51]

    A.; Shechtman, E.; and Wang, O

    Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 586--595

    2018

  44. [52]

    Zhang, Z.; Hu, Z.; Deng, W.; Fan, C.; Lv, T.; and Ding, Y. 2023 b . Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video. In AAAI Conference on Artificial Intelligence, 3543--3551

    2023

  45. [53]

    Zhong, W.; Fang, C.; Cai, Y.; Wei, P.; Zhao, G.; Lin, L.; and Li, G. 2023. Identity-preserving talking face generation with landmark and appearance priors. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9729--9738

    2023

  46. [54]

    Zhuang, Y.; Cheng, B.; Cheng, Y.; Jin, Y.; Liu, R.; Li, C.; Cheng, X.; Liao, J.; and Lin, J. 2024. Learn2Talk: 3D Talking Face Learns from 2D Talking Face. IEEE Transactions on Visualization and Computer Graphics, 1--13

    2024

  47. [55]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.