Pith. sign in

REVIEW 3 major objections 5 minor 86 references

JOLT3D: Joint Learning of Talking Heads and 3DMM Parameters with Application to Lip-Sync

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Jointly trained 3D face model sharpens talking heads and lip-sync

desk verdict A substantial 3DMM-based talking-head and lip-sync system with a genuinely new joint-training idea, but the central claim that joint training is responsible for the gains is not isolated for lack of a frozen-ReconNet control. read the letter →

arxiv 2507.20452 v1 pith:5LIVD3TP submitted 2025-07-28 cs.CV

Sungjoon Park , Minsik Park , Haneol Lee , Jaesub Yun , Donggeon Lee This is my paper
classification cs.CV
keywords 3Dmorphablemodeltalkingheadsynthesislip-syncFACSblendshapesjointtrainingfacereenactmentdiffusionICT-FaceKit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

JOLT3D claims that the 3D morphable face model (3DMM) is not the limiting factor in audio-driven talking-head synthesis; the limiting factor is how the 3DMM parameters are extracted. Instead of fitting landmarks or relying on pretrained reconstruction networks, the paper trains the face-reconstruction network jointly with the talking-head generator, so the blendshapes it outputs are optimized for re-rendering faces. On public benchmarks, this joint training improves image quality, identity preservation, and lip-sync metrics over 3DMM-based reenactment baselines. The resulting FACS-based blendshape representation also enables a lip-sync pipeline that edits only mouth-related blendshapes, re-renders the face twice, and blends the results so the chin contour can change without flicker.

What carries the argument

The load-bearing object is the pair of the face-reconstruction network (ReconNet) and the talking-head generator, trained jointly. ReconNet predicts identity, blendshape, head-pose, and eyeball-rotation parameters; those parameters are rendered into a vertex position map and a face sketch, which FlowNet uses to predict a warping mask and a raw flow field. The key modification is the warping parameterization $I_{\text{warped}} = K(F_{\text{raw}} \star I) + (1-K)I$, which prevents the mask $K$ from collapsing to zero during training. ICT-FaceKit's FACS-based blendshapes provide the mouth-localized control used for lip-sync, and a diffusion model converts audio and style embeddings into 35 mouth blendshapes. The final lip-sync mechanism is double reenactment: one pass with a fixed reference frame supplies the mouth interior, and one pass with the current frame adjusts the chin contour before blending.

What would settle it

Retrain the JOLT3D generator and FlowNet with the ReconNet's weights frozen after the landmark-pretraining stage, keeping every other loss and architectural change, and compare L1, PSNR, FID, LSE-D, and CSIM on HDTF and VoxCeleb2. If the frozen ReconNet matches or beats joint training, the paper's attribution of improvement to joint training is falsified; if joint training clearly wins, the claim is supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a FACS-based 3DMM representation, paired with a reconstruction network trained jointly with the generator, is sufficient for high-quality talking-head reenactment and lip-sync. The central claim is that earlier 3DMM-based talking-head systems underperformed because their parameters came from landmark fitting or pretrained reconstruction models not tuned for synthesis, not because 3DMMs are inherently limiting. With the jointly trained ReconNet, the 55 FACS blendshapes of ICT-FaceKit give localized, interpretable control including gaze, and a diffusion model maps audio plus speaking style to mouth blendshapes. The proposed lip-sync pipeline reenacts the face with both a fixed reference and the current frame, then blends the mouth from the fixed-reference reenactment into the chin-corrected current-frame reenactment, which the paper claims avoids the classic mouth-mask artifacts of a preserved chin contour and flicker.

Load-bearing premise

The result depends on the premise that the quality gains come specifically from jointly training the face-reconstruction network with the generator, rather than from the accompanying changes to normalization, SPADE layers, warping parameterization, and losses.

Editorial extensions

If this is right

  • 3DMM-based talking heads remain competitive with latent-embedding and keypoint approaches when the reconstruction network is trained for the synthesis task.
  • FACS blendshapes make mouth-region editing explicit, so lip-sync can change the chin contour instead of preserving the source face shape.
  • The joint-training scheme transfers to 3D-warping architectures by binding 3D keypoints to 3DMM vertices, as the paper's preliminary experiments indicate.
  • Because the 3DMM includes eyeball geometry, gaze control comes without additional modeling, enabling gaze-aware reenactment.
  • The double-reenactment blending strategy provides a template for other region-specific face edits beyond the mouth, such as brow or cheek changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A clean control experiment would freeze the ReconNet after landmark pretraining and retrain only the generator, which would isolate whether joint training, rather than the new normalizations, SPADE layers, warping parameterization, and losses, drives the reported gains.
  • If the joint-training result generalizes, task-specific reconstruction could become a design choice for other single-image avatar tasks such as expression transfer and gaze editing from a single photo.
  • The double-reenactment blending idea could be paired with video diffusion inpainting to handle the strong illumination changes that the paper identifies as a limitation of its blending network.
  • The FACS-based mouth-blendshape prediction could be evaluated with a perceptual lip-reading model beyond the reported LSE metrics, testing whether the reduced flicker trades off against speech readability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes JOLT3D, a framework that jointly trains a 3D face reconstruction network (ReconNet) and a talking head generator based on dense 2D warping, using the ICT-FaceKit 3DMM with FACS-based blendshapes. The authors argue that previous 3DMM-based talking head methods were limited by suboptimal parameter extraction from pretrained reconstruction models or landmark fitting, and that jointly optimizing the parameters for synthesis makes 3DMM a suitable representation. They also propose an audio-to-blendshape diffusion model and a lip-sync pipeline that reenacts the face twice (fixed and current reference) before blending, claiming reduced flicker and chin-contour artifacts. Experiments on HDTF and VoxCeleb2 compare self- and cross-reenactment against Real3D-Portrait, HeadGAN, and FADM, and lip-sync against LatentSync, MuseTalk, and JoyGen.

Significance. The paper's strengths include unusually detailed training recipes, pseudocode for gaze blendshape mapping and 3DMM encoding, evaluation on external benchmarks (FID, LSE-D/LSE-C, CSIM), and an honest limitations section. The joint-training objective for 3DMM parameters is a potentially valuable idea, and the FACS-based blendshape control with eyeball geometry is a practical contribution. If the causal claim about joint training is verified, the paper would challenge the prevailing move away from 3DMMs in talking head synthesis. The lip-sync pipeline's decoupling of the chin contour is also creative. However, the current evidence does not isolate the joint-training mechanism, and the lip-sync advantages are largely qualitative.

major comments (3)
  1. [§5.3, Table 1; §3.3; Appendix B/C] The central claim that joint training of ReconNet and the generator causes the reported improvements is not supported by a controlled comparison. Relative to HeadGAN, the method changes the warping parameterization (Eq. 2), replaces InstanceNorm with GroupNorm, modifies SPADE layer count and channel widths, removes AdaIN audio conditioning (Appendix B), and adds perceptual, feature-matching, locality, ArcFace, flow-consistency, and smoothness losses (Appendix C.2). Table 1 compares the entire package against HeadGAN, FADM, and Real3D-Portrait. Without a frozen-ReconNet baseline that uses the same generator, losses, and training data, the gains cannot be attributed to joint training rather than to any of these other components. Please add an ablation in which ReconNet is pretrained and frozen while all other components and losses are kept identical.
  2. [§4.3, Table 2; Fig. 7] The lip-sync contribution is also missing the key ablation. The two-pass pipeline (I_FF from a fixed reference plus I_CF from the current reference) is claimed to reduce flicker and allow chin-contour modification, but no comparison is reported against single-pass variants (I_CF-only or I_FF-only with the same blending). The chin-contour metric ∆CL is introduced but only reported for the full pipeline; flicker is only demonstrated qualitatively in Fig. 7. Without these ablations and a quantitative artifact metric, the claimed advantages of the double-reenactment mechanism remain untested.
  3. [Table 1 and Table 2] All metric comparisons are reported without error bars or statistical significance tests. In particular, the HDTF evaluation uses only 52 randomly sampled videos, and several entries in Table 1 are close (e.g., CSIM 0.949 vs 0.931 for FADM on HDTF), so it is unclear whether the improvements are significant. For the lip-sync results, the claim that the method reduces visual artifacts would be strengthened by reporting a quantitative flicker metric and by noting that LSE-D/LSE-C are worse than the original video's scores on VoxCeleb2.
minor comments (5)
  1. [§5.1 and Table 1] The dataset names 'A VSpeech' and 'V oxCeleb2' contain spacing typos; the same issue appears in the Table 1 header.
  2. [Table 1 and Table 2] The paired values (self- vs cross-reenactment in Table 1; original vs different audio in Table 2) are not clearly labeled in the column headers, forcing the reader to infer the meaning from the text; explicit annotations would improve readability.
  3. [Appendix A] In the pseudocode for computing F_3dmm, the variable name 'packed_proj_verts_s' is used where the reference vertices are intended; please rename for clarity and consistency with the surrounding code.
  4. [Section F] The first paragraph of the Limitations section contains a typographical artifact: 'the architecture of‘ the HeadGAN framework' uses an unmatched quotation mark.
  5. [Figure 5] The caption lists many blendshape names, but some, such as 'mouthClose+jawOpen', appear to be a composite expression that is not described in the text; please clarify its semantic meaning or separate the components.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation chain: central claims are benchmarked against external metrics; the only self-citation is a non-load-bearing data-filtering SyncNet reference.

full rationale

The paper's causal claims are not derived from its inputs by construction. The central assertion that 3DMM parameters become suitable for talking-head synthesis when the ReconNet is jointly trained with the generator is tested against external benchmarks: LSE-D/LSE-C from Wav2Lip [45], CSIM from ArcFace [7], FID, and standard image-similarity metrics. The audio-to-blendshape model uses an internally trained sync-loss with its own Blendshape Encoder, but the reported lip-sync metrics come from the independent Wav2Lip expert, so the evaluation metric is not identical to a training objective. The identity-preservation loss uses ArcFace cosine similarity and CSIM also uses ArcFace embeddings; this is an aligned objective, but CSIM is computed on held-out videos and is not a fitted prediction. The only self-citation is [42], used in Sec. 5.1 to filter training videos by sync quality; this does not enter the evaluation and is not load-bearing for the main claims. Finally, the absence of a frozen-ReconNet control and the lack of single-pass ablations for the double-reenactment pipeline are missing controls rather than circular reductions; they weaken causal attribution but do not make the reported numbers equivalent to the paper's inputs.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central result depends on a collection of hand-set hyperparameters, domain assumptions about the 3DMM and warping architecture, and the external validity of the chosen metrics. No new physical entities are introduced.

free parameters (6)
  • Data filtering thresholds = quality > 0.4, sync probability >= 0.8, offscreen ratio <= 0.354
    Hand-chosen thresholds in Section 5.1 determine the training distribution and therefore affect all downstream results.
  • Gaze angle limits th_max, tv_max = about 30 degrees
    Chosen by convention from [74] in the supplemental material; defines the coupling between gaze direction and eyeball blendshapes.
  • Joint training loss weights = lambda_locality=20, lambda_arc=1, lambda_pixel=50, etc.
    Manually tuned in SM C.2; the disentanglement and locality behavior, which are central to the lip-sync claims, depend on these weights.
  • Audio-to-blendshape loss weights and guidance scale = lambda_simple=1, lambda_vel=7.5, lambda_sync=1, guidance=1.2
    Hand-tuned in SM C.3; directly shapes the diffusion model that predicts the mouth blendshapes used for lip-sync.
  • Number of denoising steps = T=1000 training, 50 inference
    Discrete choice in SM C.3; affects lip-sync quality and inference speed.
  • Mouth blendshape subset = 35 of 55 ICT-FaceKit blendshapes
    Selection made in Section 4.1; the lip-sync representation and its expressiveness depend on this subset.
assumptions (6)
  • domain assumption ICT-FaceKit FACS blendshapes can represent the mouth and expression variations needed for lip-sync.
    The entire pipeline relies on this; the paper's own Figure 5 shows several blendshapes, such as cheekPuff, respond inadequately, and Section F admits the learned parameters are not 3D consistent.
  • domain assumption 2D feature warping is sufficient for faithful talking head synthesis.
    The HeadGAN-style warping architecture is adopted without proof; limitations with occlusions, teeth, and iris are acknowledged in Section F.
  • domain assumption The external metrics LSE-D/LSE-C, FID, and CSIM are valid proxies for lip-sync quality, image quality, and identity preservation.
    All quantitative conclusions rest on these metrics; no user study is provided.
  • domain assumption The joint training objective converges to a useful optimum on the chosen datasets.
    Stability is demonstrated empirically only through the reported runs; there is no theoretical guarantee, and the paper describes the mask-collapse failure mode that motivated a reparameterization.
  • domain assumption MediaPipe landmark labels and the FaRL face parser provide sufficient supervision.
    ReconNet pretraining uses MediaPipe [19] and mouth masks use FaRL [80]; errors in either tool will propagate into the learned parameters and final composite.
  • domain assumption The authors' own Interpretable Convolutional SyncNet [42] correctly identifies audio-visual synchronization.
    The training set is filtered with this model in Section 5.1; if its notion of sync is biased, the lip-sync training distribution will be biased accordingly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JOLT3D: Joint Learning of Talking Heads and 3DMM Parameters with Application to Lip-Sync." pith.science (2026). https://pith.science/paper/5LIVD3TP

@misc{pith2026250720452,
  author       = {Pith},
  title        = {Pith review of: JOLT3D: Joint Learning of Talking Heads and 3DMM Parameters with Application to Lip-Sync},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5LIVD3TP}},
  note         = {Machine review of arXiv:2507.20452}
}
read the original abstract

In this work, we revisit the effectiveness of 3DMM for talking head synthesis by jointly learning a 3D face reconstruction model and a talking head synthesis model. This enables us to obtain a FACS-based blendshape representation of facial expressions that is optimized for talking head synthesis. This contrasts with previous methods that either fit 3DMM parameters to 2D landmarks or rely on pretrained face reconstruction models. Not only does our approach increase the quality of the generated face, but it also allows us to take advantage of the blendshape representation to modify just the mouth region for the purpose of audio-based lip-sync. To this end, we propose a novel lip-sync pipeline that, unlike previous methods, decouples the original chin contour from the lip-synced chin contour, and reduces flickering near the mouth.

Figures

Figures reproduced from arXiv: 2507.20452 by the authors.

Figure 1
Figure 1. Joint training of ReconNet and Generator. The ReconNet extracts 3DMM param [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The Audio-to-Blendshape model. The Blendshape Encoder is an auxiliary model [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. BlendingNet takes the mouth region of the predicted face, the region outside the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Left: lip-sync pipeline. Right: reenactment with fixed and current references. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Responses to changes in blendshapes. The mouth blendshapes are highlighted. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Results of self-reenactment (above) and cross-reenactment (below). [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: First row: Large mouth mask in LatentSync can cause flickering near the mouth. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: (a) Simplified geometry (b) Original geometry after triangulation. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: (a) Style Encoder. (b) Blendshape Encoder. (c) FlowNet and Generator. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: The effect of locality loss. When locality loss is not imposed, visible flickering [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Importance of properly parameterizing the warping operation can be seen from [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

86 extracted references · 49 canonical work pages

  1. [42]

    Interpretable Convolutional SyncNet

    Sungjoon Park, Jaesub Yun, Donggeon Lee, and Minsik Park. Interpretable convolu- tional syncnet.arXiv preprint arXiv:2409.00971, 2024

    work page Pith review arXiv 2024

  2. [1]

    Facewarehouse: A 3d facial expression database for visual computing.IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013

    Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing.IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013

    2013

  3. [2]

    Hiface: High-fidelity 3d face reconstruction by learning static and dynamic details

    Zenghao Chai, Tianke Zhang, Tianyu He, Xu Tan, Tadas Baltrusaitis, HsiangTao Wu, Runnan Li, Sheng Zhao, Chun Yuan, and Jiang Bian. Hiface: High-fidelity 3d face reconstruction by learning static and dynamic details. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9087–9098, 2023

    2023

  4. [3]

    IQA-PyTorch: Pytorch toolbox for image qual- ity assessment

    Chaofeng Chen and Jiadi Mo. IQA-PyTorch: Pytorch toolbox for image qual- ity assessment. [Online]. Available:https://github.com/chaofengc/ IQA-PyTorch, 2022

    2022

  5. [4]

    Topiq: A top-down approach from semantics to distortions for image quality assessment.IEEE Transactions on Image Processing, 2024

    Chaofeng Chen, Jiadi Mo, Jingwen Hou, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Topiq: A top-down approach from semantics to distortions for image quality assessment.IEEE Transactions on Image Processing, 2024

    2024

  6. [5]

    V oxceleb2: Deep speaker recognition.arXiv preprint arXiv:1806.05622, 2018

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. V oxceleb2: Deep speaker recognition.arXiv preprint arXiv:1806.05622, 2018

    arXiv 2018

  7. [6]

    Emoca: Emotion driven monoc- ular face capture and animation

    Radek Dan ˇeˇcek, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monoc- ular face capture and animation. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 20311–20322, 2022

    2022

  8. [7]

    Arcface: Additive angular margin loss for deep face recognition

    Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 4690–4699, 2019

    2019

Show all 86 references
  1. [8]

    Ac- curate 3d face reconstruction with weakly-supervised learning: From single image to image set

    Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Ac- curate 3d face reconstruction with weakly-supervised learning: From single image to image set. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0–0, ...

    2019

  2. [9]

    Headgan: One-shot neural head synthesis and editing

    Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. InProceedings of the IEEE/CVF Interna- tional conference on Computer Vision, pages 14398–14407, 2021

    2021

  3. [10]

    Free-headgan: Neural talking head synthesis with explicit gaze control

    Michail Christos Doukas, Evangelos Ververas, Viktoriia Sharmanska, and Stefanos Zafeiriou. Free-headgan: Neural talking head synthesis with explicit gaze control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):9743–9756, 2023

    2023

  4. [11]

    Megaportraits: One-shot megapixel neural head avatars

    Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lem- pitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663– 2671, 2022

    2022

  5. [12]

    Emoportraits: Emotion-enhanced multimodal one-shot head avatars

    Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos V ougioukas, Zoe Land- graf, Stavros Petridis, and Maja Pantic. Emoportraits: Emotion-enhanced multimodal one-shot head avatars. InProceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, page...

    2024

  6. [13]

    3d morphable face models—past, present, and future.ACM Transactions on Graphics (ToG), 39(5):1–38, 2020

    Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoe- fer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romd- hani, et al. 3d morphable face models—past, present, and future.ACM Transactions on Graphics (ToG), 39(5):1–38, 2020

    2020

  7. [14]

    Facial action coding system.Environmental Psy- chology & Nonverbal Behavior, 1978

    Paul Ekman and Wallace V Friesen. Facial action coding system.Environmental Psy- chology & Nonverbal Behavior, 1978

    1978

  8. [15]

    Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.arXiv preprint arXiv:1804.03619, 2018

    Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Has- sidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.arXiv preprint arXiv:1804.03619, 2018

    2018 arXiv

  9. [16]

    Learning an animatable detailed 3d face model from in-the-wild images.ACM Transactions on Graphics (ToG), 40(4):1–13, 2021

    Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the-wild images.ACM Transactions on Graphics (ToG), 40(4):1–13, 2021

    2021

  10. [17]

    Surface simplification using quadric error met- rics

    Michael Garland and Paul S Heckbert. Surface simplification using quadric error met- rics. InProceedings of the 24th annual conference on Computer graphics and interac- tive techniques, pages 209–216, 1997

    1997

  11. [18]

    Morphable face models-an open frame- work

    Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter. Morphable face models-an open frame- work. In2018 13th IEEE international conference on automatic face & gesture recog- nition (FG 2018), pages 75–82. IEEE, 2018

    2018

  12. [19]

    Attention mesh: High-fidelity face mesh prediction in real-time

    Ivan Grishchenko, Artsiom Ablavatski, Yury Kartynnik, Karthik Raveendran, and Matthias Grundmann. Attention mesh: High-fidelity face mesh prediction in real-time. arXiv preprint arXiv:2006.10962, 2020

    2006 arXiv

  13. [20]

    Liveportrait: Efficient portrait animation with stitching and retar- geting control.arXiv preprint arXiv:2407.03168, 2024

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait animation with stitching and retar- geting control.arXiv preprint arXiv:2407.03168, 2024. 12PARK ET AL.: JOLT3D

    2024 arXiv

  14. [21]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  15. [22]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    2017

  16. [23]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  17. [24]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  18. [25]

    Image-to-image trans- lation with conditional adversarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image trans- lation with conditional adversarial networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017

    2017

  19. [26]

    Realtalk: Real-time and realistic audio-driven face generation with 3d facial prior-guided identity alignment network

    Xiaozhong Ji, Chuming Lin, Zhonggan Ding, Ying Tai, Junwei Zhu, Xiaobin Hu, Donghao Luo, Yanhao Ge, and Chengjie Wang. Realtalk: Real-time and realistic audio-driven face generation with 3d facial prior-guided identity alignment network. arXiv preprint arXiv:2406.18284, 2024

    2024 arXiv

  20. [27]

    Eamm: One-shot emotional talking face via audio-based emotion-aware motion model

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. Eamm: One-shot emotional talking face via audio-based emotion-aware motion model. InACM SIGGRAPH 2022 conference proceedings, pages 1–10, 2022

    2022

  21. [28]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  22. [29]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  23. [30]

    Photo-realistic single image super-resolution using a generative adversarial network

    Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE c...

    2017

  24. [31]

    Latentsync: Audio conditioned latent diffusion models for lip sync

    Chunyu Li, Chao Zhang, Weikai Xu, Jinghui Xie, Weiguo Feng, Bingyue Peng, and Weiwei Xing. Latentsync: Audio conditioned latent diffusion models for lip sync. arXiv preprint arXiv:2412.09262, 2024

    2024 arXiv

  25. [32]

    Learning formation of physically-based face attributes

    Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xi- ang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al. Learning formation of physically-based face attributes. InProceedings of the IEEE/CVF conference on computer vision and pattern re...

    2020

  26. [33]

    Geometric gan.arXiv preprint arXiv:1705.02894, 2017

    Jae Hyun Lim and Jong Chul Ye. Geometric gan.arXiv preprint arXiv:1705.02894, 2017. PARK ET AL.: JOLT3D13

    2017 arXiv

  27. [34]

    Robust high- resolution video matting with temporal guidance

    Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. Robust high- resolution video matting with temporal guidance. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 238–247, 2022

    2022

  28. [35]

    Anitalker: animate vivid and diverse talking faces through identity-decoupled facial motion encoding

    Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, and Kai Yu. Anitalker: animate vivid and diverse talking faces through identity-decoupled facial motion encoding. InProceedings of the 32nd ACM International Conference on Mul- timedia, pages 6696–6705, 2024

    2024

  29. [36]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  30. [37]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022

    2022

  31. [38]

    Implicit warping for animation with image sets.Advances in Neural Information Processing Systems, 35:22438–22450, 2022

    Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu. Implicit warping for animation with image sets.Advances in Neural Information Processing Systems, 35:22438–22450, 2022

    2022

  32. [39]

    Sidgan: High-resolution dubbed video generation via shift-invariant learning

    Urwa Muaz, Wondong Jang, Rohun Tripathi, Santhosh Mani, Wenbin Ouyang, Ravi Teja Gadde, Baris Gecer, Sergio Elizondo, Reza Madad, and Naveen Nair. Sidgan: High-resolution dubbed video generation via shift-invariant learning. InProceedings of the IEEE/CVF International Conferen...

    2023

  33. [40]

    Said: Speech-driven blendshape facial animation with diffusion.arXiv preprint arXiv:2401.08655, 2023

    Inkyu Park and Jaewoong Cho. Said: Speech-driven blendshape facial animation with diffusion.arXiv preprint arXiv:2401.08655, 2023

    2023 arXiv

  34. [41]

    Synctalk- face: Talking face generation with precise lip-syncing via audio-lip memory

    Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, and Yong Man Ro. Synctalk- face: Talking face generation with precise lip-syncing via audio-lip memory. InPro- ceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2062– 2070, 2022

    2022

  35. [43]

    Semantic image synthesis with spatially-adaptive normalization

    Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 2337–2346, 2019

    2019

  36. [44]

    A 3d face model for pose and illumination invariant face recognition

    Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In2009 sixth IEEE international conference on advanced video and signal based surveillance, pages 296–301. Ieee, 2009

    2009

  37. [45]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. InProceedings of the 28th ACM international conference on multimedia, pages 484–492, 2020

    2020

  38. [46]

    Accelerating 3d deep learning with pytorch3d

    Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv:2007.08501, 2020. 14PARK ET AL.: JOLT3D

    2007 arXiv

  39. [47]

    Pirenderer: Control- lable portrait image generation via semantic neural rendering

    Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Control- lable portrait image generation via semantic neural rendering. InProceedings of the IEEE/CVF international conference on computer vision, pages 13759–13768, 2021

    2021

  40. [48]

    U-net: Convolutional net- works for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional net- works for biomedical image segmentation. InMedical image computing and computer- assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, par...

    2015

  41. [49]

    Palette: Image-to-image diffusion mod- els

    Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Sali- mans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion mod- els. InACM SIGGRAPH 2022 conference proceedings, pages 1–10, 2022

    2022

  42. [50]

    Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

    2016

  43. [51]

    pytorch-fid: FID Score for PyTorch.https://github.com/ mseitzer/pytorch-fid, August 2020

    Maximilian Seitzer. pytorch-fid: FID Score for PyTorch.https://github.com/ mseitzer/pytorch-fid, August 2020. Version 0.3.0

    2020

  44. [52]

    First order motion model for image animation.Advances in neural informa- tion processing systems, 32, 2019

    Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation.Advances in neural informa- tion processing systems, 32, 2019

    2019

  45. [53]

    Very deep convolutional networks for large- scale image recognition.arXiv preprint arXiv:1409.1556, 2014

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large- scale image recognition.arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  46. [54]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational confer- ence on machine learning, pages 2256–2265. pmlr, 2015

    2015

  47. [55]

    Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  48. [56]

    Uniavatar: Taming lifelike audio-driven talking head generation with comprehensive motion and lighting control.arXiv preprint arXiv:2412.19860, 2024

    Wenzhang Sun, Xiang Li, Donglin Di, Zhuding Liang, Qiyuan Zhang, Hao Li, Wei Chen, and Jianxun Cui. Uniavatar: Taming lifelike audio-driven talking head generation with comprehensive motion and lighting control.arXiv preprint arXiv:2412.19860, 2024

    2024 arXiv

  49. [57]

    Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models.ACM Transactions on Graphics (TOG), 43(4): 1–9, 2024

    Zhiyao Sun, Tian Lv, Sheng Ye, Matthieu Lin, Jenny Sheng, Yu-Hui Wen, Minjing Yu, and Yong-jin Liu. Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models.ACM Transactions on Graphics (TOG), 43(4): 1–9, 2024

    2024

  50. [58]

    Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction

    Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. InProceedings of the IEEE international conference on compute...

    2017

  51. [59]

    Instance normalization: The missing ingredient for fast stylization.arXiv preprint arXiv:1607.08022, 2016

    Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization.arXiv preprint arXiv:1607.08022, 2016. PARK ET AL.: JOLT3D15

    2016 arXiv

  52. [60]

    Seeing what you said: Talking face generation guided by a lip reading expert

    Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. Seeing what you said: Talking face generation guided by a lip reading expert. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14653– 14662, 2023

    2023

  53. [61]

    Joygen: Audio-driven 3d depth-aware talking-face video editing.arXiv preprint arXiv:2501.01798, 2025

    Qili Wang, Dajiang Wu, Zihang Xu, Junshi Huang, and Jun Lv. Joygen: Audio-driven 3d depth-aware talking-face video editing.arXiv preprint arXiv:2501.01798, 2025

    2025 arXiv

  54. [62]

    One-shot free-view neural talking- head synthesis for video conferencing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking- head synthesis for video conferencing. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10039–10049, 2021

    2021

  55. [63]

    Im- itating arbitrary talking style for realistic audio-driven talking face synthesis

    Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng. Im- itating arbitrary talking style for realistic audio-driven talking face synthesis. InPro- ceedings of the 29th ACM International Conference on Multimedia, pages 1478–1486, 2021

    2021

  56. [64]

    Group normalization

    Yuxin Wu and Kaiming He. Group normalization. InProceedings of the European conference on computer vision (ECCV), pages 3–19, 2018

    2018

  57. [65]

    Vfhq: A high-quality dataset and benchmark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 657– 666, 2022

    2022

  58. [66]

    High-fidelity generalized emotional talking face generation with multi-modal emotion space learning

    Chao Xu, Junwei Zhu, Jiangning Zhang, Yue Han, Wenqing Chu, Ying Tai, Chengjie Wang, Zhifeng Xie, and Yong Liu. High-fidelity generalized emotional talking face generation with multi-modal emotion space learning. InProceedings of the IEEE/CVF conference on computer vision and ...

    2023

  59. [67]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    2024 arXiv

  60. [68]

    Vasa-1: Lifelike audio-driven talking faces generated in real time.Advances in Neural Information Processing Systems, 37: 660–684, 2025

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time.Advances in Neural Information Processing Systems, 37: 660–684, 2025

    2025

  61. [69]

    Real3d-portrait: One-shot real- istic 3d talking portrait synthesis.arXiv preprint arXiv:2401.08503, 2024

    Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, et al. Real3d-portrait: One-shot real- istic 3d talking portrait synthesis.arXiv preprint arXiv:2401.08503, 2024

    2024 arXiv

  62. [70]

    Dynamic neural textures: Generating talking-face videos with continuously control- lable expressions.arXiv preprint arXiv:2204.06180, 2022

    Zipeng Ye, Zhiyao Sun, Yu-Hui Wen, Yanan Sun, Tian Lv, Ran Yi, and Yong-Jin Liu. Dynamic neural textures: Generating talking-face videos with continuously control- lable expressions.arXiv preprint arXiv:2204.06180, 2022

    2022 arXiv

  63. [71]

    Audio-driven talk- ing face video generation with learning-based personalized head pose.arXiv preprint arXiv:2002.10137, 2020

    Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu. Audio-driven talk- ing face video generation with learning-based personalized head pose.arXiv preprint arXiv:2002.10137, 2020. 16PARK ET AL.: JOLT3D

    2002 arXiv

  64. [72]

    Face animation with an attribute-guided diffusion model

    Bohan Zeng, Xuhui Liu, Sicheng Gao, Boyu Liu, Hong Li, Jianzhuang Liu, and Baochang Zhang. Face animation with an attribute-guided diffusion model. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 628–637, 2023

    2023

  65. [73]

    Facial: Synthesizing dynamic talking face with implicit attribute learning

    Chenxu Zhang, Yifan Zhao, Yifei Huang, Ming Zeng, Saifeng Ni, Madhukar Budagavi, and Xiaohu Guo. Facial: Synthesizing dynamic talking face with implicit attribute learning. InProceedings of the IEEE/CVF international conference on computer vi- sion, pages 3867–3876, 2021

    2021

  66. [74]

    Refa: Real-time egocentric facial animations for virtual reality

    Qiang Zhang, Tong Xiao, Haroun Habeeb, Larissa Laich, Sofien Bouaziz, Patrick Snape, Wenjing Zhang, Matthew Cioffi, Peizhao Zhang, Pavel Pidlypenskyi, et al. Refa: Real-time egocentric facial animations for virtual reality. InProceedings of the IEEE/CVF Conference on Computer ...

    2024

  67. [75]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2023

  68. [76]

    Musetalk: Real-time high quality lip synchro- nization with latent space inpainting.arXiv preprint arXiv:2410.10122, 2024

    Yue Zhang, Minhao Liu, Zhaokang Chen, Bin Wu, Yubin Zeng, Chao Zhan, Yingjie He, Junxin Huang, and Wenjiang Zhou. Musetalk: Real-time high quality lip synchro- nization with latent space inpainting.arXiv preprint arXiv:2410.10122, 2024

    arXiv

  69. [77]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661– 3670, 2021

    2021

  70. [78]

    Dinet: Deformation inpainting network for realistic face visually dubbing on high res- olution video

    Zhimeng Zhang, Zhipeng Hu, Wenjin Deng, Changjie Fan, Tangjie Lv, and Yu Ding. Dinet: Deformation inpainting network for realistic face visually dubbing on high res- olution video. InProceedings of the AAAI conference on artificial intelligence, vol- ume 37, pages 3543–3551, 2023

    2023

  71. [79]

    Avid: Any-length video inpainting with diffusion model

    Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. Avid: Any-length video inpainting with diffusion model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7162–7...

    2024

  72. [80]

    General facial representation learn- ing in a visual-linguistic manner

    Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learn- ing in a visual-linguistic manner. InProceedings of the IEEE/CVF conference on com- puter vision and pattern recogni...

    2022

  73. [81]

    Identity-preserving talking face generation with landmark and appearance priors

    Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, and Guanbin Li. Identity-preserving talking face generation with landmark and appearance priors. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2...

    2023

  74. [82]

    On the continuity of rotation representations in neural networks

    Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. InProceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 5745–5753, 2019

    2019

  75. [83]

    Celebv-hq: A large-scale video facial attributes dataset

    Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale video facial attributes dataset. In European conference on computer vision, pages 650–667. Springer, 2022

    2022

  76. [84]

    Face alignment across large poses: A 3d solution

    Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z Li. Face alignment across large poses: A 3d solution. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 146–155, 2016. 18PARK ET AL.: JOLT3D (a) (b) Figure 8: (a) Simplified geomet...

    2016

  77. [85]

    We also include identity consistency lossL id =∥α 1 −α 2∥2 2, where 1 and 2 indicate identity param- eters extracted from the same video but from different frames

    Note thatl label consists of 68 standard facial landmarks supplemented by 10 additional landmarks for the iris, which are extracted using MediaPipe [19]. We also include identity consistency lossL id =∥α 1 −α 2∥2 2, where 1 and 2 indicate identity param- eters extracted from t...

  78. [86]

    Recall from Sec

    We also include the velocity lossL vel =∥(b 0 1:49 −b 0 0:48)−( ˆb0 1:49 − ˆb0:48)∥2 2, and the smoothness loss Lsmooth =∥ ˆb0 2:49 −2 ˆb0 1:48 + ˆb0 0:47∥2 2.To help train the Audio Encoder, we also include the sync-loss as follows. Recall from Sec. 4.1 that the Mel spectrogr...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.