Pith. sign in

REVIEW 4 major objections 5 minor 47 references

MoDA: Multi-modal Diffusion Architecture for Talking Head Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MoDA claims talking-head generation improves by predicting motion in a compact space where audio, emotion, and identity interact rather than entangling them with pixels.

desk verdict Competent two-stage talking-head system with a genuinely interesting coarse-to-fine fusion schedule, but the HDTF evaluation probably leaks identity and the quantitative claims are overstated. read the letter →

arxiv 2507.03256 v3 pith:VVDREWAL submitted 2025-07-04 cs.GR cs.CV

Xinyang Li , Gen Li , Zhihui Lin , Yichen Qian , GongXin Yao , Weinan Jia , Aowen Wang , Weihua Chen
show 1 more author
Fan Wang
This is my paper
classification cs.GRcs.CV
keywords talkingheadgenerationaudio-drivenanimationdiffusiontransformerrectifiedflowmulti-modalfusionmotion-spacelipsynchronizationreal-timeinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MoDA claims that talking-head generation improves if the model predicts motion in a compact, identity-agnostic parameter space instead of generating pixels or VAE latents directly, and if the diffusion model lets noisy motion, audio, emotion, and identity interact through joint attention. The paper argues that existing single-stage diffusion methods suffer from slow inference and visual artifacts because appearance and motion are entangled in the latent space, while two-stage methods lose expressiveness because they rely on shallow cross-attention between a mixed condition and motion. MoDA replaces that with a coarse-to-fine fusion that first separates modalities, then merges audio with emotion and identity, then unifies everything. On HDTF, CelebV-HQ, and an in-the-wild set, it reports lower FVD/FID, better lip-sync, and higher motion smoothness than six recent baselines, and reaches real-time inference with 10 denoising steps. The reason to care: if right, expressive and controllable digital humans can run interactively without costly pixel-space diffusion.

What carries the argument

The load-bearing object is the joint parameter space: a 70-dimensional vector $(R, \delta, t, S)$ from a frozen face-reenactment motion extractor, representing pose, expression deformation, translation, and scale in a form independent of who is speaking. MoDA trains a diffusion transformer to denoise this vector with rectified flow, so the generative target is motion, not pixels; the compactness of the space is what allows 10-step ODE integration and real-time inference. The other mechanism is the coarse-to-fine fusion schedule: four modality-specific streams, then a shared stream merging audio with emotion and identity, then one unified stream, with joint attention and aligned rotational positional encoding across all tokens at every stage.

What would settle it

Take a held-out set of clips with large head yaw, roughly more than 45 degrees, or with heavy headwear, run MoDA and its two-stage baselines under identical conditions, and compare FVD and lip-sync distance; if MoDA's advantage disappears on this subset, the reported gains come from the frozen renderer rather than from the multi-modal motion predictor.

Watch

Extended reading notes

Core claim

At its core, the paper establishes that inter-modal inconsistency—the stylistic gap between speech audio, a person's identity, and portrait dynamics—is the bottleneck for audio-driven talking heads, and that a motion-space diffusion model can remove it. MoDA freezes a face-reenactment extractor that represents the source and target as 70-dimensional motion parameters (pose, expression deformation, translation, scale), and trains a rectified-flow diffusion transformer to predict those parameters from audio, conditioned on emotion, identity, and previous motion. The transformer uses joint attention across all modalities with aligned rotational positional encoding, and a three-stage coarse-to-fine fusion that progressively shares weights across streams, cutting parameters from 904M to 370M while improving quality. The paper reports that this design outperforms EchoMimic, JoyHallo, Hallo, Hallo2, Ditto, and JoyVASA in most quality and synchronization metrics, with its best FVD on HDTF at 174.6 versus 207.0 for the closest compared method, and an optional lip-sync expert that trades a little diversity for stronger synchronization.

Load-bearing premise

MoDA assumes the frozen face-reenactment pipeline's compact motion representation is expressive enough for every target identity, pose, and audio condition, and the paper itself notes it degrades on large pose changes and complex headwear.

Editorial extensions

If this is right

  • If MoDA is right, motion-space generation becomes the preferred two-stage design: the expensive renderer is frozen, and a small diffusion network trained on a 70-dimensional vector handles all expression and head dynamics.
  • The reported 10-step rectified-flow denoising with real-time inference implies audio-driven avatars can run in conversational loops on a single GPU without specialized acceleration.
  • Treating emotion and identity as catalysts that adapt audio features means the same audio can drive different identities with different facial behavior, which matters for dubbing, virtual characters, and assistive communication.
  • The coarse-to-fine fusion result, cutting parameters from 904M to 370M while improving metrics, implies progressive weight sharing is both a quality and an efficiency win for multi-modal transformers in low-dimensional spaces.
  • The optional lip-sync expert shows a user-selectable trade-off: stronger synchronization at a small cost in diversity, or richer motion without the expert.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The replace-image ablation suggests cross-attention's failure is structural rather than a matter of capacity, meaning audio must be conditioned on identity before attention; the same progressive-fusion recipe may transfer to other audio-conditioned animation tasks such as gesture, gaze, or full-body motion.
  • Because the motion space is inherited from a frozen reenactment pipeline, MoDA's ceiling on large poses and occluding accessories is set by that pipeline, and a test on out-of-distribution headwear would likely show the reported advantage shrinking even if the motion predictor is flawless.
  • The evaluation relies on FVD/FID and lip-sync metrics, so a human perceptual study on the in-the-wild set would be a natural next check, since score gains do not always track perceived naturalness.
  • The same coarse-to-fine schedule could be applied to latent video diffusion models; if inter-modal inconsistency is the true bottleneck, the strategy should help there too, not only in 70-dimensional motion spaces.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MoDA, a two-stage diffusion-based framework for one-shot talking head generation. It maps audio and auxiliary conditions (identity, emotion, initial motion) into LivePortrait's low-dimensional motion space using a flow-matching DiT with joint attention, and then renders video with a frozen LivePortrait decoder. The architecture is organized as a coarse-to-fine fusion of modality-specific streams, with an optional Adaptive Lip-motion sync Expert (ALSE) loss. Experiments on HDTF, CelebV-HQ, and a 20-clip in-the-wild set compare against six baselines using FVD, FID, F-SIM, lip-sync metrics, smoothness, and RTF. The paper reports better FVD/FID and competitive real-time factors, and includes ablations of the attention mechanism and fusion strategy.

Significance. If the reported results hold, MoDA is a useful practical contribution: it shows that a compact disentangled motion space plus rectified-flow DiT with progressive multimodal fusion can produce competitive talking-head quality at near-real-time speed, and the ablation study supports the architectural choices. The paper also explicitly acknowledges the limitations of the frozen LivePortrait backbone for large poses and complex headwear, which is a fair statement of scope. However, the evaluation currently has load-bearing gaps: the HDTF test set is not identity-disjoint from training, no confidence intervals or significance tests are reported on the 50-clip test sets, the velocity loss in Eq. (5) is underspecified, and no code is released. These issues limit how strongly the central 'arbitrary identities' and state-of-the-art claims can be accepted.

major comments (4)
  1. [Section 4.1.1, Table 1] The central claim of generalization to arbitrary identities is not supported by the current evaluation protocol. MoDA is trained on HDTF (Section 4.1.1), and the HDTF test set consists of 50 randomly sampled HDTF clips with no stated identity-disjoint split. If identities overlap between training and test, the reported HDTF margins (FVD 174.6 vs. 207.9; FID 28.2 vs. 29.6) may partly reflect memorization of identity-specific dynamics rather than generalization to unseen people. In addition, no per-clip variance, bootstrap confidence intervals, or significance tests are reported for either 50-clip public test set, so the statistical reliability of the margins over the second-best methods is unknown. Please provide an identity-disjoint evaluation protocol, report the degree of identity overlap, and add confidence intervals or significance tests for the headline metrics.
  2. [Section 3.2.4, Eq. (5)] The velocity loss Lvel is underspecified to the point of non-reproducibility. The text defines m as the output of v_theta but then states that "m'', m'' denote the first-order and second-order derivatives of m", repeated the same symbol for both orders. The quantities x' and x'' are never defined, and it is unclear whether derivatives are taken with respect to the flow time t, the discrete denoising step, or the temporal sequence index. Because Lvel is part of the final loss in Eq. (6) and is not isolated in the ablations, please define the exact computation of each term, including how derivatives are estimated on discrete motion sequences.
  3. [Section 4.2.1, Table 1] The prose summary of the CelebV-HQ results is internally inconsistent with Table 1. The text states that MoDA "outperforms the six baseline methods in all metrics except Sync-D and Smo," but the Ours row reports Sync-C = 5.878, which is lower than JoyHallo's 6.041; only Ours (sync) exceeds JoyHallo on Sync-C. Please correct the claim and clarify which configuration (Ours or Ours (sync)) is being described in each sentence.
  4. [Section 3.2.4, Eq. (7)] The ALSE loss is a named contribution and is used in the Ours (sync) variant reported in Tables 1 and 2, but its architecture, pretraining data, and the audio-motion alignment it consumes are deferred entirely to the supplementary material. The gating rule in Eq. (7) with threshold tau = 0.4 is not justified, and it is not stated how the gate is evaluated during training. Please include the ALSE formulation and training details in the main text or make the supplementary material available for review.
minor comments (5)
  1. [Section 1, second paragraph] "methods like Hallo2 [5] remain two major limitations persist" is ungrammatical and should be rewritten.
  2. [Section 3.3.1, Eq. (8)] The notation C|c = ∅ is ambiguous. Please specify that it denotes replacing one condition with its dropout token while keeping the other conditions, and state the dropout schedule used for each condition at inference.
  3. [Section 4.3.1, Figure 7] The "Replace audio" and "Replace image" experiments are described only in prose with a single visual example and no quantitative results. Adding metrics for these variants would strengthen the causal claim that cross-attention fails to adapt audio features to different identity conditions.
  4. [Throughout] The paper repeatedly refers to the supplementary material for ALSE details, long-video comparisons, and implementation details. Several of these references, especially the ALSE formulation, should be expanded in the main text if the supplementary is not part of the submitted manuscript.
  5. [Table 2] The RTF comparison would be more interpretable if the hardware and software configuration were reported; Ditto's TensorRT acceleration is mentioned as the reason for its lower RTF, so a like-for-like setup is needed to support the efficiency claim.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild circularity: HDTF test clips are sampled from the same dataset MoDA trains on, with no identity-disjoint split stated, so the headline FVD/FID 'arbitrary identities' numbers partly re-measure the training distribution; the architecture claims themselves rest on clean ablations and external baselines.

  1. fitted input called prediction [Section 4.1.1 (Dataset and Metrics); Table 1 (HDTF block)]
    "MoDA is primarily trained on three publicly available datasets: HDTF [46], CelebV-Text [43], and MEAD [36]. For evaluation, we conducted experiments on three distinct test sets. The first two are derived from public datasets, CelebV-HQ [47] and HDTF, each consisting of 50 randomly sampled clips ranging from 3 to 10 seconds in length."

    The HDTF evaluation set is drawn from the same dataset used for training, with no identity-disjoint split stated. Training on HDTF while reporting FVD/FID on '50 randomly sampled clips' from HDTF lets the motion predictor exploit memorized identity-specific dynamics, so the headline HDTF gains (FVD 174.6 vs. 207.9; FID 28.2 vs. 29.6, Table 1) partially re-measure the training distribution rather than the central 'arbitrary identities' generalization claim. This is a dataset-level instance of the fitted-input-called-prediction pattern: the test quantity is statistically favored by construction of the split, not by the method's generality. The relative ranking vs.

full rationale

The paper's derivation chain is otherwise self-contained. The rectified-flow objective (Eqs. 1-4) is the standard flow-matching regression target x = ϵ - z0 defined from the same (z0, ϵ) used to construct zt; this is the ordinary conditional denoising setup, not a result smuggled in through its own definition. Motion targets come from the frozen, externally published LivePortrait extractor [11]; audio from wav2vec [24]; emotion labels from HSEmotion [23]; and the MMDiT/rectified-flow machinery from SD3 [8] and rectified-flow papers [17, 20] — none authored by the present team, and no load-bearing self-citation is present. The optional ALSE loss (Eqs. 6-7) and its threshold τ = 0.4 are internal training choices; the reported Sync-C/Sync-D metrics come from the external SyncNet [4], so the synchronization claim is not measured by its own training signal. Ablations (Table 3) are controlled (CABA, w/o C2F, MAF vs. full model) and support the coarse-to-fine fusion claim. Section 4.4 candidly states the frozen disentanglement stage can 'degrade on large pose changes or when the subject wears complex head accessories' and names alternatives (MegaPortraits, Face Vid2Vid), ruling out an imported-uniqueness pattern. The single genuine circularity concern is dataset-level: MoDA trains on HDTF and the HDTF test set is '50 randomly sampled clips' from the same dataset with no stated identity-disjoint split (the same risk applies to CelebV-HQ via CelebV-Text). The headline HDTF gains may therefore partly reflect memorized identity-specific dynamics rather than generalization to arbitrary identities, and the 20-clip in-the-wild set is too small and heterogeneous to fully compensate. Relative comparisons against baselines evaluated on the same clips remain informative, and the central architectural contributions are independently supported, so this is a mild blemish (2/10), not a circular derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim rests mainly on the frozen LivePortrait renderer, pretrained audio and emotion encoders, and a small set of hand-chosen inference constants. The architectural contribution is a weight-sharing schedule across modality streams, not a new mathematical principle.

free parameters (3)
  • Per-condition CFG scales lambda_c = not reported
    Equation (8) defines a separate classifier-free guidance scale for each condition, but the main text gives no values. The reported lip-sync and diversity trade-offs depend on these unlisted scales.
  • ALSE gate threshold tau = 0.4
    Equation (7) activates the optional sync loss only when LALSE is below 0.4. No sensitivity analysis is provided, so the reported Sync gains may be threshold-dependent.
  • Denoising steps N = 10
    Section 3.3 states that 10-step ODE integration is used instead of 50. The efficiency claim rests on this inference choice, and no ablation with varying N is shown in the main text.
assumptions (5)
  • domain assumption LivePortrait's R70 joint parameter space is identity-agnostic and expressive enough to represent the full range of target motions.
    Section 3.2.1 adopts the frozen LivePortrait motion and rendering backbone. If this space cannot represent large poses or complex accessories, MoDA inherits the failure, as acknowledged in Section 4.4.
  • ad hoc to paper Audio, emotion, and identity cues are semantically compatible enough to be merged into one stream before fusion with motion.
    The coarse-to-fine ordering is justified in Section 4.3.2 by the argument that speech audio naturally carries emotion and identity signals. This is a design hypothesis rather than an independently verified property.
  • domain assumption wav2vec audio features and the HSEMotion emotion labels contain the information needed for lip-sync and expression control.
    Section 3.2.1 treats these pretrained encoders as reliable conditioning sources. Errors or biases in them propagate directly into the generated motion.
  • standard math Rectified flow training converges to a model of the conditional motion distribution.
    The paper relies on known flow-matching results from [17,20] without re-deriving them. The stated objective in Eq (1) and (4) is standard, and no convergence proof is needed for the empirical claim.
  • domain assumption FVD, FID, F-SIM, Sync-C, Sync-D, and Smo are valid proxies for realism, identity preservation, and lip-sync quality.
    All conclusions rest on these metrics computed over 50-clip test sets in Section 4.1.1. No human evaluation is reported, so the metrics are the only evidence for the quality claims.
invented entities (1)
  • Adaptive Lip-motion sync Expert (ALSE)
    purpose: An optional pretrained auxiliary network whose loss LALSE is added to enforce lip-sync after a threshold is met.
    The paper defers all ALSE details to the supplementary materials and releases no checkpoint, so its contribution to the reported Sync-C and Sync-D gains cannot be checked outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoDA: Multi-modal Diffusion Architecture for Talking Head Generation." pith.science (2026). https://pith.science/paper/VVDREWAL

@misc{pith2026250703256,
  author       = {Pith},
  title        = {Pith review of: MoDA: Multi-modal Diffusion Architecture for Talking Head Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VVDREWAL}},
  note         = {Machine review of arXiv:2507.03256}
}
read the original abstract

Talking head generation with arbitrary identities and speech audio remains a crucial problem in the realm of the virtual metaverse. Recently, diffusion models have become a popular generative technique in this field with their strong generation capabilities. However, several challenges remain for diffusion-based methods: 1) inefficient inference and visual artifacts caused by the implicit latent space of Variational Auto-Encoders (VAE), which complicates the diffusion process; 2) a lack of authentic facial expressions and head movements due to inadequate multi-modal information fusion. In this paper, MoDA handles these challenges by: 1) defining a joint parameter space that bridges motion generation and neural rendering, and leveraging flow matching to simplify diffusion learning; 2) introducing a multi-modal diffusion architecture to model the interaction among noisy motion, audio, and auxiliary conditions, enhancing overall facial expressiveness. In addition, a coarse-to-fine fusion strategy is employed to progressively integrate different modalities, ensuring effective feature fusion. Experimental results demonstrate that MoDA improves video diversity, realism, and efficiency, making it suitable for real-world applications. Project Page: https://lixinyyang.github.io/MoDA.github.io/

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 28 canonical work pages

  1. [1]

    A morphable model for the synthesis of 3d faces

    Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Pro- ceedings of the 26th annual conference on Com- puter graphics and interactive techniques - SIG- GRAPH ’99, Jan 1999

    work page 1999

  2. [2]

    Joyvasa: Portrait and animal image animation with diffusion-based audio-driven facial dynam- ics and head motion generation.arXiv preprint arXiv:2411.09209, 2024

    Xuyang Cao, Guoxin Wang, Sheng Shi, Jun Zhao, Yang Yao, Jintao Fei, and Minyu Gao. Joyvasa: Portrait and animal image animation with diffusion-based audio-driven facial dynam- ics and head motion generation.arXiv preprint arXiv:2411.09209, 2024

    arXiv 2024

  3. [3]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136, 2024

    arXiv 2024

  4. [4]

    Out of time: automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. InCom- puter Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017

    work page 2016

  5. [5]

    Hallo2: Long-duration and high-resolution audio-driven portrait image animation

    Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duration and high-resolution audio-driven portrait image animation. arXiv preprint arXiv:2410.07718, 2024

    arXiv 2024

  6. [6]

    Hallo3: Highly dynamic and realistic portrait image anima- tion with diffusion transformer networks.arXiv preprint arXiv:2412.00733, 2024

    Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image anima- tion with diffusion transformer networks.arXiv preprint arXiv:2412.00733, 2024

    arXiv 2024

  7. [7]

    Megaportraits: One- shot megapixel neural head avatars

    Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempit- sky, and Egor Zakharov. Megaportraits: One- shot megapixel neural head avatars. InProceed- ings of the 30th ACM International Conference on Multimedia, pages 2663–2671, 2022

    work page 2022

  8. [8]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning, 2024

    2024

Show all 47 references
  1. [9]

    Gen- erative adversarial nets.Advances in neural in- formation processing systems, 27, 2014

    Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, AaronCourville, andYoshuaBengio. Gen- erative adversarial nets.Advances in neural in- formation processing systems, 27, 2014

    2014

  2. [10]

    Stylesync: High-fidelity generalized and personalized lip sync in style-based genera- tor

    Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Stylesync: High-fidelity generalized and personalized lip sync in style-based genera- tor. InProceedings of the IEEE/CVF Conference on Co...

    2023

  3. [11]

    Liveportrait: Efficient portrait ani- mation with stitching and retargeting control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Liveportrait: Efficient portrait ani- mation with stitching and retargeting control. arXiv preprint arXiv:2407.03168, 2024

    2024 arXiv

  4. [12]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chan- paisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceed- ings of the IEEE/CVF Conference on Computer Vision and ...

    2024

  5. [13]

    Loopy: Taming audio-driven portrait avatar with long- term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long- term motion dependency. In The Thirteenth International Conference on Learning Represen- tations, 2024

    2024

  6. [14]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  7. [15]

    Ditto: Motion- space diffusion for controllable realtime talking head synthesis.arXiv preprint arXiv:2411.19509, 2024

    Tianqi Li, Ruobing Zheng, Minghui Yang, Jing- dong Chen, and Ming Yang. Ditto: Motion- space diffusion for controllable realtime talking head synthesis.arXiv preprint arXiv:2411.19509, 2024

    2024

  8. [16]

    Learning a model of facial shape and expression from 4d scans.ACM Trans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans.ACM Trans. Graph., 36(6):194–1, 2017

    2017

  9. [17]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022. 10

    2022 arXiv

  10. [18]

    Sora: A review on background, technology, limitations, and opportunities of large vision models.arXiv preprint arXiv:2402.17177, 2024

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models.arXiv preprint arXiv:2402.17177, 2024

    2024 arXiv

  11. [19]

    Scalable diffu- sion models with transformers

    William Peebles and Saining Xie. Scalable diffu- sion models with transformers. Dec 2022

    2022

  12. [20]

    Multisample flow matching: Straightening flows with minibatch couplings

    Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky TQ Chen. Multisample flow matching: Straightening flows with minibatch couplings. arXiv preprint arXiv:2304.14772, 2023

    2023 arXiv

  13. [21]

    Learning indi- vidual speaking styles for accurate lip to speech synthesis

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. Learning indi- vidual speaking styles for accurate lip to speech synthesis. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recogni- tion, pages 13796–13805, 2020

    2020

  14. [22]

    A lip sync ex- pert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync ex- pert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484–492, 2020

    2020

  15. [23]

    Savchenko

    Andrey V. Savchenko. Hsemotion: High-speed emotion recognition library.Software Impacts, page 100433, Dec 2022

    2022

  16. [24]

    wav2vec: Unsuper- vised pre-training for speech recognition

    Steffen Schneider, Alexei Baevski, Ronan Col- lobert, and Michael Auli. wav2vec: Unsuper- vised pre-training for speech recognition. In Interspeech 2019, Sep 2019

    2019

  17. [25]

    pytorch-fid: Fid score for pytorch, 2020

    Maximilian Seitzer. pytorch-fid: Fid score for pytorch, 2020

    2020

  18. [26]

    Joyhallo: Digital human model for man- darin

    Sheng Shi, Xuyang Cao, Jun Zhao, and Guoxin Wang. Joyhallo: Digital human model for man- darin. arXiv preprint arXiv:2409.13268, 2024

    2024 arXiv

  19. [27]

    First order motion model for image animation

    Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing sys- tems, 32, 2019

    2019

  20. [28]

    Weiss, Niru Ma- heswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, EricA. Weiss, Niru Ma- heswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium ther- modynamics. arXiv: Learning,arXiv: Learning, Mar 2015

    2015

  21. [29]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Er- mon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  22. [30]

    Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior

    Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Daiheng Gao, Liefeng Bo, and Xun Cao. Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.01841, 2023

    2023 arXiv

  23. [31]

    Synthesizing obama: learning lip sync from audio.ACM Transactions on Graphics (ToG), 36(4):1–13, 2017

    Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learning lip sync from audio.ACM Transactions on Graphics (ToG), 36(4):1–13, 2017

    2017

  24. [32]

    Emo2: End-effector guided audio- driven avatar video generation.arXiv preprint arXiv:2501.10687, 2025

    Linrui Tian, Siqi Hu, Qi Wang, Bang Zhang, and Liefeng Bo. Emo2: End-effector guided audio- driven avatar video generation.arXiv preprint arXiv:2501.10687, 2025

    2025 arXiv

  25. [33]

    Emo: Emote portrait alive generating ex- pressive portrait videos with audio2video diffu- sion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating ex- pressive portrait videos with audio2video diffu- sion model under weak conditions. InEuropean Conference on Computer Vision, pages 244–260. Springer, 2024

    2024

  26. [34]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michal- ski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019

    2019

  27. [35]

    Namboodiri, and Rajesh M Hegde

    Munender Varshney, Ravindra Yadav, Vinay P. Namboodiri, and Rajesh M Hegde. Learning speaker-specificlip-to-speechgeneration. In 2022 26th International Conference on Pattern Recog- nition (ICPR), pages 491–498, 2022

    2022

  28. [36]

    Mead: A large-scale audio-visual dataset for emotional talking-face generation

    Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuo- qian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. InEuropean conference on computer vision, pages 700–717. Springer, 2020

    2020

  29. [37]

    One-shot free-view neural talking- head synthesis for video conferencing

    Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking- head synthesis for video conferencing. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021

    2021

  30. [38]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 11

    2024 arXiv

  31. [39]

    Vasa-1: Lifelike audio-driven talking faces generated in real time

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiao- long Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. Advances in Neural Information Pro- cessing Systems, 37:660–684, 2024

    2024

  32. [40]

    Cogvideox: Text-to-video diffu- sion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffu- sion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  33. [41]

    Geneface: Gener- alized and high-fidelity audio-driven 3d talking face synthesis.arXiv preprint arXiv:2301.13430, 2023

    Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. Geneface: Gener- alized and high-fidelity audio-driven 3d talking face synthesis.arXiv preprint arXiv:2301.13430, 2023

    2023 arXiv

  34. [42]

    Audio-driven talking face video generation with dynamic convolution ker- nels

    Zipeng Ye, Mengfei Xia, Ran Yi, Juyong Zhang, Yu-Kun Lai, Xuwei Huang, Guoxin Zhang, and Yong-jin Liu. Audio-driven talking face video generation with dynamic convolution ker- nels. IEEE Transactions on Multimedia, 25:2033– 2046, 2022

    2022

  35. [43]

    Celebv-text: A large-scale facial text-video dataset

    Jianhui Yu, Hao Zhu, Liming Jiang, Chen Change Loy, Weidong Cai, and Wayne Wu. Celebv-text: A large-scale facial text-video dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14805–14814, 2023

    2023

  36. [44]

    Fast bi- layer neural synthesis of one-shot realistic head avatars

    Egor Zakharov, Aleksei Ivakhnenko, Aliaksan- dra Shysheya, and Victor Lempitsky. Fast bi- layer neural synthesis of one-shot realistic head avatars. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part XII 16, pages 524–

    2020

  37. [45]

    Sadtalker: Learning realistic 3d mo- tion coefficients for stylized audio-driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d mo- tion coefficients for stylized audio-driven single image talking face animation. InProceedings of the IEEE/CVF conference on computer vision and p...

    2023

  38. [46]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. InProceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 3661–3670, 2021

    2021

  39. [47]

    Celebv-hq: A large-scale video facial attributes dataset

    Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale video facial attributes dataset. In European conference on computer vision, pages 650–667. Springer, 2022. 12

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.