Pith. sign in

REVIEW 5 major objections 6 minor 18 cited by

Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pretrained video diffusion transformer can be adapted to animate a single portrait into long, dynamic, audio-driven video with preserved identity, even from profile angles.

desk verdict Hallo3's real advance is switching portrait animation to a DiT backbone, but its HDTF evaluation lacks a stated train/test split and needs hardening. read the letter →

arxiv 2412.00733 v4 pith:DF6UGGHY submitted 2024-12-01 cs.CV cs.GRcs.LG

Jiahao Cui , Hui Li , Yun Zhan , Hanlin Shang , Kaihui Cheng , Yuqi Ma , Shan Mu , Hang Zhou
show 2 more authors
Jingdong Wang Siyu Zhu
This is my paper · ORCID
classification cs.CVcs.GRcs.LG
keywords portraitimageanimationaudio-driventalkingheaddiffusiontransformeridentityreferencenetworkvideoextrapolationlipsynchronizationdynamicscenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a pretrained diffusion-transformer video generator can be repurposed for portrait image animation, and that this substitution of backbone solves problems that U-Net-based talking-head systems leave unsolved: non-frontal head poses, moving foreground objects, and lively backgrounds. The authors build on the CogVideoX transformer and add three components: an identity reference network that encodes the source portrait through a causal 3D VAE and 42 transformer layers, audio conditioning via cross-attention on wav2vec features, and a motion-frame mechanism that lets the model continue a video beyond its 49-frame training window. On established benchmarks and a newly collected wild dataset, the method reports better FID and FVD than SadTalker, DreamTalk, AniPortrait, and Hallo, while producing visibly more motion in both the subject and the background. The significance is that the task shifts from a static 'talking head in front of a camera' to full portrait animation in dynamic scenes, closer to real film and social-media use cases.

What carries the argument

The load-bearing mechanism is the identity reference network: the source portrait is passed through the same causal 3D VAE that encodes the video latents, and its output is run through 42 transformer layers; at each denoising layer, the vision tokens of those layers are concatenated into the self-attention of the denoising network. Because both branches share VAE weights and layer counts, the identity tokens stay in the same semantic and scale space as the video tokens, which is what preserves facial identity over long sequences. The other two mechanisms are audio cross-attention, where frame-aligned wav2vec embeddings act as keys and values for the denoising latents, and motion frames, where the last two frames of a completed clip are zero-padded and concatenated with noise to seed the next clip.

What would settle it

Recompute the HDTF FID and FVD with a strictly disjoint train/test split, or compare identity embeddings of test frames against training frames; overlap large enough to explain the reported gap would falsify the claimed generalization.

Watch

Extended reading notes

Core claim

The central discovery is that a pretrained DiT-based video diffusion model can serve as the backbone for portrait animation, provided identity, audio, and temporal continuity are re-inserted on top of it. Previous identity-conditioning methods designed for U-Net architectures do not transfer, so the paper introduces an identity reference network in which the same causal 3D VAE and the same 42-layer transformer stack encode the reference image and the denoising latents, letting the model inject appearance tokens directly into self-attention. Speech audio is most effectively added with cross-attention rather than adaptive normalization or self-attention, and repeated use of the last generated frames as motion frames enables long-term video extrapolation. The authors report that this combination gives improved FID and FVD on HDTF and Celeb-V, the strongest lip synchronization on their wild dataset, and stable identity over long video clips.

Load-bearing premise

The HDTF numbers in Table 1 assume that the identities and clips used for testing were not used in training, but the paper lists HDTF as a training source and never documents the train/test split, so any overlap would shrink the reported generalization gains.

Editorial extensions

If this is right

  • Portrait animation no longer requires a frontal, centered reference: the DiT backbone's generalization carries through to profile, overhead, and low-angle views.
  • Dynamic foreground objects and backgrounds can be animated together with the person, so a still image plus audio can produce a scene-level video rather than a face-only animation.
  • Long videos, beyond the 49-frame training window, can be generated by chaining clips through motion frames, with identity maintained across the chain.
  • Because the backbone is a pretrained video transformer, future improvements in video diffusion models could be inherited by portrait animation systems without redesigning the identity and audio layers from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the DiT backbone's generalization is the real source of the gain, then swapping the backbone for a larger or more recent video transformer should improve portrait animation roughly for free, without new identity or audio machinery.
  • Beyond the paper: the motion-frame extrapolation scheme could be reused as a testbed for long-video consistency in other single-image-to-video tasks, such as object animation, where identity drift is less severe and fewer identity constraints might be needed.
  • Beyond the paper: the ablations suggest a trade-off between lip-sync accuracy and motion dynamics; a testable extension would measure whether training on a mix of static talking-head data and dynamic scene data can recover the Sync-C gap without losing background motion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper adapts CogVideoX, a pretrained DiT-based video diffusion model, to the task of portrait image animation. The main contributions are an identity reference network built from a causal 3D VAE and stacked transformer layers, a cross-attention audio conditioning mechanism based on wav2vec features, and a motion-frame extrapolation strategy for long video generation. The method is evaluated on HDTF, CelebV, and a newly collected 34-case wild dataset, with ablations of audio conditioning, identity conditioning, and motion-frame count. The paper claims to be the first application of a pretrained DiT video backbone to portrait animation and reports improved FID/FVD over U-Net baselines while acknowledging a lip-synchronization tradeoff.

Significance. The architectural direction is timely and plausible: replacing the U-Net backbone with a pretrained video diffusion transformer is a natural step for portrait animation, and the proposed identity reference network is a reasonable way to inject appearance information into a DiT denoiser. The ablations are internally consistent and support the main design choices. If the quantitative claims were properly substantiated, the paper would be a useful system contribution and a reference point for follow-up DiT-based talking-head work. However, the strongest evidence for the central 'substantial improvements' claim rests on comparisons with no uncertainty estimates, on a benchmark that also appears in the training set without a stated split, and on a wild dataset whose metrics are not defined. These issues are fixable and do not invalidate the architectural proposal, but they currently prevent acceptance of the paper's comparative claims.

major comments (5)
  1. [Section 3.5 and Section 4.2, Table 1] Section 3.5 states that the training data includes 6 hours of high-quality HDTF data, yet Section 4.2 reports benchmark results on HDTF without stating any train/test split or identity exclusion. Since Table 1 is the principal quantitative evidence for the claim of strong generalization, please specify the exact evaluation protocol: how many HDTF clips are used, whether evaluation identities and clips are disjoint from the training set, and how the same held-out split is applied to all baselines. Without this information, the reported FID/FVD numbers may reflect training-set familiarity rather than generalization, and the comparison to baselines trained on different data is potentially unfair.
  2. [Section 4.2, Tables 1 and 2] The headline numbers are reported without error bars, repeated runs, or significance tests. On HDTF, the FID gap between the proposed method and Hallo is only 20.359 vs. 20.545, and the Sync-C and Sync-D scores are actually worse than several baselines. Please report results over multiple seeds or sampling runs with confidence intervals, and state whether the observed FID/FVD differences are statistically meaningful. The phrase 'substantial improvements over prior methods' is not supported by the current table alone.
  3. [Section 4.2, Table 3] The wild-dataset evaluation is based on only 34 cases, and the metrics SubjectDynamic, BackgroundDynamic, SubjectFVD, and BackgroundFVD are introduced in the table without definitions or implementation details. Please define each metric, describe the evaluation pipeline, and report per-case variance or confidence intervals. Given the small sample size, a human perceptual study or at least a larger evaluation set would substantially strengthen the claim that the method handles dynamic foregrounds and backgrounds better than prior work.
  4. [Section 4.3, Tables 4 to 6] The ablation study does not state which dataset or split is used. The FID/FVD values in Tables 4 and 5 are around 23.458/242.602, which differ noticeably from the HDTF numbers in Table 1 (20.359/160.838). Please clarify whether the ablations are on a different subset of HDTF, on CelebV, or on the wild dataset, and explain why the numbers differ. Also, Table 5 reports a 'Subject consistency' metric that is not defined; please provide its computation.
  5. [Section 3.3, 'Temporal Motion Frames'] The motion-frame extrapolation description is ambiguous: the text says 'We apply zero padding to the subsequent (l - n) frames and concatenate them with l frames of Gaussian noise.' It is unclear whether the motion-frame latent is concatenated with the noise latent along the channel dimension, the frame dimension, or both. Please give the exact tensor shapes and the masking/denoising procedure. Additionally, Table 6 shows that n = 1 gives the best Sync-C and Sync-D, while the inference description uses n = 2; the choice of n = 2 deserves an explicit justification, such as a tradeoff with temporal consistency.
minor comments (6)
  1. [Section 1 and Figure 6] The dataset name is spelled 'HTDF' in Section 1 and in the Figure 6 caption; it should be 'HDTF'.
  2. [Section 2, Related Work] The formatting 'V ASA-1' and 'V ASA' appears with an extra space; please correct to 'VASA-1' and 'VASA'.
  3. [Section 3.5, Data Sources] The text says HDTF contains 8 hours of raw video but the final training data includes 6 hours of high-quality HDTF. Please clarify whether the 6 hours is the subset remaining after filtering, and if so, state the filtering retention rate explicitly.
  4. [Section 3.5 and Section 4.1] Section 3.5 says videos are cropped to a 3:2 aspect ratio, while Section 4.1 states the training resolution is 480 x 720, which is 2:3. One of these is likely a typo; please reconcile the two statements.
  5. [Section 3.1, Baseline Network] The description of conditioning says 'expert adaptive layer normalization' is used, but the term 'expert' is not defined. Since CogVideoX's architecture is not widely known to all readers, a brief explanation of the expert transformer blocks would improve reproducibility.
  6. [Figure 4] The caption uses the abbreviation 'FE' (face encoder) without defining it in the caption or in the main text at that location.

Circularity Check

1 steps flagged · score 4.0 of 10

HDTF is both training data (Sec. 3.5) and headline benchmark (Sec. 4.2) with no stated split, so the Table 1 generalization claims are not independent; otherwise the derivation is self-contained.

  1. fitted input called prediction [Section 3.5 (Data Statistics) and Section 4.2 (Comparison on HDTF/Celeb-V, Table 1)]
    "Finally, our training data contains about 134 hours training data, including 6 hours of high-quality data from HDTF dataset, 72 hours YouTube videos, and 56 hours movie videos."

    The model is trained on HDTF (6 hours are in the 134-hour training set) and the headline generalization numbers in Table 1 are reported on HDTF; Section 4.2 never states a train/test identity or clip split. Absent an explicit held-out split, the FID/FVD improvements on HDTF are measurements on the fitted dataset, not independent predictions of generalization, so this portion of the evidence reduces to the training input by construction. The CelebV and wild-dataset results are separate and give the central claim independent support, but the strongest quantitative table is affected.

full rationale

Score 4 reflects a partial empirical circularity, not a derivation-level one. Section 3.5 explicitly includes 6 hours of HDTF in the training corpus; Section 4.2 then presents HDTF FID/FVD (Table 1) as evidence of strong generalization capabilities without stating a train/test identity or clip split, so the paper's most load-bearing numbers are not demonstrably out-of-sample. No equation in the paper defines a target in terms of an input: the identity reference network, audio cross-attention, and motion-frame extrapolation are implemented as stated and validated by ablations; CogVideoX, wav2vec2, and T5 are external pretrained models; and the Hallo/Hallo2 self-citations are used as baselines rather than as justifications. Because the wild-dataset and CelebV evaluations are separate and the methodological contribution is described concretely, the circularity is localized rather than total.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or ontological entities. Its central claim rests on pretrained models (CogVideoX, wav2vec2, T5, InsightFace), on hyperparameters chosen without sensitivity analysis, and on the untested assumption that the HDTF evaluation identities were not seen in training.

free parameters (4)
  • Number of motion frames n = 2
    Ablation in Table 6 shows n=2 was chosen for the best FID/FVD balance; n=1 gives better lip sync, so the choice involves a metric tradeoff.
  • Training steps per stage = 20000
    Both training phases run for 20,000 steps; the paper does not report a convergence analysis or a sweep over this value.
  • Learning rate = 1e-5
    The learning rate is given in Section 4.1 without sensitivity analysis.
  • Audio drop probability and motion mask probability = 0.05 and 0.25
    These regularization probabilities are chosen during training without reported ablation.
assumptions (5)
  • domain assumption CogVideoX is an adequate pretrained DiT backbone for portrait animation.
    The method builds directly on CogVideoX as the baseline in Section 3.1; if this backbone cannot represent facial detail, the whole pipeline inherits that limitation.
  • domain assumption wav2vec2 audio embeddings capture phonetics and prosody sufficient for lip synchronization.
    Section 3.2 concatenates the final twelve wav2vec2 layers and uses them as the sole audio condition.
  • domain assumption The causal 3D VAE latent space preserves identity details from a reference image.
    Section 3.3 injects reference latent codes from the same 3D VAE into the denoising network, assuming semantic and scale consistency.
  • domain assumption FID, FVD, Sync-C, Sync-D, and the dynamic degree metrics are valid proxies for video quality and lip sync.
    Section 4.2 uses these as the only quantitative measures; the paper does not report user studies or perceptual validation.
  • domain assumption The HDTF evaluation set does not overlap with the HDTF training set.
    Section 3.5 lists 6 hours of HDTF in the training data, while Section 4.2 reports results on HDTF; no train/test split is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer." pith.science (2026). https://pith.science/paper/DF6UGGHY

@misc{pith2026241200733,
  author       = {Pith},
  title        = {Pith review of: Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DF6UGGHY}},
  note         = {Machine review of arXiv:2412.00733}
}
read the original abstract

Existing methodologies for animating portrait images face significant challenges, particularly in handling non-frontal perspectives, rendering dynamic objects around the portrait, and generating immersive, realistic backgrounds. In this paper, we introduce the first application of a pretrained transformer-based video generative model that demonstrates strong generalization capabilities and generates highly dynamic, realistic videos for portrait animation, effectively addressing these challenges. The adoption of a new video backbone model makes previous U-Net-based methods for identity maintenance, audio conditioning, and video extrapolation inapplicable. To address this limitation, we design an identity reference network consisting of a causal 3D VAE combined with a stacked series of transformer layers, ensuring consistent facial identity across video sequences. Additionally, we investigate various speech audio conditioning and motion frame mechanisms to enable the generation of continuous video driven by speech audio. Our method is validated through experiments on benchmark and newly proposed wild datasets, demonstrating substantial improvements over prior methods in generating realistic portraits characterized by diverse orientations within dynamic and immersive scenes. Further visualizations and the source code are available at: https://fudan-generative-vision.github.io/hallo3/.

Figures

Figures reproduced from arXiv: 2412.00733 by the authors.

Figure 1
Figure 1. Demonstration of the proposed approach. Given a reference image, an audio sequence, and a textual prompt, the method generates [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Different strategies of audio con￾ditioning. (a) self-attention; (b) adaptive norm; (c) cross-attention. (a) (b) (c) (d) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. llustration of the dataset, including the flow of data processing, data distribution across different metric, and the visualization of [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (4 more)
Figure 6
Figure 6. Figure 6: Qualitative comparison on the HTDF (left) and CelebV (right) data-set. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Complex facial identity with dynamic accessories subjects and different pose orientation. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Complex scenes with dynamic foreground or background and various head poses. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 10
Figure 10. Figure 10: Qualitative comparison of different strategies for iden [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    SyncBreaker jointly attacks image and audio streams with Multi-Interval Sampling and Cross-Attention Fooling to degrade speech-driven talking head generation more than single-modality baselines.

  2. Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MultiTalk is the first framework to generate multi-person conversational videos from multi-stream audio, using Label Rotary Position Embedding to bind each voice to the correct person.

  3. HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characters

    cs.CV 2025-05 conditional novelty 6.0 of 10

    HunyuanVideo-Avatar is an audio-driven video generator that enables emotion-controllable and multi-character animation by injecting character images, routing audio via face masks, and transferring emotion from referen...

  4. OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A mixed-condition training recipe, combining text, audio, and pose signals, lets a single diffusion transformer animate photos into realistic talking or singing videos at unprecedented data scale.

  5. InfinityHuman: Towards Long-Term Audio-Driven Human

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A coarse-to-fine audio-driven animation framework that uses pose-guided refinement and hand-specific reward learning to generate long, identity-stable talking videos.

  6. MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generation

    cs.CV 2025-08 reject novelty 5.0 of 10

    A new autoregressive video-generation framework for interactive digital humans, with a 64x compression autoencoder and a diffusion renderer, claims real-time multimodal control but is only demonstrated for audio input.

  7. InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Sparse-frame dubbing with adjacent-chunk keyframe sampling lets a streaming audio-video model produce full-body motion synchronized to new audio while preserving identity and camera motion.

  8. EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis

    cs.CV 2025-08 conditional novelty 5.0 of 10

    EDTalk++ disentangles talking-head video into four orthogonal motion banks (mouth, pose, eyes, expression) and drives them from either video or audio inputs.

  9. MoDA: Multi-modal Diffusion Architecture for Talking Head Generation

    cs.GR 2025-07 conditional novelty 5.0 of 10

    MoDA uses flow matching in a compact face-motion space with a progressively fused multi-modal transformer to generate expressive, lip-synced talking-head videos from a single image and audio.

  10. FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Cases

    cs.CV 2025-07 conditional novelty 5.0 of 10

    FixTalk adds two modules to a real-time GAN talking-head model, decoupling identity from motion to stop identity leakage while using a memory to recover details and reduce artifacts.

  11. Seeing Voices: Generating A-Roll Video from Audio with Mirage

    cs.CV 2025-06 reject novelty 5.0 of 10

    Mirage generates photorealistic A-roll videos of people speaking directly from audio, using only joint self-attention over audio, text, and video tokens.

  12. TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models

    cs.SD 2025-06 conditional novelty 5.0 of 10

    A pretrained 14B image-to-video diffusion model is adapted into a two-step autoregressive audio-driven avatar generator that can stream indefinitely in real time.

  13. SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    An audio-conditioned video diffusion transformer that animates portraits from image, video, text, and audio inputs with a sliding-window fusion for long videos.

  14. FaceEditTalker: Controllable Talking Head Generation with Facial Attribute Editing

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A single framework can edit predefined facial attributes in audio-synchronized talking head videos while preserving identity and lip-sync quality.

  15. Preview WB-DH: Towards Whole Body Digital Human Bench for the Generation of Whole-body Talking Avatar Videos

    cs.CV 2025-08 reject novelty 4.0 of 10

    The paper previews a claimed 2M-clip multimodal benchmark for whole-body talking avatar video generation, with standard metrics and an initial evaluation of eight open-source models.

  16. JWB-DH-V1: Benchmark for Joint Whole-Body Talking Avatar and Speech Generation Version 1

    cs.CV 2025-07 reject novelty 4.0 of 10

    A paper announcing a large-scale whole-body talking avatar benchmark and evaluation protocol, but with insufficient details to verify the dataset or the joint audio-video evaluation.

  17. Livatar-1: Real-Time Talking Heads Generation with Tailored Flow Matching

    cs.CV 2025-07 reject novelty 4.0 of 10

    Livatar claims the best lip-sync score in its comparison table and real-time throughput, but the preprint does not describe the model or release any code, data, or weights.

  18. LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    Using consistency distillation, INT8 quantization, and pipeline parallelism, the LLIA system generates portrait video from audio at 78 FPS, with 140 ms initial latency.

Reference graph

Works this paper leans on

40 extracted references · 15 canonical work pages · cited by 18 Pith papers

  1. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural infor- mation processing systems, 33:12449–12460, 2020. 6

    2020

  2. [2]

    Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models

    Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024. 2

    arXiv 2024

  3. [3]

    Face recognition based on fitting a 3d morphable model

    V olker Blanz and Thomas Vetter. Face recognition based on fitting a 3d morphable model. IEEE Transactions on pattern analysis and machine intelligence , 25(9):1063–1074, 2003. 1

    work page 2003

  4. [4]

    pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe

    Herv ´e Bredin. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In Proc. INTER- SPEECH 2023, 2023. 6

    work page 2023

  5. [5]

    J. S. Chung and A. Zisserman. Out of time: automated lip sync in the wild. In Workshop on Multi-view Lip-reading, ACCV, 2016. 6

    2016

  6. [6]

    Vlogger: Multimodal diffusion for embodied avatar synthe- sis

    Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthe- sis. arXiv preprint arXiv:2403.08764, 2024. 1

    arXiv 2024

  7. [7]

    Hallo2: Long-duration and high-resolution audio-driven portrait im- age animation

    Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duration and high-resolution audio-driven portrait im- age animation. arXiv preprint arXiv:2410.07718, 2024. 1, 3

    arXiv

  8. [8]

    High-fidelity and freely controllable talking head video generation

    Yue Gao, Yuan Zhou, Jinglu Wang, Xiao Li, Xiang Ming, and Yan Lu. High-fidelity and freely controllable talking head video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5609–5619, 2023. 1

    work page 2023

Show all 40 references
  1. [9]

    Livepor- trait: Efficient portrait animation with stitching and retarget- ing control

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control. arXiv preprint arXiv:2407.03168, 2024. 1, 3

    2024 arXiv

  2. [10]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 3

    2023 arXiv

  3. [11]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.02634, 2024. 3

    2024 arXiv

  4. [12]

    Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos

    Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos. In Proc. arXiv:2410.11831, 2024. 6

    2024 arXiv

  5. [13]

    Co- tracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker: It is better to track together. In Proc. ECCV, 2024. 6

    2024

  6. [14]

    Anitalker: Animate vivid and di- verse talking faces through identity-decoupled facial motion encoding

    Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, and Kai Yu. Anitalker: Animate vivid and di- verse talking faces through identity-decoupled facial motion encoding. arXiv preprint arXiv:2405.03121, 2024. 1

    2024 arXiv

  7. [15]

    Sora: A review on background, technology, limitations, and opportunities of large vision models

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024. 2

    2024 arXiv

  8. [16]

    Dreamtalk: When expressive talking head generation meets diffusion probabilistic models

    Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767, 2023. 2, 6

    2023 arXiv

  9. [17]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 4195–4205, 2023. 3

    2023

  10. [18]

    Powerset multi-class cross entropy loss for neural speaker diarization

    Alexis Plaquet and Herv ´e Bredin. Powerset multi-class cross entropy loss for neural speaker diarization. In Proc. INTER- SPEECH 2023, 2023. 6

    2023

  11. [19]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  12. [20]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia (ACM MM), pages 484–492, 2020. 2

    2020

  13. [21]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. 3

    2023

  14. [22]

    Facial geometric de- tail recovery via implicit representation

    Xingyu Ren, Alexandros Lattas, Baris Gecer, Jiankang Deng, Chao Ma, and Xiaokang Yang. Facial geometric de- tail recovery via implicit representation. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), 2023. 6

    2023

  15. [23]

    Pirenderer: Controllable portrait image generation via semantic neural rendering

    Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13759–13768, 2021. 1

    2021

  16. [24]

    wav2vec: Unsupervised pre-training for speech recognition

    Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862, 2019. 3

    1904 arXiv

  17. [25]

    Difftalk: Crafting diffusion models for generalized audio-driven portraits animation

    Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 1982–1991, 2023. 2

    1982

  18. [26]

    First order motion model for im- age animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for im- age animation. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019. 1

    2019

  19. [27]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  20. [28]

    Vividtalk: One-shot audio-driven talk- ing head generation based on 3d hybrid prior

    Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Daiheng Gao, Liefeng Bo, and Xun Cao. Vividtalk: One-shot audio-driven talk- ing head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.01841, 2023. 2

    2023 arXiv

  21. [29]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485, 2024. 1, 2

    2024 arXiv

  22. [30]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024. 3

    2024

  23. [31]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024. 1, 3, 6

    2024 arXiv

  24. [32]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation, 2024. 1, 3, 6

    2024

  25. [33]

    Vasa-1: Lifelike audio-driven talking faces generated in real time

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404.10667 ,

    arXiv

  26. [34]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 2, 3

    2024 arXiv

  27. [35]

    Fast bi-layer neural synthesis of one- shot realistic head avatars

    Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of one- shot realistic head avatars. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16 , pages 524–540....

    2020

  28. [36]

    Metaportrait: Identity-preserving talking head gener- ation with fast personalized adaptation

    Bowen Zhang, Chenyang Qi, Pan Zhang, Bo Zhang, Hsiang- Tao Wu, Dong Chen, Qifeng Chen, Yong Wang, and Fang Wen. Metaportrait: Identity-preserving talking head gener- ation with fast personalized adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2023

  29. [37]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  30. [38]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  31. [39]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021. 6

    2021

  32. [40]

    Magicvideo: Efficient video generation with latent diffusion models

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 3

    2022 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.