Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SkyReels-Audio generates and edits talking portraits from audio plus text, image, or video inputs, claiming better lip-sync, identity consistency, and natural motion than current open baselines.

desk verdict Solid system paper whose own Table 3 undercuts the 'superior performance' claim in the abstract. read the letter →

arxiv 2506.00830 v1 pith:JZYGYM3X submitted 2025-06-01 cs.CV

Zhengcong Fei , Hao Jiang , Di Qiu , Baoxuan Gu , Youqiang Zhang , Jiahua Wang , Jialin Bai , Debang Li
show 3 more authors
Mingyuan Fan Guibin Chen Yahui Zhou
This is my paper
classification cs.CV
keywords audio-conditionedtalkingportraitsvideodiffusiontransformerslip-syncinfinite-lengthgenerationbidirectionallatentfusionmultimodalconditioningportraitanimationandeditingflowmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SkyReels-Audio is a unified system for generating and editing talking portraits, built on a pretrained video diffusion transformer and driven by audio together with text, image, or video controls. The paper claims it produces better lip-sync accuracy, identity consistency, and natural facial dynamics than open-source baselines such as Hallo3 and FantacyTalking, on both the public HDTF dataset and the authors' internal benchmark. The model treats audio as the main controller of articulation, uses a hybrid curriculum to jointly learn image animation and video editing, and adds a facial-mask loss plus audio-guided classifier-free guidance to keep lips aligned without losing global consistency. A tuning-free sliding-window scheme, Bidirectional Latent Fusion, stitches overlapping denoised latents so output can be extended indefinitely. If the claims hold, one model could replace separate pipelines for animating a single photo, editing an existing video, and producing long-form portraits from speech, singing, or rap in multiple languages.

What carries the argument

The load-bearing mechanism is the audio-conditioned DiT: a pretrained 3D video VAE compresses frames, a pretrained video diffusion transformer backbone supplies the spatio-temporal prior, and a speech encoder turns the waveform into audio tokens that are fused in dedicated cross-attention layers with 1D rotary position embeddings. Training uses a hybrid curriculum where image-animation and video-editing tasks are distinguished by a binary temporal mask, and the flow-matching loss is reweighted by facial masks: $L_{\mathrm{joint}} = w_1 \, V_m^{\mathrm{downsample}} \odot L_{\mathrm{mse}} + w_2 \, (1 - V_m^{\mathrm{downsample}}) \odot L_{\mathrm{msk}}$, with a probabilistic gate that focuses on lip regions. At inference, audio-guided classifier-free guidance with separate, time-scheduled audio and text scales sharpens synchronization, and Bidirectional Latent Fusion overlaps adjacent windows during denoising and linearly interpolates the overlapping latents to remove stitching artifacts in long videos.

What would settle it

Run SkyReels-Audio and the open baselines on a held-out set of speakers and languages not seen in its training data, compute Sync-C and Sync-D with bootstrapped confidence intervals, and add a forced-choice human listening study; if the HDTF Sync-C edge (6.06 versus 5.75) collapses within noise or reverses on fast speech or non-verbal vocalizations, the superiority claim fails. Alternatively, a long-video continuity test in which BLF is replaced by no-overlap or unidirectional fusion should show visible seam jumps if BLF is doing the claimed work.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single audio-conditioned video diffusion transformer can outperform specialized talking-head systems on lip-sync accuracy, identity preservation, and motion realism, while also handling arbitrary-length generation. The reported evidence: on HDTF, SkyReels-Audio reaches Sync-C 6.06 and Sync-D 9.12, ahead of Hallo3 (5.75/10.12) and FantacyTalking (5.36/11.68), with lower FID and FVD; on the internal benchmark it beats open baselines though it trails the closed OmniHuman-1 on some metrics; and a user study on the internal set rates it above LatentSync on audio-visual consistency and visual quality. The design that carries the claim is audio injection through cross-attention with 1D rotary position embeddings, joint image and video training under a mask-weighted flow-matching loss, inference-time audio classifier-free guidance, and Bidirectional Latent Fusion for stitching long videos.

Load-bearing premise

The conclusion that SkyReels-Audio is superior rests on the assumption that SyncNet-based Sync-C and Sync-D scores on the HDTF clips and the authors' 50-image, 30-audio internal benchmark are faithful, unbiased measures of real lip-sync quality, and that the 1K-hour filtered training set does not silently favor the model's biases.

Editorial extensions

If this is right

  • A single pretrained DiT can serve both image-animation and video-editing use cases, with audio as the primary driver of lip motion and auxiliary modalities controlling expression, gesture, and background.
  • With Audio CFG at 4.5 the model trades a small amount of visual quality for noticeably better sync metrics; lowering CFG to 1 or 3 degrades Sync-C and Sync-D on the internal benchmark.
  • Removing audio RoPE drops Sync-C from 6.75 to 5.58 on the internal benchmark, indicating that positional alignment of audio tokens within cross-attention is a necessary component.
  • TeaCache and USP together cut the time for 80 frames from about 23.6 minutes to under 1 minute on 8 A800 GPUs, making the approach practical for long-form and interactive use.
  • Bidirectional Latent Fusion produces smooth window transitions without extra training, so infinite-length portrait videos can be stitched online during denoising.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the Sync-C deltas over open baselines are small (for instance 6.06 versus 5.75 on HDTF), so the practical win may lie more in long-form stability and multimodal control than in raw lip-sync; that split is testable by comparing single-shot clips separately from multi-window videos.
  • Beyond the paper: BLF is stated as tuning-free and window-agnostic, which suggests it could be lifted into other windowed DiT video models; a drop-in study would establish whether the fusion weights transfer.
  • Beyond the paper: the data funnel from 10K to 1K hours, with manual annotation and a sub-5% bad-case target, positions data curation as a first-class contribution; an ablation that trains the same architecture on unfiltered versus filtered data would quantify how much of the gain comes from the pipeline rather than the model.
  • Beyond the paper: the benchmark mixes anime, sculpture, and photorealistic portraits but trains only on real videos, so the generalization claim could be stress-tested by adding out-of-domain portrait types the pipeline never saw.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes SkyReels-Audio, a unified audio-conditioned talking-portrait generation and editing framework built on the SkyReels-V2 video diffusion transformer. Audio is encoded by Whisper and injected via cross-attention layers with 1D RoPE; a hybrid curriculum trains image animation and video editing jointly with a facial-region-weighted loss; inference uses audio/text classifier-free guidance, a Bidirectional Latent Fusion (BLF) sliding-window mechanism for infinite-length generation, and TeaCache/USP acceleration. The authors describe a multi-stage data pipeline that filters 10K hours of video down to 1K hours of audio-video-text triplets. The evaluation reports quantitative metrics on HDTF and an internal benchmark, a 20-participant user study, and ablations of Audio CFG and Audio RoPE.

Significance. If the central claim of superior lip-sync accuracy, identity consistency, and facial dynamics were supported, the paper would be a useful systems contribution to audio-conditioned portrait animation, particularly for its multimodal conditioning and long-video inference strategy. The manuscript has genuine strengths: it provides a detailed data curation pipeline, an algorithmic specification of BLF, and ablations that isolate the effects of Audio CFG and RoPE. It also gives concrete inference-acceleration measurements. However, the empirical evidence as presented is not sufficient to support the headline claims: the internal benchmark is not released, Tables 1-3 report single numbers without variance or significance testing, and the paper's own Table 3 contradicts the abstract's 'superior performance' claim. The central contribution is plausible and the framework is well-motivated, but the claims need substantial revision and the evaluation needs strengthening.

major comments (4)
  1. [Abstract and Section 3.2, Table 3] The claim of 'superior performance' is contradicted by Table 3 on the paper's own internal benchmark. OmniHuman-1 beats SkyReels-Audio on all four metrics (Sync-C 7.50 vs. 6.75, Sync-D 7.47 vs. 8.32, IQA 4.66 vs. 4.42, ASE 3.19 vs. 2.91), and MagicInfinite also exceeds SkyReels-Audio on IQA (4.56 vs. 4.42) and ASE (3.00 vs. 2.91). The Section 3.2 phrasing 'comparable with close-source models' is accurate, but the Abstract and parts of the text claim consistent superiority. This is an internal inconsistency that directly affects the paper's central claim and must be resolved by either revising the claims or providing additional evidence that supports superiority.
  2. [Algorithm 1, line 10] The BLF update in Algorithm 1, line 10, is written as z_{t-1}^{[s,s+o]} = w * z_{t-1}^{[s,s+o]} + (w-1) * z_{t-1}^{[e_prev-o,e_prev]}. Since w is defined as a vector from 0 to 1, (w-1) is negative or zero, so this is not a convex combination and at w=0 the previous-window latent is subtracted rather than fused. This appears to be a sign error; the intended formula is likely (1-w) * z_prev. As written, the algorithm does not implement the 'weighted fusion' described in the text, so the correctness of the BLF mechanism is not established.
  3. [Section 3.1 and Tables 1-3] All quantitative results are reported as single numbers without error bars, standard deviations, multiple seeds, or significance tests. For differences like HDTF Sync-C 6.06 vs. 5.75 in Table 1, it is impossible to determine whether the improvement is meaningful. This is especially problematic because the internal benchmark is neither released nor described in sufficient detail to be reproduced, and the Table 3 baseline numbers for close-source models are not independently verifiable. The evaluation should include variance estimates or, at minimum, clearly specify the evaluation protocol and release the benchmark materials.
  4. [Section 3.2 and Table 2] The user study has only 20 participants and the table reports no inter-rater agreement or significance analysis. The heading 'HDTF User Study(Internal)' is also confusing because HDTF is a public dataset while the study is described as being on the Internal dataset. Since the user study is used as evidence for perceptual superiority, the small sample size and unclear protocol substantially weaken this evidence. Please provide the study protocol, participant instructions, and statistical treatment.
minor comments (5)
  1. [Throughout] The manuscript contains multiple typographical errors that should be corrected, including 'denosing network' (Section 2.2), 'iamge' (Section 2.4), 'seto' (Section 3.1), 'seuqences' (Section 2.4), 'out contributions' (Section 1), and 'claim naturally consistent video results' (Figure 8 caption).
  2. [Equation (2)] The loss notation is unclear: L_msk appears in Equation (2) but is not formally defined; presumably it is the flow-matching loss restricted to non-masked areas, but the distinction between L_mse and L_msk should be stated explicitly.
  3. [Section 2.4] The post-processing step for color unification is mentioned but never described; since the authors identify a color-darkening problem during long-video inference, they should specify the color-unification procedure for reproducibility.
  4. [Table 3] The 'Audio CFG=1' row is effectively a no-audio-CFG ablation but is not labeled as such; also, the row order in the table should be clarified so that the default configuration is identified unambiguously.
  5. [Section 3.2] The description of the internal benchmark says it comprises '50+ audio-driven scenarios' and also '50+ portrait images, 30 audio segments, 20 textual prompts'; the exact composition of the benchmark should be stated more precisely to allow interpretation of the results.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the empirical comparisons, especially on external HDTF, are not forced by the paper's equations or by self-citation; evaluation caveats exist but are not derivation-level circularity.

full rationale

SkyReels-Audio is an empirical systems paper, not a derivation. Section 2 specifies a flow-matching training objective (Eq. 1), a masked joint loss (Eq. 2), and inference-time guidance (Eqs. 3-4); none of the reported lip-sync or identity metrics is algebraically implied by these equations, and no parameter is renamed as a prediction. The central quantitative claims are direct model-vs-model comparisons. Table 1 is measured on HDTF, an external public dataset stated to be excluded from training, which gives independent content to the Sync-C/Sync-D/IQA/ASE results. Self-citations to SkyReels-V2 [6] and SkyCaptioner-V1 [6] are transparent reuse of the authors' prior backbone and captioner; they are load-bearing in an engineering sense but are not used as a forced premise or uniqueness argument, so they do not make the present claim circular. Two evaluation caveats should be recorded: (i) the data pipeline filters training clips using a SyncNet sync-confidence score [11], and evaluation also uses SyncNet-derived Sync-C/Sync-D, so the internal-benchmark lip-sync metric is not fully independent of the training-set selection criterion; and (ii) the default audio-CFG value 4.5 is chosen by inspecting metrics on the same internal dataset reported in Table 3, so those numbers are partly tuned rather than purely predicted. These are methodological weaknesses that may overstate the internal-benchmark margin, and the Table 3 reversal against OmniHuman-1 is an internal inconsistency in the 'superior performance' claim; however, they do not exhibit an equation-level reduction of a prediction to its input. Accordingly, no significant circularity is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on pretrained components (SkyReels-V2, Whisper, DWPose) and on several hand-chosen hyperparameters that are not fully reported. No new physical or conceptual entities are introduced.

free parameters (6)
  • audio_cfg_scale = 4.5
    Chosen by balancing Sync-C/Sync-D against visual quality in Table 3 (values 1, 3, 4.5 tested).
  • text_cfg_scale = not reported
    Appears in Eq. 3 for text guidance but no value or ablation is given.
  • teacache_alpha = 0.3
    Threshold for caching denoising steps, used in Table 4.
  • mask_threshold_p_mask = not reported
    Probabilistic gating threshold in the facial mask loss L_face; no value or ablation.
  • loss_weights_w1_w2 = not reported
    Weights in Eq. 2 for masked and non-masked regions; no values reported.
  • blf_overlap_length = not reported
    Overlap size 'o' in Algorithm 1; no value or sensitivity analysis given.
assumptions (5)
  • domain assumption The pretrained SkyReels-V2 backbone and its causal 3D VAE provide a sufficient representation space for portrait video generation.
    Adopted as core backbone in Section 2.1 without independent verification in this paper.
  • domain assumption Whisper audio embeddings capture the articulatory features needed for lip sync.
    Used as the sole audio encoder in Section 2.2; no comparison to other audio representations.
  • domain assumption DWPose landmark detection is accurate enough to generate mouth masks that improve training.
    Used in Section 2.3 to create masks; errors in pose detection would propagate to the loss.
  • domain assumption SyncNet-based Sync-C and Sync-D are valid and sufficient proxies for lip-sync quality.
    Used as the primary evaluation metrics in Section 3.1 without reporting error bars or human correlation.
  • standard math Flow matching with linear interpolation (Eq. 1) is a valid training objective for this model.
    Standard framework from Lipman et al., cited in Section 2.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers." pith.science (2026). https://pith.science/paper/JZYGYM3X

@misc{pith2026250600830,
  author       = {Pith},
  title        = {Pith review of: SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JZYGYM3X}},
  note         = {Machine review of arXiv:2506.00830}
}
read the original abstract

The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos. Built upon pretrained video diffusion transformers, our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs. We employ a hybrid curriculum learning strategy to progressively align audio with facial motion, enabling fine-grained multimodal control over long video sequences. To enhance local facial coherence, we introduce a facial mask loss and an audio-guided classifier-free guidance mechanism. A sliding-window denoising approach further fuses latent representations across temporal segments, ensuring visual fidelity and temporal consistency across extended durations and diverse identities. More importantly, we construct a dedicated data pipeline for curating high-quality triplets consisting of synchronized audio, video, and textual descriptions. Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior performance in lip-sync accuracy, identity consistency, and realistic facial dynamics, particularly under complex and challenging conditions.

Figures

Figures reproduced from arXiv: 2506.00830 by the authors.

Figure 1
Figure 1. Given a portrait image, text, or video along with audio input, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of SkyReels-Audio. Whisper encodes resampled audio and fuse video tokens with cross-attention layers. Image and video controls are joint featured with VAE before combine with input noise to provide a video identity and environment priors. is employed to extract latent visual features, which are concatenated with a noise tensor along the channel dimension. To distinguish between static and dynamic inputs, we… view at source ↗
Figure 3
Figure 3. Illustration of BLF. BLF is a tuning-free overlapping sliding window strategy, performing bidirectional fusion of the latents within adjacent windows in the same denoising step. using DWPose and generate lower-face bounding boxes. These frames are then masked based on the BBox to produce the corresponding video and mask sequences Vs = (I0, Imask 1 , ..., Imask n ), Vm = (Mones, Mmouth, ..., Mmouth). Subsequently, th… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Data Processing Pipeline. This is a data funnel to filter high-quality video data. To enhance the quality of model training, we constructed a data processing pipeline as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparisons with other audio-driven talking portrait methods. Our approach produce more accurate lip synchronization with naturalness. acceleration. Notably, despite being trained solely on real-world portrait videos, SkyReels-Audio demonstrates strong gene…
Figure 6
Figure 6. Figure 6: Qualitative comparisons with SoTA lip-sync methods. We can see that SkyReels-Audio create better audio lip alignment compared with the baseline. consistency (AV Consist.) and visual quality. A total of 20 participants rated each aspect on a scale for 0 to 2 (from bad t…
Figure 7
Figure 7. Figure 7: Qualitative comparison of different latent fusion methods. BLF generates long videos with [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: More generated results of SkyReels-Audio. Our approach can handle reference images of different objectives, sizes, and styles, and claim naturally consistent video results. [3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do￾minik …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Sparse-frame dubbing with adjacent-chunk keyframe sampling lets a streaming audio-video model produce full-body motion synchronized to new audio while preserving identity and camera motion.

Reference graph

Works this paper leans on

83 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://www.prnewswire.com/news-releases/deepbrain-ai-delivers-ai-avatar-to empower-people-with-disabilities-302026965.html

    DeepBrain AI. https://www.prnewswire.com/news-releases/deepbrain-ai-delivers-ai-avatar-to empower-people-with-disabilities-302026965.html. InOnline, 2024

    2024

  2. [2]

    Efficient 3d implicit head avatar with mesh-anchored hash table blendshapes

    Ziqian Bai, Feitong Tan, Sean Fanello, Rohit Pandey, Mingsong Dou, Shichen Liu, Ping Tan, and Yinda Zhang. Efficient 3d implicit head avatar with mesh-anchored hash table blendshapes. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1975–1984, 2024. 12 Figure 8:More generated results of SkyReels-Audio.Our approach...

    1975

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  4. [4]

    Lipnerf: What is the right feature space to lip-sync a nerf

    Aggelina Chatziagapi, ShahRukh Athar, Abhinav Jain, Rohith Mysore Vijaya Kumar, Vimal Bhat, and Dimitris Samaras. Lipnerf: What is the right feature space to lip-sync a nerf. In International Conference on Automatic Face and Gesture Recognition 2023, 2023

    2023

  5. [5]

    Gstalker: Real-time audio-driven talking face generation via deformable gaussian splatting.arXiv preprint arXiv:2404.19040, 2024

    Bo Chen, Shoukang Hu, Qi Chen, Chenpeng Du, Ran Yi, Yanmin Qian, and Xie Chen. Gstalker: Real-time audio-driven talking face generation via deformable gaussian splatting.arXiv preprint arXiv:2404.19040, 2024

    arXiv 2024

  6. [6]

    Skyreels-v2: Infinite-length film generative model.arXiv preprint arXiv:2504.13074, 2025

    Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Juncheng Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengchen Ma, et al. Skyreels-v2: Infinite-length film generative model.arXiv preprint arXiv:2504.13074, 2025

    arXiv 2025

  7. [7]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    arXiv

  8. [8]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions.arXiv preprint arXiv:2407.08136, 2024

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions.arXiv preprint arXiv:2407.08136, 2024. 13

    arXiv 2024

Show all 83 references
  1. [9]

    Yolo-world: Real-time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024

    2024

  2. [10]

    Gaussiantalker: Real-time high-fidelity talking head synthesis with audio-driven 3d gaussian splatting.arXiv preprint arXiv:2404.16012, 2024

    Kyusun Cho, Joungbin Lee, Heeji Yoon, Yeobin Hong, Jaehoon Ko, Sangjun Ahn, and Seun- gryong Kim. Gaussiantalker: Real-time high-fidelity talking head synthesis with audio-driven 3d gaussian splatting.arXiv preprint arXiv:2404.16012, 2024

    2024 arXiv

  3. [11]

    Out of time: automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017

    2016

  4. [12]

    Hallo3: Highly dynamic and realistic portrait image animation with diffusion transformer networks.arXiv preprint arXiv:2412.00733, 2024

    Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with diffusion transformer networks.arXiv preprint arXiv:2412.00733, 2024

    2024 arXiv

  5. [13]

    Adam: A method for stochastic optimization.(No Title), 2014

    P Kingma Diederik. Adam: A method for stochastic optimization.(No Title), 2014

    2014

  6. [14]

    Scaling rectified flow transform- ers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. InForty-first International Conference on Machine...

    2024

  7. [15]

    Motioncharacter: Identity- preserving and motion controllable human video generation.arXiv preprint arXiv:2411.18281, 2024

    Haopeng Fang, Di Qiu, Binjie Mao, Pengfei Yan, and He Tang. Motioncharacter: Identity- preserving and motion controllable human video generation.arXiv preprint arXiv:2411.18281, 2024

    2024

  8. [16]

    A unified sequence parallelism approach for long context generative ai.arXiv preprint arXiv:2405.07719, 2024

    Jiarui Fang and Shangchun Zhao. A unified sequence parallelism approach for long context generative ai.arXiv preprint arXiv:2405.07719, 2024

    2024 arXiv

  9. [17]

    Scalable diffusion models with state space backbone.arXiv preprint arXiv:2402.05608, 2024

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Scalable diffusion models with state space backbone.arXiv preprint arXiv:2402.05608, 2024

    2024 arXiv

  10. [18]

    Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models.arXiv preprint arXiv:2404.04478, 2024

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models.arXiv preprint arXiv:2404.04478, 2024

    2024 arXiv

  11. [19]

    Skyreels-a2: Compose anything in video diffusion transform- ers.arXiv preprint arXiv:2504.02436, 2025

    Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, et al. Skyreels-a2: Compose anything in video diffusion transform- ers.arXiv preprint arXiv:2504.02436, 2025

    2025 arXiv

  12. [20]

    Ingredients: Blending custom photos with video diffusion transformers.arXiv preprint arXiv:2501.01790, 2025

    Zhengcong Fei, Debang Li, Di Qiu, Changqian Yu, and Mingyuan Fan. Ingredients: Blending custom photos with video diffusion transformers.arXiv preprint arXiv:2501.01790, 2025

    2025 arXiv

  13. [21]

    Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

    2020

  14. [22]

    Stylesync: High-fidelity generalized and personalized lip sync in style-based generator

    Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Stylesync: High-fidelity generalized and personalized lip sync in style-based generator. InProceedings of the IEEE/CVF Conference on Comp...

    2023

  15. [23]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  16. [24]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    2017

  17. [25]

    Sonic: Shifting focus to global audio perception in portrait animation.arXiv preprint arXiv:2411.16331, 2024

    Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, et al. Sonic: Shifting focus to global audio perception in portrait animation.arXiv preprint arXiv:2411.16331, 2024

    2024 arXiv

  18. [26]

    Audio-driven emotional video portraits

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio-driven emotional video portraits. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14080–14089, 2021

    2021

  19. [27]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency.arXiv preprint arXiv:2409.02634, 2024

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency.arXiv preprint arXiv:2409.02634, 2024. 14

    2024 arXiv

  20. [28]

    Assessing empathy and managing emotions through interactions with an affective avatar.Health informatics journal, 24(2):182–193, 2018

    Esperanza Johnson, Ramón Hervás, Carlos Gutiérrez López de la Franca, Tania Mondéjar, Ser- gio F Ochoa, and Jesús Favela. Assessing empathy and managing emotions through interactions with an affective avatar.Health informatics journal, 24(2):182–193, 2018

    2018

  21. [29]

    Educational virtual reality game design for film and animation

    Oytun Kal and Yavuz Samur. Educational virtual reality game design for film and animation. Encyclopedia of Computer Graphics and Games, pages 621–636, 2024

    2024

  22. [30]

    3d gaussian splatting for real-time radiance field rendering.ACM Trans

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Trans. Graph., 42(4):139–1, 2023

    2023

  23. [31]

    Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013

    Diederik P Kingma. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  24. [32]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  25. [33]

    Latentsync: Audio conditioned latent diffusion models for lip sync.arXiv preprint arXiv:2412.09262, 2024

    Chunyu Li, Chao Zhang, Weikai Xu, Jinghui Xie, Weiguo Feng, Bingyue Peng, and Weiwei Xing. Latentsync: Audio conditioned latent diffusion models for lip sync.arXiv preprint arXiv:2412.09262, 2024

    2024 arXiv

  26. [34]

    Ae- nerf: Audio enhanced neural radiance field for few shot talking head synthesis

    Dongze Li, Kang Zhao, Wei Wang, Bo Peng, Yingya Zhang, Jing Dong, and Tieniu Tan. Ae- nerf: Audio enhanced neural radiance field for few shot talking head synthesis. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3037–3045, 2024

    2024

  27. [35]

    Openhumanvid: A large-scale high-quality dataset for enhancing human-centric video generation.arXiv preprint arXiv:2412.00115, 2024

    Hui Li, Mingwang Xu, Yun Zhan, Shan Mu, Jiaye Li, Kaihui Cheng, Yuxuan Chen, Tan Chen, Mao Ye, Jingdong Wang, et al. Openhumanvid: A large-scale high-quality dataset for enhancing human-centric video generation.arXiv preprint arXiv:2412.00115, 2024

    2024 arXiv

  28. [36]

    Talkinggaus- sian: Structure-persistent 3d talking head synthesis via gaussian splatting.arXiv preprint arXiv:2404.15264, 2024

    Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Talkinggaus- sian: Structure-persistent 3d talking head synthesis via gaussian splatting.arXiv preprint arXiv:2404.15264, 2024

    2024 arXiv

  29. [37]

    Learning a model of facial shape and expression from 4d scans.ACM Trans

    Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans.ACM Trans. Graph., 36(6):194–1, 2017

    2017

  30. [38]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models, 2025

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models, 2025

    2025

  31. [39]

    Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  32. [40]

    Timestep embedding tells: It’s time to cache for video diffusion model.arXiv preprint arXiv:2411.19108, 2024

    Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It’s time to cache for video diffusion model.arXiv preprint arXiv:2411.19108, 2024

    2024 arXiv

  33. [41]

    Moda: Mapping-once audio- driven portrait animation with dual attentions

    Yunfei Liu, Lijian Lin, Fei Yu, Changyin Zhou, and Yu Li. Moda: Mapping-once audio- driven portrait animation with dual attentions. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 23020–23029, 2023

    2023

  34. [42]

    Live speech portraits: real-time photorealistic talking-head animation.ACM Transactions on Graphics (ToG), 40(6):1–17, 2021

    Yuanxun Lu, Jinxiang Chai, and Xun Cao. Live speech portraits: real-time photorealistic talking-head animation.ACM Transactions on Graphics (ToG), 40(6):1–17, 2021

    2021

  35. [43]

    Pixel codec avatars

    Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li, Fernando De La Torre, and Yaser Sheikh. Pixel codec avatars. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 64–73, 2021

    2021

  36. [44]

    Styletalk: One-shot talking head generation with controllable speaking styles

    Yifeng Ma, Suzhen Wang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding, Zhidong Deng, and Xin Yu. Styletalk: One-shot talking head generation with controllable speaking styles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1896–1904, 2023

    1904

  37. [45]

    Dreamtalk: When expressive talking head generation meets diffusion probabilistic models

    Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767, 2(3), 2023

    2023 arXiv

  38. [46]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021

    2021

  39. [47]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023. 15

    2023

  40. [48]

    Synctalk: The devil is in the synchronization for talking head synthesis

    Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan. Synctalk: The devil is in the synchronization for talking head synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 666...

    2024

  41. [49]

    Emotalk: Speech-driven emotional disentanglement for 3d face animation

    Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech-driven emotional disentanglement for 3d face animation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20687– 20697, 2023

    2023

  42. [50]

    Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720, 2024

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models.arXiv preprint arXiv:2410.13720, 2024

    2024 arXiv

  43. [51]

    Moviecharacter: A tuning-free framework for controllable character video synthesis.arXiv preprint arXiv:2410.20974, 2024

    Di Qiu, Zheng Chen, Rui Wang, Mingyuan Fan, Changqian Yu, Junshi Huang, and Xiang Wen. Moviecharacter: A tuning-free framework for controllable character video synthesis.arXiv preprint arXiv:2410.20974, 2024

    2024 arXiv

  44. [52]

    Skyreels-a1: Expressive portrait animation in video diffusion transformers.arXiv preprint arXiv:2502.10841, 2025

    Di Qiu, Zhengcong Fei, Rui Wang, Jialin Bai, Changqian Yu, Mingyuan Fan, Guibin Chen, and Xiang Wen. Skyreels-a1: Expressive portrait animation in video diffusion transformers.arXiv preprint arXiv:2502.10841, 2025

    2025 arXiv

  45. [53]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  46. [54]

    What role can avatars play in e-mental health interventions? exploring new models of client–therapist interaction.Frontiers in Psychiatry, 7:186, 2016

    Imogen C Rehm, Emily Foenander, Klaire Wallace, Jo-Anne M Abbott, Michael Kyrios, and Neil Thomas. What role can avatars play in e-mental health interventions? exploring new models of client–therapist interaction.Frontiers in Psychiatry, 7:186, 2016

    2016

  47. [55]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  48. [56]

    Audio-driven dubbing for user generated contents via style-aware semi-parametric synthesis.IEEE Transactions on Circuits and Systems for Video Technology, 33(3):1247–1261, 2022

    Linsen Song, Wayne Wu, Chaoyou Fu, Chen Change Loy, and Ran He. Audio-driven dubbing for user generated contents via style-aware semi-parametric synthesis.IEEE Transactions on Circuits and Systems for Video Technology, 33(3):1247–1261, 2022

    2022

  49. [57]

    Audio-driven high-resolution seamless talking head video editing via stylegan.arXiv preprint arXiv:2407.05577, 2024

    Jiacheng Su, Kunhong Liu, Liyan Chen, Junfeng Yao, Qingsong Liu, and Dongdong Lv. Audio-driven high-resolution seamless talking head video editing via stylegan.arXiv preprint arXiv:2407.05577, 2024

    2024 arXiv

  50. [58]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  51. [59]

    Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions.arXiv preprint arXiv:2402.17485, 2024

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions.arXiv preprint arXiv:2402.17485, 2024

    2024 arXiv

  52. [60]

    Nonlinear 3d face morphable model

    Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7346–7355, 2018

    2018

  53. [61]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019

    2019

  54. [62]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pande...

    2025 arXiv

  55. [63]

    V-express: Conditional dropout for progressive training of portrait video generation.arXiv preprint arXiv:2406.02511, 2024

    Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation.arXiv preprint arXiv:2406.02511, 2024. 16

    2024 arXiv

  56. [64]

    Seeing what you said: Talking face generation guided by a lip reading expert

    Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. Seeing what you said: Talking face generation guided by a lip reading expert. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14653–14662, 2023

    2023

  57. [65]

    Fantasytalking: Realistic talking portrait generation via coherent motion synthesis

    Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, and Mu Xu. Fantasytalking: Realistic talking portrait generation via coherent motion synthesis. arXiv preprint arXiv:2504.04842, 2025

    2025 arXiv

  58. [66]

    One-shot free-view neural talking-head synthesis for video conferencing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10039–10049, 2021

    2021

  59. [67]

    Magicvideo-v2: Multi-stage high-aesthetic video generation.arXiv preprint arXiv:2401.04468, 2024

    Weimin Wang, Jiawei Liu, Zhijie Lin, Jiangqiao Yan, Shuo Chen, Chetwin Low, Tuyen Hoang, Jie Wu, Jun Hao Liew, Hanshu Yan, et al. Magicvideo-v2: Multi-stage high-aesthetic video generation.arXiv preprint arXiv:2401.04468, 2024

    2024 arXiv

  60. [68]

    Panda: A gigapixel-level human-centric video dataset

    Xueyang Wang, Xiya Zhang, Yinheng Zhu, Yuchen Guo, Xiaoyun Yuan, Liuyu Xiang, Zerun Wang, Guiguang Ding, David Brady, Qionghai Dai, and Lu Fang. Panda: A gigapixel-level human-centric video dataset. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2020

  61. [69]

    Aniportrait: Audio-driven synthesis of photore- alistic portrait animation.arXiv preprint arXiv:2403.17694, 2024

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photore- alistic portrait animation.arXiv preprint arXiv:2403.17694, 2024

    2024 arXiv

  62. [70]

    Q-align: Teaching lmms for visual scoring via discrete text-defined levels.arXiv preprint arXiv:2312.17090, 2023

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching lmms for visual scoring via discrete text-defined levels.arXiv preprint arXiv:2312.17090, 2023

    2023 arXiv

  63. [71]

    Codetalker: Speech-driven 3d facial animation with discrete motion prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12780–12790, 2023

    2023

  64. [72]

    Easyanimate: A high-performance long video generation method based on transformer architecture.arXiv preprint arXiv:2405.18991, 2024

    Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture.arXiv preprint arXiv:2405.18991, 2024

    2024

  65. [73]

    Megactor-sigma: Unlocking flexible mixed-modal control in portrait animation with diffusion transformer.arXiv preprint arXiv:2408.14975, 2024

    Shurong Yang, Huadong Li, Juhao Wu, Minhao Jing, Linze Li, Renhe Ji, Jiajun Liang, Haoqiang Fan, and Jin Wang. Megactor-sigma: Unlocking flexible mixed-modal control in portrait animation with diffusion transformer.arXiv preprint arXiv:2408.14975, 2024

    2024 arXiv

  66. [74]

    Effective whole-body pose estimation with two-stages distillation

    Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4210–4220, 2023

    2023

  67. [75]

    Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  68. [76]

    Real3d-portrait: One-shot realistic 3d talking portrait synthesis.arXiv preprint arXiv:2401.08503, 2024

    Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, et al. Real3d-portrait: One-shot realistic 3d talking portrait synthesis.arXiv preprint arXiv:2401.08503, 2024

    2024 arXiv

  69. [77]

    Magicinfinite: Generating infinite talking videos with your words and voice.arXiv preprint arXiv:2503.05978, 2025

    Hongwei Yi, Tian Ye, Shitong Shao, Xuancheng Yang, Jiantong Zhao, Hanzhong Guo, Terrance Wang, Qingyu Yin, Zeke Xie, Lei Zhu, et al. Magicinfinite: Generating infinite talking videos with your words and voice.arXiv preprint arXiv:2503.05978, 2025

    2025 arXiv

  70. [78]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2023

  71. [79]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021

    2021

  72. [80]

    Memo: Memory-guided diffusion for expressive talking video generation.arXiv preprint arXiv:2412.04448, 2024

    Longtao Zheng, Yifan Zhang, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An, and Shuicheng Yan. Memo: Memory-guided diffusion for expressive talking video generation.arXiv preprint arXiv:2412.04448, 2024. 17

    2024 arXiv

  73. [81]

    Long and short guidance in score identity distillation for one-step text-to-image generation.ArXiv 2406.01561, 2024

    Mingyuan Zhou, Zhendong Wang, Huangjie Zheng, and Hai Huang. Long and short guidance in score identity distillation for one-step text-to-image generation.ArXiv 2406.01561, 2024

    2024 arXiv

  74. [82]

    Allegro: Open the black box of commercial-level video generation model.arXiv preprint arXiv:2410.15458, 2024

    Yuan Zhou, Qiuyue Wang, Yuxuan Cai, and Huan Yang. Allegro: Open the black box of commercial-level video generation model.arXiv preprint arXiv:2410.15458, 2024

    2024 arXiv

  75. [83]

    Learn2talk: 3d talking face learns from 2d talking face

    Yixiang Zhuang, Baoping Cheng, Yao Cheng, Yuntao Jin, Renshuai Liu, Chengyang Li, Xuan Cheng, Jing Liao, and Juncong Lin. Learn2talk: 3d talking face learns from 2d talking face. arXiv preprint arXiv:2404.12888, 2024. 18

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.