Pith. sign in

REVIEW 4 major objections 6 minor 32 cited by

OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OmniHuman claims that mixing weak and strong motion conditions during training makes human animation data scalable, yielding one audio-driven model that handles any body proportion, image style, and pose-driven or combined driving.

desk verdict A credible data-scaling recipe for human animation, with ablations that support the core idea but comparisons that need error bars before the superiority claims can be trusted. read the letter →

arxiv 2502.01061 v3 pith:ACKR4SU3 submitted 2025-02-03 cs.CV

Gaojie Lin , Jianwen Jiang , Jiaqi Yang , Zerong Zheng , Chao Liang This is my paper · ORCID
classification cs.CV
keywords humananimationaudio-drivenvideogenerationdiffusiontransformermixed-conditiontrainingdatascalingpose-conditionedimage-to-videotalkinghead
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that audio-driven human animation has failed to scale not because video data is scarce, but because single-condition training wastes most of it: audio correlates mainly with facial expression, so audio-only training forces aggressive filtering that discards data containing valuable body motion. Its proposed fix, the omni-conditions training strategy, mixes text, audio, and pose conditions during training so that data unusable for audio can still train the shared model under weaker conditions, while stronger conditions are trained less often. If the claim is right, one Diffusion Transformer model can animate faces, half-bodies, and full bodies from audio, support pose-driven and combined driving, and handle cartoon and stylized inputs. A sympathetic reader should care because the paper is offering a general training recipe for scaling conditioned generation when the primary condition is only weakly tied to the motion to be generated.

What carries the argument

The mechanism that carries the argument is the omni-conditions training strategy, defined by two scheduling principles: stronger-conditioned tasks reuse weaker-conditioned data to scale up training, and stronger conditions receive lower training ratios so they do not dominate weaker ones; in the final stage the ratios are text 90 percent, audio 50 percent, and pose 25 percent, with pose introduced last. The model itself is an MMDiT-based Diffusion Transformer, where MMDiT is a multimodal Diffusion Transformer backbone, in which audio features from wav2vec are compressed and injected as frame-wise cross-attention tokens, skeleton pose features from a pose guider are stacked with the noisy latents along the channel dimension, text goes through the original text branch, and the reference image is encoded by reusing the denoising backbone with zeroed temporal RoPE (rotary position embedding) for reference tokens, letting reference and video tokens interact through self-attention without extra parameters. This shared-backbone design is what allows the mixed-condition schedule to train a single model across image-text-to-video, image-text-audio-to-video, and image-text-audio-pose-to-video tasks.

What would settle it

Re-run the evaluation with multiple training seeds to obtain confidence intervals, and add a forced-choice human preference test between OmniHuman and its closest baselines on the same reference images and audio; if the intervals for FVD and Sync-C overlap the baselines' intervals, or human judges show no significant preference on portraits and body poses, the claimed superiority of the mixed-condition recipe is not established.

Watch

Extended reading notes

Core claim

The central claim is that data scaling for human animation becomes feasible when motion-related conditions are mixed in training instead of isolated. The paper argues that a weak condition such as audio is primarily associated with facial expressions and lip motion and has little correlation with body pose, background motion, or camera movement, so audio-only training forces costly cleaning that retains under ten percent of collected data. By training one model with text, audio, and pose conditions jointly, data that fails audio-conditioned filtering can still contribute under a weaker text condition, and different conditions complement one another during inference. The paper states two training principles — stronger-conditioned tasks may leverage weaker-conditioned data, and the stronger the condition, the lower its training ratio should be — and builds OmniHuman, a Diffusion Transformer (DiT) based one-stage model, around them. It claims this makes OmniHuman the first audio-driven solution that accepts input images with any body proportion and image style and also supports auxiliary pose driving, outperforming specialized portrait and body animation baselines.

Load-bearing premise

The claim's load-bearing premise is that the automatic metrics used for comparison — FID, FVD, Sync-C, Q-Align IQA and ASE, and hand-keypoint scores, reported without error bars or a human preference study — track what a viewer actually perceives as realistic and well-synchronized motion; the paper itself notes that no single metric captures motion-style preferences across scenarios.

Editorial extensions

If this is right

  • One trained model can switch among audio-driven, pose-driven, and combined audio-plus-pose driving without task-specific fine-tuning, covering face close-ups, portraits, half-body, and full-body inputs.
  • Training data no longer needs the strict audio-only cleaning that the paper says keeps under ten percent of collected data; text-conditioned data can be reused, expanding the usable corpus to 18.7K hours and improving gesture richness and hand quality.
  • Because stronger conditions are trained less often, pose-conditioned training does not suppress audio learning; the paper reports that the hybrid-driven model decouples hand motion from the audio track and reduces exaggerated gestures.
  • The same backbone handles stylized, cartoon, and even anthropomorphic non-human inputs, so input flexibility becomes a property of the training recipe rather than of dataset filtering.
  • The two principles give a general curriculum for adding new driving modalities to a pretrained video diffusion model, which the paper frames as the actual contribution rather than a single model's numbers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the recipe generalizes, other weakly conditioned generation tasks, such as music-driven dance or audio-driven scene video, could adopt the same mix: reuse data under weaker conditions, lower the ratio of stronger conditions, and add strong conditions late in training.
  • The fixed ratio schedule (text 90 percent, audio 50 percent, pose 25 percent) is a heuristic; a testable extension is an adaptive schedule or per-batch loss weighting that tunes condition strength continuously instead of by fixed halving.
  • The paper's observation that image-quality scores can decrease while video-distance metrics improve suggests the model learns to match the input image distribution rather than the filtered training distribution; a perceptual study could verify whether this trade-off is genuine or a metric artifact.
  • The parameter-free reference encoding, which reuses the DiT backbone with zeroed temporal RoPE, implies appearance conditioning may scale with backbone size without a separate reference network; one could test whether this holds at larger model scales.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes OmniHuman, a Diffusion Transformer-based framework for human animation that is trained with mixed conditioning signals (text, audio, pose) together with an appearance reference image. The central claim is that an 'omni-conditions' training strategy, governed by two principles (weaker-conditioned tasks can supply data to stronger-conditioned tasks, and stronger conditions should have lower training ratios), enables scaling up training data and yields a single model that outperforms specialized baselines on both portrait and body animation, while also supporting arbitrary body proportions, image styles, and multiple driving modalities. The method is validated with ablations on the training principles, comparisons against seven portrait and three body animation baselines, and qualitative examples.

Significance. If the claims are substantiated, this is a significant advance: it is, to my knowledge, the first one-stage audio-driven human animation model that handles arbitrary body proportions and image styles, and the idea of using mixed weak/strong conditions to unlock larger training corpora is a plausible and important direction for the field. The ablations are systematic and directly test the two proposed principles, including the ordering of pose/audio introduction and the training-ratio balance, which is a strength. The paper also reports a large set of metrics and provides qualitative evidence of generalization. However, the evaluation is weakened by the absence of error bars, statistical tests, and human evaluation; several of the reported superiority gaps are small and could be within metric noise. The training dataset and code are not released, which limits reproducibility, though that is not by itself a flaw. Overall, the core idea is promising but the evidence for the central comparative claim is not yet conclusive.

major comments (4)
  1. [§4.3, Tables 2–3] The claim that OmniHuman outperforms existing audio-driven baselines rests on single point estimates of FID, FVD, IQA, ASE, and Sync-C with no error bars, no multiple seeds, and no significance tests. On RAVDESS the FVD gap to Loopy is 0.228 (15.906 vs 16.134), and on the CyberHost test set the FVD gap is 0.972 (27.031 vs 28.003); these are within typical run-to-run variability for FVD in video generation. The paper itself acknowledges in §4.3 that "the model's varying preferences for motion styles across different scenarios complicate performance measurement using a single metric," yet the entire superiority claim is based on unreplicated point estimates. Please provide confidence intervals, multiple seeds, or a human preference study to establish that the reported differences are not noise.
  2. [§4.2, Table 1 (upper part)] The T-Data sweep varies both the type of conditioning (text vs. none) and the total amount of training data at the same time. Since the 0% T-Data configuration necessarily has less data overall, the observed improvements from 0% to 100% T-Data could be driven by sheer data quantity rather than by the text condition specifically. To isolate the effect of Principle 1, the total training data hours should be held fixed across configurations, for example by comparing text-conditioned data against an equal volume of weakly or unlabeled data. Without such a control, the evidence that weaker conditions are what enable scaling is confounded.
  3. [§4.1, §4.2, Table 1] The Q-Align IQA and ASE results are reported without specifying the exact prompt and protocol used, which is critical for no-reference metrics since they are highly sensitive to prompt wording. Moreover, the paper observes in §4.2 that IQA decreases with more text-conditioned data while FVD and Sync-C improve, and attributes this to the model adhering to the input image distribution rather than the training distribution. Without the Q-Align configuration or a validation that the metric aligns with human perception in this setting, the reader cannot assess whether the reported quality scores are meaningful. Please provide the prompt details and, ideally, a human evaluation to corroborate quality and motion naturalness.
  4. [§4.3, Table 4] The CyberHost row in Table 4 reports FVD as 7.7178, which appears to be a typographical error (likely 77.178). As printed, this makes the comparison misleading, because 7.7178 would be substantially better than OmniHuman's 7.3184, whereas if the intended value is 77.178 the comparison reverses. Please correct the table and re-evaluate the conclusions drawn from that comparison.
minor comments (6)
  1. [§2.2] There is a typo: "challanges" should be "challenges".
  2. [Abstract] The abstract contains a LaTeX artifact: "ttfamily project page" should be "project page".
  3. [§4.1 and Table 3] The baseline name is written inconsistently: "DiffGest. [82]+MomicMo. [76]" in Table 3, while the text and references use "DiffGest" and "MimicMotion". Please make the names consistent.
  4. [§3.3] The training ratios T=90%, A=50%, P=25% and the CFG scale of 6.5 are presented as outcomes of the two principles, but the principles themselves are formulated after observing which configuration works best. The ablations in the paper provide some support, but the generality of the exact ratios is unclear. A sentence acknowledging that these ratios are empirical choices tuned on the authors' own model would clarify the scope of the claim.
  5. [§4.1] The paper states that "less than 10% of the data is retained" after filtering, citing no specific source; please add a reference or clarify whether this is the authors' own estimate.
  6. [§4.3] In Table 4, CyberHost's FVD value appears to be missing a decimal or contains an extra digit; also, the table column alignment could be improved to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the omni-conditions training recipe is ablated and benchmarked externally, not derived from its own outputs.

full rationale

I walked the claimed derivation chain. The two training principles are stated as design hypotheses (Sec. 3.3) and are tested against explicit alternatives in Table 1 and Supplementary Section 5: the text-data proportion sweep (0/25/50/100% T-Data) tests Principle 1, and the IPA vs IAP ordering and A<P vs A>P ratio comparisons test Principle 2. The adopted ratios (T=90%, A=50%, P=25%) and CFG scale 6.5 are selected hyperparameters, reported as such, not predictions derived from the metrics they are used to produce; there is no fitted parameter that is renamed as a predicted outcome. The comparison against baselines uses public test sets (CelebV-HQ, RAVDESS, CyberHost's test set) and standard metrics; the self-citations to Loopy and CyberHost supply architecture context, metrics, and a test set, but the claimed superiority is not logically inherited from those citations—it rests on the reported measurements. No equation in the paper reduces to its inputs by construction, and the limitation section ('uncoordinated or overly expressive movements still occur...') is an honest caveat rather than a circular step. The post hoc flavor of some explanations (e.g., IQA decreasing with more text data because the model 'follows the quality of the input images') is interpretive, not circular, and the central claim retains independent empirical content. Concerns about missing error bars or metric noise are validity/evidence issues, not circularity.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The model builds on a pretrained text-to-video diffusion transformer, assumed as background. The training ratios and CFG scale are tuned on the authors' data, making the 'principles' partly post hoc. No new physical or mathematical entities are introduced.

free parameters (5)
  • Text data training ratio T = 90%
    Selected via ablations in Section 4.2 and Supplementary Section 5; T=90% was fixed while varying audio ratio.
  • Audio data training ratio A = 50%
    A=50% reported as balanced; too high or too low degrades performance (Supplementary Section 5).
  • Pose data training ratio P = 25%
    Lower pose ratio consistent with Principle 2; A<P (P=75%) degrades metrics in Table 1.
  • CFG scale for audio and text = 6.5
    Set by manual tuning in Section 3.4; pose CFG is disabled.
  • Fraction of data passing lipsync/pose filters = 13%
    Only 13% of the 18.7K hours had usable audio and pose conditions; this determines the amount of strongly conditioned data.
assumptions (3)
  • domain assumption A pretrained general text-to-video MMDiT model provides a valid base for human animation
    The method starts from a pretrained MMDiT without studying how much this base contributes to final quality; Section 3.1.
  • domain assumption Condition strength ordering text < audio < pose is valid and stable across datasets
    This ordering underpins both training principles in Section 3.3 but is justified only by the authors' intuition and their own ablations.
  • domain assumption Mixing weaker-condition data into strongly-conditioned training does not degrade the strong-condition capability
    The central premise of Principle 1; tested only on the authors' in-house data in Table 1, with no analysis of failure modes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models." pith.science (2026). https://pith.science/paper/ACKR4SU3

@misc{pith2026250201061,
  author       = {Pith},
  title        = {Pith review of: OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ACKR4SU3}},
  note         = {Machine review of arXiv:2502.01061}
}
read the original abstract

End-to-end human animation, such as audio-driven talking human generation, has undergone notable advancements in the recent few years. However, existing methods still struggle to scale up as large general video generation models, limiting their potential in real applications. In this paper, we propose OmniHuman, a Diffusion Transformer-based framework that scales up data by mixing motion-related conditions into the training phase. To this end, we introduce two training principles for these mixed conditions, along with the corresponding model architecture and inference strategy. These designs enable OmniHuman to fully leverage data-driven motion generation, ultimately achieving highly realistic human video generation. More importantly, OmniHuman supports various portrait contents (face close-up, portrait, half-body, full-body), supports both talking and singing, handles human-object interactions and challenging body poses, and accommodates different image styles. Compared to existing end-to-end audio-driven methods, OmniHuman not only produces more realistic videos, but also offers greater flexibility in inputs. It also supports multiple driving modalities (audio-driven, video-driven and combined driving signals). Video samples are provided on the ttfamily project page (https://omnihuman-lab.github.io)

Figures

Figures reproduced from arXiv: 2502.01061 by the authors.

Figure 1
Figure 1. The video frames generated by OmniHuman based on input audio and image. The generated results feature head and gesture movements, as well as facial expressions, that match the audio. OmniHuman generates realistic videos with any aspect ratio and body proportion, and significantly improves gesture generation and object interaction over existing methods, due to the data scaling up enabled by omni-conditions training. … view at source ↗
Figure 2
Figure 2. The framework of OmniHuman. It consists of two parts: (1) the OmniHuman model, which is based on the DiT architecture and supports simultaneous conditioning with multiple modalities including text, image, audio, and pose. To support long video continuation, we concatenate the latents of the last generated frames with noise latents, which are omitted for simplicity. (2) the omni-conditions training strategy, which em… view at source ↗
Figure 5
Figure 5. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The videos generated by OmniHuman based on in￾put audio and images. OmniHuman is compatible with stylized humanoid and 2D cartoon characters, and can even animate non￾human images in an anthropomorphic manner. verse input images and maintains the motion style of the in…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 32 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment

    cs.CV 2026-07 conditional novelty 6.0 of 10

    AgentHOI generates human-object interaction videos from text plus one human image and one object image, using multi-agent action planning and implicit text-to-motion feature alignment inside a video diffusion model.

  2. 3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Presents a scene-adaptive 3D human image animation framework using ground-adaptive motion retargeting and viewpoint-adaptive latent fusion to control human and camera trajectories, claiming improvements on two benchmarks.

  3. Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation

    cs.LG 2026-01 conditional novelty 6.0 of 10

    A causal diffusion-forcing model generates interactive head-avatar motion with 500ms motion-generation latency and learns expressive reactions via DPO with synthetic negative samples.

  4. OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A single autoregressive diffusion model, trained on a new 286-hour SMPL-X dataset, generates whole-body motion from text, audio, and spatial-temporal control signals, with reference-motion conditioning.

  5. UniVerse-1: Unified Audio-Video Generation via Stitching of Experts

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A unified audio-video generator built by stitching pre-trained video and music diffusion models, trained on 7,600 hours of data, with a new evaluation benchmark.

  6. OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    OmniHuman-1.5 combines MLLM-based planning with a multimodal diffusion transformer and pseudo-last-frame identity conditioning to generate context-aware avatar videos from audio and a reference image.

  7. FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animation

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    A three-part system, Talking-Critic, Talking-NSQ, and TLPO, aligns diffusion portrait animation models to human preferences and improves lip-sync, motion naturalness, and visual quality.

  8. Extension of generalized KYP lemma: from LTI systems to LPV systems

    math.DS 2025-08 unverdicted novelty 6.0 of 10

    The abstract claims a gKYP lemma extension for LPV systems via frequency-range enlargement, but the submitted full text is an unrelated video generation paper.

  9. SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    SpeakerVid-5M provides 5.2 million audio-visual human clips (8,743 hours) with rich annotations and a dyadic interaction benchmark for training interactive virtual humans.

  10. AnyAni: An Interactive System with Generative AI for Animation Effect Creation and Code Understanding in Web Development

    cs.HC 2025-06 conditional novelty 6.0 of 10

    AnyAni combines LLM generation, a version tree, and video-based checking to help front-end developers create and understand web animations; a nine-person study reports usability gains over a chatbot baseline.

  11. Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-mask-based Embedding Router

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Bind-Your-Avatar generates videos of multiple co-present talking characters by using a learned 3D-mask router to bind each voice to the correct character.

  12. DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformers

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A diffusion transformer model generates human-product demonstration videos from paired human and product images while preserving both identities through masked cross-attention and motion template guidance.

  13. HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    HunyuanVideo-HOMA generates human-object interaction videos from weak, sparse inputs: one arm pose, an object center dot, a human photo, and an object photo.

  14. Audio-Sync Video Generation with Multi-Stream Temporal Control

    cs.CV 2025-06 reject novelty 6.0 of 10

    MTV splits audio into speech, effects, and music to separately drive lip sync, event timing, and visual mood in video generation, trained on a new 392K-clip dataset.

  15. Identity Deepfake Threats to Biometric Authentication Systems: Public and Expert Perspectives

    cs.HC 2025-06 conditional novelty 6.0 of 10

    A UK-based mixed-methods study finds a wide gap between public trust in biometrics and expert concern about deepfake spoofing, and proposes a tri-layer mitigation framework.

  16. DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DreamDance animates a single character artwork by reconstructing its background as a 3D Gaussian scene and then inpainting the animated character into the rendered video.

  17. Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MultiTalk is the first framework to generate multi-person conversational videos from multi-stream audio, using Label Rotary Position Embedding to bind each voice to the correct person.

  18. AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A diffusion model animates a character into arbitrary dynamic backgrounds by conditioning on a rendered 3D-avatar video, reframing open-domain animation as a restoration problem.

  19. HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characters

    cs.CV 2025-05 conditional novelty 6.0 of 10

    HunyuanVideo-Avatar is an audio-driven video generator that enables emotion-controllable and multi-character animation by injecting character images, routing audio via face masks, and transferring emotion from referen...

  20. RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Simple conditioning patches and training tricks on the Wan-2.1 model outperform specialized reference-network methods for controllable character animation, according to the paper's benchmarks.

  21. Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    The paper offers the first focused review of MLLM-based video translation organized by a three-role taxonomy of Semantic Reasoner, Expressive Performer, and Visual Synthesizer, plus open challenges.

  22. JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing

    cs.CV 2025-12 conditional novelty 5.0 of 10

    A dual-branch diffusion transformer with joint video-audio self-attention and a keypoint-based mouth-area loss reports top lip-sync and speech metrics on two benchmarks.

  23. InfinityHuman: Towards Long-Term Audio-Driven Human

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A coarse-to-fine audio-driven animation framework that uses pose-guided refinement and hand-specific reward learning to generate long, identity-stable talking videos.

  24. MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generation

    cs.CV 2025-08 reject novelty 5.0 of 10

    A new autoregressive video-generation framework for interactive digital humans, with a 64x compression autoencoder and a diffusion renderer, claims real-time multimodal control but is only demonstrated for audio input.

  25. InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Sparse-frame dubbing with adjacent-chunk keyframe sampling lets a streaming audio-video model produce full-body motion synchronized to new audio while preserving identity and camera motion.

  26. StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A diffusion-based avatar generator that uses a timestep-aware audio adapter, audio-adaptive guidance, and weighted sliding-window fusion to produce audio-synced talking-head videos several minutes long with reduced id...

  27. OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    An audio-driven avatar generator that injects Wav2Vec2 audio features as additive latents into multiple DiT layers of a LoRA-fine-tuned Wan2.1 model, improving lip-sync and enabling prompt-controlled full-body animation.

  28. FramePrompt: In-context Controllable Animation with Zero Structural Changes

    cs.GR 2025-06 conditional novelty 5.0 of 10

    FramePrompt turns character animation into a video-continuation task by concatenating reference image, skeleton frames, and target frames into one sequence, then training the pretrained Wan-I2V model to generate only ...

  29. AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Timestep-segment preference optimization with separate motion and fidelity LoRAs improves audio-driven human animation quality and allows a 3.3x inference speedup.

  30. Seeing Voices: Generating A-Roll Video from Audio with Mirage

    cs.CV 2025-06 reject novelty 5.0 of 10

    Mirage generates photorealistic A-roll videos of people speaking directly from audio, using only joint self-attention over audio, text, and video tokens.

  31. FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    FullDiT2 accelerates FullDiT-style in-context conditioning for video by dynamic token selection and selective context caching, cutting per-step time by 2-3x with minimal quality loss.

  32. Wan-S2V: Audio-Driven Cinematic Video Generation

    cs.CV 2025-08 reject novelty 4.0 of 10

    Wan-S2V is an audio-driven video generator built on Wan, claiming better cinematic character animation than prior systems, though the evaluation is limited.

Reference graph

Works this paper leans on

81 extracted references · 29 canonical work pages · cited by 32 Pith papers

  1. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural infor- mation processing systems, 33:12449–12460, 2020. 4

    2020

  2. [2]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 3

    arXiv 2023

  3. [3]

    Lumiere: A space- time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Her- rmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space- time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024. 2

    arXiv 2024

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  5. [5]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023

    2023

  6. [6]

    Generating long videos of dynamic scenes

    Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. Advances in Neural Information Processing Systems, 35:31769–31781, 2022. 2

    work page 2022

  7. [7]

    Pixart-delta: Fast and controllable image generation with latent consistency models, 2024

    Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-delta: Fast and controllable image generation with latent consistency models, 2024. 3

    work page 2024

  8. [8]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13320–13331, 2024. 5

    work page 2024

Show all 81 references
  1. [9]

    Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven por- trait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136, 2024. 2, 5, 7

    2024 arXiv

  2. [10]

    Out of time: auto- mated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: auto- mated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017. 5

    2016

  3. [11]

    Vlogger: Multimodal diffusion for embodied avatar synthe- sis

    Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthe- sis. arXiv preprint arXiv:2403.08764, 2024. 3

    2024 arXiv

  4. [12]

    Hallo3: Highly dynamic and realistic portrait image an- imation with diffusion transformer networks

    Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image an- imation with diffusion transformer networks. arXiv preprint arXiv:2412.00733, 2024. 3, 5, 7

    2024 arXiv

  5. [13]

    Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdul- mohsin, et al. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neu...

    2024

  6. [14]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  7. [15]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  8. [16]

    Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis

    Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural ra- diance fields for talking head synthesis. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5784–5794, 2021. 3

    2021

  9. [17]

    Animatediff: Animate your personalized text- to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text- to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2, 3

    2023 arXiv

  10. [18]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2023. 2

    2023 arXiv

  11. [19]

    Gaia: Zero-shot talking avatar generation

    Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, Han Hu, et al. Gaia: Zero-shot talking avatar generation. arXiv preprint arXiv:2311.15230, 2023. 2

    2023 arXiv

  12. [20]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 5

    2017

  13. [21]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In Advances in Neural Infor- mation Processing Systems, pages 6840–6851. Curran Asso- ciates, Inc., 2020. 2

    2020

  14. [22]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022. 2 9

    2022

  15. [23]

    Diffted: One-shot audio-driven ted talk video generation with diffusion-based co-speech ges- tures

    Steven Hogue, Chenxu Zhang, Hamza Daruger, Yapeng Tian, and Xiaohu Guo. Diffted: One-shot audio-driven ted talk video generation with diffusion-based co-speech ges- tures. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 1922–1931,

    1922

  16. [24]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 3

    2022 arXiv

  17. [25]

    Animate anyone: Consistent and controllable image- to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024. 3, 4, 7

    2024

  18. [26]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.02634, 2024. 2, 3, 4, 5, 6, 7

    2024 arXiv

  19. [27]

    Mobile- portrait: Real-time one-shot neural head avatars on mobile devices

    Jianwen Jiang, Gaojie Lin, Zhengkun Rong, Chao Liang, Yongming Zhu, Jiaqi Yang, and Tianyun Zhong. Mobile- portrait: Real-time one-shot neural head avatars on mobile devices. arXiv preprint arXiv:2407.05712, 2024. 3

    2024 arXiv

  20. [28]

    Ravdess emotional speech audio

    Kaggle. Ravdess emotional speech audio. https://www. kaggle.com/datasets/uwrfkaggler/ravdess- emotional-speech-audio. 5

  21. [29]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. 2

    2022

  22. [30]

    Videopoet: A large language model for zero-shot video gen- eration

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video gen- eration. arXiv preprint arXiv:2312.14125, 2023. 3

    2023 arXiv

  23. [31]

    Hunyuanvideo: A systematic framework for large video generative models

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 3

    2024 arXiv

  24. [32]

    Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2023. 3

    2023

  25. [33]

    Video generation from text

    Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. InProceedings of the AAAI conference on artificial intelligence, 2018. 2

    2018

  26. [34]

    Cyberhost: A one-stage diffusion framework for audio-driven talking body generation

    Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Ji- aqi Yang, Zerong Zheng, and Yanbo Zheng. Cyberhost: A one-stage diffusion framework for audio-driven talking body generation. In The Thirteenth International Conference on Learning Representations, 2025. 2, 3, 4, 5, 6, 7

    2025

  27. [35]

    Diffusion adversarial post-training for one-step video generation

    Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversarial post-training for one-step video generation. arXiv preprint arXiv:2501.08316,

  28. [36]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 3

    2024

  29. [37]

    Flow straight and fast: Learning to generate and transfer data with rectified flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. ArXiv, abs/2209.03003, 2022. 2

    2022 arXiv

  30. [38]

    Echomimicv2: Towards striking, simplified, and semi-body human animation

    Rang Meng, Xingyu Zhang, Yuming Li, and Chenguang Ma. Echomimicv2: Towards striking, simplified, and semi-body human animation. arXiv preprint arXiv:2411.10061, 2024. 3, 6

    2024

  31. [39]

    V oxceleb: a large- scale speaker identification dataset

    A Nagrani, J Chung, and A Zisserman. V oxceleb: a large- scale speaker identification dataset. Interspeech 2017, 2017. 3

    2017

  32. [40]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Ad- vances in neural information processing systems, 35:...

    2022

  33. [41]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  34. [42]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  35. [43]

    Fatezero: Fusing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.09535, 2023. 3

    2023 arXiv

  36. [44]

    wav2vec: Unsupervised pre-training for speech recognition

    Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862, 2019. 4

    1904 arXiv

  37. [45]

    Human4dit: Free-view human video generation with 4d diffusion transformer

    Ruizhi Shao, Youxin Pang, Zerong Zheng, Jingxiang Sun, and Yebin Liu. Human4dit: Free-view human video generation with 4d diffusion transformer. arXiv preprint arXiv:2405.17405, 2024. 3

    2024 arXiv

  38. [46]

    First order motion model for image animation

    Aliaksandr Siarohin, St ´ephane Lathuili`ere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019. 3

    2019

  39. [47]

    Motion representations for ar- ticulated animation

    Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for ar- ticulated animation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 13653–13662, 2021. 3

    2021

  40. [48]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  41. [49]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2021. 2

    2021

  42. [50]

    Score-based generative modeling through stochastic differential equa- tions

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions. arXiv preprint arXiv:2011.13456, 2020. 2 10

    2011 arXiv

  43. [51]

    Diffused heads: Diffusion models beat gans on talking-face genera- tion

    Michal Stypulkowski, Konstantinos V ougioukas, Sen He, Maciej Zieba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face genera- tion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5091–5100...

    2024

  44. [52]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  45. [53]

    Emo2: End-effector guided audio-driven avatar video generation

    Linrui Tian, Siqi Hu, Qi Wang, Bang Zhang, and Liefeng Bo. Emo2: End-effector guided audio-driven avatar video generation. arXiv preprint arXiv:2501.10687, 2025. 3

    2025 arXiv

  46. [54]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pages 244–260. Springer, 2025. 2, 3, 4

    2025

  47. [55]

    Video generation models as world simulators

    Brooks Tim, Peebles Bill, Connorm Holmes, DePue Will, Yufeim Guo, Jing Li, Schnurr David, Taylor Joe, Luhman Troy, Luhman Eric, Ng Clarence, Wang Ricky, and Ramesh Aditya. Video generation models as world simulators. 2024. Accessed: 2024-02-15. 3

    2024

  48. [56]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  49. [57]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 5

  50. [58]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Le...

    2022

  51. [60]

    V-express: Conditional dropout for progres- sive training of portrait video generation

    Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progres- sive training of portrait video generation. arXiv preprint arXiv:2406.02511, 2024. 2, 5, 6

    2024 arXiv

  52. [61]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023. 2, 3

    2023 arXiv

  53. [62]

    Disco: Disentangled control for realistic human dance generation

    Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung- Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,...

    2024

  54. [63]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 10039–10049,

  55. [64]

    Imaginator: Conditional spatio-temporal gan for video generation

    Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. Imaginator: Conditional spatio-temporal gan for video generation. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision , pages 1160–1169, 2020. 2

    2020

  56. [65]

    Q-align: Teaching lmms for visual scoring via discrete text-defined levels

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching lmms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.17090, 2023. 5

    2023 arXiv

  57. [66]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  58. [67]

    Vfhq: A high-quality dataset and bench- mark for video face super-resolution

    Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and bench- mark for video face super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 657–666, 2022. 3

    2022

  59. [68]

    Hallo: Hierarchical audio-driven vi- sual synthesis for portrait image animation

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Li- wei Zhang, Ce Liu, Jingdong Wang, Luc Van Gool, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven vi- sual synthesis for portrait image animation. arXiv preprint arXiv:2406.08801, 2024. 2, 5, 7

    2024 arXiv

  60. [69]

    Vasa-1: Lifelike audio-driven talking faces generated in real time

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404.10667 ,

    arXiv

  61. [71]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 3

    2024 arXiv

  62. [72]

    Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis

    Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. In The Eleventh In- ternational Conference on Learning Representations , 2022. 3

    2022

  63. [73]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 3

    arXiv

  64. [74]

    Make pixels dance: High- 11 dynamic video generation

    Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High- 11 dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8850–8860, 2024. 3

    2024

  65. [75]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2023

  66. [76]

    Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance

    Yuang Zhang, Jiaxi Gu, Li-Wen Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mim- icmotion: High-quality human motion video generation with confidence-aware pose guidance. arXiv preprint arXiv:2406.19680, 2024. 3, 5, 7

    2024 arXiv

  67. [77]

    Thin-plate spline motion model for image animation

    Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3657–3666, 2022. 3

    2022

  68. [78]

    Open-sora: Democratizing efficient video production for all, 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. 3, 4

    2024

  69. [79]

    Fada: Fast diffusion avatar syn- thesis with mixed-supervised multi-cfg distillation

    Tianyun Zhong, Chao Liang, Jianwen Jiang, Gaojie Lin, Ji- aqi Yang, and Zhou Zhao. Fada: Fast diffusion avatar syn- thesis with mixed-supervised multi-cfg distillation. arXiv preprint arXiv:2412.16915, 2024. 3

    2024 arXiv

  70. [80]

    Magicvideo: Efficient video generation with latent diffusion models

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 2, 3

    2022 arXiv

  71. [81]

    Celebv- hq: A large-scale video facial attributes dataset

    Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv- hq: A large-scale video facial attributes dataset. InEuropean conference on computer vision , pages 650–667. Springer,

  72. [82]

    Taming diffusion models for audio- driven co-speech gesture generation

    Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for audio- driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10544–10553, 2023. 5, 7

    2023

  73. [83]

    Champ: Controllable and consistent human image an- imation with 3d parametric guidance

    Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Zilong Dong, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image an- imation with 3d parametric guidance. In European Confer- ence on Computer Vision , pages 145–162. Springer, 2025. 3 12

    2025

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.