Pith. sign in

REVIEW 12 cited by

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.10667 v2 pith:TVLHT3D2 submitted 2024-04-16 cs.CV

Sicheng Xu , Guojun Chen , Yu-Xiao Guo , Jiaolong Yang , Chong Li , Zhenyu Zang , Yizhong Zhang , Xin Tong
show 1 more author
Baining Guo
This is my paper
classification cs.CV
keywords facialheadlifelikeaudiodynamicsfacefacesgenerating
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce VASA, a framework for generating lifelike talking faces with appealing visual affective skills (VAS) given a single static image and a speech audio clip. Our premiere model, VASA-1, is capable of not only generating lip movements that are exquisitely synchronized with the audio, but also producing a large spectrum of facial nuances and natural head motions that contribute to the perception of authenticity and liveliness. The core innovations include a holistic facial dynamics and head movement generation model that works in a face latent space, and the development of such an expressive and disentangled face latent space using videos. Through extensive experiments including evaluation on a set of new metrics, we show that our method significantly outperforms previous methods along various dimensions comprehensively. Our method not only delivers high video quality with realistic facial and head dynamics but also supports the online generation of 512x512 videos at up to 40 FPS with negligible starting latency. It paves the way for real-time engagements with lifelike avatars that emulate human conversational behaviors.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Hallo-Live achieves 20.38 FPS real-time text-to-audio-video avatar generation with 0.94s latency using asynchronous dual-stream diffusion and HP-DMD preference distillation, matching teacher model quality at 16x highe...

  2. ViDS: Video Diffusion Shader using 3D Face Tracking

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Fine-tuning a video diffusion model on dense 3DMM normal maps from Pixel3DMM lets a single portrait photo be animated with a driving video's expressions and pose, surpassing landmark- and latent-based portrait animati...

  3. Conversational Human Audio-visual Talking Dialogue Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    CHAT generates mutually responsive dyadic audio-visual dialogue clips from a single text prompt and yields a 50k synthetic pre-training set that improves facial reaction models on REACT 2024.

  4. Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Wan-Streamer is a unified end-to-end Transformer for low-latency streaming audio-visual interaction using block-causal attention on interleaved multimodal tokens.

  5. Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation

    cs.CV 2024-11 unverdicted novelty 6.0 of 10

    LetsTalk combines a multimodal diffusion transformer, noise-regularized memory bank, deep compression autoencoder, and symbiotic/direct fusion schemes to achieve state-of-the-art quality and efficiency in long-duratio...

  6. Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Wan-Streamer is a unified Transformer model for low-latency streaming audio-visual interaction that jointly handles perception, reasoning, generation, and timing without external modules.

  7. Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Wan-Streamer presents a unified end-to-end Transformer for low-latency multimodal streaming interaction without external modules.

  8. EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis

    cs.CV 2025-08 conditional novelty 5.0 of 10

    EDTalk++ disentangles talking-head video into four orthogonal motion banks (mouth, pose, eyes, expression) and drives them from either video or audio inputs.

  9. HunyuanVideo: A Systematic Framework For Large Video Generative Models

    cs.CV 2024-12 unverdicted novelty 5.0 of 10

    HunyuanVideo presents a 13B-parameter open-source video generative model with integrated data, architecture, training, and inference systems whose professional evaluations show it outperforming prior SOTA models inclu...

  10. JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation

    cs.CV 2024-11 unverdicted novelty 5.0 of 10

    JoyVASA decouples static 3D facial representations from identity-independent dynamic motion sequences generated by a diffusion transformer to produce audio-driven animations for humans and animals.

  11. Wan-Streamer v0.2: Higher Resolution, Same Latency

    cs.CV 2026-07 conditional novelty 4.0 of 10

    Wan-Streamer v0.2 upgrades native-streaming audio-visual interaction to 640×368 at 25 FPS with unchanged ~200 ms model-side latency via a single-GPU thinker and multi-GPU Ulysses-style performer.

  12. Human Motion Video Generation: A Survey

    cs.CV 2025-09 conditional novelty 4.0 of 10

    A comprehensive survey with a five-phase pipeline model for human motion video generation, covering over 200 papers and adding a new benchmark comparison of nine pose-guided methods.

Pith tools