pith. sign in

arxiv: 2606.01620 · v1 · pith:UWGUOYEXnew · submitted 2026-06-01 · 💻 cs.CV

Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs

Sicheng Xu , Yu Deng , Shoukang Hu , Yichuan Wang , Yizhong Zhang , Zhan Chen , Jiaolong Yang , Baining Guo This is my paper

Pith reviewed 2026-06-28 15:40 UTC · model grok-4.3

classification 💻 cs.CV
keywords talking portrait videocausal VAEreal-time video generationstreamable videoreference-guided compressionautoregressive denoisingrectified flow transformer
0
0 comments X

The pith

A causal VAE guided by reference images enables real-time streamable talking portrait video from audio.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a framework for generating talking portrait videos in real time, conditioned on speech audio and reference images, specifically for streaming use. It relies on a causal video VAE that performs deep latent compression while taking a variable number of reference images to emphasize motion over static appearance. An autoregressive latent denoising model built on a Rectified Flow Transformer then produces video latents block by block. The design targets the high compute cost of diffusion models that currently blocks interactive applications.

Core claim

Integrating variable reference images as guidance into a causal video VAE lets the network focus on dynamic information, which improves compression efficacy and reconstruction quality for streaming; extending residual auto-encoding further supports spatial-temporal causality, and the Rectified Flow Transformer generator produces latents in a blockwise autoregressive manner to reach real-time speeds.

What carries the argument

Reference-guided causal video VAE for deep latent compression, with extended residual auto-encoding for causality handling and a Rectified Flow Transformer for blockwise autoregressive latent denoising.

If this is right

  • Real-time generation of talking portraits becomes practical for interactive applications.
  • Generation speed exceeds that of prior diffusion-based models while quality remains comparable or higher.
  • Streaming pipelines gain from improved latent compression without loss of temporal coherence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same reference-guidance pattern could be tested on non-portrait conditional video tasks such as scene animation.
  • Blockwise autoregressive decoding may support generation of longer continuous video streams without full-sequence buffering.
  • Lower latency from the causal design could be measured in end-to-end audio-to-video pipelines for live use.

Load-bearing premise

That feeding a variable number of reference images lets the VAE separate dynamic content from static appearance enough to raise both compression rate and output quality in streaming settings.

What would settle it

A side-by-side timing and perceptual-quality test on standard talking-head datasets that measures frames per second and realism scores against the large diffusion baselines the paper compares to.

Figures

Figures reproduced from arXiv: 2606.01620 by Baining Guo, Jiaolong Yang, Shoukang Hu, Sicheng Xu, Yichuan Wang, Yizhong Zhang, Yu Deng, Zhan Chen.

Figure 1
Figure 1. Figure 1: Our method synthesizes streamable talking portrait videos given speech audio and one or multiple reference images, enabling [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our framework. Left: The proposed reference-guided causal video VAE. Qfz and Kfz are query–key pairs from fz, upsampled from video latent z via D1. Right: Rectified flow transformer with block-wise causal attention for modeling the probabilistic distribution of the compact video latents. enable streaming video decoding to prevent temporal dis￾continuity and minimize latency during real-time syn… view at source ↗
Figure 3
Figure 3. Figure 3: Causal residual video auto-encoding. We apply sepa [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The causal block￾wise attention mask. Blockwise attention and autoregressive generation. To facilitate streamable gen￾eration and reduce latency, we configure G to generate latents in an autoregressive manner. In this setup, the content to be generated at each time only attends to the generated context in the past, thus preserving temporal causality. To improve computa￾tional efficiency, we partition the l… view at source ↗
Figure 5
Figure 5. Figure 5: Talking portrait videos generated by our method, which naturally and comprehensively capture portrait video dynamics. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
read the original abstract

Video diffusion models have significantly advanced portrait video generation, yet their high computational demands limit their use in interactive applications. This work presents a framework for streamable talking portrait video generation conditioned on speech audio and reference images. Designed meticulously for streaming scenarios, it features a causal video VAE for deep latent compression and an autoregressive latent denoising model. Our causal VAE integrates a variable number of reference images as guidance, allowing the network to focus on dynamic information rather than static appearance, thereby enhancing compression efficacy and reconstruction quality. Additionally, we extend the residual auto-encoding paradigm to improve spatial-temporal causality handling in our VAE. The generator is based on a Rectified Flow Transformer architecture and produces video latents in a blockwise auto-regressive manner. Our method enables the real-time generation of high-quality talking portrait videos, achieving speeds significantly faster than baseline models. Furthermore, comprehensive experiments demonstrate that it is on par with or even outperforms these large models in realism, vividness, and video quality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper presents a framework for real-time streamable talking portrait video generation conditioned on speech audio and reference images. It introduces a causal video VAE that incorporates a variable number of reference images for deep latent compression (focusing the network on dynamic information) and extends the residual auto-encoding paradigm for improved spatial-temporal causality. Video latents are generated blockwise autoregressively via a Rectified Flow Transformer. The central claims are that the method achieves real-time speeds significantly faster than baseline models while matching or outperforming them in realism, vividness, and video quality, as supported by comprehensive experiments.

Significance. If the performance claims hold with rigorous quantitative validation, the work would address a key limitation of video diffusion models by enabling interactive streaming applications for talking portraits, with the reference-guided causal VAE and blockwise autoregressive design offering a practical path to low-latency generation.

major comments (1)
  1. [Abstract] Abstract: the performance claims (real-time speeds 'significantly faster than baseline models' and quality 'on par with or even outperforms') are stated without any referenced quantitative results, tables, figures, error bars, or ablation studies in the provided text, making it impossible to assess whether the experiments support the claims.
minor comments (2)
  1. The description of how the variable number of reference images is integrated into the causal VAE (e.g., conditioning mechanism or embedding strategy) lacks architectural specifics that would clarify the claimed focus on dynamic information.
  2. Clarify the exact blockwise autoregressive generation process in the Rectified Flow Transformer, including any details on latency or streaming constraints.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their review and for highlighting the need for clearer linkage between abstract claims and supporting evidence. We address the single major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the performance claims (real-time speeds 'significantly faster than baseline models' and quality 'on par with or even outperforms') are stated without any referenced quantitative results, tables, figures, error bars, or ablation studies in the provided text, making it impossible to assess whether the experiments support the claims.

    Authors: The abstract is a concise summary by design and does not embed specific numbers or citations. The full manuscript supports the claims with quantitative results in Section 4 (Experiments), including speed comparisons (Table 1), quality metrics with error bars (Table 2), qualitative figures, and ablation studies (Section 4.3). To address the concern directly, we will revise the abstract to include brief references to these results (e.g., 'achieving 25+ FPS, over 2x faster than baselines per Table 1, with FID and user-study scores on par or superior as shown in Table 2 and Figure 4'). revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper presents an engineering framework for streamable talking portrait video generation via a causal VAE (with reference guidance and extended residual encoding) plus Rectified Flow Transformer for blockwise autoregressive latents. All performance claims (real-time speed, quality parity or superiority) are explicitly tied to experimental validation against baselines rather than any internal derivation, fitted parameter renamed as prediction, or self-citation chain. No equations or load-bearing steps in the abstract or description reduce to self-definitional inputs or prior author work by construction; the contribution is self-contained as an empirical architecture.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are identifiable from the abstract alone.

pith-pipeline@v0.9.1-grok · 5726 in / 1052 out tokens · 31807 ms · 2026-06-28T15:40:10.338169+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

78 extracted references · 26 canonical work pages · 12 internal anchors

  1. [1]

    Gpt-4o system card. 2024. 2, 8

    2024

  2. [2]

    Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023. 2, 8

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    V oice puppetry

    Matthew Brand. V oice puppetry. InProceedings of the 26th annual conference on Computer graphics and interac- tive techniques, pages 21–28, 1999. 3

    1999

  4. [4]

    Video rewrite: Driving visual speech with audio

    Christoph Bregler, Michele Covell, and Malcolm Slaney. Video rewrite: Driving visual speech with audio. InSem- inal Graphics Papers: Pushing the Boundaries, Volume 2, pages 715–722. 2023. 3

    2023

  5. [5]

    Diffusion forcing: Next-token prediction meets full-sequence diffu- sion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Sim- chowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffu- sion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024. 3

    2024

  6. [6]

    arXiv preprint arXiv:2410.10733 , year=

    Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffu- sion models.arXiv preprint arXiv:2410.10733, 2024. 2, 3, 4

    work page arXiv 2024

  7. [7]

    Dc-videogen: Efficient video generation with deep compression video autoencoder

    Junyu Chen, Wenkun He, Yuchao Gu, Yuyang Zhao, Jincheng Yu, Junsong Chen, Dongyun Zou, Yujun Lin, Zhekai Zhang, Muyang Li, et al. Dc-videogen: Efficient video generation with deep compression video autoencoder. arXiv preprint arXiv:2509.25182, 2025. 3

    work page arXiv 2025

  8. [8]

    Lip movements generation at a glance

    Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements generation at a glance. In European Conference on Computer Vision, pages 520–535,

  9. [9]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. InPro- ceedings of the AAAI Conference on Artificial Intelligence, pages 2403–2410, 2025. 3, 8

    2025

  10. [10]

    Out of time: auto- mated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: auto- mated lip sync in the wild. InAsian Conference on Computer Vision Workshops, pages 251–263. Springer, 2017. 8

    2017

  11. [11]

    V oxceleb2: Deep speaker recognition.arXiv preprint arXiv:1806.05622, 2018

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. V oxceleb2: Deep speaker recognition.arXiv preprint arXiv:1806.05622, 2018. 7

    work page arXiv 2018

  12. [12]

    Hallo2: Long-duration and high-resolution audio-driven portrait im- age animation

    Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duration and high-resolution audio-driven portrait im- age animation. InThe Thirteenth International Conference on Learning Representations. 8

  13. [13]

    Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer

    Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu. Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 21086–21095, 2025. 2, 3, 8

    2025

  14. [14]

    Emoportraits: Emotion-enhanced multimodal one-shot head avatars.arXiv preprint arXiv:2404.19110, 2024

    Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos V ougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pan- tic. Emoportraits: Emotion-enhanced multimodal one-shot head avatars.arXiv preprint arXiv:2404.19110, 2024. 3

    work page arXiv 2024

  15. [15]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning,

  16. [16]

    Introducing gemini 2.5 flash image: Our state-of-the-art image model.https://developers

    Alisa Fortin, Guillaume Vernade, Kat Kampf, and Am- maar Reshi. Introducing gemini 2.5 flash image: Our state-of-the-art image model.https://developers. googleblog.com/en/introducing- gemini- 2- 5-flash-image/, 2025. Google Developer Blog. 1, 8

    2025

  17. [17]

    Liveportrait: Efficient portrait animation with stitching and retargeting control,

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Livepor- trait: Efficient portrait animation with stitching and retarget- ing control.arXiv preprint arXiv:2407.03168, 2024. 2

    work page arXiv 2024

  18. [18]

    Space: Speech-driven portrait an- imation with controllable expression

    Siddharth Gururani, Arun Mallya, Ting-Chun Wang, Rafael Valle, and Ming-Yu Liu. Space: Speech-driven portrait an- imation with controllable expression. InProceedings of the ieee/cvf international conference on computer vision, pages 20914–20923, 2023. 3

    2023

  19. [19]

    LTX-Video: Realtime Video Latent Diffusion

    Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion.arXiv preprint arXiv:2501.00103,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Classifier-Free Diffusion Guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. 13

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  21. [21]

    Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  22. [22]

    Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion

    Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train- test gap in autoregressive video diffusion.arXiv preprint arXiv:2506.08009, 2025. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    Eamm: One-shot emotional talking face via audio-based emotion-aware motion model

    Xinya Ji, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Wayne Wu, Feng Xu, and Xun Cao. Eamm: One-shot emotional talking face via audio-based emotion-aware motion model. InACM SIGGRAPH 2022 conference proceedings, pages 1– 10, 2022. 3

    2022

  24. [24]

    Sonic: Shifting focus to global audio perception in portrait anima- tion

    Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chum- ing Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, Qinglin Lu, and Chengjie Wang. Sonic: Shifting focus to global audio perception in portrait anima- tion. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 193– 203, 2025. 2, 3, 8

    2025

  25. [25]

    Auto-encoding vari- ational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding vari- ational bayes, 2013. 5

    2013

  26. [26]

    Tokenmotion: Decoupled motion control via token disentanglement for human-centric video generation

    Ruineng Li, Daitao Xing, Huiming Sun, Yuanzhou Ha, Jinglin Shen, and Chiuman Ho. Tokenmotion: Decoupled motion control via token disentanglement for human-centric video generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 1951–1961, 2025. 3

    1951

  27. [27]

    Expressive talking head generation with granular audio-visual control

    Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, 9 and Jingdong Wang. Expressive talking head generation with granular audio-visual control. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3387–3396, 2022. 3

    2022

  28. [28]

    Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models.arXiv preprint arXiv:2502.01061, 2025

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang. Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models.arXiv preprint arXiv:2502.01061, 2025. 2, 3

    work page arXiv 2025

  29. [29]

    Diffusion adversarial post-training for one-step video generation.arXiv preprint arXiv:2501.08316, 2025a

    Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversarial post-training for one-step video generation.arXiv preprint arXiv:2501.08316,

    work page arXiv

  30. [30]

    Autoregressive adversarial post-training for real-time inter- active video generation.arXiv preprint arXiv:2506.09350,

    Shanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang, Yuxi Ren, Xin Xia, Yang Zhao, Xuefeng Xiao, and Lu Jiang. Autoregressive adversarial post-training for real-time inter- active video generation.arXiv preprint arXiv:2506.09350,

    work page arXiv

  31. [31]

    Flow Matching for Generative Modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022. 2, 3, 5

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  32. [32]

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022. 3, 5

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  33. [33]

    Talkingmachines: Real- time audio-driven facetime-style video via autoregressive diffusion models.arXiv preprint arXiv:2506.03099, 2025

    Chetwin Low and Weimin Wang. Talkingmachines: Real- time audio-driven facetime-style video via autoregressive diffusion models.arXiv preprint arXiv:2506.03099, 2025. 3

    work page arXiv 2025

  34. [34]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  35. [35]

    Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k

    Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young, Xinying Guo, Binluo Wang, Hang Xu, Hongxin Liu, Mingyan Jiang, Wenjun Li, et al. Open-sora 2.0: Training a commercial-level video generation model in $200 k.arXiv preprint arXiv:2503.09642, 2025. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nambood- iri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. InACM International Conference on Multimedia, pages 484–492, 2020. 2, 3, 5, 15

    2020

  37. [37]

    Chatanyone: Stylized real-time portrait video generation with hierarchical motion diffusion model

    Jinwei Qi, Chaonan Ji, Sheng Xu, Peng Zhang, Bang Zhang, and Liefeng Bo. Chatanyone: Stylized real-time portrait video generation with hierarchical motion diffusion model. arXiv preprint arXiv:2503.21144, 2025. 3

    work page arXiv 2025

  38. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 3, 5, 8

    2022

  39. [39]

    Fast high- resolution image synthesis with latent adversarial diffusion distillation

    Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high- resolution image synthesis with latent adversarial diffusion distillation. InSIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024. 3

    2024

  40. [40]

    Adversarial diffusion distillation

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. InEuropean Conference on Computer Vision, pages 87–103. Springer,

  41. [41]

    Score-Based Generative Modeling through Stochastic Differential Equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions.arXiv preprint arXiv:2011.13456, 2020. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2011

  42. [42]

    Diffused heads: Diffusion models beat gans on talking-face genera- tion

    Michał Stypułkowski, Konstantinos V ougioukas, Sen He, Maciej Zi˛ eba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face genera- tion. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5091–5100, 2024. 2

    2024

  43. [43]

    Synthesizing obama: learn- ing lip sync from audio.ACM Transactions on Graphics, 36 (4):1–13, 2017

    Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing obama: learn- ing lip sync from audio.ACM Transactions on Graphics, 36 (4):1–13, 2017. 3

    2017

  44. [44]

    MAGI-1: Autoregressive Video Generation at Scale

    Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Mingqiu Tang, Shuai Han, Tianning Zhang, WQ Zhang, Weifeng Luo, et al. Magi-1: Autoregressive video genera- tion at scale.arXiv preprint arXiv:2505.13211, 2025. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  45. [45]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision, pages 244–260. Springer, 2024. 2, 3

    2024

  46. [46]

    Reducio! generating 1k video within 16 seconds using extremely compressed mo- tion latents

    Rui Tian, Qi Dai, Jianmin Bao, Kai Qiu, Yifan Yang, Chong Luo, Zuxuan Wu, and Yu-Gang Jiang. Reducio! generating 1k video within 16 seconds using extremely compressed mo- tion latents. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19237–19247, 2025. 3

    2025

  47. [47]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019. 8

    2019

  48. [48]

    Conditional image genera- tion with pixelcnn decoders.Advances in neural information processing systems, 29, 2016

    Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image genera- tion with pixelcnn decoders.Advances in neural information processing systems, 29, 2016. 3

    2016

  49. [49]

    Pixel recurrent neural networks

    Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. InInterna- tional conference on machine learning, pages 1747–1756. PMLR, 2016. 3

    2016

  50. [50]

    Wan: Open and Advanced Large-Scale Video Generative Models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianx- iao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jin- gren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fan...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  51. [51]

    Progressive disentangled representation 10 learning for fine-grained controllable talking head synthesis

    Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, and Baoyuan Wang. Progressive disentangled representation 10 learning for fine-grained controllable talking head synthesis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17979–17989, 2023. 2, 3

    2023

  52. [52]

    Echoshot: Multi-shot portrait video generation

    Jiahao Wang, Hualian Sheng, Sijia Cai, Weizhan Zhang, Caixia Yan, Yachuang Feng, Bing Deng, and Jieping Ye. Echoshot: Multi-shot portrait video generation. InThe Thirty-ninth Annual Conference on Neural Information Pro- cessing Systems, 2025. 3

    2025

  53. [53]

    Fanta- sytalking: Realistic talking portrait generation via coherent motion synthesis

    Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yun- peng Zhang, Yonggang Qi, Kun Zhao, and Mu Xu. Fanta- sytalking: Realistic talking portrait generation via coherent motion synthesis. InProceedings of the 33rd ACM Interna- tional Conference on Multimedia, pages 9891–9900, 2025. 2, 3, 8

    2025

  54. [54]

    Audio2head: Audio-driven one-shot talking-head gener- ation with natural head motion

    Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. Audio2head: Audio-driven one-shot talking-head gener- ation with natural head motion. InInternational Joint Con- ference on Artificial Intelligence, 2021. 2, 3

    2021

  55. [55]

    One-shot free-view neural talking-head synthesis for video conferenc- ing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head synthesis for video conferenc- ing. InIEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10039–10049, 2021. 2

    2021

  56. [56]

    Aniportrait: Audio- driven synthesis of photorealistic portrait animation,

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024. 2, 3

    work page arXiv 2024

  57. [57]

    A learning algorithm for continually running fully recurrent neural networks.Neu- ral computation, 1(2):270–280, 1989

    Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks.Neu- ral computation, 1(2):270–280, 1989. 5

    1989

  58. [58]

    Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024

    Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, and Siyu Zhu. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation.arXiv preprint arXiv:2406.08801, 2024. 2, 8

    work page arXiv 2024

  59. [59]

    Vasa-1: Lifelike audio-driven talking faces generated in real time.Advances in Neural Information Pro- cessing Systems, 37:660–684, 2024

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time.Advances in Neural Information Pro- cessing Systems, 37:660–684, 2024. 2, 3, 8, 13

    2024

  60. [60]

    VideoGPT: Video Generation using VQ-VAE and Transformers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and trans- formers.arXiv preprint arXiv:2104.10157, 2021. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  61. [61]

    CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024. 3, 5, 8

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  62. [62]

    Real3d-portrait: One-shot realistic 3d talking portrait synthesis.arXiv preprint arXiv:2401.08503,

    Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, et al. Real3d-portrait: One-shot realistic 3d talking portrait synthesis.arXiv preprint arXiv:2401.08503,

    work page arXiv

  63. [63]

    Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan

    Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. InEuropean Conference on Computer Vision, pages 85–101, 2022. 3

    2022

  64. [64]

    Im- proved distribution matching distillation for fast image syn- thesis

    Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Im- proved distribution matching distillation for fast image syn- thesis. InNeurIPS, 2024. 2, 3

    2024

  65. [65]

    One-step diffusion with distribution matching distillation

    Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shecht- man, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 6613–6623, 2024. 3

    2024

  66. [66]

    One-step diffusion with distribution matching distillation

    Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shecht- man, Frédo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In CVPR, 2024. 2

    2024

  67. [67]

    From slow bidirectional to fast autoregressive video diffusion mod- els

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T Free- man, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion mod- els. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 22963–22974, 2025. 3

    2025

  68. [68]

    An image is worth 32 tokens for reconstruction and generation.Advances in Neural Information Processing Systems, 37:128940– 128966, 2024

    Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation.Advances in Neural Information Processing Systems, 37:128940– 128966, 2024. 3

    2024

  69. [69]

    Efficient video diffusion mod- els via content-frame motion-latent decomposition.arXiv preprint arXiv:2403.14148, 2024

    Sihyun Yu, Weili Nie, De-An Huang, Boyi Li, Jinwoo Shin, and Anima Anandkumar. Efficient video diffusion mod- els via content-frame motion-latent decomposition.arXiv preprint arXiv:2403.14148, 2024. 3

    work page arXiv 2024

  70. [70]

    Talking head generation with probabilistic audio-to-visual diffusion priors

    Zhentao Yu, Zixin Yin, Deyu Zhou, Duomin Wang, Finn Wong, and Baoyuan Wang. Talking head generation with probabilistic audio-to-visual diffusion priors. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 7645–7655, 2023. 3

    2023

  71. [71]

    Identity- preserving text-to-video generation by frequency decompo- sition

    Shenghai Yuan, Jinfa Huang, Xianyi He, Yunyang Ge, Yu- jun Shi, Liuhan Chen, Jiebo Luo, and Li Yuan. Identity- preserving text-to-video generation by frequency decompo- sition. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 12978–12988, 2025. 3

    2025

  72. [72]

    Root mean square layer nor- malization

    Biao Zhang and Rico Sennrich. Root mean square layer nor- malization. InAdvances in Neural Information Processing Systems. Curran Associates, Inc., 2019. 4

    2019

  73. [73]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 5

    2018

  74. [74]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio- driven single image talking face animation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8652–8661, 2023. 2, 3, 13

    2023

  75. [75]

    Flow-guided one-shot talking face generation with a high- resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high- resolution audio-visual dataset. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661– 3670, 2021. 7

    2021

  76. [76]

    11 Taming teacher forcing for masked autoregressive video gen- eration

    Deyu Zhou, Quan Sun, Yuang Peng, Kun Yan, Runpei Dong, Duomin Wang, Zheng Ge, Nan Duan, and Xiangyu Zhang. 11 Taming teacher forcing for masked autoregressive video gen- eration. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 7374–7384, 2025. 2

    2025

  77. [77]

    Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation

    Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose-controllable talking face generation by implicitly modularized audio-visual rep- resentation. InProceedings of the IEEE/CVF Conference on computer Vision and Pattern Recognition, pages 4176–4186,

  78. [78]

    split-first

    Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevar- ria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation.ACM Transactions On Graphics (TOG), 39(6):1–15, 2020. 3 12 A. More Results and Comparisons Audio-driven talking head comparison.We also com- pare with several audio-driven talking head methods, in- cluding SadT...

    2020