Pith. sign in

REVIEW 4 major objections 5 minor 60 references

Multi-human Interactive Talking Dataset

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces MIT, the first large-scale dataset for multi-human talking video generation, and CovOG, a baseline that outperforms single-person methods.

desk verdict A genuinely new full-body multi-person talking dataset with a workable baseline, but the benchmark's reliability rests on annotation alignment that the paper asserts without quantifying. read the letter →

arxiv 2508.03050 v1 pith:FO75J4TI submitted 2025-08-05 cs.CV

Zeyu Zhu , Weijia Wu , Mike Zheng Shou This is my paper
classification cs.CV
keywords multi-humantalkingvideogenerationdatasetmulti-speakerinteractionposeannotationspeakingscoreaudio-drivendiffusionmodelconversationalbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that multi-human talking video generation is a distinct and tractable task, and that the field now has a benchmark to work with. It introduces MIT, a 12-hour, high-resolution collection of two-to-four-speaker conversational clips with full-body pose annotations and per-speaker speaking scores, built by an automatic pipeline. It also presents CovOG, a diffusion-based baseline that encodes each speaker's pose separately and uses speaking scores to modulate audio-driven facial motion, and reports that it beats single-person baselines on the new test splits. A sympathetic reader would care because realistic conversations in interviews, panels, and shows require exactly these multi-party interaction dynamics, which previous datasets and models did not capture.

What carries the argument

The dataset side rests on an alignment-by-bounding-box scheme: YOLOv7 boxes give each individual a shared spatial reference, pose keypoints are assigned to the box containing the most keypoints, and face tracks are matched to the box with the highest overlap frequency, so pose and speaking score stay attached to one person. The model side rests on two modules: the Multi-Human Pose Encoder (MPE), a shared convolutional network that extracts each person's pose embedding independently and sums them to handle a variable number of speakers, and the Interactive Audio Driver (IAD), which scales each speaker's audio embedding by their speaking score and applies masked cross-attention to facial features to drive speaking-listening transitions. Together they carry the argument that per-person control plus a speaking-status signal is enough to extend single-person animation to multi-person dialogue.

What would settle it

Manually label who is speaking and where each person is in roughly 50 randomly selected MIT clips that contain overlapping speech or occlusions, then measure how often the pipeline's per-person assignment disagrees with the manual labels; if disagreement exceeds a few percent of frames, the dataset labels and CovOG's reported gains lose reliability. Alternatively, train CovOG with shuffled speaking-score assignments and check whether its audio-visual alignment score changes; if it does not, the score channel is not carrying the claimed interaction signal.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that an automatic pipeline can turn raw talk-show footage into a reliable multi-person talking-video benchmark: shot segmentation isolates continuous clips, a pose estimator supplies 2D skeletons, a speech-activity model supplies per-speaker speaking scores, and person bounding boxes align the two signals to the correct individual. The paper further claims that CovOG, by aggregating per-person pose embeddings and cross-attending audio features weighted by speaking scores onto facial regions, produces natural turn-taking and listening reactions, and that it outperforms AnimateAnyone and ControlSVD on quantitative metrics and in user studies.

Load-bearing premise

The load-bearing premise is that the automatic pipeline assigns every pose and speaking score to the correct person in every frame, a claim validated on only 20 videos and without a reported pose-alignment error rate.

Editorial extensions

If this is right

  • Existing single-person animation frameworks can be extended to multi-person scenes by per-person pose encoding, without redesigning the generation backbone.
  • The speaking-score channel becomes a practical control signal for driving both who is speaking and how listeners react, applicable to dyadic and small-group avatars.
  • The automatic collection pipeline means the 12-hour dataset can be scaled up from additional talk-show or interview footage with minimal manual effort.
  • The benchmark's easy and challenging test splits give the community a standard protocol for comparing multi-human talking-video models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the speaking score alone, without emotion labels, may carry enough signal to drive listener reactions; this could be tested by ablating score magnitude versus score sign.
  • The overlap-based alignment heuristic likely weakens exactly where conversations get interesting—overlapping speech, side views, and occlusions—so future work should report per-frame identity-assignment accuracy on those segments.
  • Because the data comes from two static-camera talk shows, the benchmark measures interaction rather than cinematography; a natural extension is testing how well models trained on MIT transfer to multi-camera edited content after shot-boundary filtering.
  • Since standard lip-sync metrics fail on side-facing speakers, a curated side-view speaking subset derived from MIT could become a testbed for interactive lip-synchronization evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces MIT, a dataset of 12 hours of high-resolution two-to-four-speaker conversational videos with per-person body pose annotations and speaking scores, built through an automated pipeline (WhisperV for shot segmentation, Sapiens-2B for poses, TalkNet for speech activity, YOLOv7 for spatial alignment). The authors also propose CovOG, an AnimateAnyone-based baseline with a Multi-Human Pose Encoder (MPE) and Interactive Audio Driver (IAD), and evaluate it against AnimateAnyone and ControlSVD using SSIM/PSNR/FVD, a user study, and a cross-modal VBench experiment. The paper claims that this is the first full-body multi-speaker benchmark for talking video generation and that CovOG outperforms both baselines across all metrics.

Significance. If the dataset and its annotations are reliable, MIT would be a useful first benchmark for full-body multi-human talking video generation, filling a clear gap relative to single-person and head-only datasets. The automatic collection pipeline and the explicit modeling of interaction patterns are valuable contributions. The baseline is described clearly, the code is promised, and the cross-modal experiment is a sensible addition. However, the central value depends on trust in the per-person pose/speech alignment, which is currently supported by very thin validation; the evaluation tables also lack variance and have an incomplete baseline comparison. The dataset itself is not yet publicly accessible, which limits immediate verification by the community.

major comments (4)
  1. [§3.1–3.2] The pose-to-person assignment in §3.1 is based on which YOLOv7 box contains the most Sapiens keypoints, and the face-to-person match is based on aggregate overlap frequency; these heuristics can produce identity swaps during rapid turn-taking, overlapping speech, or partial occlusion. The validation reported in §3.2 is limited to 20 testing videos, provides no quantitative pose-error or identity-assignment error numbers, gives no protocol for the claim that 'pose–speaking alignments of all samples are correct,' and reports the 0.1 s transition error without sample size or variance. Because both the benchmark labels and the CovOG performance numbers depend on this alignment, the paper needs a quantitative alignment study on a larger and more diverse sample, with per-condition error rates and illustrative failure cases.
  2. [Table 2 and §5.3] The claim that CovOG 'consistently outperforms both baselines across all metrics' is not fully supported: ControlSVD has no multi-human or all-test entries, and the table reports only point estimates without variance, confidence intervals, or the number of clips in each condition. The SSIM differences are small (e.g., 0.62 vs 0.60), so statistical significance should be reported. Please complete the ControlSVD comparison on the multi-human split and add sample counts and variance measures.
  3. [§1 and Abstract] The dataset is the central contribution, but the paper only provides a code URL and does not state where or how the MIT dataset can be accessed, downloaded, or licensed. Without dataset availability, the benchmark cannot be used or verified by others. The manuscript should include a clear dataset-release statement with a URL, download instructions, license, and any access restrictions.
  4. [Table 3 and §5.3 User Study] The user study uses seven raters and 10 randomly selected samples per method, with no inter-rater agreement, confidence intervals, or significance tests reported. Since the paper relies on the user study to compensate for the lack of an appropriate lip-sync metric, this evidence is too thin as presented. Please report per-rater variance and agreement statistics, or increase the rater and sample counts.
minor comments (5)
  1. [Throughout] There are several typos, including 'Mulit-human' in the abstract, 'furthur', 'avalibale', 'Datset' in the Section 3 title, 'Multi-huamn' in Section 5.6, and 'exiting' in the introduction; these should be corrected.
  2. [References [3]] Reference [3] is cited for TalkNet, but the cited paper is a speech synthesis model; the active-speaker-detection TalkNet has a different source. Please correct this citation for reproducibility.
  3. [References [36]] Reference [36] is cited for MovieGen in Section 2.1, but the listed paper is 'Mediconfusion'; please replace it with the correct MovieGen reference.
  4. [Table 2 and §5.1] Section 5.1 defines easy and challenging test cases, but Table 2 does not report them separately; please clarify how the 'Two Human' and 'Multiple Human' splits relate to the easy/challenging distinction.
  5. [Tables 2–4] The quantitative tables would benefit from standard deviations or confidence intervals; consider adding them to support the reported differences.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: dataset construction and baseline evaluation are self-contained against external tools and metrics.

full rationale

The paper's derivation chain is not circular. The MIT dataset is built by an automatic pipeline whose components (Sapiens-2B pose estimation, TalkNet speech activity detection, YOLOv7 bounding boxes, and WhisperV shot/face-track segmentation) are external systems or openly available tools; none of their outputs is fitted to the downstream benchmark claim. The central comparison claim that CovOG 'consistently outperforms both baselines across all metrics' is evaluated with SSIM, PSNR, FVD, VBench, and user studies, all external to the model's training objectives. The MPE/IAD ablations compare model variants on the same held-out test splits, so the reported differences are not forced by construction. The use of WhisperV [17], a same-lab tool, is an implementation dependency in data preprocessing rather than a load-bearing self-citation or uniqueness argument; no equation reduces a prediction to its own input. The unquantified pose-speech alignment accuracy discussed in Section 3.2 is a validity and robustness concern that could affect label quality, but it is not circularity: the paper does not define alignment accuracy in terms of the model's reported performance, nor does it rename a fitted parameter as a prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No scalar free parameters are fitted in the paper: model weights are learned and no explicit threshold or scaling constant is reported. The central claims depend on the four domain assumptions above, all concerning automatic annotation quality. No new physical or conceptual entities are introduced.

assumptions (4)
  • domain assumption Sapiens-2B keypoints are accurate enough on talk-show footage to serve as ground truth for pose annotation.
    Section 3.1 Pose Annotation states these are pseudo-labels and says validation on 20 videos found them sufficiently accurate, but no quantitative accuracy numbers are reported.
  • domain assumption TalkNet speaking scores correctly identify speaking and listening states for each tracked face.
    Section 3.1 Speaking Score uses TalkNet; Section 3.2 reports manual transition-point checks below 0.1 second without giving sample sizes or error distributions.
  • domain assumption YOLOv7 human boxes plus face-track overlap uniquely and correctly assign poses and speaking scores to each individual.
    Section 3.1 Pose-Speech Alignment: assignments are made by highest keypoint count and most frequent overlap; the paper asserts all alignments are correct but gives no failure statistics for occluded or overlapping speakers.
  • domain assumption Three head keypoints and audio features are sufficient to represent facial dynamics for multi-human talking generation.
    Section 3.1 Pose Annotation deliberately keeps only three head keypoints because finer facial expressions are said to be audio-driven; this design choice is not independently validated for side-facing speech.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-human Interactive Talking Dataset." pith.science (2026). https://pith.science/paper/FO75J4TI

@misc{pith2026250803050,
  author       = {Pith},
  title        = {Pith review of: Multi-human Interactive Talking Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FO75J4TI}},
  note         = {Machine review of arXiv:2508.03050}
}
read the original abstract

Existing studies on talking video generation have predominantly focused on single-person monologues or isolated facial animations, limiting their applicability to realistic multi-human interactions. To bridge this gap, we introduce MIT, a large-scale dataset specifically designed for multi-human talking video generation. To this end, we develop an automatic pipeline that collects and annotates multi-person conversational videos. The resulting dataset comprises 12 hours of high-resolution footage, each featuring two to four speakers, with fine-grained annotations of body poses and speech interactions. It captures natural conversational dynamics in multi-speaker scenario, offering a rich resource for studying interactive visual behaviors. To demonstrate the potential of MIT, we furthur propose CovOG, a baseline model for this novel task. It integrates a Multi-Human Pose Encoder (MPE) to handle varying numbers of speakers by aggregating individual pose embeddings, and an Interactive Audio Driver (IAD) to modulate head dynamics based on speaker-specific audio features. Together, these components showcase the feasibility and challenges of generating realistic multi-human talking videos, establishing MIT as a valuable benchmark for future research. The code is avalibale at: https://github.com/showlab/Multi-human-Talking-Video-Dataset.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 31 canonical work pages

  1. [1]

    GitHub repository

    Moore-animateanyone. GitHub repository. URL https://github.com/MooreThreads/ Moore-AnimateAnyone

    work page

  2. [2]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460, 2020

    work page 2020

  3. [3]

    TalkNet: Fully-Convolutional Non-Autoregressive Speech Synthesis Model

    Stanislav Beliaev, Yurii Rebryk, and Boris Ginsburg. Talknet: Fully-convolutional non- autoregressive speech synthesis model. arXiv preprint arXiv:2005.05514, 2020

    work page Pith review arXiv 2005

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  5. [5]

    Magicdance: Realistic human dance video generation with motions & facial expressions transfer

    Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Xiao Yang, and Mohammad Soleymani. Magicdance: Realistic human dance video generation with motions & facial expressions transfer. CoRR, 2023

    work page 2023

  6. [6]

    Videocrafter1: Open diffusion models for high-quality video generation

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512, 2023

    arXiv

  7. [7]

    Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions

    Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. Echomimic: Lifelike audio-driven portrait animations through editable landmark conditions. arXiv preprint arXiv:2407.08136, 2024

    arXiv 2024

  8. [8]

    Out of time: automated lip sync in the wild

    Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 251–263. Springer, 2017

    2016

Show all 60 references
  1. [9]

    V oxceleb2: Deep speaker recognition

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. V oxceleb2: Deep speaker recognition. arXiv preprint arXiv:1806.05622, 2018

    2018 arXiv

  2. [10]

    Hallo2: Long-duration and high-resolution audio-driven portrait image animation

    Jiahao Cui, Hui Li, Yao Yao, Hao Zhu, Hanlin Shang, Kaihui Cheng, Hang Zhou, Siyu Zhu, and Jingdong Wang. Hallo2: Long-duration and high-resolution audio-driven portrait image animation. arXiv preprint arXiv:2410.07718, 2024

    2024 arXiv

  3. [11]

    Dreamoving: A human dance video generation framework based on diffusion models

    Mengyang Feng, Jinlin Liu, Kai Yu, Yuan Yao, Zheng Hui, Xiefan Guo, Xianhui Lin, Haolan Xue, Chen Shi, Xiaowen Li, et al. Dreamoving: A human dance video generation framework based on diffusion models. arXiv preprint arXiv:2312.05107, 2023

    2023 arXiv

  4. [12]

    Affective faces for goal-driven dyadic communication

    Scott Geng, Revant Teotia, Purva Tendulkar, Sachit Menon, and Carl V ondrick. Affective faces for goal-driven dyadic communication. arXiv preprint arXiv:2301.10939, 2023

    2023 arXiv

  5. [13]

    Learning individual styles of conversational gesture

    Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Ma- lik. Learning individual styles of conversational gesture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3497–3506, 2019

    2019

  6. [14]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  7. [15]

    Co-speech gesture video generation via motion- decoupled diffusion model

    Xu He, Qiaochu Huang, Zhensong Zhang, Zhiwei Lin, Zhiyong Wu, Sicheng Yang, Minglei Li, Zhiyi Chen, Songcen Xu, and Xiaofei Wu. Co-speech gesture video generation via motion- decoupled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  8. [16]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153–8163, 2024

    2024

  9. [17]

    Whisperv

    Siyuan Hu, Kevin Qinghong Lin, Difei Gao, and Mike Zheng Shou. Whisperv. GitHub repository, 2023. URL https://github.com/showlab/whisperV. 10

    2023

  10. [18]

    Perceptual conversational head generation with regularized driver and enhanced renderer

    Ailin Huang, Zhewei Huang, and Shuchang Zhou. Perceptual conversational head generation with regularized driver and enhanced renderer. In Proceedings of the 30th ACM international conference on multimedia, pages 7050–7054, 2022

    2022

  11. [19]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  12. [20]

    Loopy: Taming audio-driven portrait avatar with long-term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.02634, 2024

    2024 arXiv

  13. [21]

    Text2performer: Text-driven human video generation

    Yuming Jiang, Shuai Yang, Tong Liang Koh, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2performer: Text-driven human video generation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 22747–22757, 2023

    2023

  14. [22]

    Whole-body human pose estimation in the wild

    Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16 , pages 196–214. Springer, 2020

    2020

  15. [23]

    Sapiens: Foundation for human vision models

    Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models. In European Conference on Computer Vision, pages 206–228. Springer, 2024

    2024

  16. [24]

    A comprehensive survey on human video generation: Challenges, methods, and insights

    Wentao Lei, Jinting Wang, Fengji Ma, Guanjie Huang, and Li Liu. A comprehensive survey on human video generation: Challenges, methods, and insights. arXiv preprint arXiv:2407.08428, 2024

    2024 arXiv

  17. [25]

    Openhumanvid: A large-scale high-quality dataset for enhancing human-centric video generation

    Hui Li, Mingwang Xu, Yun Zhan, Shan Mu, Jiaye Li, Kaihui Cheng, Yuxuan Chen, Tan Chen, Mao Ye, Jingdong Wang, et al. Openhumanvid: A large-scale high-quality dataset for enhancing human-centric video generation. arXiv preprint arXiv:2412.00115, 2024

    2024 arXiv

  18. [26]

    Tango: Co-speech gesture video reenactment with hierarchical audio motion embedding and diffusion interpolation

    Haiyang Liu, Xingchao Yang, Tomoya Akiyama, Yuantian Huang, Qiaoge Li, Shigeru Kuriyama, and Takafumi Taketomi. Tango: Co-speech gesture video reenactment with hierarchical audio motion embedding and diffusion interpolation. arXiv preprint arXiv:2410.04221, 2024

    arXiv

  19. [27]

    Customlistener: Text-guided responsive interaction for user-friendly listening head generation

    Xi Liu, Ying Guo, Cheng Zhen, Tong Li, Yingying Ao, and Pengfei Yan. Customlistener: Text-guided responsive interaction for user-friendly listening head generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2415–2424, 2024

    2024

  20. [28]

    Learning hierarchical cross-modal association for co-speech gesture generation

    Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2022

  21. [29]

    Follow your pose: Pose-guided text-to-video generation using pose-free videos

    Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4117–4125, 2024

    2024

  22. [30]

    Learning to listen: Modeling non-deterministic dyadic facial motion

    Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. Learning to listen: Modeling non-deterministic dyadic facial motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20395–20405, 2022

    2022

  23. [31]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  24. [32]

    Con- trolnext: Powerful and efficient control for image and video generation

    Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming-Chang Yang, and Jiaya Jia. Con- trolnext: Powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.06070, 2024. 11

    2024 arXiv

  25. [33]

    A lip sync expert is all you need for speech to lip generation in the wild

    KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484–492, 2020

    2020

  26. [34]

    Speech drives templates: Co- speech gesture synthesis with learned templates

    Shenhan Qian, Zhi Tu, Yihao Zhi, Wen Liu, and Shenghua Gao. Speech drives templates: Co- speech gesture synthesis with learned templates. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11077–11086, 2021

    2021

  27. [35]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  28. [36]

    Mediconfusion: Can you trust your ai radiologist? probing the reliability of multimodal medical foundation models

    Mohammad Shahab Sepehri, Zalan Fabian, Maryam Soltanolkotabi, and Mahdi Soltanolkotabi. Mediconfusion: Can you trust your ai radiologist? probing the reliability of multimodal medical foundation models. arXiv preprint arXiv:2409.15477, 2024

    2024 arXiv

  29. [37]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022

    2022 arXiv

  30. [38]

    Lip reading sentences in the wild

    Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6447–6456, 2017

    2017

  31. [39]

    Directorllm for human-centric video generation

    Kunpeng Song, Tingbo Hou, Zecheng He, Haoyu Ma, Jialiang Wang, Animesh Sinha, Sam Tsai, Yaqiao Luo, Xiaoliang Dai, Li Chen, et al. Directorllm for human-centric video generation. arXiv preprint arXiv:2412.14484, 2024

    2024 arXiv

  32. [40]

    Diffused heads: Diffusion models beat gans on talking-face generation

    Michał Stypułkowski, Konstantinos V ougioukas, Sen He, Maciej Zi˛ eba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5091–5100, 2024

    2024

  33. [41]

    Multitalk: Enhancing 3d talking head generation across languages with multilingual video dataset

    Kim Sung-Bin, Lee Chae-Yeon, Gihun Son, Oh Hyun-Bin, Janghoon Ju, Suekyeong Nam, and Tae-Hyun Oh. Multitalk: Enhancing 3d talking head generation across languages with multilingual video dataset. arXiv preprint arXiv:2406.14272, 2024

    2024 arXiv

  34. [42]

    Edtalk: Efficient disentanglement for emotional talking head synthesis

    Shuai Tan, Bin Ji, Mengxiao Bi, and Ye Pan. Edtalk: Efficient disentanglement for emotional talking head synthesis. In European Conference on Computer Vision, pages 398–416. Springer, 2024

    2024

  35. [43]

    Dyadic interaction modeling for social behavior generation

    Minh Tran, Di Chang, Maksim Siniukov, and Mohammad Soleymani. Dyadic interaction modeling for social behavior generation. arXiv preprint arXiv:2403.09069, 2024

    2024 arXiv

  36. [44]

    Realistic speech-driven facial animation with gans

    Konstantinos V ougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial animation with gans. International Journal of Computer Vision, 128(5):1398–1413, 2020

    2020

  37. [45]

    Agentavatar: Disentangling planning, driving and rendering for photorealistic avatar agents

    Duomin Wang, Bin Dai, Yu Deng, and Baoyuan Wang. Agentavatar: Disentangling planning, driving and rendering for photorealistic avatar agents. arXiv preprint arXiv:2311.17465, 2023

    2023 arXiv

  38. [46]

    Emotivetalk: Expressive talking head generation through audio information decoupling and emotional video diffusion

    Haotian Wang, Yuzhe Weng, Yueyan Li, Zilu Guo, Jun Du, Shutong Niu, Jiefeng Ma, Shan He, Xiaoyan Wu, Qiming Hu, et al. Emotivetalk: Expressive talking head generation through audio information decoupling and emotional video diffusion. arXiv preprint arXiv:2411.16726, 2024

    2024 arXiv

  39. [47]

    Mead: A large-scale audio-visual dataset for emotional talking-face generation

    Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. In European Conference on Computer Vision, pages 700–717. Springer, 2020

    2020

  40. [48]

    Draganything: Motion control for anything using entity representation

    Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In European Conference on Computer Vision, pages 331–348, 2024. 12

    2024

  41. [49]

    Magicanimate: Temporally consistent human image animation using diffusion model

    Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  42. [50]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. CoRR, 2024

    2024

  43. [51]

    Show-1: Marrying pixel and latent diffusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. International Journal of Computer Vision, pages 1–15, 2024

    2024

  44. [52]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023

    2023

  45. [53]

    Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation

    Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2023

  46. [54]

    Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset

    Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021

    2021

  47. [55]

    Responsive listening head generation: a benchmark dataset and baseline

    Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei. Responsive listening head generation: a benchmark dataset and baseline. In European Conference on Computer Vision, pages 124–142. Springer, 2022

    2022

  48. [56]

    Interactive conversational head generation

    Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, and Tiejun Zhao. Interactive conversational head generation. arXiv preprint arXiv:2307.02090, 2023

    2023 arXiv

  49. [57]

    Audio-driven neural gesture reenactment with video motion graphs

    Yang Zhou, Jimei Yang, Dingzeyu Li, Jun Saito, Deepali Aneja, and Evangelos Kalogerakis. Audio-driven neural gesture reenactment with video motion graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3418–3428, 2022

    2022

  50. [58]

    Celebv-hq: A large-scale video facial attributes dataset

    Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale video facial attributes dataset. In European conference on computer vision, pages 650–667. Springer, 2022

    2022

  51. [59]

    Taming diffusion models for audio-driven co-speech gesture generation

    Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for audio-driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10544–10553, 2023

    2023

  52. [60]

    Infp: Audio-driven interactive head generation in dyadic conversations

    Yongming Zhu, Longhao Zhang, Zhengkun Rong, Tianshu Hu, Shuang Liang, and Zhipeng Ge. Infp: Audio-driven interactive head generation in dyadic conversations. arXiv preprint arXiv:2412.04037, 2024. 13

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.