Pith. sign in

REVIEW 2 major objections 2 minor 3 cited by

ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body

T0 review · 2 major / 2 minor · reviewed 2026-05-16 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ViBES builds a 3D conversational agent that jointly plans language, prosody, and body movements from speech or text inputs.

desk verdict ViBES tries joint language-and-body planning in a 3D conversational agent via modality-routed experts, but the abstract gives no numbers to show it works. read the letter →

arxiv 2512.14234 v2 submitted 2025-12-16 cs.CV

Juze Zhang , Changan Chen , Xin Chen , Heng Yu , Tiange Xiang , Ali Sartaz Khan , Shrinidhi K. Lakshmikanth , Ehsan Adeli This is my paper · ORCID
classification cs.CV
keywords conversationalagent3Dvirtualbodymultimodalgenerationco-speechmotionmixtureofexpertsspeech-language-behaviormodelagenticinteraction
checked against Cost.FunctionalEquationembed_strictMono_of_one_lt
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents ViBES as a speech-language-behavior model that moves past isolated text-to-motion or co-speech gesture generation. It uses a mixture-of-modality-experts transformer where speech, facial, and body experts process interleaved tokens with hard routing per modality and cross-expert attention to share context. This design lets the agent decide when to move, adapt across dialogue turns, and respond to mixed user inputs like spoken words, typed text, or mid-conversation body directives. The result is measured through improved alignment between generated dialogue and 3D actions on multi-turn benchmarks.

What carries the argument

Mixture-of-modality-experts (MoME) backbone that applies hard routing by modality to separate transformer experts for speech, facial expression, and body motion while sharing information via cross-expert attention on interleaved token streams.

What would settle it

A multi-turn dialogue test where the agent produces body motions that contradict the spoken content or timing after three or more turns, showing loss of joint planning.

Watch

Extended reading notes

Core claim

ViBES jointly generates language and 3D body actions by processing interleaved multimodal token streams through modality-partitioned transformer experts connected by cross-expert attention, enabling agentic planning of when and how to act during conversation rather than mapping fixed utterances to motion clips.

Load-bearing premise

Hard routing by modality plus cross-expert attention on interleaved tokens is enough to keep language and body actions coherent across multiple dialogue turns without losing cross-modal context.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces ViBES, a conversational 3D agent based on a speech-language-behavior (SLB) model with a mixture-of-modality-experts (MoME) backbone. Modality-partitioned transformer experts handle speech, facial expression, and body motion on interleaved token streams using hard routing by modality and cross-expert attention. The system jointly plans language and movement for multi-turn dialogue, supports mixed-initiative input, and claims consistent gains over co-speech gesture and text-to-motion baselines on dialogue-motion alignment and behavior quality metrics, advancing beyond isolated translation tasks toward agentic virtual bodies.

Significance. If the empirical results hold, the work would advance integrated multimodal conversational agents by combining pretrained speech-language components with controllable 3D behavior generation, addressing brittle timing and fragmented modality stacks in prior systems.

major comments (2)
  1. [Abstract] Abstract: the claim of 'consistent gains over strong co-speech and text-to-motion baselines' on dialogue-motion alignment metrics is unsupported by any numerical values, error bars, data-split details, or baseline implementation descriptions, which is load-bearing for the central superiority claim.
  2. [Model description] Model section (MoME backbone): hard routing splits parameters per expert while cross-expert attention is the sole sharing mechanism; no ablations on routing or long-horizon multi-turn coherence metrics are reported, leaving unverified whether this suffices for joint language-body planning without context loss.
minor comments (2)
  1. [Abstract] Abstract: the phrase 'Code and data will be made available' should include a specific repository URL or DOI for reproducibility.
  2. [Introduction] The terms 'controllable behavior hooks' and 'streaming responses' are introduced without precise definitions or interface specifications.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive review. We appreciate the focus on strengthening the empirical claims and model analysis. We address each major comment below and have revised the manuscript to incorporate the suggested improvements where feasible.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim of 'consistent gains over strong co-speech and text-to-motion baselines' on dialogue-motion alignment metrics is unsupported by any numerical values, error bars, data-split details, or baseline implementation descriptions, which is load-bearing for the central superiority claim.

    Authors: We agree that the abstract should explicitly support the superiority claim with quantitative details. In the revised manuscript, we will update the abstract to include specific numerical gains on dialogue-motion alignment metrics (drawn from the results in Section 4), along with error bars, data-split information, and pointers to the baseline implementation details provided in the supplementary material. The full experimental comparisons, including all metrics and baselines, remain unchanged in the body of the paper. revision: yes

  2. Referee: [Model description] Model section (MoME backbone): hard routing splits parameters per expert while cross-expert attention is the sole sharing mechanism; no ablations on routing or long-horizon multi-turn coherence metrics are reported, leaving unverified whether this suffices for joint language-body planning without context loss.

    Authors: The MoME backbone employs hard routing by modality to partition parameters for efficiency while relying on cross-expert attention for inter-modality information sharing during joint language-body planning. We acknowledge the value of ablations; however, the current work prioritizes end-to-end system evaluation over isolated routing studies. We will expand the model section with additional justification for the design and include any long-horizon coherence metrics already computed as part of our multi-turn dialogue experiments. Comprehensive routing ablations are not added at this stage due to computational scope. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity in architectural description or empirical evaluation

full rationale

The paper describes a multimodal SLB model with MoME backbone built from pretrained speech-language components, using hard routing and cross-expert attention for interleaved tokens. All claims rest on empirical benchmarks for dialogue-motion alignment rather than any mathematical derivations, fitted parameters renamed as predictions, or self-citation chains. No equations appear that reduce outputs to inputs by construction, and the architecture is presented as an engineering composition evaluated externally. This matches the default expectation of a self-contained system description.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The abstract relies on the existence of strong pretrained speech-language models and assumes that modality-partitioned experts with cross-attention can integrate interleaved streams; no explicit free parameters, new axioms, or invented entities are stated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body." pith.science (2026). https://pith.science/paper/2512.14234

@misc{pith2026251214234,
  author       = {Pith},
  title        = {Pith review of: ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2512.14234}},
  note         = {Machine review of arXiv:2512.14234}
}
read the original abstract

Human communication is inherently multimodal and social: words, prosody, and body language jointly carry intent. Yet most prior systems model human behavior as a translation task co-speech gesture or text-to-motion that maps a fixed utterance to motion clips-without requiring agentic decision-making about when to move, what to do, or how to adapt across multi-turn dialogue. This leads to brittle timing, weak social grounding, and fragmented stacks where speech, text, and motion are trained or inferred in isolation. We introduce ViBES (Voice in Behavioral Expression and Synchrony), a conversational 3D agent that jointly plans language and movement and executes dialogue-conditioned body actions. Concretely, ViBES is a speech-language-behavior (SLB) model with a mixture-of-modality-experts (MoME) backbone: modality-partitioned transformer experts for speech, facial expression, and body motion. The model processes interleaved multimodal token streams with hard routing by modality (parameters are split per expert), while sharing information through cross-expert attention. By leveraging strong pretrained speech-language models, the agent supports mixed-initiative interaction: users can speak, type, or issue body-action directives mid-conversation, and the system exposes controllable behavior hooks for streaming responses. We further benchmark on multi-turn conversation with automatic metrics of dialogue-motion alignment and behavior quality, and observe consistent gains over strong co-speech and text-to-motion baselines. ViBES goes beyond "speech-conditioned motion generation" toward agentic virtual bodies where language, prosody, and movement are jointly generated, enabling controllable, socially competent 3D interaction. Code and data will be made available at: ai.stanford.edu/~juze/ViBES/

Figures

Figures reproduced from arXiv: 2512.14234 by the authors.

Figure 1
Figure 1. We present a novel speech–language–behavior (SLB) model with a mixture–of–modality–experts (MoME) architecture that [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Model overview. The model adopts an autoregressive structure that converts all modalities into a unified token space. It consists [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative examples of conversational behavior. We [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison with prior methods on the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison for text-to-motion. While [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Overview of our YouTube data processing pipeline. The [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Application: driving video generation with ViBES. We [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Word cloud visualization of our Converse3D data from [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Additional qualitative comparisons with prior methods [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Additional qualitative examples for text-to-motion generation. Given a text caption, we compare the 3D motion generated by our [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: System prompt for motion-conditioned conversational answer generation. We instruct the model to generate answers as if the [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: System Prompt for Conversational Agent Behavior Evaluation. We instruct the model to assess semantic alignment, content–motion match, and social appropriateness on videos rendered from motion sequences. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

Discussion (0). Continue with ORCID to comment.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-based Humanoid Control

    cs.GR 2026-05 unverdicted novelty 6.0 of 10

    SCRIPT presents a scalable diffusion policy with JAST-DiT architecture, nonlinear history conditioning, and RLHR post-training that claims to outperform prior methods on text alignment, motion quality, and physical re...

  2. IAM: Identity-Aware Human Motion and Shape Joint Generation

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    IAM jointly synthesizes motion sequences and body shape parameters conditioned on multimodal identity signals to achieve more realistic and identity-consistent human motions.

  3. PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation

    cs.RO 2026-01 unverdicted novelty 6.0 of 10

    PALM improves long-horizon robotic manipulation success by distilling affordance representations for object interaction and predicting within-subtask progress in a VLA model.

Reference graph

Works this paper leans on

154 extracted references · 154 canonical work pages · cited by 3 Pith papers

  1. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    work page Pith review arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716–23736, 2022

    work page 2022

  3. [3]

    Vlmo: Unified vision- language pre-training with mixture-of-modality-experts

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision- language pre-training with mixture-of-modality-experts. Advances in neural information processing systems, 35: 32897–32912, 2022

    work page 2022

  4. [4]

    $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control

    Kevin Black, Noah Brown, Danny Driess, Adnan Es- mail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π0: A vision- language-action flow model for general robot control. corr, abs/2410.24164, 2024. doi: 10.48550.arXiv preprint ARXIV .2410.24164

    work page Pith review arXiv 2024

  5. [5]

    Higgs Audio V2: Redefining Expressiveness in Audio Generation.https://github.com/boson- ai/higgs- audio, 2025

    Boson AI. Higgs Audio V2: Redefining Expressiveness in Audio Generation.https://github.com/boson- ai/higgs- audio, 2025. GitHub repository. Release blog available athttps://www.boson.ai/blog/ higgs-audio-v2

    work page 2025

  6. [6]

    RT-1: Robotics Transformer for Real-World Control at Scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world con- trol at scale.arXiv preprint arXiv:2212.06817, 2022

    work page Pith review arXiv 2022

  7. [7]

    Language models are few-shot learners.Advances in neu- ral information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neu- ral information processing systems, 33:1877–1901, 2020

    work page 1901

  8. [8]

    Digital life project: Au- tonomous 3d characters with social intelligence

    Zhongang Cai, Jianping Jiang, Zhongfei Qing, Xinying Guo, Mingyuan Zhang, Zhengyu Lin, Haiyi Mei, Chen Wei, Ruisi Wang, Wanqi Yin, et al. Digital life project: Au- tonomous 3d characters with social intelligence. InCVPR, pages 582–592, 2024

    work page 2024

Show all 154 references
  1. [9]

    Enabling synergistic full-body control in prompt-based co-speech motion generation

    Bohong Chen, Yumeng Li, Yao-Xiang Ding, Tianjia Shao, and Kun Zhou. Enabling synergistic full-body control in prompt-based co-speech motion generation. InProceedings of the 32nd ACM International Conference on Multimedia, pages 6774–6783, 2024

    2024

  2. [10]

    The language of motion: Unifying verbal and non-verbal language of 3d human motion

    Changan Chen, Juze Zhang, Shrinidhi K Lakshmikanth, Yusu Fang, Ruizhi Shao, Gordon Wetzstein, Li Fei-Fei, and Ehsan Adeli. The language of motion: Unifying verbal and non-verbal language of 3d human motion. InProceedings of the Computer Vision and Pattern Recognition Confer- e...

    2025

  3. [11]

    Talkcuts: A large-scale dataset for multi-shot human speech video generation.arXiv preprint arXiv:2510.07249, 2025

    Jiaben Chen, Zixin Wang, Ailing Zeng, Yang Fu, Xueyang Yu, Siyuan Cen, Julian Tanke, Yihang Chen, Koichi Saito, Yuki Mitsufuji, et al. Talkcuts: A large-scale dataset for multi-shot human speech video generation.arXiv preprint arXiv:2510.07249, 2025

    2025

  4. [12]

    Rapverse: Coherent vocals and whole-body motion generation from text

    Jiaben Chen, Xin Yan, Yihang Chen, Siyuan Cen, Zixin Wang, Qinwei Ma, Haoyu Zhen, Kaizhi Qian, Lie Lu, and Chuang Gan. Rapverse: Coherent vocals and whole-body motion generation from text. InICCV, pages 10097–10107, 2025

    2025

  5. [13]

    Executing your commands via motion diffusion in latent space

    Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. InCVPR, pages 18000–18010, 2023

    2023

  6. [14]

    Artalk: Speech-driven 3d head animation via autoregressive model.arXiv preprint arXiv:2502.20323, 2025

    Xuangeng Chu, Nabarun Goswami, Ziteng Cui, Hanqin Wang, and Tatsuya Harada. Artalk: Speech-driven 3d head animation via autoregressive model.arXiv preprint arXiv:2502.20323, 2025

    2025

  7. [15]

    Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  8. [16]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic cap...

    2025 arXiv

  9. [17]

    Weakly su- pervised 3d multi-person pose estimation for large-scale scenes based on monocular camera and single lidar

    Peishan Cong, Yiteng Xu, Yiming Ren, Juze Zhang, Lan Xu, Jingya Wang, Jingyi Yu, and Yuexin Ma. Weakly su- pervised 3d multi-person pose estimation for large-scale scenes based on monocular camera and single lidar. InPro- ceedings of the AAAI Conference on Artificial Intellige...

    2023

  10. [18]

    Supervising 3d talking head avatars with analysis-by-audio-synthesis.arXiv preprint arXiv:2504.13386, 2025

    Radek Dan ˇeˇcek, Carolin Schmitt, Senya Polikovsky, and Michael J Black. Supervising 3d talking head avatars with analysis-by-audio-synthesis.arXiv preprint arXiv:2504.13386, 2025

    2025

  11. [19]

    Moshi: a speech-text foundation model 9 for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    Alexandre D ´efossez, Laurent Mazar´e, Manu Orsini, Am´elie Royer, Patrick P ´erez, Herv ´e J ´egou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model 9 for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  12. [20]

    Emerging properties in unified multi- modal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multi- modal pretraining.arXiv preprint arXiv:2505.14683, 2025

    2025 arXiv

  13. [21]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. 2023

    2023

  14. [22]

    Cosyvoice: A scalable multilingual zero-shot text- to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text- to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  15. [23]

    Cosyvoice 2: Scalable stream- ing speech synthesis with large language models.arXiv preprint arXiv:2412.10117, 2024

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xi- ang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable stream- ing speech synthesis with large language models.arXiv preprint arXiv:2412.10117, 2024

    2024 arXiv

  16. [24]

    Cosyvoice 3: Towards in-the-wild speech gen- eration via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

    Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Xian Shi, Keyu An, et al. Cosyvoice 3: Towards in-the-wild speech gen- eration via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

    2025 arXiv

  17. [25]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InForty-first international conference on machi...

    2024

  18. [26]

    Unitalker: Scaling up audio-driven 3d facial anima- tion through a unified model

    Xiangyu Fan, Jiaqi Li, Zhiqian Lin, Weiye Xiao, and Lei Yang. Unitalker: Scaling up audio-driven 3d facial anima- tion through a unified model. InECCV, pages 204–221. Springer, 2024

    2024

  19. [27]

    Filntisis, George Retsinas, Foivos Paraperas- Papantoniou, Athanasios Katsamanis, Anastasios Roussos, and Petros Maragos

    Panagiotis P. Filntisis, George Retsinas, Foivos Paraperas- Papantoniou, Athanasios Katsamanis, Anastasios Roussos, and Petros Maragos. Visual speech-aware perceptual 3d facial expression reconstruction from videos, 2022

    2022

  20. [28]

    Zeroeggs: Zero-shot example-based gesture generation from speech

    Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F Troje, and Marc-Andr ´e Carbonneau. Zeroeggs: Zero-shot example-based gesture generation from speech. InCom- puter Graphics Forum, pages 206–216. Wiley Online Li- brary, 2023

    2023

  21. [29]

    Duetgen: Music driven two-person dance generation via hierarchical masked modeling

    Anindita Ghosh, Bing Zhou, Rishabh Dabral, Jian Wang, Vladislav Golyanik, Christian Theobalt, Philipp Slusallek, and Chuan Guo. Duetgen: Music driven two-person dance generation via hierarchical masked modeling. InProceed- ings of the Special Interest Group on Computer Graphic...

    2025

  22. [30]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  23. [31]

    Humans in 4D: Reconstructing and tracking humans with transformers

    Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023

    2023

  24. [32]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. InCVPR, pages 5152–5161, 2022

    2022

  25. [33]

    Tm2t: Stochastic and tokenized modeling for the reciprocal gen- eration of 3d human motions and texts

    Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal gen- eration of 3d human motions and texts. InECCV, pages 580–597. Springer, 2022

    2022

  26. [34]

    Momask: Generative masked mod- eling of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked mod- eling of 3d human motions. InCVPR, pages 1900–1910, 2024

    1900

  27. [35]

    Live- portrait: Efficient portrait animation with stitching and re- targeting control.arXiv preprint arXiv:2407.03168, 2024

    Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. Live- portrait: Efficient portrait animation with stitching and re- targeting control.arXiv preprint arXiv:2407.03168, 2024

    2024

  28. [36]

    Learning speech-driven 3d conversational gestures from video

    Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed El- gharib, and Christian Theobalt. Learning speech-driven 3d conversational gestures from video. InProceedings of the 21st ACM international conference on intelligent vir- ...

    2021

  29. [37]

    Video-bench: Human-aligned video gen- eration benchmark

    Hui Han, Siyuan Li, Jiaqi Chen, Yiwen Yuan, Yuling Wu, Yufan Deng, Chak Tou Leong, Hanwen Du, Junchen Fu, Youhua Li, et al. Video-bench: Human-aligned video gen- eration benchmark. InCVPR, pages 18858–18868, 2025

    2025

  30. [38]

    Motionverse: A unified multimodal framework for motion comprehension, generation and edit- ing.arXiv preprint arXiv:2509.23635, 2025

    Ruibing Hou, Mingshuang Luo, Hongyu Pan, Hong Chang, and Shiguang Shan. Motionverse: A unified multimodal framework for motion comprehension, generation and edit- ing.arXiv preprint arXiv:2509.23635, 2025

    2025

  31. [39]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3...

    2021

  32. [40]

    Step-audio: Unified understanding and generation in intelligent speech interaction.arXiv preprint arXiv:2502.11946, 2025

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Min- grui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction.arXiv preprint arXiv:2502.11946, 2025

    2025

  33. [41]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InCVPR, pages 21807–21818, 2024

    2024

  34. [42]

    Beat-it: Beat- synchronized multi-condition 3d dance generation

    Zikai Huang, Xuemiao Xu, Cheng Xu, Huaidong Zhang, Chenxi Zheng, Jing Qin, and Shengfeng He. Beat-it: Beat- synchronized multi-condition 3d dance generation. InEu- ropean conference on computer vision, pages 273–290. Springer, 2024

    2024

  35. [43]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 10

    2024 arXiv

  36. [44]

    Motiongpt: Human motion as a foreign lan- guage.NeurIPS, 36:20067–20079, 2023

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign lan- guage.NeurIPS, 36:20067–20079, 2023

    2023

  37. [45]

    Loopy: Taming audio- driven portrait avatar with long-term motion dependency

    Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio- driven portrait avatar with long-term motion dependency. arXiv preprint arXiv:2409.02634, 2024

    2024

  38. [46]

    Solami: Social vision- language-action modeling for immersive interaction with 3d autonomous characters

    Jianping Jiang, Weiye Xiao, Zhengyu Lin, Huaizhong Zhang, Tianxiang Ren, Yang Gao, Zhiqian Lin, Zhongang Cai, Lei Yang, and Ziwei Liu. Solami: Social vision- language-action modeling for immersive interaction with 3d autonomous characters. InCVPR, 2025

    2025

  39. [47]

    Omnihuman-1.5: Instilling an active mind in avatars via cognitive simulation.arXiv preprint arXiv:2508.19209, 2025

    Jianwen Jiang, Weihong Zeng, Zerong Zheng, Jiaqi Yang, Chao Liang, Wang Liao, Han Liang, Yuan Zhang, and Mingyuan Gao. Omnihuman-1.5: Instilling an active mind in avatars via cognitive simulation.arXiv preprint arXiv:2508.19209, 2025

    2025

  40. [48]

    Open- vla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Open- vla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  41. [49]

    Talking with hands 16.2 m: A large-scale dataset of synchronized body- finger motion and audio for conversational motion analy- sis and synthesis

    Gilwoo Lee, Zhiwei Deng, Shugao Ma, Takaaki Shiratori, Siddhartha S Srinivasa, and Yaser Sheikh. Talking with hands 16.2 m: A large-scale dataset of synchronized body- finger motion and audio for conversational motion analy- sis and synthesis. InProceedings of the IEEE/CVF Int...

    2019

  42. [50]

    Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders

    Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders. InICCV, pages 11293–11302, 2021

    2021

  43. [51]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023

    2023

  44. [52]

    Genmo: A GENer- alist model for human MOtion

    Jiefeng Li, Jinkun Cao, Haotian Zhang, Davis Rempe, Jan Kautz, Umar Iqbal, and Ye Yuan. Genmo: A GENer- alist model for human MOtion. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025

    2025

  45. [53]

    Ross, and Angjoo Kanazawa

    Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. InICCV, 2021

    2021

  46. [54]

    Finedance: A fine-grained choreography dataset for 3d full body dance generation

    Ronghui Li, Junfan Zhao, Yachao Zhang, Mingyang Su, Zeping Ren, Han Zhang, Yansong Tang, and Xiu Li. Finedance: A fine-grained choreography dataset for 3d full body dance generation. InICCV, pages 10234–10243, 2023

    2023

  47. [55]

    Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and ex- pression from 4D scans.ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6):194:1–194:17, 2017

    2017

  48. [56]

    Infinityhuman: Towards long-term audio-driven hu- man.arXiv preprint arXiv:2508.20210, 2025

    Xiaodi Li, Pan Xie, Yi Ren, Qijun Gan, Chen Zhang, Fangyuan Kong, Xiang Yin, Bingyue Peng, and Zehuan Yuan. Infinityhuman: Towards long-term audio-driven hu- man.arXiv preprint arXiv:2508.20210, 2025

    2025

  49. [57]

    Llava-slt: Visual language tuning for sign language transla- tion.arXiv preprint arXiv:2412.16524, 2024

    Han Liang, Chengyu Huang, Yuecheng Xu, Cheng Tang, Weicai Ye, Juze Zhang, Xin Chen, Jingyi Yu, and Lan Xu. Llava-slt: Visual language tuning for sign language transla- tion.arXiv preprint arXiv:2412.16524, 2024

    2024

  50. [58]

    Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv preprint arXiv:2411.04996, 2024

    Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.arXiv preprint arXiv:2411.04996, 2024

    2024

  51. [59]

    Omnihuman-1: Re- thinking the scaling-up of one-stage conditioned human an- imation models

    Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, Chao Liang, Yuan Zhang, and Jingtuo Liu. Omnihuman-1: Re- thinking the scaling-up of one-stage conditioned human an- imation models. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 13847– 1...

    2025

  52. [60]

    The quest for generalizable motion gen- eration: Data, model, and evaluation.arXiv preprint arXiv:2510.26794, 2025

    Jing Lin, Ruisi Wang, Junzhe Lu, Ziqi Huang, Guorui Song, Ailing Zeng, Xian Liu, Chen Wei, Wanqi Yin, Qing- ping Sun, et al. The quest for generalizable motion gen- eration: Data, model, and evaluation.arXiv preprint arXiv:2510.26794, 2025

    2025

  53. [61]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  54. [62]

    Disco: Dis- entangled implicit content and rhythm learning for di- verse co-speech gestures synthesis

    Haiyang Liu, Naoya Iwamoto, Zihao Zhu, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Disco: Dis- entangled implicit content and rhythm learning for di- verse co-speech gestures synthesis. InProceedings of the 30th ACM International Conference on Multimedia, pages 3764–3773, 2022

    2022

  55. [63]

    Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis

    Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. InECCV, pages 612–630. Springer, 2022

    2022

  56. [64]

    Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis

    Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. InECCV, 2022

    2022

  57. [65]

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J. Black. Emage: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. InCVPR, 2024

    2024

  58. [66]

    Mimicparts: Part-aware style injec- tion for speech-driven 3d motion generation.arXiv preprint arXiv:2510.13208, 2025

    Lianlian Liu, YongKang He, Zhaojie Chu, Xiaofen Xing, and Xiangmin Xu. Mimicparts: Part-aware style injec- tion for speech-driven 3d motion generation.arXiv preprint arXiv:2510.13208, 2025

    2025

  59. [67]

    Mosa: Motion generation with scalable autoregressive modeling.arXiv preprint arXiv:2511.01200, 2025

    Mengyuan Liu, Sheng Yan, Yong Wang, Yingjie Li, Gui- Bin Bian, and Hong Liu. Mosa: Motion generation with scalable autoregressive modeling.arXiv preprint arXiv:2511.01200, 2025

    2025

  60. [68]

    Learning hierarchical cross-modal associa- tion for co-speech gesture generation

    Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and 11 Bolei Zhou. Learning hierarchical cross-modal associa- tion for co-speech gesture generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recogni...

    2022

  61. [69]

    Gcdance: Genre-controlled 3d full body dance generation driven by music.arXiv preprint arXiv:2502.18309, 2025

    Xinran Liu, Xu Dong, Diptesh Kanojia, Wenwu Wang, and Zhenhua Feng. Gcdance: Genre-controlled 3d full body dance generation driven by music.arXiv preprint arXiv:2502.18309, 2025

    2025 arXiv

  62. [70]

    Dgfm: Full body dance generation driven by mu- sic foundation models.arXiv preprint arXiv:2502.20176, 2025

    Xinran Liu, Zhenhua Feng, Diptesh Kanojia, and Wenwu Wang. Dgfm: Full body dance generation driven by mu- sic foundation models.arXiv preprint arXiv:2502.20176, 2025

    2025

  63. [71]

    Matthew Loper, Naureen Mahmood, Javier Romero, Ger- ard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model.ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, 2015

    2015

  64. [72]

    Diversemotion: Towards diverse human motion generation via discrete diffusion.arXiv preprint arXiv:2309.01372, 2023

    Yunhong Lou, Linchao Zhu, Yaxiong Wang, Xiaohan Wang, and Yi Yang. Diversemotion: Towards diverse human motion generation via discrete diffusion.arXiv preprint arXiv:2309.01372, 2023

    2023

  65. [73]

    Vil- bert: Pretraining task-agnostic visiolinguistic representa- tions for vision-and-language tasks.Advances in neural information processing systems, 32, 2019

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vil- bert: Pretraining task-agnostic visiolinguistic representa- tions for vision-and-language tasks.Advances in neural information processing systems, 32, 2019

    2019

  66. [74]

    Build llm-based zero-shot streaming tts system with cosyvoice

    Xiang Lyu, Yuxuan Wang, Tianyu Zhao, Hao Wang, Huadai Liu, and Zhihao Du. Build llm-based zero-shot streaming tts system with cosyvoice. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–2. IEEE, 2025

    2025

  67. [75]

    Follow-your- emoji-faster: Towards efficient, fine-controllable, and ex- pressive freestyle portrait animation.arXiv preprint arXiv:2509.16630, 2025

    Yue Ma, Zexuan Yan, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeung Shum, et al. Follow-your- emoji-faster: Towards efficient, fine-controllable, and ex- pressive freestyle portrait animation.arXiv preprint arXiv:2509.16630, 2025

    2025

  68. [76]

    Troje, Gerard Pons-Moll, and Michael J

    Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. InInternational Con- ference on Computer Vision, pages 5442–5451, 2019

    2019

  69. [77]

    Embody 3d: A large-scale multimodal motion and behavior dataset.arXiv preprint arXiv:2510.16258, 2025

    Claire McLean, Makenzie Meendering, Tristan Swartz, Orri Gabbay, Alexandra Olsen, Rachel Jacobs, Nicholas Rosen, Philippe de Bree, Tony Garcia, Gadsden Merrill, et al. Embody 3d: A large-scale multimodal motion and behavior dataset.arXiv preprint arXiv:2510.16258, 2025

    2025

  70. [78]

    Convofusion: Multi-modal conversational dif- fusion for co-speech gesture synthesis

    Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, and Christian Theobalt. Convofusion: Multi-modal conversational dif- fusion for co-speech gesture synthesis. InProceedings of the IEEE/CVF conference on computer vision and pattern recogn...

    2024

  71. [79]

    Multimodal con- trastive learning with limoe: the language-image mixture of experts.Advances in Neural Information Processing Systems, 35:9564–9576, 2022

    Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal con- trastive learning with limoe: the language-image mixture of experts.Advances in Neural Information Processing Systems, 35:9564–9576, 2022

    2022

  72. [80]

    Ges- ture generation (still) needs improved human evaluation practices: Insights from a community-driven state-of-the- art benchmark.arXiv preprint arXiv:2511.01233, 2025

    Rajmund Nagy, Hendric V oss, Thanh Hoang-Minh, Mi- hail Tsakov, Teodor Nikolov, Zeyi Zhang, Tenglong Ao, Sicheng Yang, Shaoli Huang, Yongkang Cheng, et al. Ges- ture generation (still) needs improved human evaluation practices: Insights from a community-driven state-of-the- ar...

    2025 arXiv

  73. [81]

    From audio to photoreal embodiment: Synthesiz- ing humans in conversations

    Evonne Ng, Javier Romero, Timur Bagautdinov, Shao- jie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthesiz- ing humans in conversations. InCVPR, pages 1001–1010, 2024

    2024

  74. [82]

    Scantalk: 3d talking heads from unregistered scans

    Federico Nocentini, Thomas Besnier, Claudio Ferrari, Syl- vain Arguillere, Stefano Berretti, and Mohamed Daoudi. Scantalk: 3d talking heads from unregistered scans. In ECCV, pages 19–36. Springer, 2024

    2024

  75. [83]

    Generative agents: Interactive simulacra of human behav- ior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behav- ior. InProceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023

    2023

  76. [84]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. InProceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  77. [85]

    Re- constructing hands in 3D with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Re- constructing hands in 3D with transformers. InCVPR, 2024

    2024

  78. [86]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. InICCV, pages 4195–4205, 2023

    2023

  79. [87]

    Selftalk: A self- supervised commutative training diagram to comprehend 3d talking faces

    Ziqiao Peng, Yihao Luo, Yue Shi, Hao Xu, Xiangyu Zhu, Hongyan Liu, Jun He, and Zhaoxin Fan. Selftalk: A self- supervised commutative training diagram to comprehend 3d talking faces. InACMMM, pages 5292–5301, 2023

    2023

  80. [88]

    Vibevoice technical report.arXiv preprint arXiv:2508.19205, 2025

    Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, et al. Vibevoice technical report.arXiv preprint arXiv:2508.19205, 2025

    2025

  81. [89]

    Co3 gesture: Towards coherent concurrent co-speech 3d gesture generation with interactive diffusion

    Xingqun Qi, Yatian Wang, Hengyuan Zhang, Jiahao Pan, Wei Xue, Shanghang Zhang, Wenhan Luo, Qifeng Liu, and Yike Guo. Co3 gesture: Towards coherent concurrent co-speech 3d gesture generation with interactive diffusion. ICLR, 2025

    2025

  82. [90]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018

    2018

  83. [91]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsu- pervised multitask learners. 2019

    2019

  84. [92]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInter- national conference on machine learning, pages 28492– 28518. PMLR, 2023

    2023

  85. [93]

    The candor 12 corpus: Insights from a large multimodal dataset of nat- uralistic conversation.Science Advances, 9(13):eadf3197, 2023

    Andrew Reece, Gus Cooney, Peter Bull, Christine Chung, Bryn Dawson, Casey Fitzpatrick, Tamara Glazer, Dean Knox, Alex Liebscher, and Sebastian Marin. The candor 12 corpus: Insights from a large multimodal dataset of nat- uralistic conversation.Science Advances, 9(13):eadf3197, 2023

    2023

  86. [94]

    Meshtalk: 3d face an- imation from speech using cross-modality disentanglement

    Alexander Richard, Michael Zollh ¨ofer, Yandong Wen, Fer- nando De la Torre, and Yaser Sheikh. Meshtalk: 3d face an- imation from speech using cross-modality disentanglement. InProceedings of the IEEE/CVF international conference on computer vision, pages 1173–1182, 2021

    2021

  87. [95]

    Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bod- ies together.ACM Transactions on Graphics, (Proc. SIG- GRAPH Asia), 36(6), 2017

    2017

  88. [96]

    Runway: Ai video generation platform

    Runway AI, Inc. Runway: Ai video generation platform. https://runwayml.com, 2025. Accessed: 2025-11- 20

    2025

  89. [97]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  90. [98]

    Bailando: 3d dance gen- eration by actor-critic gpt with choreographic memory

    Li Siyao, Xintong Yu, and et al. Bailando: 3d dance gen- eration by actor-critic gpt with choreographic memory. In CVPR, 2022

    2022

  91. [99]

    Facediffuser: Speech-driven 3d facial animation synthesis using diffusion

    Stefan Stan, Kazi Injamamul Haque, and Zerrin Yumak. Facediffuser: Speech-driven 3d facial animation synthesis using diffusion. InProceedings of the 16th ACM SIG- GRAPH Conference on Motion, Interaction and Games, pages 1–11, 2023

    2023

  92. [100]

    Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models.ACM Transactions on Graphics (TOG), 43(4):1–9, 2024

    Zhiyao Sun, Tian Lv, Sheng Ye, Matthieu Lin, Jenny Sheng, Yu-Hui Wen, Minjing Yu, and Yong-jin Liu. Diffposetalk: Speech-driven stylistic 3d facial animation and head pose generation via diffusion models.ACM Transactions on Graphics (TOG), 43(4):1–9, 2024

    2024

  93. [101]

    Mul- titalk: Enhancing 3d talking head generation across lan- guages with multilingual video dataset.arXiv preprint arXiv:2406.14272, 2024

    Kim Sung-Bin, Lee Chae-Yeon, Gihun Son, Oh Hyun-Bin, Janghoon Ju, Suekyeong Nam, and Tae-Hyun Oh. Mul- titalk: Enhancing 3d talking head generation across lan- guages with multilingual video dataset.arXiv preprint arXiv:2406.14272, 2024

    2024

  94. [102]

    Lxmert: Learning cross- modality encoder representations from transformers.arXiv preprint arXiv:1908.07490, 2019

    Hao Tan and Mohit Bansal. Lxmert: Learning cross- modality encoder representations from transformers.arXiv preprint arXiv:1908.07490, 2019

    1908

  95. [103]

    Is someone speaking? exploring long-term temporal features for audio-visual ac- tive speaker detection

    Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li. Is someone speaking? exploring long-term temporal features for audio-visual ac- tive speaker detection. InACMMM, page 3927–3935, 2021

    2021

  96. [104]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  97. [105]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  98. [106]

    Human motion diffu- sion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffu- sion model. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  99. [107]

    Closd: Closing the loop between simula- tion and diffusion for multi-task character control.arXiv preprint arXiv:2410.03441, 2024

    Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue Bin Peng, Amit H Bermano, and Michiel van de Panne. Closd: Closing the loop between simula- tion and diffusion for multi-task character control.arXiv preprint arXiv:2410.03441, 2024

    2024

  100. [108]

    Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions

    Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. InECCV, pages 244–260. Springer, 2024

    2024

  101. [109]

    Edge: Editable dance generation from music

    Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 448–458, 2023

    2023

  102. [110]

    Dancechat: Large language model-guided music-to-dance generation.arXiv preprint arXiv:2506.10574, 2025

    Qing Wang, Xiaohang Yang, Yilan Dong, Naveen Raj Govindaraj, Gregory Slabaugh, and Shanxin Yuan. Dancechat: Large language model-guided music-to-dance generation.arXiv preprint arXiv:2506.10574, 2025

    2025

  103. [111]

    One- shot free-view neural talking-head synthesis for video con- ferencing

    Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One- shot free-view neural talking-head synthesis for video con- ferencing. InCVPR, 2021

    2021

  104. [112]

    Motiongpt-2: A general-purpose motion- language model for motion generation and understanding

    Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. Motiongpt-2: A general-purpose motion- language model for motion generation and understanding. arXiv preprint arXiv:2410.21747, 2024

    2024

  105. [113]

    Aniportrait: Audio-driven synthesis of photorealistic portrait animation

    Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694, 2024

    2024

  106. [114]

    Dual-stream diffusion for world-model augmented vision-language-action model.arXiv preprint arXiv:2510.27607, 2025

    John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, and Jinwoo Shin. Dual-stream diffusion for world-model augmented vision-language-action model.arXiv preprint arXiv:2510.27607, 2025

    2025

  107. [115]

    Mote: Learning motion-text diffusion model for multiple generation tasks.arXiv preprint arXiv:2411.19786, 2024

    Yiming Wu, Wei Ji, Kecheng Zheng, Zicheng Wang, and Dong Xu. Mote: Learning motion-text diffusion model for multiple generation tasks.arXiv preprint arXiv:2411.19786, 2024

    2024

  108. [116]

    X-streamer: Unified hu- man world modeling with audiovisual interaction.arXiv preprint arXiv:2509.21574, 2025

    You Xie, Tianpei Gu, Zenan Li, Chenxu Zhang, Guox- ian Song, Xiaochen Zhao, Chao Liang, Jianwen Jiang, Hongyi Xu, and Linjie Luo. X-streamer: Unified hu- man world modeling with audiovisual interaction.arXiv preprint arXiv:2509.21574, 2025

    2025

  109. [117]

    Codetalker: Speech-driven 3d facial animation with discrete motion prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. InCVPR, pages 12780–12790, 2023

    2023

  110. [118]

    Omnimotion-x: Versatile multimodal whole-body motion generation.arXiv preprint arXiv:2510.19789, 2025

    Guowei Xu, Yuxuan Bian, Ailing Zeng, Mingyi Shi, Shaoli Huang, Wen Li, Lixin Duan, and Qiang Xu. Omnimotion-x: Versatile multimodal whole-body motion generation.arXiv preprint arXiv:2510.19789, 2025

    2025

  111. [119]

    Qwen2.5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Jun- yang Lin. Qwen2.5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  112. [120]

    Vasa-1: Lifelike audio-driven talking faces 13 generated in real time.Advances in Neural Information Processing Systems, 37:660–684, 2024

    Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces 13 generated in real time.Advances in Neural Information Processing Systems, 37:660–684, 2024

    2024

  113. [121]

    Hunyuanportrait: Implicit condition control for enhanced portrait animation

    Zunnan Xu, Zhentao Yu, Zixiang Zhou, Jun Zhou, Xiaoyu Jin, Fa-Ting Hong, Xiaozhong Ji, Junwei Zhu, Chengfei Cai, Shiyu Tang, et al. Hunyuanportrait: Implicit condition control for enhanced portrait animation. InCVPR, pages 15909–15919, 2025

    2025

  114. [122]

    Diffusestylegesture: Stylized audio-driven co-speech ges- ture generation with diffusion models

    Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao. Diffusestylegesture: Stylized audio-driven co-speech ges- ture generation with diffusion models. InProceedings of the Thirty-Second International Joint Conference on Artifi- ...

    2023

  115. [123]

    Videogen-eval: Agent-based system for video generation evaluation.arXiv preprint arXiv:2503.23452, 2025

    Yuhang Yang, Ke Fan, Shangkun Sun, Hongxiang Li, Ailing Zeng, FeiLin Han, Wei Zhai, Wei Liu, Yang Cao, and Zheng-Jun Zha. Videogen-eval: Agent-based system for video generation evaluation.arXiv preprint arXiv:2503.23452, 2025

    2025

  116. [124]

    Generating holistic 3d human motion from speech

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yan- dong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. InCVPR, pages 469–480, 2023

    2023

  117. [125]

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yan- dong Wen, Timo Bolkart, Dacheng Tao, and Michael J. Black. Generating holistic 3d human motion from speech. InCVPR, 2023

    2023

  118. [126]

    Speech ges- ture generation from the trimodal context of text, audio, and speaker identity.ACM Transactions on Graphics (TOG), 39 (6):1–16, 2020

    Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech ges- ture generation from the trimodal context of text, audio, and speaker identity.ACM Transactions on Graphics (TOG), 39 (6):1–16, 2020

    2020

  119. [127]

    Socialgen: Modeling multi-human social interaction with language models.arXiv preprint arXiv:2503.22906, 2025

    Heng Yu, Juze Zhang, Changan Chen, Tiange Xiang, Yusu Fang, Juan Carlos Niebles, and Ehsan Adeli. Socialgen: Modeling multi-human social interaction with language models.arXiv preprint arXiv:2503.22906, 2025

    2025

  120. [128]

    Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation.arXiv preprint arXiv:2411.18138, 2024

    Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xi- aohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation.arXiv preprint arXiv:2411.18138, 2024

    2024

  121. [129]

    Glm-4-voice: Towards intelligent and human-like end-to- end spoken chatbot.arXiv preprint arXiv:2412.02612, 2024

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to- end spoken chatbot.arXiv preprint arXiv:2412.02612, 2024

    2024 arXiv

  122. [130]

    Anygpt: Unified multimodal llm with discrete sequence modeling.arXiv preprint arXiv:2402.12226, 2024

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with discrete sequence modeling.arXiv preprint arXiv:2402.12226, 2024

    2024

  123. [131]

    Mutual adaptive reasoning for monocular 3d multi-person pose estimation

    Juze Zhang, Jingya Wang, Ye Shi, Fei Gao, Lan Xu, and Jingyi Yu. Mutual adaptive reasoning for monocular 3d multi-person pose estimation. InProceedings of the 30th ACM international conference on multimedia, pages 1788– 1796, 2022

    2022

  124. [132]

    Neuraldome: A neural modeling pipeline on multi- view human-object interactions

    Juze Zhang, Haimin Luo, Hongdi Yang, Xinru Xu, Qianyang Wu, Ye Shi, Jingyi Yu, Lan Xu, and Jingya Wang. Neuraldome: A neural modeling pipeline on multi- view human-object interactions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8...

    2023

  125. [133]

    Ikol: Inverse kinematics optimization layer for 3d human pose and shape estimation via gauss-newton differentiation

    Juze Zhang, Ye Shi, Yuexin Ma, Lan Xu, Jingyi Yu, and Jingya Wang. Ikol: Inverse kinematics optimization layer for 3d human pose and shape estimation via gauss-newton differentiation. InProceedings of the AAAI conference on artificial intelligence, pages 3454–3462, 2023

    2023

  126. [134]

    Generating human motion from textual descrip- tions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descrip- tions with discrete representations. InCVPR, pages 14730– 14740, 2023

    2023

  127. [135]

    Hoi-mˆ 3: Capture multiple humans and objects in- teraction within contextual environment

    Juze Zhang, Jingyan Zhang, Zining Song, Zhanhe Shi, Chengfeng Zhao, Ye Shi, Jingyi Yu, Lan Xu, and Jingya Wang. Hoi-mˆ 3: Capture multiple humans and objects in- teraction within contextual environment. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  128. [136]

    Large motion model for unified multi-modal motion generation

    Mingyuan Zhang, Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, and Ziwei Liu. Large motion model for unified multi-modal motion generation. InECCV, 2024

    2024

  129. [137]

    Unimumo: Uni- fied text, music and motion generation.arXiv preprint arXiv:2410.04534, 2024

    Ruichen Zhang, Wen Wang, et al. Unimumo: Uni- fied text, music and motion generation.arXiv preprint arXiv:2410.04534, 2024

    2024

  130. [138]

    Both2hands: Infer- ring 3d hands from both text prompts and body dynamics

    Wenqian Zhang, Molin Huang, Yuxuan Zhou, Juze Zhang, Jingyi Yu, Jingya Wang, and Lan Xu. Both2hands: Infer- ring 3d hands from both text prompts and body dynamics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2393–2404, 2024

    2024

  131. [139]

    Motiongpt: Finetuned llms are general-purpose motion generators

    Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned llms are general-purpose motion generators. InProceedings of the AAAI Conference on Ar- tificial Intelligence, pages 7368–7376, 2024

    2024

  132. [140]

    Motion anything: Any to motion generation.arXiv preprint arXiv:2503.06955, 2025

    Zeyu Zhang, Yiran Wang, Wei Mao, Danning Li, Rui Zhao, Biao Wu, Zirui Song, Bohan Zhuang, Ian Reid, and Richard Hartley. Motion anything: Any to motion generation.arXiv preprint arXiv:2503.06955, 2025

    2025

  133. [141]

    Social agent: Mastering dyadic nonverbal behavior generation via conversational llm agents.arXiv preprint arXiv:2510.04637, 2025

    Zeyi Zhang, Yanju Zhou, Heyuan Yao, Tenglong Ao, Xi- aohang Zhan, and Libin Liu. Social agent: Mastering dyadic nonverbal behavior generation via conversational llm agents.arXiv preprint arXiv:2510.04637, 2025

    2025

  134. [142]

    I’m hoi: Inertia-aware monocular capture of 3d human-object interactions

    Chengfeng Zhao, Juze Zhang, Jiashen Du, Ziwei Shan, Junye Wang, Jingyi Yu, Jingya Wang, and Lan Xu. I’m hoi: Inertia-aware monocular capture of 3d human-object interactions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 729– 741, 2024

    2024

  135. [143]

    Image and video tokenization with binary spherical quantization

    Yue Zhao, Yuanjun Xiong, and Philipp Kr ¨ahenb¨uhl. Image and video tokenization with binary spherical quantization. arXiv preprint arXiv:2406.07548, 2024. 14

    2024

  136. [144]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  137. [145]

    Mo- tiongpt3: Human motion as a second modality.arXiv preprint arXiv:2506.24086, 2025

    Bingfan Zhu, Biao Jiang, Sunyi Wang, Shixiang Tang, Tao Chen, Linjie Luo, Youyi Zheng, and Xin Chen. Mo- tiongpt3: Human motion as a second modality.arXiv preprint arXiv:2506.24086, 2025

    2025

  138. [146]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023. 15

    2023

  139. [147]

    Supplementary Material In this supplementary material, we provide additional de- tails about:

  140. [148]

    Supplementary video for qualitative examples

  141. [149]

    Application cases of ViBES

  142. [150]

    Additional implementation details of ViBES

  143. [151]

    Additional details on the YouTube data processing pipeline (referenced in Sec. 4)

  144. [152]

    Additional details on constructing conversational motion from the AMASS dataset (referenced in Sec. 4)

  145. [153]

    Additional details on benchmark for conversational be- havior (referenced in Sec. 5.1)

  146. [154]

    yep”, “ok

    Additional qualitative example of talking head genera- tion and text-to-motion. 7.1. Supplementary Video We provide a supplementary video to illustrate our method and results. The video presents: 1) the background and motivation of this work; 2) an explanation of the overall f...

    2024

Pith tools

Reviewed May 16, 2026 · model on record in the stance chip above.