REVIEW 5 cited by
EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements. To achieve this, we first introduce BEAT2 (BEAT-SMPLX-FLAME), a new mesh-level holistic co-speech dataset. BEAT2 combines a MoShed SMPL-X body with FLAME head parameters and further refines the modeling of head, neck, and finger movements, offering a community-standardized, high-quality 3D motion captured dataset. EMAGE leverages masked body gesture priors during training to boost inference performance. It involves a Masked Audio Gesture Transformer, facilitating joint training on audio-to-gesture generation and masked gesture reconstruction to effectively encode audio and body gesture hints. Encoded body hints from masked gestures are then separately employed to generate facial and body movements. Moreover, EMAGE adaptively merges speech features from the audio's rhythm and content and utilizes four compositional VQ-VAEs to enhance the results' fidelity and diversity. Experiments demonstrate that EMAGE generates holistic gestures with state-of-the-art performance and is flexible in accepting predefined spatial-temporal gesture inputs, generating complete, audio-synchronized results. Our code and dataset are available https://pantomatrix.github.io/EMAGE/
Forward citations
Cited by 5 Pith papers
-
InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios
InterAct provides 241 long, acted two-person scenes with synchronized audio, body motion, and facial ARKit parameters, plus a hierarchical diffusion baseline that generates both people's motion from speech.
-
M3G: Multi-Granular Gesture Generator for Audio-Driven Full-Body Human Motion Synthesis
A multi-granular VQ-VAE that encodes and reconstructs motion at several time scales, paired with an audio-to-token predictor, lowers FGD and beats EMAGE in perceptual A/B tests on BEAT2 full-body gesture generation.
-
GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modeling
GestureLSM combines residual-vector-quantized body-region tokens, spatial-temporal attention, and latent shortcut sampling to generate coherent full-body co-speech gestures at real-time speed with state-of-the-art FGD...
-
Human-like Nonverbal Behavior with MetaHumans in Real-World Interaction Studies: An Architecture Using Generative Methods and Motion Capture
A distributed MetaHuman-based architecture combines conversational AI, camera-based user detection, and generative plus motion-captured nonverbal behavior for real-world human-agent interaction studies.
-
Generative AI for Character Animation: A Comprehensive Survey of Techniques, Applications, and Future Directions
A comprehensive survey that unifies generative AI techniques for character animation across facial, gesture, motion, and 3D asset generation, with a shared taxonomy and resource list.
Discussion (0). Continue with ORCID to comment.