REVIEW 5 cited by
DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models
read the original abstract
The art of communication beyond speech there are gestures. The automatic co-speech gesture generation draws much attention in computer animation. It is a challenging task due to the diversity of gestures and the difficulty of matching the rhythm and semantics of the gesture to the corresponding speech. To address these problems, we present DiffuseStyleGesture, a diffusion model based speech-driven gesture generation approach. It generates high-quality, speech-matched, stylized, and diverse co-speech gestures based on given speeches of arbitrary length. Specifically, we introduce cross-local attention and self-attention to the gesture diffusion pipeline to generate better speech matched and realistic gestures. We then train our model with classifier-free guidance to control the gesture style by interpolation or extrapolation. Additionally, we improve the diversity of generated gestures with different initial gestures and noise. Extensive experiments show that our method outperforms recent approaches on speech-driven gesture generation. Our code, pre-trained models, and demos are available at https://github.com/YoungSeng/DiffuseStyleGesture.
Forward citations
Cited by 5 Pith papers
-
Reality Check: How Avatar and Face Representation Affect the Perceptual Evaluation of Synthesized Gestures
Avatar and face representation systematically shift perceptual judgments of synthesized co-speech gestures.
-
Reality Check: How Avatar and Face Representation Affect the Perceptual Evaluation of Synthesized Gestures
Avatar appearance and facial presentation systematically bias perceptual judgments of synthesized co-speech gestures.
-
Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech
Lightweight transformer predicts iconic gesture placement and intensity from text and emotion for robot co-speech, outperforming GPT-4o on BEAT2 without audio input.
-
Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech
A lightweight transformer predicts iconic gesture placement and intensity from text and emotion alone, outperforming GPT-4o on the BEAT2 dataset for real-time robot deployment.
-
Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech
A compact transformer predicts iconic gesture placement and intensity from text and emotion alone, outperforming GPT-4o on the BEAT2 dataset for robot co-speech use.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.