Pith. sign in

REVIEW 4 major objections 6 minor 53 references

SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SemTalk argues that co-speech motion should be generated as a rhythm-locked base layer plus a sparse, semantically gated overlay that is fused adaptively, and reports state-of-the-art quality on BEAT2 and SHOW.

desk verdict Solid BEAT2 evidence for a genuinely new semantic-gated architecture, but the SHOW half relies on self-generated labels, so the two-dataset claim is not independent. read the letter →

arxiv 2412.16563 v3 pith:L3XQMTKN submitted 2024-12-21 cs.CV

Xiangyue Zhang , Jianfang Li , Jiaxu Zhang , Ziqiang Dang , Jianqiang Ren , Liefeng Bo , Zhigang Tu This is my paper · ORCID
classification cs.CV
keywords co-speechmotiongenerationsemanticemphasisgesturesynthesisrhythmicconsistencygateRVQ-VAEframe-levelscoreholistic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SemTalk is a framework for generating full-body co-speech motion that treats rhythmic gestures and semantic gestures as separate problems. The paper's core idea is that most frames follow speech rhythm and only a few frames carry meaning, so the model learns a rhythm-aligned base motion and then learns a semantic score that highlights sparse, meaningful gestures. A coarse-to-fine cross-attention module and local-global rhythmic consistency losses build the base motion, while a learned semantic gate produces a frame-level score that adaptively fuses the semantic overlay into the base. On BEAT2 the method reports lower FGD, MSE, and LVD than prior models, and on SHOW it reports lower FGD and higher beat correlation.

What carries the argument

The load-bearing object is the semantic score ψ produced by the sem-gate: it is a learned, frame-level scalar that decides when the sparse semantic codes are activated and how strongly they are blended into the rhythm-aligned base codes. The base motion itself is built by a hierarchical Coarse2Fine Cross-Attention module that propagates guidance from face to hands to upper body to lower body, plus a local and global rhythmic consistency loss based on InfoNCE that aligns latent motion features with HuBERT audio features at both frame and sequence level.

What would settle it

Re-evaluate SemTalk on SHOW using human-annotated or independently produced semantic labels instead of the sem-gate's own predictions, then compare FGD and BC against EMAGE and TalkSHOW; if the reported gains disappear with independent labels, the semantic emphasis advantage is largely an artifact of self-labeling.

Watch

Extended reading notes

Core claim

The paper claims that separately learning rhythm-related base motion and semantic-aware sparse motion, then fusing them through a learned frame-level semantic score, yields more natural and more semantically expressive co-speech gestures than jointly conditioned baselines. On BEAT2, SemTalk reports FGD 4.278 versus EMAGE's 5.512, MSE 6.153 versus 7.680, and LVD 6.938 versus 7.556, and on SHOW it reports FGD 20.18 versus 22.12 and BC 8.304 versus 8.280. The key mechanism is the semantic gate, which outputs a per-frame score that both weights the sparse semantic features and supervises the loss, allowing the model to emphasize key moments such as the hand raise for 'my opinion' while preserving the stable rhythmic base elsewhere.

Load-bearing premise

The cleanest results stand on the assumption that the frame-level semantic labels in BEAT2 are reliable and that a sem-gate trained on them transfers to SHOW without bias; on SHOW the paper generates its own semantic labels with the very model under test.

Editorial extensions

If this is right

  • Sparse semantic gestures that prior rhythm-dominated models dilute can be recovered explicitly, because the model does not have to compromise base motion to express meaning.
  • The learned semantic score provides a frame-level control signal that could be used for editing generated motion by changing which frames are emphasized.
  • By separating rhythm and semantics, the same base motion block can be reused across speakers, with only the semantic overlay modulated by content and emotion.
  • The method produces smoother facial and full-body motion than VQ-VAE baselines, as measured by lower LVD on both BEAT2 and SHOW.
  • The semantic score adapts to emotional tone even when the transcribed text stays the same, indicating that the model captures delivery-dependent emphasis rather than purely word-level cues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The decomposition principle may transfer to other co-speech or expressive-motion tasks, such as sign-language nuance or public-speaking rehearsal, where most motion is prosodic but a few moments carry the message.
  • A testable extension is to use SemTalk's semantic score as a pseudo-labeler for new unlabeled motion datasets, then measure whether downstream models trained on those labels match the quality of models trained on manual annotations.
  • If the semantic score is stable under speaker identity changes, it could serve as an interpretable diagnostic for whether a gesture-generation model is actually 'listening' to content rather than only to prosody.
  • The self-generated labels on SHOW leave open the question of whether the reported semantic-consistency gain survives under independent annotations, which would be the decisive check of the semantic emphasis claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript introduces SemTalk, a framework for holistic co-speech motion generation that decomposes motion into a rhythm-related base component and a sparse semantic component. The base motion is generated with a coarse-to-fine cross-attention module and local-global rhythmic consistency (InfoNCE) losses, while the sparse motion is produced from text, speech, emotion, and HuBERT features and gated by a learned semantic score. The final motion is obtained by replacing base codes with sparse semantic codes on frames where the semantic score exceeds a threshold. The method is evaluated on BEAT2 and SHOW against several recent baselines, with ablations, a user study, and qualitative comparisons.

Significance. If the claims hold, the explicit separation of rhythm-related base motion and semantic sparse motion with frame-level gating is a useful and plausible contribution to co-speech motion generation. The BEAT2 experiments are internally consistent and show improvements over strong baselines on FGD, MSE, and LVD, and the ablation study supports the importance of the proposed components. The idea of using a learned semantic score for adaptive fusion is clearly presented and worth pursuing. However, the SHOW validation is seriously weakened by the use of the model's own sem-gate to generate the semantic labels used for training, and the lack of error bars or significance tests limits confidence in the quantitative claims.

major comments (4)
  1. [Sec. 4.1] The SHOW experiments are circular for the semantic component. The text states: 'Since it lacks frame-level semantic labels, we use the sem-gate from SemTalk, pre-trained on BEAT2, to generate them.' This means the model under evaluation supplies the semantic supervision used to train it on SHOW, and the qualitative claims of enhanced semantic richness on SHOW are judged against the gate's own notion of emphasis. The global metrics (FGD, BC, MSE, LVD) are computed against ground-truth motion and are therefore not directly leaked, but the training signal gives SemTalk an extra supervisory input unavailable to the baselines, and the semantic-claim on SHOW is not independent. This is load-bearing for the abstract's claim of outperforming the state of the art on 'two public datasets.' Please either add human-annotated semantic keyframes on a SHOW subset, use an external labeler shared by all methods, or reframe the SHOW results as a domain-transfer demonstration without semantic-claim independence.
  2. [Sec. 4.3, Tables 1-3] All quantitative results are reported as single numbers with no error bars, confidence intervals, or significance tests. Many differences are small in absolute terms (e.g., BEAT2 BC 7.770 vs. 7.724, and SHOW BC 8.304 vs. 8.280), and without repeated runs it is impossible to tell whether these differences are meaningful. Please report mean and standard deviation over at least three random seeds for all metrics and, where appropriate, a paired significance test (e.g., bootstrap or Wilcoxon signed-rank).
  3. [Table 3] The ablation table contains a surprising inconsistency that the text does not explain. Adding RVQ alone increases MSE from 8.239 to 13.882 and LVD from 7.831 to 15.42, and the row with C2F+RVQ has MSE 13.416 and LVD 15.72, while the full model achieves 6.153 and 6.938. The text says RVQ 'slightly increases MSE and LVD,' but the increase is large. Please clarify whether the RVQ rows are valid configurations, whether the reported scales are consistent across rows, and how RC (and SE) interact with RVQ to produce the final values.
  4. [Sec. 3.3, Eq. (4)] The paper defines local and global rhythmic consistency losses, L_(Rhy)^(L) and L_(Rhy)^(G), but only Eq. (4) is given. It is not clear how the global sentence-level loss differs from the local frame-level loss, how the two are combined into the total loss, or whether the same temperature tau = 0.1 is used for both. Since rhythmic consistency is a central component of the base-motion claim, please provide the explicit form of L_(Rhy)^(G) and the total loss combination.
minor comments (6)
  1. [Sec. 3.2, 3.4, 3.5] Notation is inconsistent for the base-motion codes: the text switches between qb and qr, and Eq. (5) uses fs and fb while the surrounding text uses qs and qb. Please unify the notation.
  2. [Sec. 4.1 and Table 1] The caption of Table 1 mentions 'FMD' but the metric is FGD. Also, the scaling factors (FGD×10−1, BC×10−1, MSE×10−8, LVD×10−5) should be stated in the table header or in the text before the caption, and the definitions of FGD, BC, DIV, MSE, and LVD should be given explicitly.
  3. [Sec. 3.4] The two weighting methods Wf and Wl are described only verbally. Please provide the equations for how the semantic score ψ is computed and how feature weighting and loss weighting are applied, including the classification loss L_cls^G and the code classification loss L_cls.
  4. [Table 3] The row labels in Table 3 are ambiguous: the baseline row with no components presumably still uses some VQ-VAE, but the 'RVQ' row is described as adding RVQ-VAE. Please clarify what the default quantizer is and what 'w/o RVQ' means.
  5. [Sec. 4.2, Fig. 10] The user study reports only aggregate preferences without sample size per condition, effect sizes, or statistical tests. Please report the number of participants per video, the distribution of rankings, and significance of the preference differences.
  6. [Eq. (4)] The symbol N is described as 'the number of frames (or the batch size),' which is ambiguous. Please state once whether the InfoNCE loss is computed over frames within a sequence or over samples in a batch.

Circularity Check

1 steps flagged · score 6.0 of 10

SHOW semantic labels are generated by SemTalk's own sem-gate and then used to train and evaluate SemTalk, making the SHOW semantic-emphasis claim circular; BEAT2 remains an independent test.

  1. self definitional [Section 4.1, Datasets (SHOW); see also Section 3.4 Semantic Emphasis Learning]
    "Since it lacks frame-level semantic labels, we use the sem-gate from SemTalk, pre-trained on BEAT2, to generate them."

    SHOW has no ground-truth frame-level semantic labels, so the model under test, SemTalk's sem-gate, produces the labels. In Section 3.4, the paper states that 'loss weighting is applied by supervising ψ, with a classification loss LG_cls based on semantic labels', meaning the generated labels are used as supervision for the same semantic-score component. The SHOW qualitative claim of enhanced semantic consistency is therefore judged against SemTalk's own notion of semantic importance, not an independent annotation.

full rationale

The derivation chain has one genuinely circular link, and it affects the SHOW half of the two-dataset claim. BEAT2 provides external frame-level semantic labels, and the ablation in Table 2 shows SemTalk's sem-gate uses those labels for the classification loss; the BEAT2 results in Table 1 are therefore an independent test of the method. For SHOW, however, Section 4.1 states that SemTalk's own sem-gate, pre-trained on BEAT2, is used to generate the missing semantic labels. Since Section 3.4 makes the same sem-gate's output ψ the target of a classification loss based on semantic labels, the pseudo-labels produced by the model are then used as supervision for the same model component. Consequently, the SHOW semantic-consistency claim and any qualitative 'semantic richness' judgments reduce to agreement between SemTalk and its own gate rather than agreement with an external standard. The SHOW FGD/MSE/LVD metrics do not depend on the pseudo-labels and still show competitive motion quality, but they do not validate the frame-level semantic emphasis mechanism itself. Because the paper's central claim is an advantage in semantic emphasis validated on two datasets, and one of those datasets is circular in exactly the component under test, the overall result is partially circular rather than fully independent. There is no load-bearing self-citation chain; the circularity is data-label self-generation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and modular. It depends on pretrained feature encoders, on the validity of BEAT2 semantic labels, on the decomposition into base and sparse motion, and on pseudo-labels for SHOW generated by the model itself. Free parameters are limited to hyperparameters; no new physical entities are introduced.

free parameters (2)
  • Fusion threshold beta = 0.5
    Set empirically in Implementation Details ('beta = 0.5 empirically'); controls when sparse codes replace base codes in the fusion, directly affecting semantic emphasis strength.
  • InfoNCE temperature tau = 0.1
    Set in Implementation Details; governs sharpness of the rhythmic consistency loss in Eq. 4.
assumptions (4)
  • domain assumption HuBERT features encode sufficient rhythm and audio information for frame-level motion alignment.
    Used throughout Base Motion Generation and Rhythm Consistency Learning, Eq. 4; relies on pretrained HuBERT rather than a formal guarantee.
  • domain assumption BEAT2 frame-level semantic labels are reliable ground truth for semantic emphasis.
    Used to supervise the sem-gate classification loss and to compute Acc in Table 2; the paper does not audit label quality or inter-annotator agreement.
  • domain assumption Co-speech motion can be decomposed into a rhythm-related base and a sparse semantic layer, and this decomposition is sufficient for holistic generation.
    Key design premise in Section 3.2, inspired by TM2D; if gestures do not decompose this way, the two-branch architecture is not justified.
  • ad hoc to paper SHOW semantic labels generated by the paper's own sem-gate are valid for training and evaluating semantic emphasis on SHOW.
    Section 4.1: 'we use the sem-gate from SemTalk, pre-trained on BEAT2, to generate them.' This makes SHOW evaluation dependent on the model being evaluated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis." pith.science (2026). https://pith.science/paper/L3XQMTKN

@misc{pith2026241216563,
  author       = {Pith},
  title        = {Pith review of: SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L3XQMTKN}},
  note         = {Machine review of arXiv:2412.16563}
}
read the original abstract

A good co-speech motion generation cannot be achieved without a careful integration of common rhythmic motion and rare yet essential semantic motion. In this work, we propose SemTalk for holistic co-speech motion generation with frame-level semantic emphasis. Our key insight is to separately learn base motions and sparse motions, and then adaptively fuse them. In particular, coarse2fine cross-attention module and rhythmic consistency learning are explored to establish rhythm-related base motion, ensuring a coherent foundation that synchronizes gestures with the speech rhythm. Subsequently, semantic emphasis learning is designed to generate semantic-aware sparse motion, focusing on frame-level semantic cues. Finally, to integrate sparse motion into the base motion and generate semantic-emphasized co-speech gestures, we further leverage a learned semantic score for adaptive synthesis. Qualitative and quantitative comparisons on two public datasets demonstrate that our method outperforms the state-of-the-art, delivering high-quality co-speech motion with enhanced semantic richness over a stable base motion.

Figures

Figures reproduced from arXiv: 2412.16563 by the authors.

Figure 1
Figure 1. On the left, we analyze semantic labels from the BEAT2 dataset [31] and visualize frame-level motion, revealing that semantically relevant motions are rare and sparse, aligning with real-life observations. On the right, this observation drives the design of SemTalk, which establishes a rhythm-aligned base motion and dynamically emphasizes sparse semantic gestures at the frame-level. In this example, SemTalk amplifie… view at source ↗
Figure 2
Figure 2. An overview of the SemTalk pipeline. SemTalk gener￾ates holistic co-speech motion by first constructing rhythm-aligned q r in fr, guided by rhythmic consistency loss LRhy. Meanwhile, fs produce frame-level semantic codes q s , activated selectively by the semantic score ψ. Finally, q m is achieved by fusing q r and q s based on ψ, with motion decoder, yielding synchronized and contextually enriched motions. γh, extr… view at source ↗
Figure 3
Figure 3. Architecture of SemTalk. SemTalk generates holistic co-speech motion in three stages. (a) Base Motion Generation uses rhythmic consistency learning to produce rhythm-aligned codes q b , conditioned on rhythmic features γb, γh. (b) Sparse Motion Generation employs semantic emphasis learning to generate semantic codes q s , activated by semantic score ψ. (c) Adaptively Fusion automatically combines q b and q s based o… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Concept comparison with LivelySpeaker [52]. (Top) LivelySpeaker generates semantic gestures with CLIP embeddings in SAG and refines rhythm-related gestures separately using diffu￾sion, causing potential jitter. (Bottom) SemTalk integrates text and speech, uses a semant…
Figure 5
Figure 5. Figure 5: Comparison on BEAT2 [31] Dataset. SemTalk* refers to the model trained solely on the Base Motion Generation stage, capturing rhythmic alignment but lacking semantic gestures. In contrast, SemTalk successfully emphasized sparse yet vivid motions. For instance, when sayi…
Figure 6
Figure 6. Figure 6: Comparison on SHOW [48] Dataset. Our method performs better in motion diversity and semantic richness. discrepancies between GT and generated facial vertices. 4.2. Qualitative Results Qualitative Comparisons. We encourage readers to watch our demo video for a clearer u…
Figure 7
Figure 7. Figure 7: Facial Comparison on the BEAT2 [31] Dataset [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Qualitative study on semantic score. Semantic score aligns with keywords, influencing gesture intensity. emphasis learning. In facial comparisons ( [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 46 canonical work pages

  1. [1]

    Low-resource adaptation for personalized co- speech gesture generation

    Chaitanya Ahuja, Dong Won Lee, and Louis-Philippe Morency. Low-resource adaptation for personalized co- speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20566–20576, 2022. 2

    work page 2022

  2. [2]

    Listen, denoise, action! audio-driven motion synthesis with diffusion models

    Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–20, 2023. 2

    work page 2023

  3. [3]

    Gesturediffuclip: Gesture diffusion model with clip latents.ACM Transactions on Graphics (TOG), 42(4):1–18, 2023

    Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents.ACM Transactions on Graphics (TOG), 42(4):1–18, 2023. 3

    work page 2023

  4. [4]

    Audiolm: a language modeling approach to audio genera- tion

    Zal ´an Borsos, Rapha ¨el Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio genera- tion. IEEE/ACM transactions on audio, speech, and lan- guage processing, 31:2523–2533, 2023. 3

    work page 2023

  5. [5]

    Animated conversation: rule- based generation of facial expression, gesture & spoken in- tonation for multiple conversational agents

    Justine Cassell, Catherine Pelachaud, Norman Badler, Mark Steedman, Brett Achorn, Tripp Becket, Brett Douville, Scott Prevost, and Matthew Stone. Animated conversation: rule- based generation of facial expression, gesture & spoken in- tonation for multiple conversational agents. In Proceedings of the 21st annual conference on Computer graphics and in- ter...

    work page 1994

  6. [6]

    Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information

    Justine Cassell, David McNeill, and Karl-Erik McCullough. Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information. Pragmatics & cognition, 7(1):1–34, 1999. 1

    work page 1999

  7. [7]

    Beat: the behavior expression animation toolkit

    Justine Cassell, Hannes H ¨ogni Vilhj´almsson, and Timothy Bickmore. Beat: the behavior expression animation toolkit. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 477–486, 2001. 2

    work page 2001

  8. [8]

    Enabling synergistic full-body control in prompt-based co-speech motion generation

    Bohong Chen, Yumeng Li, Yao-Xiang Ding, Tianjia Shao, and Kun Zhou. Enabling synergistic full-body control in prompt-based co-speech motion generation. In Proceedings of the 32nd ACM International Conference on Multimedia , pages 6774–6783, 2024. 2

    work page 2024

Show all 53 references
  1. [9]

    Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and ges- ture generation

    Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and ges- ture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  2. [10]

    Emotional speech-driven 3d body animation via disentan- gled latent diffusion

    Kiran Chhatre, Nikos Athanasiou, Giorgio Becherini, Christopher Peters, Michael J Black, Timo Bolkart, et al. Emotional speech-driven 3d body animation via disentan- gled latent diffusion. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, ...

    1942

  3. [11]

    Envi- ronmental sound recognition with time–frequency audio fea- tures

    Selina Chu, Shrikanth Narayanan, and C-C Jay Kuo. Envi- ronmental sound recognition with time–frequency audio fea- tures. IEEE Transactions on Audio, Speech, and Language Processing, 17(6):1142–1158, 2009. 3

    2009

  4. [12]

    Emotional speech-driven animation with content-emotion disentangle- ment

    Radek Dan ˇeˇcek, Kiran Chhatre, Shashank Tripathi, Yan- dong Wen, Michael Black, and Timo Bolkart. Emotional speech-driven animation with content-emotion disentangle- ment. In SIGGRAPH Asia 2023 Conference Papers , pages 1–13, 2023. 2

    2023

  5. [13]

    Faceformer: Speech-driven 3d facial anima- tion with transformers

    Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech-driven 3d facial anima- tion with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18770–18780, 2022. 8

    2022

  6. [14]

    The role of gesture in communica- tion and thinking

    Susan Goldin-Meadow. The role of gesture in communica- tion and thinking. Trends in cognitive sciences, 3(11):419– 429, 1999. 1

    1999

  7. [15]

    Tm2d: Bimodality driven 3d dance generation via music-text integration

    Kehong Gong, Dongze Lian, Heng Chang, Chuan Guo, Zi- hang Jiang, Xinxin Zuo, Michael Bi Mi, and Xinchao Wang. Tm2d: Bimodality driven 3d dance generation via music-text integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9942–9952, 2023. 3

    2023

  8. [16]

    Momask: Generative masked model- ing of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked model- ing of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1900–1910, 2024. 3

    1900

  9. [17]

    Learning speech-driven 3d conversational gestures from video

    Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed El- gharib, and Christian Theobalt. Learning speech-driven 3d conversational gestures from video. In Proceedings of the 21st ACM International Conference on Intelligent Virt...

    2021

  10. [18]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:...

    2021

  11. [19]

    Robot behavior toolkit: generating effective social behaviors for robots

    Chien-Ming Huang and Bilge Mutlu. Robot behavior toolkit: generating effective social behaviors for robots. In Proceed- ings of the seventh annual ACM/IEEE international confer- ence on Human-Robot Interaction, pages 25–32, 2012. 2

    2012

  12. [20]

    Gesture: Visible action as utterance

    Adam Kendon. Gesture: Visible action as utterance. Cam- bridge University Press, 2004. 1

    2004

  13. [21]

    Gesture generation by imitation: From hu- man behavior to computer character animation

    Michael Kipp. Gesture generation by imitation: From hu- man behavior to computer character animation. Universal- Publishers, 2005. 2

    2005

  14. [22]

    Towards a common framework for multimodal generation: The behavior markup language

    Stefan Kopp, Brigitte Krenn, Stacy Marsella, Andrew N Marshall, Catherine Pelachaud, Hannes Pirker, Kristinn R Th´orisson, and Hannes Vilhj ´almsson. Towards a common framework for multimodal generation: The behavior markup language. In Intelligent Virtual Agents: 6th Internat...

    2006

  15. [23]

    Gesticulator: A framework for semantically-aware speech-driven gesture generation

    Taras Kucherenko, Patrik Jonell, Sanne Van Waveren, Gustav Eje Henter, Simon Alexandersson, Iolanda Leite, and Hedvig Kjellstr ¨om. Gesticulator: A framework for semantically-aware speech-driven gesture generation. In Proceedings of the 2020 international conference on mul- ti...

    2020

  16. [24]

    A large, crowdsourced eval- uation of gesture generation systems on common data: The genea challenge 2020

    Taras Kucherenko, Patrik Jonell, Youngwoo Yoon, Pieter Wolfert, and Gustav Eje Henter. A large, crowdsourced eval- uation of gesture generation systems on common data: The genea challenge 2020. In Proceedings of the 26th Interna- tional Conference on Intelligent User Interface...

    2020

  17. [25]

    A formal semantic analysis of gesture

    Alex Lascarides and Matthew Stone. A formal semantic analysis of gesture. Journal of Semantics , 26(4):393–449,

  18. [26]

    Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders

    Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Audio2gestures: Generating diverse gestures from speech audio with conditional varia- tional autoencoders. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 112...

    2021

  19. [27]

    Ai choreographer: Music conditioned 3d dance generation with aist++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 13401– 13412, 2021. 5

    2021

  20. [28]

    Seeg: Semantic energized co-speech gesture generation

    Yuanzhi Liang, Qianyu Feng, Linchao Zhu, Li Hu, Pan Pan, and Yi Yang. Seeg: Semantic energized co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10473– 10482, 2022. 2

    2022

  21. [29]

    Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthesis

    Haiyang Liu, Naoya Iwamoto, Zihao Zhu, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthesis. In Proceedings of the 30th ACM inter- national conference on multimedia, pages 3764–377...

    2022

  22. [30]

    Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis

    Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis. In European confer- ence on computer vision, pages 612–630. Springer, 2...

    2022

  23. [31]

    Emage: Towards unified holis- tic co-speech gesture generation via expressive masked audio gesture modeling

    Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J Black. Emage: Towards unified holis- tic co-speech gesture generation via expressive masked audio gesture modeling. In Proceedings of the IEEE/CVF ...

    2024

  24. [32]

    Learning hierarchical cross-modal association for co- speech gesture generation

    Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross-modal association for co- speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2022

  25. [33]

    Towards variable and coordinated holistic co-speech motion generation

    Yifei Liu, Qiong Cao, Yandong Wen, Huaiguang Jiang, and Changxing Ding. Towards variable and coordinated holistic co-speech motion generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1566–1576, 2024. 2

    2024

  26. [34]

    Humantomato: Text-aligned whole-body motion generation

    Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung Shum. Humantomato: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310.12978, 2023. 2

    2023 arXiv

  27. [35]

    emotion2vec: Self- supervised pre-training for speech emotion representation

    Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self- supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185, 2023. 4

    2023 arXiv

  28. [36]

    Virtual character perfor- mance from speech

    Stacy Marsella, Yuyu Xu, Margaux Lhommet, Andrew Feng, Stefan Scherer, and Ari Shapiro. Virtual character perfor- mance from speech. In Proceedings of the 12th ACM SIG- GRAPH/Eurographics symposium on computer animation , pages 25–35, 2013. 2

    2013

  29. [37]

    From audio to photoreal embodiment: Synthesizing humans in conversations

    Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard. From audio to photoreal embodiment: Synthesizing humans in conversations. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , ...

    2024

  30. [38]

    On-line integration of semantic information from speech and gesture: Insights from event-related brain po- tentials

    Aslı ¨Ozy¨urek, Roel M Willems, Sotaro Kita, and Peter Ha- goort. On-line integration of semantic information from speech and gesture: Insights from event-related brain po- tentials. Journal of cognitive neuroscience, 19(4):605–616,

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  32. [40]

    Real-time gesture animation generation from speech for vir- tual human interaction

    Manuel Rebol, Christian G ¨utl, and Krzysztof Pietroszek. Real-time gesture animation generation from speech for vir- tual human interaction. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1–4, 2021. 2

    2021

  33. [41]

    Difftalk: Crafting diffusion models for generalized talking head synthesis.arXiv preprint arXiv:2301.03786, 2(4):5, 2023

    Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized talking head synthesis.arXiv preprint arXiv:2301.03786, 2(4):5, 2023. 2

    2023 arXiv

  34. [42]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 2, 5

    2017

  35. [43]

    Learning product codebooks using vector-quantized autoencoders for image retrieval

    Hanwei Wu and Markus Flierl. Learning product codebooks using vector-quantized autoencoders for image retrieval. In 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 1–5. IEEE, 2019. 2

    2019

  36. [44]

    Codetalker: Speech-driven 3d facial animation with discrete motion prior

    Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with discrete motion prior. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12780–12790, 2023. 8

    2023

  37. [45]

    Chain of generation: Multi-modal gesture syn- thesis via cascaded conditional control

    Zunnan Xu, Yachao Zhang, Sicheng Yang, Ronghui Li, and Xiu Li. Chain of generation: Multi-modal gesture syn- thesis via cascaded conditional control. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6387– 6395, 2024. 3

    2024

  38. [46]

    Unifiedgesture: A unified ges- ture synthesis model for multiple skeletons

    Sicheng Yang, Zilin Wang, Zhiyong Wu, Minglei Li, Zhen- song Zhang, Qiaochu Huang, Lei Hao, Songcen Xu, Xiaofei Wu, Changpeng Yang, et al. Unifiedgesture: A unified ges- ture synthesis model for multiple skeletons. In Proceedings of the 31st ACM International Conference on Mul...

    2023

  39. [47]

    Diffusestylegesture: Stylized audio-driven co-speech ges- ture generation with diffusion models

    Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao. Diffusestylegesture: Stylized audio-driven co-speech ges- ture generation with diffusion models. arXiv preprint arXiv:2305.04919, 2023. 2, 5, 8

    2023 arXiv

  40. [48]

    Generating holistic 3d human motion from speech

    Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black. Generating holistic 3d human motion from speech. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 469–480, 2023. 1, 2, 5, 6, 8

    2023

  41. [49]

    Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots

    Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jae- hong Kim, and Geehyuk Lee. Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots. In 2019 International Conference on Robotics and Automation (ICRA) , pages 4303–4309. IEEE,

    2019

  42. [50]

    Speech ges- ture generation from the trimodal context of text, audio, and speaker identity

    Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech ges- ture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39 (6):1–16, 2020. 5

    2020

  43. [51]

    Soundstream: An end- to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end- to-end neural audio codec. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing , 30:495–507, 2021. 3

    2021

  44. [52]

    Livelyspeaker: Towards semantic-aware co-speech gesture generation

    Yihao Zhi, Xiaodong Cun, Xuelin Chen, Xi Shen, Wen Guo, Shaoli Huang, and Shenghua Gao. Livelyspeaker: Towards semantic-aware co-speech gesture generation. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 20807–20817, 2023. 2, 5, 8

    2023

  45. [53]

    Taming diffusion models for audio- driven co-speech gesture generation

    Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for audio- driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10544–10553, 2023. 2

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.