{"id":"13989850-5e08-4226-af19-3aa4b06b559e","arxiv_id":"2507.19359","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-stage VQ-VAE and crossmodal transformer with coherence and relevance losses produces semantically aware co-speech gestures, beating four baselines on BEAT and TED Expressive for FGD, diversity, and SRGR.","lead":"SemGes is a two-stage model that generates 3D gestures for a virtual avatar from speech audio, transcript text, and speaker identity, aiming for gestures that match meaning, not just rhythm. The authors report better naturalness, diversity, and semantic relevance than four prior systems on BEAT and TED Expressive, which matters for building expressive conversational avatars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic coherence loss in Eq. 6 is not contrastive and is never directly evaluated; a retrieval test is needed before claiming discourse-level semantic grounding.","rationale":"The reader identified the BEAT semantic-relevance annotations as the weakest assumption, focusing on Eq. 9-10 and the circularity of SRGR. I agree that the SRGR metric is not an independent validation of semantic relevance because it shares labels with the training loss. However, I find a more direct and internal problem: the semantic coherence loss, which is half of the paper's central novelty, is not implemented as a contrastive objective. Eq. 6 has only positive-pair terms, despite the text claiming mismatched pairs are also minimized. This is an internal inconsistency, not merely a concern about external annotation quality. The ablation study does not resolve the issue because it only reports FGD, Diversity, and SRGR, none of which measure whether the learned text-motion space is discriminative across meanings. The proposed retrieval test would directly probe whether the coherence loss creates a semantic space in which a transcript can be matched to its corresponding gesture among distractors. If retrieval is at chance, the 'discourse-level context' claim is unsupported, and the paper should either correct Eq. 6 or soften the claim. The paper has real strengths: a two-stage VQ-VAE design, a user study, and public code/model releases, and the FGD/Diversity gains may still hold. The issue is fixable by adding explicit negative pairs and reporting retrieval metrics, so the reader's CONDITIONAL verdict remains appropriate; no change in verdict is needed.","tokens_in":14059,"tokens_out":6757,"duration_ms":70978,"concrete_test":"Run a held-out retrieval probe on the BEAT test set using the trained encoders: for each transcript in the test split, encode it with Es and encode K candidate gesture clips with the frozen Em (hand and body), then rank candidates by cos(Zs, Zh) + cos(Zs, Zb). Report R@1/R@5 and mean reciprocal rank. If retrieval accuracy is at or near chance (1/K), Eq. 6 does not build a discriminative semantic space and the coherence claim fails. As a control, train the same architecture with explicit in-batch negatives added to Eq. 6 (e.g., InfoNCE-style); if retrieval jumps while FGD stays similar, the published loss is the reason the coherence claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SemGes jointly captures discourse-level context via a semantic coherence loss rests on Eq. 6. The text in Section 3.2.1 says the loss maximizes similarity of correct (gesture, transcript) pairs and minimizes it for mismatched pairs, but Eq. 6 contains only positive-pair terms: L = (1 - cos(Zh, Zs)) + (1 - cos(Zb, Zs)). There is no negative term, no margin, and no temperature-scaled softmax. Minimizing these terms can be accomplished by mapping every transcript to a single vector or by whatever per-sample mapping reduces positive cosine distance; nothing in the objective penalizes a mismatched transcript being similar to a gesture. Thus the discourse-level context claimed by the paper is not a formal consequence of Eq. 6 as written. Moreover, no experiment isolates the coherence module: the ablation in Table 2 reports FGD, Diversity, and SRGR, but SRGR is computed from the same BEAT semantic-relevance labels used to train the relevance loss (Eq. 9), and FGD and Diversity do not measure semantic discriminability. The distinct contribution of the coherence loss is therefore under-specified and unevaluated. This is not an objection to using positive alignment as a regularizer; it is an objection to claiming that a non-contrastive objective produces semantic coherence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SemGes, a two-stage framework for co-speech gesture generation. Stage 1 trains separate VQ-VAEs for hand and body motions; Stage 2 uses a cross-modal Transformer conditioned on HuBERT audio features, FastText-based transcript embeddings, and speaker identity. Training combines a semantic coherence loss (Eq. 6), a multimodal quantization consistency loss (Eq. 7), and a semantic relevance loss (Eq. 9), with a clip-overlap long-sequence inference strategy. On BEAT and TED Expressive, the authors report improved FGD, Diversity, and SRGR over four baselines, plus user-study ratings, and provide ablations of the two semantic losses.","tokens_in":14341,"tokens_out":4264,"duration_ms":39872,"significance":"If the results hold, SemGes would be a useful practical advance: it is a complete two-stage system with code/data links, ablations, a user study, and state-of-the-art objective numbers on two benchmarks. The strongest assets are the attempt to combine global text-motion alignment with local relevance weighting and the evaluation on two datasets. However, the evidence for the central semantic-coherence claim is currently weak: Eq. (6) is not contrastive as claimed, the SRGR metric is tied to the training annotations, and no direct semantic alignment test is provided. These issues are fixable in revision, so the work is conditionally positioned rather than rejected.","major_comments":[{"comment":"The text states that the loss 'maximize[s] the similarity of correct (gesture, transcript) pairs and minimize[s] it for mismatched pairs,' but Eq. (6) contains only positive terms: (1 - cos(Z_h, Z_s)) + (1 - cos(Z_b, Z_s)). There are no negative samples, no margin, and no temperature-scaled softmax over negatives. Minimizing Eq. (6) can be achieved by mapping all transcripts to a single point or by any update that raises positive cosine similarity; nothing in the objective penalizes a mismatched transcript being similar to a gesture. Consequently, the claimed 'discourse-level context' and 'semantic coherence' are not formal consequences of the stated objective. I recommend adding a true contrastive term (e.g., in-batch negatives with a margin or temperature) and, more importantly, evaluating the module with a cross-modal retrieval or alignment test rather than only with global FGD and Diversity.","section":"Section 3.2.1, Eq. (6)"},{"comment":"The Semantic-Relevant Gesture Recall (SRGR) metric is computed from the same BEAT semantic-relevance annotations that weight the semantic relevance loss in Eq. (9). The SRGR gains in Table 1 may therefore reflect learning where annotated moments occur rather than generalizable semantic alignment; the paper does not report a direct semantic-alignment test that is independent of the training labels. In addition, the annotation relevance factor lambda in Eq. (9) is never specified, so the strength of the up-weighting and its effect on SRGR cannot be assessed. Please report a decoupled evaluation (e.g., human judgments of iconicity or metaphoricity on held-out unannotated moments, or a retrieval test on novel text-motion pairs) and specify the value or schedule of lambda.","section":"Section 5 and Eq. (9)"},{"comment":"The reported gains over baselines are presented without error bars, confidence intervals, or significance tests for any objective metric. Given that the central SOTA claim rests on FGD 4.467 versus 6.623 for DiffSheg, a single seed without variance is difficult to assess. Moreover, the Beat Consistency of SemGes (0.453) is far below all baselines (0.797–0.922), and the paper's explanation that this is 'expected' is supported only by a separate beat-dominant evaluation reported in prose (0.689), not in the main table. Please provide results over at least three seeds with variance and a significance test, and include the beat-dominant experiment in the tables.","section":"Table 1, Section 5"},{"comment":"The ablation for the semantic coherence module reports only FGD, Diversity, and SRGR, none of which measures whether matched text-motion pairs are closer than mismatched pairs. Thus the claimed contribution of the coherence loss is not independently verified. A simple retrieval precision@1 score or a contrastive alignment metric on held-out clips would directly test the module; without such a test, the 'w/o Semantic Coherence' row cannot distinguish the effect of semantic coherence from the effect of simply adding another text encoder to the model.","section":"Table 2, Section 5"}],"minor_comments":[{"comment":"The heading 'Quantitive Objective Evaluations' should read 'Quantitative Objective Evaluations.'","section":"Section 5 heading"},{"comment":"The stop-gradient notation in Eq. (4) is written as 'sgr' and is never defined; please define it explicitly or replace it with the standard 'sg' notation with a one-line explanation.","section":"Eq. (4)"},{"comment":"Algorithm 1 overwrites the first four frames of each clip with the last four frames of the previous clip, but the pseudocode does not describe any blending or smoothing at the seam, so it is unclear how 'smooth continuity' is achieved.","section":"Algorithm 1"},{"comment":"The main text repeatedly refers to supplementary sections for implementation details, additional evaluations, and limitations, but the supplementary material is not included with the arXiv submission; it should be made available for review.","section":"Supplementary material"},{"comment":"The baseline name is inconsistent: the method is called 'DiffGesture' in Table 1 but 'DiffGes' in Section 4.2; please use one consistent name throughout.","section":"Section 4.2 and Table 1"},{"comment":"The TED-Expressive column for 'w/o Semantic Relevance Module' is marked with dashes, but the text does not explicitly state in the table caption or notes that this is because the semantic relevance loss was not used for TED-Expressive; please add that explanation.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is a reasonable fit for the venue and the authors are clearly familiar with the relevant literature. My main concern is not the novelty of the individual components but that the central claims—especially semantic coherence and semantic relevance—are not yet supported by the stated objective functions and metrics. The revision should include a true contrastive objective or a clear justification for the non-contrastive one, a direct semantic-alignment evaluation, and variance reporting for the objective metrics. I would also like the supplementary material to be available during review. These are fixable within the scope of the manuscript, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: SemGes is a workmanlike two-stage gesture generator that beats four baselines on FGD and diversity, and the authors actually release code and data. The main claim to 'discourse-level semantic coherence' doesn't survive contact with Eq. 6: it contains only positive-pair cosine terms, so the text's promise to also minimize mismatched pairs is not realized in the objective. That is more than a wording issue — it means the semantic coherence loss is just an alignment regularizer, not a discriminative grounding mechanism, and no experiment evaluates whether the model retrieves the correct text for a gesture (or vice versa). The stress-test note holds up.\n\nWhat's solid: the VQ-VAE motion prior, the crossmodal transformer, the relevance weighting of semantically annotated moments, and the overlap-combine inference for long sequences. The ablations show removing either semantic loss hurts FGD/diversity, though the effect is modest. The user study with 30 participants is above baselines on naturalness, diversity, and alignment, with p<0.05.\n\nSoft spots, in proportion: (1) Eq. 6 is the clearest issue. A positive-only cosine loss can be satisfied by mapping everything to the same latent region; it does not enforce coherence. The authors need a contrastive term or a retrieval evaluation to support their central claim. (2) SRGR is computed from the same BEAT semantic-relevance annotations used to train the relevance loss in Eq. 9, so the SRGR gains are partly circular. A held-out annotation set or a text-motion retrieval metric would make the result believable. (3) Objective metrics are reported without error bars or significance tests, so the FGD gap to DiffSheg (4.467 vs 6.623) could be within noise; the paper needs multiple seeds or confidence intervals. (4) Lambda in Eq. 9 is never specified, which blocks reproduction.\n\nMinor but worth noting: BC is much lower than all baselines, and the explanation about jitter feels hand-wavy. If the model truly sacrifices beat alignment, that should be owned more directly.\n\nBottom line: this is a real, careful empirical system with an honest architecture, but the semantic grounding claims are currently under-supported. It deserves peer review — an editor should send it out, and a referee should ask for the contrastive/retrieval fix, error bars, and lambda.","headline":"Solid two-stage gesture generation with real gains, but Eq. 6 doesn't deliver the claimed semantic coherence and SRGR is built on the same annotations as the loss.","tokens_in":14825,"tokens_out":2498,"would_cite":true,"duration_ms":25570,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SemGes claims that aligning gesture and text embeddings at two semantic levels produces more realistic and more diverse co-speech gestures than existing methods.","keywords":["co-speech gesture generation","semantic coherence","semantic relevance","VQ-VAE","cross-modal transformer","multimodal gesture synthesis","speaker identity","semantic gestures"],"falsifier":"Have a fresh annotator mark semantic gestures on the BEAT test split and recompute SRGR, or rerun the user study, for SemGes with and without the semantic-relevance loss; if the advantage disappears under fresh annotations, the loss is fitting annotation patterns rather than generalizable semantics.","tokens_in":13903,"feed_emoji":"👐","tokens_out":6833,"duration_ms":63134,"temperature":0.7,"pith_summary":"The paper is trying to establish that co-speech gesture generation can be made semantically grounded by combining two complementary signals: a global alignment between text and motion latents and a local emphasis on moments annotated as carrying semantic meaning. It argues that current models mostly produce rhythmic beat gestures and miss representational gestures such as iconic or metaphoric ones. SemGes builds a two-stage system, a VQ-VAE motion prior for hands and body followed by a cross-modal transformer that fuses audio, text semantics, and speaker identity, and reports that the semantic coherence and relevance losses improve realism, diversity, and semantic recall. If the result holds, avatars could gesture with meaning rather than only rhythm.","feed_headline":"SemGes makes avatar gestures match the words being spoken","feed_subtitle":"Global and local semantic losses push co-speech gestures beyond beat-synchronized motion.","key_machinery":"The load-bearing mechanism is a two-stage latent-space design with two semantics-specific losses. The VQ-VAE stage discretizes body and hand motion into separate codebooks, giving a compositional motion prior. The semantic coherence loss then pulls a trainable text encoder into the frozen motion latent space via cosine similarity, aligning discourse-level meaning with movement. The semantic relevance loss, a piecewise quadratic-to-linear penalty scaled by an annotation factor, makes the decoder pay extra attention to temporally local moments labeled as semantically meaningful. Together with the multimodal quantization consistency loss, these objectives ensure that the final generated gestures match both word-level and global semantics.","core_discovery":"The central claim is that semantic gestures are best generated by keeping global and local semantics in one framework. In Stage 1, separate VQ-VAEs learn discrete codebooks for hand and body motion. In Stage 2, a cross-modal transformer takes HuBERT audio features, FastText word embeddings, and speaker identity; a semantic coherence loss maximizes cosine similarity between text encoder outputs and frozen motion encodings, while a semantic relevance loss up-weights reconstruction error at annotated semantic-gesture moments with a Huber-style piecewise penalty. A multimodal quantization consistency loss aligns the fused representation with the learned motion codebooks before decoding. On BEAT, SemGes reports an FGD of 4.467 versus 6.623 for the best baseline, an SRGR of 0.256, and a Diversity of 305.706; on TED Expressive it reports an FGD of 7.263 and a Diversity of 302.772. The paper also reports a lower Beat Consistency score and attributes this to the semantic focus rather than to rhythmic alignment.","pith_inferences":["A natural stress test is to relabel a held-out portion of BEAT with fresh annotations: if SRGR gains depend on matching the original annotator's label positions rather than on word meaning, the advantage should shrink.","The same coherence-plus-relevance recipe could transfer to other sparse-annotation generation tasks, such as templated explainer agents or sign-language animation, where only a few moments carry meaning.","Because the text encoder is trained against a frozen motion encoder, rare or abstract words with weak FastText embeddings may yield weaker iconic gestures; testing on low-frequency semantic words would reveal this.","A stronger test of generalization would be training on one set of topics and probing gestures for unseen topic-specific words, which the paper does not report."],"forward_implications":["Avatar systems can move beyond beat-synchronized gestures to produce iconic and metaphoric gestures tied to the words being spoken.","The overlap-and-combine inference routine lets a model trained on short clips generate long, smoothly continuous gesture sequences without additional training.","Freezing the motion encoder and training only the text-side alignment is sufficient to improve semantic recall, so the pipeline needs no extra motion supervision for semantics.","Because hands and body have separate codebooks, the framework can coordinate full-body and fine hand gestures from a single multimodal latent code."],"supporting_citations":[{"why":"Supplies the BEAT dataset, its semantic-relevance annotations used by the relevance loss and SRGR metric, and the CaMN baseline the work extends.","marker":"[27]"},{"why":"DiffSHEG, the strongest baseline in FGD on BEAT, against which SemGes reports the main objective improvement.","marker":"[10]"},{"why":"LivelySpeaker, the CLIP-based semantic-gesture baseline that motivates the global text-motion alignment branch.","marker":"[58]"},{"why":"DiffGesture, an audio-driven diffusion baseline included in the benchmark comparison.","marker":"[59]"},{"why":"Provides the VQ-VAE discrete representation learning method used for the Stage 1 motion prior.","marker":"[48]"},{"why":"HuBERT self-supervised speech features, used as the frozen audio encoder in Stage 2.","marker":"[20]"},{"why":"FastText subword word vectors, used as the frozen text embedding input for the semantic encoder.","marker":"[8]"},{"why":"Supplies the TED Expressive dataset and the pose extraction pipeline used as the second benchmark.","marker":"[33]"},{"why":"EMAGE, the two-stage VQ-VAE holistic gesture model that motivates the compositional motion-prior design.","marker":"[28]"}],"fun_headline_variants":["SemGes: avatar gestures that match spoken meaning","SemGes: semantic coherence for realistic co-speech gestures","SemGes: global and local semantics in one gesture model","SemGes: from beat gestures to semantic gestures","SemGes: semantic relevance beats rhythm in gestures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that BEAT's semantic-relevance annotations are complete and accurate enough that up-weighting reconstruction error at annotated moments teaches generalizable semantic gestures rather than merely teaching the model where annotators placed labels.","fun_headline_variants_meta":{"raw":{"variants":["SemGes: avatar gestures that match spoken meaning","SemGes: semantic coherence for realistic co-speech gestures","SemGes: global and local semantics in one gesture model","SemGes: from beat gestures to semantic gestures","SemGes: semantic relevance beats rhythm in gestures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1703,"prompt_tokens":930,"completion_tokens":773,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":698}},"tokens_in":546,"tokens_out":773,"duration_ms":7420,"temperature":1.0,"reasoning_tokens":698,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:54:07.060198+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a fresh annotator mark semantic gestures on the BEAT test split and recompute SRGR, or rerun the user study, for SemGes with and without the semantic-relevance loss; if the advantage disappears under fresh annotations, the loss is fitting annotation patterns rather than generalizable semantics.","supporting_citations":[{"cited_title":"Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis","cited_arxiv_id":null,"evidence_quote":"Supplies the BEAT dataset, its semantic-relevance annotations used by the relevance loss and SRGR metric, and the CaMN baseline the work extends."},{"cited_title":"Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expres- sion and gesture generation","cited_arxiv_id":null,"evidence_quote":"DiffSHEG, the strongest baseline in FGD on BEAT, against which SemGes reports the main objective improvement."},{"cited_title":"Livelyspeaker: Towards semantic-aware co-speech gesture generation","cited_arxiv_id":null,"evidence_quote":"LivelySpeaker, the CLIP-based semantic-gesture baseline that motivates the global text-motion alignment branch."},{"cited_title":"Taming diffusion models for audio- driven co-speech gesture generation","cited_arxiv_id":null,"evidence_quote":"DiffGesture, an audio-driven diffusion baseline included in the benchmark comparison."},{"cited_title":"Neural dis- crete representation learning","cited_arxiv_id":null,"evidence_quote":"Provides the VQ-VAE discrete representation learning method used for the Stage 1 motion prior."},{"cited_title":"Hubert: Self-supervised speech repre- sentation learning by masked prediction of hidden units","cited_arxiv_id":null,"evidence_quote":"HuBERT self-supervised speech features, used as the frozen audio encoder in Stage 2."},{"cited_title":"Enriching word vectors with subword information","cited_arxiv_id":null,"evidence_quote":"FastText subword word vectors, used as the frozen text embedding input for the semantic encoder."},{"cited_title":"Learning hierarchical cross-modal associa- tion for co-speech gesture generation","cited_arxiv_id":null,"evidence_quote":"Supplies the TED Expressive dataset and the pose extraction pipeline used as the second benchmark."},{"cited_title":"Emage: To- wards unified holistic co-speech gesture generation via expressive masked audio gesture modeling","cited_arxiv_id":null,"evidence_quote":"EMAGE, the two-stage VQ-VAE holistic gesture model that motivates the compositional motion-prior design."}],"review_version":2}