{"id":"8df199b1-2b68-43cc-beb0-860d5474c063","arxiv_id":"2412.16563","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SemTalk generates co-speech body motion by separating rhythm-based base gestures from semantically important sparse gestures and blending them with a learned frame-level semantic score.","lead":"SemTalk is a new system for generating full-body gestures from speech: it first creates a rhythm-aligned base motion, then adds sparse, emphasis gestures for important words, controlled by a learned per-frame semantic score. On the BEAT2 and SHOW benchmarks it reports lower motion errors and better beat alignment than previous methods, with the largest gains attributed to the semantic emphasis branch.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SHOW experiments use semantic labels generated by SemTalk itself, so the two-dataset claim is not independent.","rationale":"The reader's weakest assumption correctly identifies the self-generated SHOW labels as the most load-bearing concern. My read agrees: the BEAT2 comparison and component ablations (Tables 1 and 3) provide genuine support for the architecture, and the concern does not invalidate those results. However, the paper's headline differs in scope: it claims superiority on two public datasets and semantic richness. The SHOW leg of that claim is compromised because the labels that define semantic emphasis come from SemTalk itself. This is not a mathematical inconsistency, but it means the second dataset does not provide independent confirmation. The correct disposition is to keep the reader's conditional verdict: accept the BEAT2 evidence, require code release and error bars, and demand re-evaluation of SHOW with independent semantic labels before the two-dataset claim is taken at face value. I do not see a more severe, load-bearing objection in the rest of the paper; other weaknesses, such as the small user study and lack of significance testing, are secondary to this one.","tokens_in":12979,"tokens_out":4274,"duration_ms":40699,"concrete_test":"On a held-out random 50-clip subset of SHOW test data, obtain frame-level semantic-emphasis annotations from human raters (or from an independent semantic gate trained on BEAT2 ground-truth labels and frozen, never fine-tuned on SHOW). Compute agreement between SemTalk's semantic score ψ and these independent labels (e.g., ROC-AUC or F1 at the reported β=0.5). Then re-run the user study with raters blind to method and report preference with confidence intervals. If agreement is near chance, or if user preference for 'semantic consistency' disappears when the sem-gate labels are replaced with independent ones, the SHOW-based semantic-richness claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SemTalk improves holistic co-speech motion by emphasizing frame-level semantic cues, validated on two datasets. On BEAT2 the evidence is clean: ground-truth frame-level semantic labels exist, and Table 1 shows consistent gains. The load-bearing weakness is the SHOW half. Section 4.1 states: 'Since it lacks frame-level semantic labels, we use the sem-gate from SemTalk, pre-trained on BEAT2, to generate them.' This makes SHOW a self-training loop: the same model family that defines semantic emphasis also produces the supervision used to train the SHOW model. The quantitative SHOW metrics (FGD, BC, MSE, LVD) do not use pseudo-labels directly, so this is not metric leakage; but the model receives an extra supervisory signal unavailable to baselines, and the qualitative 'enhanced semantic richness' claims are only ever judged against the gate's own notion of emphasis. If the pseudo-labels are biased toward SemTalk's preferred gesture patterns, the SHOW results would improve even without genuine semantic understanding, leaving only BEAT2 as an independent test. The BEAT2 results would survive, but the abstract's 'two public datasets' claim would be overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces SemTalk, a framework for holistic co-speech motion generation that decomposes motion into a rhythm-related base component and a sparse semantic component. The base motion is generated with a coarse-to-fine cross-attention module and local-global rhythmic consistency (InfoNCE) losses, while the sparse motion is produced from text, speech, emotion, and HuBERT features and gated by a learned semantic score. The final motion is obtained by replacing base codes with sparse semantic codes on frames where the semantic score exceeds a threshold. The method is evaluated on BEAT2 and SHOW against several recent baselines, with ablations, a user study, and qualitative comparisons.","tokens_in":13197,"tokens_out":4292,"duration_ms":37589,"significance":"If the claims hold, the explicit separation of rhythm-related base motion and semantic sparse motion with frame-level gating is a useful and plausible contribution to co-speech motion generation. The BEAT2 experiments are internally consistent and show improvements over strong baselines on FGD, MSE, and LVD, and the ablation study supports the importance of the proposed components. The idea of using a learned semantic score for adaptive fusion is clearly presented and worth pursuing. However, the SHOW validation is seriously weakened by the use of the model's own sem-gate to generate the semantic labels used for training, and the lack of error bars or significance tests limits confidence in the quantitative claims.","major_comments":[{"comment":"The SHOW experiments are circular for the semantic component. The text states: 'Since it lacks frame-level semantic labels, we use the sem-gate from SemTalk, pre-trained on BEAT2, to generate them.' This means the model under evaluation supplies the semantic supervision used to train it on SHOW, and the qualitative claims of enhanced semantic richness on SHOW are judged against the gate's own notion of emphasis. The global metrics (FGD, BC, MSE, LVD) are computed against ground-truth motion and are therefore not directly leaked, but the training signal gives SemTalk an extra supervisory input unavailable to the baselines, and the semantic-claim on SHOW is not independent. This is load-bearing for the abstract's claim of outperforming the state of the art on 'two public datasets.' Please either add human-annotated semantic keyframes on a SHOW subset, use an external labeler shared by all methods, or reframe the SHOW results as a domain-transfer demonstration without semantic-claim independence.","section":"Sec. 4.1"},{"comment":"All quantitative results are reported as single numbers with no error bars, confidence intervals, or significance tests. Many differences are small in absolute terms (e.g., BEAT2 BC 7.770 vs. 7.724, and SHOW BC 8.304 vs. 8.280), and without repeated runs it is impossible to tell whether these differences are meaningful. Please report mean and standard deviation over at least three random seeds for all metrics and, where appropriate, a paired significance test (e.g., bootstrap or Wilcoxon signed-rank).","section":"Sec. 4.3, Tables 1-3"},{"comment":"The ablation table contains a surprising inconsistency that the text does not explain. Adding RVQ alone increases MSE from 8.239 to 13.882 and LVD from 7.831 to 15.42, and the row with C2F+RVQ has MSE 13.416 and LVD 15.72, while the full model achieves 6.153 and 6.938. The text says RVQ 'slightly increases MSE and LVD,' but the increase is large. Please clarify whether the RVQ rows are valid configurations, whether the reported scales are consistent across rows, and how RC (and SE) interact with RVQ to produce the final values.","section":"Table 3"},{"comment":"The paper defines local and global rhythmic consistency losses, L_(Rhy)^(L) and L_(Rhy)^(G), but only Eq. (4) is given. It is not clear how the global sentence-level loss differs from the local frame-level loss, how the two are combined into the total loss, or whether the same temperature tau = 0.1 is used for both. Since rhythmic consistency is a central component of the base-motion claim, please provide the explicit form of L_(Rhy)^(G) and the total loss combination.","section":"Sec. 3.3, Eq. (4)"}],"minor_comments":[{"comment":"Notation is inconsistent for the base-motion codes: the text switches between qb and qr, and Eq. (5) uses fs and fb while the surrounding text uses qs and qb. Please unify the notation.","section":"Sec. 3.2, 3.4, 3.5"},{"comment":"The caption of Table 1 mentions 'FMD' but the metric is FGD. Also, the scaling factors (FGD×10−1, BC×10−1, MSE×10−8, LVD×10−5) should be stated in the table header or in the text before the caption, and the definitions of FGD, BC, DIV, MSE, and LVD should be given explicitly.","section":"Sec. 4.1 and Table 1"},{"comment":"The two weighting methods Wf and Wl are described only verbally. Please provide the equations for how the semantic score ψ is computed and how feature weighting and loss weighting are applied, including the classification loss L_cls^G and the code classification loss L_cls.","section":"Sec. 3.4"},{"comment":"The row labels in Table 3 are ambiguous: the baseline row with no components presumably still uses some VQ-VAE, but the 'RVQ' row is described as adding RVQ-VAE. Please clarify what the default quantizer is and what 'w/o RVQ' means.","section":"Table 3"},{"comment":"The user study reports only aggregate preferences without sample size per condition, effect sizes, or statistical tests. Please report the number of participants per video, the distribution of rankings, and significance of the preference differences.","section":"Sec. 4.2, Fig. 10"},{"comment":"The symbol N is described as 'the number of frames (or the batch size),' which is ambiguous. Please state once whether the InfoNCE loss is computed over frames within a sequence or over samples in a batch.","section":"Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with a plausible architecture and strong BEAT2 numbers, but the SHOW pseudo-labeling issue is the main correctness risk. If the authors cannot provide an independent evaluation on SHOW, the two-dataset claim should be softened and the SHOW results presented as a domain-transfer demonstration. The lack of error bars is also a recurring weakness that should be addressed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a competent, incremental architecture paper that does one thing well and overclaims one thing. The genuinely new bit is the learned frame-level semantic gate that weights both features and loss, applied to a sparse semantic motion branch fused with a rhythm base motion. On BEAT2, where frame-level semantic labels are ground truth, the ablations are consistent and the gains over EMAGE look real (FGD 4.278 vs 5.512). That part is solid.\n\nThe soft spot is SHOW. The paper says it generates semantic labels with the SemTalk sem-gate pre-trained on BEAT2. That makes the SHOW training a self-training loop: the model's own notion of semantic emphasis supervises the SHOW model, and the qualitative 'semantic richness' claims are judged against that same gate. The metrics (FGD, MSE, LVD) don't directly use the labels, so it's not metric leakage, but the model gets an extra supervisory signal baselines don't. The BEAT2 results survive; the abstract's 'two public datasets' claim is overstated.\n\nAlso minor: no error bars anywhere, and the fusion threshold beta=0.5 is empirical. Those are minor relative to the circularity.\n\nWho's it for: people working on speech-driven gesture or holistic motion generation. The BEAT2 numbers and the semantic gate ablation are worth a close look. I'd bring it to a reading group.\n\nSend it to peer review. The BEAT2 half is a legitimate contribution; the SHOW half needs either independent labels or a clearly framed self-training variant, plus code and error bars. A serious referee should engage with it.","headline":"Solid BEAT2 evidence for a genuinely new semantic-gated architecture, but the SHOW half relies on self-generated labels, so the two-dataset claim is not independent.","tokens_in":13730,"tokens_out":1599,"would_cite":true,"duration_ms":14390,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SemTalk argues that co-speech motion should be generated as a rhythm-locked base layer plus a sparse, semantically gated overlay that is fused adaptively, and reports state-of-the-art quality on BEAT2 and SHOW.","keywords":["co-speech motion generation","semantic emphasis","gesture synthesis","rhythmic consistency","semantic gate","RVQ-VAE","frame-level semantic score","holistic motion"],"falsifier":"Re-evaluate SemTalk on SHOW using human-annotated or independently produced semantic labels instead of the sem-gate's own predictions, then compare FGD and BC against EMAGE and TalkSHOW; if the reported gains disappear with independent labels, the semantic emphasis advantage is largely an artifact of self-labeling.","tokens_in":12778,"feed_emoji":"🤲","tokens_out":2844,"duration_ms":25811,"temperature":0.7,"pith_summary":"SemTalk is a framework for generating full-body co-speech motion that treats rhythmic gestures and semantic gestures as separate problems. The paper's core idea is that most frames follow speech rhythm and only a few frames carry meaning, so the model learns a rhythm-aligned base motion and then learns a semantic score that highlights sparse, meaningful gestures. A coarse-to-fine cross-attention module and local-global rhythmic consistency losses build the base motion, while a learned semantic gate produces a frame-level score that adaptively fuses the semantic overlay into the base. On BEAT2 the method reports lower FGD, MSE, and LVD than prior models, and on SHOW it reports lower FGD and higher beat correlation.","feed_headline":"Splitting gesture into rhythm and meaning beats prior models","feed_subtitle":"A learned semantic gate highlights sparse keyframe gestures, cutting FGD on BEAT2 and SHOW below previous methods.","key_machinery":"The load-bearing object is the semantic score ψ produced by the sem-gate: it is a learned, frame-level scalar that decides when the sparse semantic codes are activated and how strongly they are blended into the rhythm-aligned base codes. The base motion itself is built by a hierarchical Coarse2Fine Cross-Attention module that propagates guidance from face to hands to upper body to lower body, plus a local and global rhythmic consistency loss based on InfoNCE that aligns latent motion features with HuBERT audio features at both frame and sequence level.","core_discovery":"The paper claims that separately learning rhythm-related base motion and semantic-aware sparse motion, then fusing them through a learned frame-level semantic score, yields more natural and more semantically expressive co-speech gestures than jointly conditioned baselines. On BEAT2, SemTalk reports FGD 4.278 versus EMAGE's 5.512, MSE 6.153 versus 7.680, and LVD 6.938 versus 7.556, and on SHOW it reports FGD 20.18 versus 22.12 and BC 8.304 versus 8.280. The key mechanism is the semantic gate, which outputs a per-frame score that both weights the sparse semantic features and supervises the loss, allowing the model to emphasize key moments such as the hand raise for 'my opinion' while preserving the stable rhythmic base elsewhere.","pith_inferences":["The decomposition principle may transfer to other co-speech or expressive-motion tasks, such as sign-language nuance or public-speaking rehearsal, where most motion is prosodic but a few moments carry the message.","A testable extension is to use SemTalk's semantic score as a pseudo-labeler for new unlabeled motion datasets, then measure whether downstream models trained on those labels match the quality of models trained on manual annotations.","If the semantic score is stable under speaker identity changes, it could serve as an interpretable diagnostic for whether a gesture-generation model is actually 'listening' to content rather than only to prosody.","The self-generated labels on SHOW leave open the question of whether the reported semantic-consistency gain survives under independent annotations, which would be the decisive check of the semantic emphasis claim."],"forward_implications":["Sparse semantic gestures that prior rhythm-dominated models dilute can be recovered explicitly, because the model does not have to compromise base motion to express meaning.","The learned semantic score provides a frame-level control signal that could be used for editing generated motion by changing which frames are emphasized.","By separating rhythm and semantics, the same base motion block can be reused across speakers, with only the semantic overlay modulated by content and emotion.","The method produces smoother facial and full-body motion than VQ-VAE baselines, as measured by lower LVD on both BEAT2 and SHOW.","The semantic score adapts to emotional tone even when the transcribed text stays the same, indicating that the model captures delivery-dependent emphasis rather than purely word-level cues."],"supporting_citations":[{"why":"Supplies the BEAT2 dataset with paired audio, text, and frame-level semantic labels, and the EMAGE baseline that SemTalk compares against.","marker":"[31]"},{"why":"Supplies the SHOW dataset and the TalkSHOW baseline, including the evaluation protocol for facial MSE and LVD.","marker":"[48]"},{"why":"Provides the LivelySpeaker baseline and its SAG component, which SemTalk replaces with its semantic gate in the comparison.","marker":"[52]"},{"why":"Supplies HuBERT features used as the rhythmic audio condition for the base motion blocks and the rhythmic consistency losses.","marker":"[18]"},{"why":"Supplies CLIP sentence-level text features that contribute to the semantic representation fed into the semantic emphasis learning.","marker":"[39]"},{"why":"Supplies emotion2vec features that capture emotional tone for the semantic gate, letting the model vary gestures for the same words.","marker":"[35]"},{"why":"Provides DiffSHEG as a holistic baseline and motivates the face-to-hands guidance used in the Coarse2Fine Cross-Attention module.","marker":"[9]"},{"why":"Supplies the VQ-VAE architecture for the RVQ-VAE codec that discretizes motion into codes for both base and sparse blocks.","marker":"[42]"},{"why":"Motivates the use of InfoNCE alignment between motion and audio features, which SemTalk extends into local and global rhythmic consistency losses.","marker":"[45]"},{"why":"Supplies the beat-detection features (amplitude, short-time energy, onset detection) that define the rhythmic beats γb.","marker":"[11]"}],"fun_headline_variants":["Split gesture into rhythm and semantic spikes","Semantic score gates sparse gestures over stable base","Frame-level semantics beat joint co-speech baselines","Two-stream motion: rhythmic base, semantic emphasis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cleanest results stand on the assumption that the frame-level semantic labels in BEAT2 are reliable and that a sem-gate trained on them transfers to SHOW without bias; on SHOW the paper generates its own semantic labels with the very model under test.","fun_headline_variants_meta":{"raw":{"variants":["Split gesture into rhythm and semantic spikes","Semantic score gates sparse gestures over stable base","Frame-level semantics beat joint co-speech baselines","Two-stream motion: rhythmic base, semantic emphasis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1342,"prompt_tokens":879,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":405}},"tokens_in":495,"tokens_out":463,"duration_ms":22462,"temperature":1.0,"reasoning_tokens":405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:27:47.655923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-evaluate SemTalk on SHOW using human-annotated or independently produced semantic labels instead of the sem-gate's own predictions, then compare FGD and BC against EMAGE and TalkSHOW; if the reported gains disappear with independent labels, the semantic emphasis advantage is largely an artifact of self-labeling.","supporting_citations":[{"cited_title":"Emage: Towards unified holis- tic co-speech gesture generation via expressive masked audio gesture modeling","cited_arxiv_id":null,"evidence_quote":"Supplies the BEAT2 dataset with paired audio, text, and frame-level semantic labels, and the EMAGE baseline that SemTalk compares against."},{"cited_title":"Generating holistic 3d human motion from speech","cited_arxiv_id":null,"evidence_quote":"Supplies the SHOW dataset and the TalkSHOW baseline, including the evaluation protocol for facial MSE and LVD."},{"cited_title":"Livelyspeaker: Towards semantic-aware co-speech gesture generation","cited_arxiv_id":null,"evidence_quote":"Provides the LivelySpeaker baseline and its SAG component, which SemTalk replaces with its semantic gate in the comparison."},{"cited_title":"Hubert: Self-supervised speech representation learning by masked prediction of hidden units","cited_arxiv_id":null,"evidence_quote":"Supplies HuBERT features used as the rhythmic audio condition for the base motion blocks and the rhythmic consistency losses."},{"cited_title":"Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and ges- ture generation","cited_arxiv_id":null,"evidence_quote":"Provides DiffSHEG as a holistic baseline and motivates the face-to-hands guidance used in the Coarse2Fine Cross-Attention module."},{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Supplies the VQ-VAE architecture for the RVQ-VAE codec that discretizes motion into codes for both base and sparse blocks."},{"cited_title":"Chain of generation: Multi-modal gesture syn- thesis via cascaded conditional control","cited_arxiv_id":null,"evidence_quote":"Motivates the use of InfoNCE alignment between motion and audio features, which SemTalk extends into local and global rhythmic consistency losses."},{"cited_title":"Envi- ronmental sound recognition with time–frequency audio fea- tures","cited_arxiv_id":null,"evidence_quote":"Supplies the beat-detection features (amplitude, short-time energy, onset detection) that define the rhythmic beats γb."}],"review_version":1}