{"id":"7bc58af5-73b1-46b5-8bf3-08e676c11600","arxiv_id":"2505.01746","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Co3Gesture generates coherent concurrent co-speech gestures for two speakers using bilateral diffusion branches with temporal interaction and mutual attention, evaluated on the new GES-Inter dataset.","lead":"This paper introduces GES-Inter, a 70-hour dataset of two-person conversational gestures, and Co3Gesture, a diffusion model that generates both speakers' upper-body gestures from their separated speech. It reports large gains over existing gesture and text-to-motion models, but the benchmark is the authors' own dataset and several design details remain underspecified.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison is only as strong as the GES-Inter pseudo-label pipeline; the paper's own Appendix A.1.1 admits pose-estimation inaccuracies and aggressive filtering that may bias the benchmark.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the GES-Inter pseudo-label pipeline may be inaccurate and biased by aggressive filtering, undermining the benchmark and the model's reported superiority. I agree with the reader's CONDITIONAL verdict. The paper's own text supports this concern: Section 4.2 claims improvement 'by a large margin' over all competitors, but the comparison is only on GES-Inter, and Appendix A.1.1 explicitly states that some inaccuracies in pose estimation are unavoidable, that SmoothNet smoothing is used to refine jittery motions, and that manually inspecting one clip per group of five leads to discarding the entire group. These choices can systematically remove fast, extreme, or discontinuous motions, which are common in interactive conversation. The reported quantitative metrics (FGD on pseudo-labels, BC computed on smoothed estimated poses, Diversity on autoencoder features trained on the same pseudo-labels) are all internal to the filtered dataset, so they do not provide external validation. The user study is also based on only 15 participants and a small number of videos, and the reported t-test analysis in Table 7 is statistically fragile with such a sample size. None of this makes the paper's method implausible; the bilateral diffusion design and temporal interaction module are reasonable, and the ablations show consistent internal dependencies. But the central claim 'outperforms all competitors by a large margin' is not yet established outside the self-contained GES-Inter pipeline. A concrete external validation of the pseudo-labels and a test on a held-out conversational dataset would be needed to make the claim robust.","tokens_in":18933,"tokens_out":1186,"duration_ms":15282,"concrete_test":"Reproduce the benchmark comparison on a small externally validated subset: take 50 GES-Inter test clips, run PyMAF-X to get pseudo-labels, and independently obtain reference motion (for example, manually annotate keypoints in the video or use a different pose estimator, and where available compare with any mocap or multi-view data). Compute the FGD between the generated motion and each version of ground truth. If the FGD ranking among methods changes materially when evaluated against the independent reference instead of the pseudo-labels, the reported superiority is an artifact of the labeling pipeline.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that Co3Gesture outperforms all SOTA competitors on GES-Inter. That claim rests entirely on the fidelity and representativeness of the GES-Inter pseudo-labels, since the same pipeline provides both training and evaluation ground truth. The paper itself concedes that PyMAF-X estimates from in-the-wild talk shows contain unavoidable inaccuracies (Appendix A.1.1) and that SmoothNet temporal smoothing is applied to refine jittery motion. The filtering rules then discard sequences where wrist Euler angles exceed 150 degrees on any axis, where inter-frame pose changes exceed 25 degrees at 15 fps, and where manual inspectors sample one clip per group of five and discard the entire group if the sampled clip is judged unnatural. These rules could systematically remove the very fast, extreme, or discontinuous motions that distinguish interactive conversation from static interview behavior. If such motions are underrepresented or absent in the evaluation set, then a model trained to reproduce the filtered, smoothed, relatively static distribution will score well on FGD and BC without actually generating the full range of concurrent conversational gestures. The reported 70 hours and 7M frames also depend on this filtered pseudo-label quality, making the dataset's claimed advantage over mocap-based alternatives conditional on the accuracy of the automatic pose estimator and the soundness of the manual sampling procedure. The paper additionally reports only self-contained user studies with 15 volunteers, and the significance analysis in Table 7 uses t-tests on 15 participants without reporting effect sizes or inter-rater agreement, so the subjective evaluation is weak independent support. Because the benchmark's ground truth is not independently validated, the quantitative superiority of Co3Gesture may reflect the filtered data distribution rather than genuine interactive gesture quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GES-Inter, a large-scale pseudo-labeled dataset of two-speaker concurrent co-speech gestures extracted from in-the-wild talk shows and interviews, containing more than 7M frames and 70 hours of SMPL-X whole-body meshes with separated per-speaker audio, transcripts, and phoneme alignments. On top of this dataset, the authors propose Co3Gesture, a bilateral diffusion framework with two cooperative branches conditioned on the separated speaker audios, a Temporal Interaction Module (TIM) that fuses mixed-audio interaction cues into each branch, and a mutual attention mechanism intended to improve inter-speaker coherence. Experiments on GES-Inter report FGD, BC, Diversity, ablations, and a user study, claiming state-of-the-art performance over single-person co-speech and text2motion baselines. The paper also explicitly acknowledges that pose-extraction noise and upper-body-only generation are limitations.","tokens_in":19200,"tokens_out":6689,"duration_ms":64671,"significance":"If the claims hold, the paper makes two useful contributions: GES-Inter is, to my knowledge, the first large-scale mesh-based whole-body concurrent two-speaker co-speech gesture dataset, and the bilateral diffusion design with temporal interaction fusion is a plausible architecture for asymmetric dyadic gesture generation. The dataset and code are promised to be public, which is a concrete asset for the community. However, the significance is currently conditional: all quantitative evidence is obtained on the authors' own dataset, the mutual-attention mechanism is not formally specified, and the adaption of single-person baselines to the two-person setting is not described. These gaps prevent the reader from verifying the central superiority claim, so the contributions are not yet fully established.","major_comments":[{"comment":"The paragraph on SOTA comparisons states that all competitors are implemented from official source codes or pretrained models and that the text2motion counterparts receive 'the same audio encoder as ours,' but it does not specify how the single-person co-speech models (TalkSHOW, ProbTalk, DiffSHEG, EMAGE) are adapted to produce two simultaneous speakers, nor how the text-driven models (MDM, InterX, InterGen) are conditioned on audio rather than text or action labels. Without these adaptation details, Table 2 is not reproducible and the claim of a fair comparison is unsupported. Please give the exact input/output configuration for each baseline (e.g., per-speaker separated audio passed independently through a single-speaker model, two-branch variants, and where the audio encoder is inserted).","section":"Sec. 4.2 (SOTA comparisons)"},{"comment":"The mutual attention mechanism is described only in words: each branch is said to take the counterpart's features as the query in multi-head attention, with shared weight update strategies, but no equations, tensor shapes, or combination rules are provided. Since mutual attention is one of the two core technical contributions (abstract, contributions, and ablation Table 3), the method is not reproducible as written. Please provide the formal definition of the mutual attention operation, including how Q, K, V are formed from the two branch features and how the attended output is merged with the existing branch embeddings.","section":"Sec. 3.3 (Mutual Attention Mechanism)"},{"comment":"The paper defines L_foot following Tevet et al. and Table 5 reports that removing it degrades FGD from 0.769 to 1.082 and BC from 0.692 to 0.675. However, Section 4.1 and Appendix A.2 state that only upper-body joints are generated, and the ablation note for Table 5 says the lower body is completed as a T-pose in the forward-kinematic function used to compute the loss. With a fixed T-pose lower body, it is unclear what physical quantity the foot contact loss regularizes or how it can affect the generated upper-body motion. Please clarify the exact loss definition and explain the mechanism by which this loss produces such a large improvement.","section":"Sec. 3.4 and Table 5 (foot contact loss)"},{"comment":"All quantitative comparisons are performed on GES-Inter, the authors' own dataset, and the FGD feature extractor is presumably trained on GES-Inter as well. Because the pseudo-label pipeline (Appendix A.1.1) aggressively filters out fast, extreme, or jittery motions (wrist Euler angles above 150 degrees rejected, inter-frame changes above 25 degrees at 15 fps rejected, plus manual group-level discarding), the measured distribution may not be representative of conversational gestures in other settings. To support the claim of general superiority, please evaluate at least one existing dyadic or conversational benchmark (e.g., TWH16.2, BEAT2, or the GENEA 2023 dyadic setting) and describe how the FGD autoencoder is trained.","section":"Sec. 4 (evaluation scope)"},{"comment":"The tables report 95% confidence intervals only for Diversity; FGD and BC are given as point estimates. FGD is stochastic and known to be sensitive to the evaluation sample, and some reported gaps (e.g., 0.769 vs 1.012 in Table 2; 0.924 vs 0.769 in Table 3) may be within noise. Please report confidence intervals or standard deviations over multiple evaluation seeds or bootstrap resamples for FGD and BC, and state the number of resamples used.","section":"Tables 2-5 (confidence intervals)"}],"minor_comments":[{"comment":"Equation (3) appears to be missing norm notation; the terms should be written as ||x_a - D(x_a^(t), C_a, C_mix, t)||_2^2 and similarly for speaker b.","section":"Eq. (3)"},{"comment":"The word 'hyper-parameter' is misspelled as 'hper-parameter' in the sentence defining sigma(t).","section":"Sec. 3.4"},{"comment":"The citation for the MANO hand model is attributed to Boukhayma et al. (2019); the canonical MANO reference is Romero et al. (2017), and the current citation should be corrected.","section":"Sec. 3.1"},{"comment":"The user study section would benefit from a statement about whether the t-test was corrected for multiple comparisons across the seven paired comparisons (one per competitor).","section":"Appendix A.3"},{"comment":"The explicit limitation statement regarding pose-extraction noise and upper-body-only generation is appreciated; in the journal version, please connect this limitation to the external-validation request in the evaluation section.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is already published at ICLR 2025. For a journal submission, the authors should ensure that the revised version adds archival value beyond the conference version, in particular the requested external evaluation, the full mutual-attention specification, and the baseline-adaptation details. The missing equations and adaptation descriptions are fixable, but they are load-bearing for the paper's central claim, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: this paper's contribution is the dataset, not the method. GES-Inter—70 hours, 7M frames, SMPL-X whole-body meshes of two speakers in conversation, with separated audio, transcripts, and phoneme alignment—is a genuinely useful resource for a task (concurrent two-person co-speech gesture generation) that previously had no large-scale mesh-based dataset. Co3Gesture is a sensible combination of known components: two diffusion branches, cross-attention to audio, plus a temporal interaction module and mutual attention. The ablations show each component helps on their own benchmark. That part is fine.\n\nThe soft spots are real and mostly about evaluation. First, both training and testing come from the same pseudo-label pipeline (PyMAF-X + SmoothNet + aggressive filtering). The appendix admits pose inaccuracies and discards clips with wrist Euler angles beyond 150 degrees on any axis, inter-frame changes above 25 degrees at 15 fps, plus a manual 'sample one out of five, discard the whole group' rule. That can systematically remove fast or extreme motions that make conversations look interactive. If the benchmark's ground truth is a smoothed, filtered distribution, the reported FGD/BC gains may not transfer to the full range of real dyadic motion. A small external validation set, or at least a comparison on existing two-person mocap data like TWH16.2, would have helped a lot.\n\nSecond, the quantitative comparison is underspecified: FGD and BC are reported without confidence intervals, and the adaptation of single-speaker baselines to two-person output is not described. Do they receive the same audio, the other speaker's audio, or a mix? This matters for interpreting the 24% FGD improvement.\n\nThird, the user-study statistics look wrong. Table 6 reports means and standard deviations for 15 raters; Table 7 gives t-statistics. For TalkSHOW's naturalness, mean 2.0 (SD 0.1) versus yours 4.4 (SD 0.2) with n=15 should give a t near 40, not 5.345. Either Table 6's error bars are standard errors or the t-tests weren't computed from those numbers. This needs a correction or an explanation.\n\nThe mutual-attention part is described in prose without equations, which will make reproduction harder, but that's a medium issue.\n\nNet: the dataset is the contribution and it deserves to be used; the model is a reasonable first baseline. I would not cite the quantitative superiority as established fact, but I would cite the dataset. It already passed ICLR review, so it's in the public record. If the code and data are released as promised, it will become a reference point for dyadic co-speech gesture work. I'd send it to peer review if it came across my desk, and I'd bring it to a reading group if anyone in the group works on gesture generation.\n\nRecommendation: use the dataset, treat the model's reported wins as provisional, and check those user-study numbers before quoting them.","headline":"The dataset is a real contribution; the model's reported wins are plausible but rest on a self-contained pseudo-label benchmark, and the user-study statistics look inconsistent.","tokens_in":19807,"tokens_out":4664,"would_cite":true,"duration_ms":42428,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-speaker co-speech gestures can be generated coherently from conversational audio, and Co3Gesture backs this with a 7M-frame whole-body dataset.","keywords":["co-speech gesture generation","concurrent two-speaker gestures","diffusion models","temporal interaction module","GES-Inter dataset","whole-body 3D mesh pose","speaker diarization","mutual attention"],"falsifier":"Collect a small validation set of true motion-capture recordings of two-person conversations, or manually cleaned pose tracks from the same videos, and compare Co3Gesture's outputs against them with the same FGD and beat-consistency metrics; if the performance gap over baselines shrinks or vanishes on clean ground truth, the pseudo-label pipeline was doing the work. Alternatively, measure motion amplitude and wrist-angle distributions in GES-Inter against an unfiltered sample to test whether the filtering selected unusually static clips.","tokens_in":18761,"feed_emoji":"👥","tokens_out":6482,"duration_ms":62380,"temperature":0.7,"pith_summary":"Co3Gesture takes on a task that previous co-speech gesture models skip: generating the gestures of both people at once during a two-person conversation, rather than animating one speaker in isolation. To support this, the paper builds GES-Inter, a dataset of more than seven million frames of two-person whole-body mesh poses with separated audio, transcripts, and phoneme timestamps extracted from talk shows and interviews. The method uses two cooperative diffusion branches, one per speaker, each conditioned on that speaker's separated audio, plus a Temporal Interaction Module that blends mixed-audio interaction cues into each branch and a mutual-attention layer that lets the two gesture streams inform each other. On GES-Inter, the paper reports lower Fréchet Gesture Distance and higher beat consistency and diversity than single-speaker co-speech and text-to-motion baselines. If those results hold, avatar animation could move from monologue gestures to realistic dyadic conversation.","feed_headline":"New model animates both speakers from conversation audio alone","feed_subtitle":"Paired diffusion branches plus an interaction module beat single-speaker baselines on the new 7M-frame GES-Inter dataset.","key_machinery":"The load-bearing mechanism is the Temporal Interaction Module (TIM), a cross-attention unit that, for each speaker, takes the speaker's own audio embedding as a query against their motion features, then builds a temporal correlation matrix between the self-conditioned motion and a motion embedding conditioned on the mixed conversation audio. A sigmoid-gated learned weight balances the two, so each branch's gesture features are softly steered by the interaction context. On top of this, a mutual attention layer exchanges query information between the two branches, and shared weight updates across the bilateral denoisers exploit the exchange symmetry of the two speakers. The full system is trained with a diffusion denoising objective plus velocity and foot-contact losses.","core_discovery":"The paper's central claim is that coherent concurrent two-speaker gestures can be generated directly from conversational audio, and that the right way to do this is to keep the two speakers' generation separate but coupled. It constructs GES-Inter, a large-scale pseudo-labelled dataset of two-person interactive whole-body poses, and proposes Co3Gesture: two transformer-based diffusion denoisers, each generating one speaker's upper-body motion from that speaker's separated voice, while a Temporal Interaction Module fuses features from the mixed conversation audio into each branch as an interaction guide. A mutual attention mechanism then lets the two denoising streams exchange information. The paper reports that this design outperforms all compared single-speaker co-speech and text-to-motion baselines on GES-Inter, and that ablations removing the interaction module, mutual attention, bilateral branches, or mixed-audio conditioning all degrade the metrics.","pith_inferences":["If the pseudo-label noise is systematic, the benchmark may reward models that reproduce smoothed, filtered motion rather than genuine conversational behavior; a small motion-capture validation split would settle this.","The soft-weight fusion idea in TIM could generalize to group conversations by chaining pairwise interaction modules across more than two speakers.","The same two-branch diffusion plus interaction-guidance recipe is a plausible template for other dyadic motion tasks, such as two-person dance or turn-taking body language, by swapping the audio condition for music or another cue.","Reporting the retention rate of the filtering pipeline, that is, how many raw clips survive each processing step, would let future work judge how much of GES-Inter's quality comes from selection rather than modeling."],"forward_implications":["Two-speaker conversational gesture synthesis becomes a tractable generation task from audio alone, so virtual avatars can gesture in dialogue rather than only in monologue.","GES-Inter gives the research community a large-scale whole-body mesh dataset with separated audio, transcripts, and phoneme timestamps, enabling downstream work on talking-face generation, behavior analysis, and dyadic motion modeling.","The bilateral-branch design with a temporal interaction module is claimed to be necessary: replacing TIM with an MLP, removing mutual attention, or collapsing to a single holistic branch all raise FGD and lower beat consistency.","Using both separated speaker audio and mixed conversation audio as conditions is claimed to preserve speech-rhythm alignment while keeping the two gesture streams interactively coherent."],"supporting_citations":[{"why":"Supplies the monocular full-body pose estimator used to extract pseudo-ground-truth whole-body 3D poses for GES-Inter.","marker":"Zhang et al. (2023a)"},{"why":"Provides the speaker-diarization tool that separates mixed conversation audio into two identity-consistent streams, a precondition for the bilateral branches.","marker":"Bredin et al. (2020)"},{"why":"Provides the speech-recognition system that yields word-level transcripts and timestamps later aligned to phonemes for multimodal annotations.","marker":"Bain et al. (2023)"},{"why":"Defines the whole-body parametric mesh representation that GES-Inter pose labels and the generated joint configurations follow.","marker":"Pavlakos et al. (2019)"},{"why":"Supplies the temporal smoothing network used to reduce jitter in the extracted pseudo-pose sequences during dataset construction.","marker":"Zeng et al. (2022)"},{"why":"Contributes the diffusion denoising formulation and the velocity and foot-contact losses adopted in the training objective.","marker":"Tevet et al. (2023)"},{"why":"Defines the Fréchet Gesture Distance metric used to compare generated gesture distributions against real ones.","marker":"Yoon et al. (2020)"},{"why":"Provides the DDIM sampling strategy used at inference to denoise the generated gesture sequences.","marker":"Song et al. (2020)"},{"why":"Supplies the beat-consistency score and the train/validation/test split convention adopted for GES-Inter.","marker":"Liu et al. (2022a)"}],"fun_headline_variants":["Co3Gesture: two-speaker gestures from conversation audio","Two-person gesture generation: Co3Gesture outperforms single-talker models","GES-Inter dataset enables concurrent co-speech gesture synthesis","Co3Gesture: diffusion-based interactive gestures for two speakers","From audio to two avatars: Co3Gesture advances co-speech animation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark rests on the assumption that the automatically extracted and heavily filtered 3D poses from talk-show videos are accurate and representative enough to serve as ground truth for both training and evaluation; if those pseudo-labels contain systematic noise or the filtering kept only unusually static clips, the reported gains may not transfer to real conversational motion.","fun_headline_variants_meta":{"raw":{"variants":["Co3Gesture: two-speaker gestures from conversation audio","Two-person gesture generation: Co3Gesture outperforms single-talker models","GES-Inter dataset enables concurrent co-speech gesture synthesis","Co3Gesture: diffusion-based interactive gestures for two speakers","From audio to two avatars: Co3Gesture advances co-speech animation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00057,"raw_usage":{"total_tokens":2734,"prompt_tokens":1017,"completion_tokens":1717,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":1628}},"tokens_in":633,"tokens_out":1717,"duration_ms":15123,"temperature":1.0,"reasoning_tokens":1628,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:10:54.277623+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a small validation set of true motion-capture recordings of two-person conversations, or manually cleaned pose tracks from the same videos, and compare Co3Gesture's outputs against them with the same FGD and beat-consistency metrics; if the performance gap over baselines shrinks or vanishes on clean ground truth, the pseudo-label pipeline was doing the work. Alternatively, measure motion amplitude and wrist-angle distributions in GES-Inter against an unfiltered sample to test whether the filtering selected unusually static clips.","supporting_citations":[{"cited_title":"Pyannote","cited_arxiv_id":null,"evidence_quote":"Provides the speaker-diarization tool that separates mixed conversation audio into two identity-consistent streams, a precondition for the bilateral branches."}],"review_version":1}