{"id":"fe346691-8ab2-410f-bf4f-aff159906d0a","arxiv_id":"2411.16729","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DiM-Gestor generates co-speech 3D gestures from Chinese speech using a Mamba-2 and AdaLN diffusion architecture, and releases the CCG dataset.","lead":"This paper introduces DiM-Gestor, a Mamba-2-based diffusion model that generates 3D full-body co-speech gestures from raw Chinese speech audio, and releases a 15.97-hour Chinese broadcast dataset. A smart generalist might read it because it claims Transformer-level gesture quality with roughly 2.4 times less memory and 2 to 4 times faster inference, which would matter for real-time virtual human applications.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline quality/efficiency claims rest on uncontrolled comparisons: English-trained baselines tested on Chinese CCG and PG benchmarked on A100 vs DiM on 4090.","rationale":"The paper's central claim is not simply that Mamba-2 can generate gestures; it is that replacing Transformer with AdaLN Mamba-2 yields comparable quality at much lower resource cost. For that claim to hold, the Transformer baseline must be evaluated under the same data and hardware conditions. The paper does not do this. This is not an attack on the architecture or dataset; the CCG dataset is a genuine contribution, and the ablation of Mamba-1/2/conv is informative. But the headline numbers cannot support the comparative conclusion as currently reported. The reader's weakest assumption points at exactly this uncontrolled comparison, and the required fix--retraining baselines on CCG and hardware-matched benchmarking--is concrete and feasible. I therefore agree with the conditional verdict: the work is publishable only after those controls are added or the claims are explicitly scaled back.","tokens_in":17185,"tokens_out":3527,"duration_ms":33634,"concrete_test":"Run a controlled benchmark: split CCG into the same train/val/test partitions; retrain PG-6blocks and PG-12blocks on those partitions using the same audio features, loss, and diffusion schedule; evaluate both retrained baselines and DiM-Gestor on the same test clips for FGD, BeatAlign, and a user study; on one GPU (e.g., a single A100 or 4090), measure peak allocated memory and median wall-clock inference for 20s/40s/60s/80s/100s gestures. If PG-12blocks retrained on CCG achieves FGD close to 17.716 or runs within 2x of DiM, the paper's quality/efficiency claims need to be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive comparison in the paper is DiM-Gestor against PG (and other Transformer baselines) for both quality and efficiency. The reader's concern is the right one: Section V.D states the baselines 'were initially trained using English speech datasets, including Trinity, ZEGGS, and BEAT,' while DiM was trained and inferred on CCG. That means the objective FGD gap (17.716 vs 100.899) and the subjective preference ratings embed two confounds: architecture and language/training-domain mismatch. A Transformer trained on English is not a valid control for judging whether Mamba-2 gives 'competitive results' on Chinese data. The efficiency claim is similarly uncontrolled: PG-12blocks uses 1.2B parameters and 12 AdaLN Transformer blocks and was run on an A100, while DiM m2 s m2 uses 535M parameters, 6 AdaLN Mamba-2 blocks, and was tested on a 4090. The reported 'approximately 2.4 times' memory reduction is simply the parameter-count ratio (1.2B/535M), not a measured peak-memory comparison, and the 2-4x speedup mixes architecture, block count, and GPU generation. If baselines are retrained on CCG and both methods are benchmarked on the same GPU, the headline gap may shrink substantially. This is load-bearing because both main contributions--competitive quality and efficiency--depend on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiM-Gestor, an end-to-end co-speech gesture generation model based on a Mamba-2 architecture augmented with Adaptive Layer Normalization (AdaLN) and a diffusion backbone. The model consists of a Mamba-2-based fuzzy feature extractor and an AdaLN Mamba-2 mapping module, and is trained and evaluated on a newly released Chinese Co-Speech Gestures (CCG) dataset comprising 15.97 hours of full-body motion from professional broadcasters. The authors report competitive subjective quality, a lower Frechet Gesture Distance than all compared baselines, and claim roughly 2.4x memory reduction and 2-4x faster inference relative to a Transformer-based Persona-Gestor model.","tokens_in":17473,"tokens_out":4395,"duration_ms":37254,"significance":"If the efficiency and quality claims were substantiated, DiM-Gestor would be a useful step toward linear-time, memory-efficient gesture generation for long sequences, and the CCG dataset would fill a gap for Chinese formal speech domains. The paper ships a new dataset and promises code release, and the ablation comparisons of Mamba-1/Mamba-2 and fuzzy extractor variants are systematic. However, the central comparative evidence is currently compromised by uncontrolled training-domain and hardware differences, so the significance of the architectural contribution is not yet established.","major_comments":[{"comment":"The objective FGD comparison is invalid as evidence of architectural superiority because all baselines (DSG+, GDC, LDA, Taming, PG-6blocks, PG-12blocks) were trained on English speech datasets (Trinity, ZEGGS, BEAT), while DiM-Gestor was trained and tested on the Chinese CCG dataset, as stated in Section V.D. The much lower FGD (17.716 vs. 100.899 for PG-12blocks) is expected from the language/domain mismatch alone. To support the claim of competitive results, all baselines must be retrained on CCG using the same training protocol and evaluated on the same test split.","section":"Section V.D, Table III"},{"comment":"The memory and inference-speed claims are not controlled. The 'approximately 2.4 times' memory reduction is computed from parameter counts (1.2B vs. 535M), not from measured peak GPU memory consumption, which also depends on activations, sequence length, and batch size. The inference-time comparison is confounded by hardware: Section V.B states that DiM-Gestor was tested on an NVIDIA GeForce 4090, while PG-12blocks was benchmarked on an A100. The authors should measure peak memory and latency for both models on the same GPU, with matched sequence lengths and batch sizes.","section":"Section VI, Tables IV and VI.A"},{"comment":"The subjective evaluation suffers from the same domain mismatch. The large advantage in style-appropriateness (1.30 vs. 0.664 for PG-12blocks) likely reflects the English training of the baselines rather than a property of the Mamba-2 architecture. The statistical results in Figure 10 show no significant difference between DiM m2 s m2 and PG-12blocks for human-likeness and appropriateness, so at most the paper can claim parity on those metrics. Retraining the baselines on CCG is required before any perceptual superiority of DiM-Gestor can be asserted.","section":"Section V.D.1, Table II"}],"minor_comments":[{"comment":"The notation is inconsistent across the paper: Eq. (1) defines X as audio and Y^0 as gesture, but Eq. (4) uses x0 and xn for original and noised gesture data, and Eq. (5) writes X0 for the gesture sequence. Please unify the notation throughout.","section":"Section III and Section IV.B"},{"comment":"In Algorithm 1, the variable X_{1:T} is used on the right-hand side before it is defined; the input line defines G_{1:T}, so the update should likely read G_{1:T} or X_{1:T}^{(0)}. Also, the second update line has unbalanced parentheses, making the formula ambiguous.","section":"Algorithm 1"},{"comment":"There is a typo in 'ANOV A' in Section V.D.1; it should be 'ANOVA'. Also, the title of Section VII spells 'DISSCUSTION' instead of 'DISCUSSION'.","section":"Section V.D"},{"comment":"Reference [27] attributes a 1908 paper to 'Paul, L.', but the cited work is by Paul Langevin on Brownian motion; the reference should be corrected to the proper author and title.","section":"Reference [27]"},{"comment":"The definition of style-appropriateness as 'the similarity between the generated gestures and their original human counterparts' appears to overlap with the definition of human-likeness. Please clarify the distinction or cite the original source of these evaluation criteria.","section":"Section V.D.1"}],"recommendation":"major_revision","confidential_remarks":"The reader's assessment is accurate: the headline quality and efficiency claims rest on uncontrolled comparisons. The English-trained baselines on a Chinese test set and the A100-vs-4090 benchmark are load-bearing issues that require re-running experiments. The CCG dataset and the Mamba-2 architectural exploration have merit, but the paper cannot be accepted in its current form. I recommend major revision with a requirement to retrain all baselines on CCG, report measured memory and latency on identical hardware, and temper the claims accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the two real contributions here are the CCG dataset (15.97 hours of professional Chinese TV broadcaster motion with speech, six styles across five scenarios) and the specific combination of AdaLN-conditioned Mamba-2 blocks inside a diffusion gesture generator. That combination is new as far as I know, and the ablations suggest Mamba-2 gives a better quality/efficiency trade-off than Mamba-1 or conv extractors within the authors' own framework. The dataset is the kind of resource the community can actually use, especially for non-English formal speech.\n\nWhat the paper does not do is show that DiM-Gestor is competitive with Transformer baselines in a controlled way. The baselines (PG-12blocks, LDA, DSG+, etc.) were trained on English data and evaluated on Chinese without retraining. The large FGD gaps (17.7 vs 100.9 for PG-12blocks) and the user-study preferences therefore conflate architecture with language/domain shift. The authors state this plainly in Section V.D, so it is not a hidden flaw, but it does mean the central comparative claim is unproven.\n\nThe efficiency claim has a similar problem. The “~2.4x memory reduction” is basically the parameter ratio (1.2B vs 535M), not a measured peak-memory comparison. And the inference speeds come from different GPUs: PG on an A100, DiM on a 4090. A fair benchmark would put both models on identical hardware, ideally after retraining baselines on CCG. My guess is the speed gap shrinks, though Mamba-2's linear complexity may still help at longer sequence lengths.\n\nMinor issues: typos and loose prose in places (e.g., “DISSCUSTION”), but nothing that undermines the method description.\n\nWho is this for? Someone working on co-speech gesture generation who wants a new non-English dataset and a candidate architecture to build on. It deserves a serious referee: the dataset and the idea are worth examining, but the current evaluation needs major rework before the quality/efficiency claims stand.\n\nRecommendation: send to review, but require the authors to retrain baselines on CCG and run controlled efficiency benchmarks on identical hardware.","headline":"A useful new Chinese gesture dataset and a plausible Mamba-2 diffusion architecture, but the headline quality and efficiency claims rest on uncontrolled comparisons.","tokens_in":18065,"tokens_out":2345,"would_cite":true,"duration_ms":20558,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a Mamba-2-based diffusion model with adaptive layer normalization generates co-speech gestures of Transformer-level quality while cutting memory by about 2.4 times and inference time by a factor of 2 to 4, validated…","keywords":["co-speech gesture generation","Mamba-2","adaptive layer normalization","diffusion model","Chinese co-speech gestures dataset","speech-driven gesture synthesis","virtual humans","efficient inference"],"falsifier":"Retrain the Transformer baselines, such as PG-12blocks, on the CCG training split and run all models on the same GPU with identical sequence lengths; if DiM-Gestor's FGD, BeatAlign, and 2-4x inference advantages largely disappear under these controlled conditions, the central efficiency claim would fail. A simpler check is to measure peak memory usage of both models on the same hardware for the same 20-second input to verify the reported 2.4 times reduction.","tokens_in":16987,"feed_emoji":"🗣️","tokens_out":5563,"duration_ms":46848,"temperature":0.7,"pith_summary":"The paper argues that the Mamba-2 state-space architecture, combined with adaptive layer normalization and a diffusion denoiser, can generate full-body co-speech gestures with quality comparable to Transformer-based systems while using roughly 2.4 times less memory and running 2 to 4 times faster at inference. To support this, the authors introduce the CCG dataset, 15.97 hours of 3D full-body skeleton motion from professional Chinese TV broadcasters across six styles and five scenarios, and evaluate their model, DiM-Gestor, on it. The best configuration reaches the lowest Fréchet Gesture Distance among all compared synthetic models and the highest user-rated style appropriateness, while matching the top Transformer baseline on human-likeness and appropriateness. The practical interest is that linear-complexity sequence models could make gesture generation feasible for real-time or long-duration virtual-human applications.","feed_headline":"Mamba-2 model matches Transformer gestures with 2.4x less memory","feed_subtitle":"New system runs 2-4x faster at inference and ships a 16-hour Chinese co-speech gesture dataset.","key_machinery":"The load-bearing mechanism is the Structured State Space Duality formulation of Mamba-2, where a 1-semiseparable Structured Masked Attention matrix replaces softmax attention, reducing the state size from linear to constant and the per-layer cost from quadratic to linear in sequence length. Around this core, the paper wraps adaptive layer normalization, which regresses the scale and shift applied to every token from the extracted speech features, and a denoising diffusion probabilistic model that learns to reverse a noising process on gesture sequences. Together, the fuzzy feature extractor supplies the condition, the AdaLN Mamba-2 blocks denoise, and the DDPM provides diverse outputs.","core_discovery":"The central claim is that replacing the Transformer backbone in a diffusion-based co-speech gesture model with Mamba-2, and conditioning it through adaptive layer normalization, preserves gesture quality while changing the scaling of inference cost from quadratic to linear in sequence length. DiM-Gestor's fuzzy feature extractor uses a Chinese pre-trained speech model plus a Mamba-2 global extractor to produce a unified latent condition, and the AdaLN Mamba-2 blocks regress per-token scale and shift parameters from that condition to guide denoising. In the authors' experiments, DiM-Gestor reaches an FGD of 17.716 on the feature space, versus 100.899 for the 12-block Persona-Gestor Transformer, with 535M parameters against 1.2B, while maintaining near-Transformer-level subjective scores. The paper also releases the CCG dataset, arguing that it fills a gap for formal, structured Chinese speech that existing English-dominated datasets do not cover.","pith_inferences":["If the Transformer baselines were retrained on the CCG dataset and evaluated on identical hardware, the reported FGD and speed gaps could shrink; the paper's comparisons mix English-trained baselines with a Chinese-trained model and use an A100 GPU for the baseline versus a 4090 GPU for DiM-Gestor.","The linear-complexity advantage should grow with sequence length, so testing beyond 100 seconds would reveal whether DiM-Gestor's inference time stays sublinear relative to Transformer baselines.","The style-appropriateness lead may partly reflect the dataset's own style distribution; a cross-dataset transfer test would separate model capability from dataset bias.","Combining the model with the accelerated diffusion sampler the paper mentions could push inference into interactive real-time range, likely at some cost in output quality."],"forward_implications":["Longer gesture sequences become much cheaper: at 100 seconds, DiM-Gestor infers in 23.27 seconds while the 12-block Persona-Gestor baseline takes 99.21 seconds in the paper's measurements.","Deployment on consumer hardware becomes easier: the model uses 535M parameters versus 1.2B for the strongest Transformer baseline, with roughly 2.4 times lower memory usage.","The CCG dataset gives the community a 15.97-hour, six-style, five-scenario Chinese corpus with full-body 3D skeleton and synchronized speech, enabling training and benchmarking on formal broadcast-style Chinese.","Style appropriateness is the configuration's strongest edge, suggesting that speaker- and style-related conditioning can come directly from raw audio without explicit style labels.","The linear-complexity backbone makes the approach a plausible candidate for real-time or interactive virtual-human systems that process long utterances."],"supporting_citations":[{"why":"Supplies the Mamba-2/Structured State Space Duality formulation that replaces Transformer attention and provides the linear-complexity backbone.","marker":"[14]"},{"why":"Defines the Persona-Gestor baseline and the fuzzy feature extractor concept that DiM-Gestor adapts to a Mamba-2 implementation.","marker":"[5]"},{"why":"Introduces AdaLN in diffusion transformers, the conditioning mechanism that the paper adapts to Mamba-2 blocks.","marker":"[6]"},{"why":"Provides the DDPM training objective and noise schedule used for the gesture diffusion process.","marker":"[25]"},{"why":"Chinese pre-trained speech model used to extract local audio features for the fuzzy feature extractor.","marker":"[23]"},{"why":"BEAT dataset, one of the English-language corpora on which baseline models were trained and a comparison point for dataset coverage.","marker":"[17]"},{"why":"Original Mamba selective state-space model, the foundation for Mamba-2's linear-time sequence modeling.","marker":"[7]"},{"why":"LDA diffusion-based gesture baseline that DiM-Gestor compares against in subjective and objective evaluations.","marker":"[4]"}],"fun_headline_variants":["Mamba-2 gestures match Transformers with 2.4x less memory","Mamba-2 gestures: same quality, 2.4x less memory","Faster co-speech gestures: Mamba-2 runs 2-4x quicker","DiM-Gestor: linear-time gestures with 2.4x memory cut"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparisons are fair even though the baseline models were trained on English speech data and evaluated on a new Chinese dataset, and even though the efficiency figures compare the baseline on an A100 GPU with DiM-Gestor on a 4090 GPU.","fun_headline_variants_meta":{"raw":{"variants":["Mamba-2 gestures match Transformers with 2.4x less memory","Mamba-2 gestures: same quality, 2.4x less memory","Faster co-speech gestures: Mamba-2 runs 2-4x quicker","DiM-Gestor: linear-time gestures with 2.4x memory cut"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000723,"raw_usage":{"total_tokens":3298,"prompt_tokens":1053,"completion_tokens":2245,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":2156}},"tokens_in":669,"tokens_out":2245,"duration_ms":14349,"temperature":1.0,"reasoning_tokens":2156,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:13:35.979648+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the Transformer baselines, such as PG-12blocks, on the CCG training split and run all models on the same GPU with identical sequence lengths; if DiM-Gestor's FGD, BeatAlign, and 2-4x inference advantages largely disappear under these controlled conditions, the central efficiency claim would fail. A simpler check is to measure peak memory usage of both models on the same hardware for the same 20-second input to verify the reported 2.4 times reduction.","supporting_citations":[{"cited_title":"Speech-driven personalized gesture synthetics: Harnessing automatic fuzzy feature inference,","cited_arxiv_id":null,"evidence_quote":"Defines the Persona-Gestor baseline and the fuzzy feature extractor concept that DiM-Gestor adapts to a Mamba-2 implementation."},{"cited_title":"chinese speech pretrain,","cited_arxiv_id":null,"evidence_quote":"Chinese pre-trained speech model used to extract local audio features for the fuzzy feature extractor."},{"cited_title":"Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis,","cited_arxiv_id":null,"evidence_quote":"BEAT dataset, one of the English-language corpora on which baseline models were trained and a comparison point for dataset coverage."}],"review_version":1}