{"id":"7598c8ff-4377-4996-aa6d-c2a0f2623599","arxiv_id":"2412.10523","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A single language model takes speech, text, or motion tokens as input and generates expressive 3D body motion, text, or emotion labels, using pretraining on unpaired audio and motion data.","lead":"This paper builds one AI system that can read and generate human body motion from speech, text, or existing motion, and can switch between these inputs and outputs. It could make animated characters in games and virtual reality respond to what a person says and how they move, using a single shared model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pre-training may include the test speaker's motion, threatening the data-scarcity and generalization claims.","rationale":"The paper's central claims are: (1) a unified multimodal language model achieves SOTA co-speech gesture generation, (2) generative pre-training enables strong novel-speaker generalization with far less paired data, and (3) novel tasks such as emotion prediction from motion. The reader's CONDITIONAL verdict identifies the weakest assumption as the possibility that pre-training motion data includes the test speaker, which would undermine the data-scarcity generalization result. I agree this is the most load-bearing concern: Fig. 5 and the 'never seeing speech-motion data' statement are the evidence for the generalization advantage, and the paper's text never rules out distribution overlap on the evaluation speaker. The concern is concrete and checkable — it is not a disagreement with consensus or an internal inconsistency, but a missing experimental control. Because the framework itself is coherent and the co-speech results are competitive, the appropriate verdict remains CONDITIONAL pending clarification of speaker exclusion and tokenizer details. The paper deserves credit for a well-designed two-stage training scheme, a reasonable ablation table, and an honest discussion of tokenization limitations; the issue is evidential, not architectural.","tokens_in":17833,"tokens_out":1913,"duration_ms":14432,"concrete_test":"Ask the authors (or check code/checkpoints when released) whether the BEATv2 motion used for spatial and temporal pre-training contains any clips from speaker-2, the evaluation speaker. A definitive test: retrain the full pipeline with pre-training restricted to BEATv2 speakers other than speaker-2 (or with speaker-2 clips explicitly held out), and rerun the Fig. 5 data-scarcity sweep; if the FGD advantage over 'w/o pre-training' at 1/32 data shrinks by more than a few percent, the generalization claim is materially weakened. Also clarify in the paper which motion tokenizer produced the tokens for the Table 1 evaluation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1 states that pre-training uses BEATv2 and Librispeech, that no audio-to-motion data is seen, and that post-training adopts the EMAGE protocol of training and testing on BEATv2 speaker-2. However, the paper never states that the BEATv2 motion used for spatial/temporal pre-training excludes speaker-2 motion. Since BEATv2 contains multiple speakers and the pre-training motion is 'unpaired' only in the sense of not being paired with audio, speaker-2 motion clips could appear in the spatial body-part translation and temporal masking tasks. If speaker-2 motion is in pre-training, then Fig. 5's comparison at 1/32 to 1/2 of post-training data is not a novel-speaker evaluation: the model has already seen the target speaker's pose statistics, body-part co-occurrences, and temporal dynamics. That would inflate the headline claims of 'never seeing speech-motion data' generalization and 'remarkable generalization' to a novel speaker. The paper also does not specify which motion tokenizer is used for Table 1 — it says 'using their motion tokenizer' referring to EMAGE, but the method section describes training four new VQ-VAEs on the compositional representation; if Table 1 uses the new tokenizer, the comparison is fairer as a method comparison, but if it uses EMAGE's tokenizer the pre-training and post-training token spaces may not match. Either way, the missing speaker-exclusion statement is the load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multimodal language model that unifies text, speech, and compositional 3D body motion (face, hands, upper body, lower body) in a single encoder-decoder T5-based architecture with discrete tokenization. The training pipeline has two stages: a generative pre-training stage with spatial and temporal body-part alignment tasks plus audio-text alignment on unpaired data, and a post-training instruction-following stage on paired audio-to-motion, text-to-motion, and emotion-related tasks. The authors report state-of-the-art FGD/BC/Diversity on the BEATv2 co-speech gesture benchmark, ablations of each pre-training task, a data-scarcity experiment showing improved FGD at small post-training data fractions, qualitative editable gesture generation with joint audio-text prompts, and a new emotion-from-motion prediction task.","tokens_in":18112,"tokens_out":2605,"duration_ms":24795,"significance":"If the results hold, the paper makes a useful contribution by showing that a single language-model backbone can handle multiple motion-related input/output modalities and that self-supervised pre-training on unpaired motion can reduce paired-data requirements for co-speech gesture generation. The compositional body-part tokenization and the explicit audio-text alignment pre-training task are reasonable design choices, and the ablation study (Table 2) gives some credit to the proposed components. The newly introduced emotion-prediction task and the qualitative editable-generation examples broaden the scope beyond standard co-speech benchmarks. The paper is clearly written and the experimental plan is generally sensible. However, the central quantitative claims rest on a few load-bearing experimental details that are currently underspecified, as detailed below.","major_comments":[{"comment":"The data-scarcity and novel-speaker generalization claims depend on whether the pre-training motion data excludes the test speaker. Section 4.1 states that pre-training uses BEATv2 and Librispeech and that no audio-to-motion data is seen, but it never states that the BEATv2 motion used for the spatial and temporal pre-training tasks excludes speaker-2, the speaker used for post-training and evaluation in the EMAGE protocol. If speaker-2 motion clips appear in the pre-training corpus, then Fig. 5's comparison at 1/32 to 1/2 of post-training data is not a clean novel-speaker evaluation: the model would already have access to the target speaker's pose statistics, body-part co-occurrences, and temporal dynamics. The authors should state explicitly whether speaker-2 motion was held out from all pre-training tasks, and if it was not, the data-efficiency and 'never seeing speech-motion data' claims must be re-framed or re-run with a clean speaker holdout.","section":"§4.1, §4.2, Fig. 5"},{"comment":"The core SOTA claim rests on a small numerical margin (FGD 5.301 vs. 5.512 for EMAGE, and only 0.107 better than the no-multimodal-pre-training ablation), and neither Table 1 nor Fig. 5 reports variance bars or significance across multiple seeds. Given the stochasticity of VQ-VAE training, autoregressive generation, and the metrics themselves, a single-run comparison of this size is not enough to establish that the improvements are not noise. The authors should provide mean and standard deviation over at least three seeds for the main comparison and for the data-scarcity curves, or otherwise justify why variance is negligible.","section":"Table 1, Fig. 5"},{"comment":"The sentence 'we adopt the same evaluation protocol as [43], i.e., training and testing on speaker-2 and using their motion tokenizer' is ambiguous about which tokenizer was used to produce Table 1. If Table 1 uses EMAGE's tokenizer, then the pre-trained model's own compositional tokenizer (Section 3.2) is not being evaluated in the main benchmark, and the connection between pre-training and downstream performance is unclear. If Table 1 uses the newly trained four-part VQ-VAEs, then the comparison with EMAGE, which was trained with its own tokenizer, may be unfair because the reconstruction quality of the tokenizers differs. Please clarify which tokenizer was used in each reported result and, if the new tokenizer is used, report tokenizer reconstruction metrics so that the comparison is interpretable.","section":"§4.1, Table 1"},{"comment":"The emotion-prediction experiment does not specify how the BEATv2 emotion labels were split between instruction-tuning and evaluation. Since the same emotion labels are used to construct the training instructions, evaluating on an overlapping set would make the reported BLEU/ROUGE/CIDEr/BERTScore numbers uninformative about generalization. Please report the exact split (e.g., speaker or clip-level holdout) and clarify whether the test emotion labels were seen during post-training. In addition, using text-similarity metrics against a single ground-truth label is unusual for a classification-style task; reporting accuracy or top-k accuracy would make the result more interpretable.","section":"§4.4, Table 3"}],"minor_comments":[{"comment":"There are several typos and spacing errors, e.g., 'modelt' at the start of Section 3.2, 'W eather' in the abstract, 'V erbal' in the title line, and 'co-speech' appearing as 'a co-speech' in several places. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The caption and text say 'as the amount of paired fine-tuning data increases, the performance reduces,' but FGD is a lower-is-better metric, so the text should say the FGD value decreases/improves. This wording inversion could confuse readers.","section":"Fig. 5"},{"comment":"The editable gesture generation results are purely qualitative; there is no quantitative metric or user study for the joint audio-text conditioning. The paper should at least state that this is a demonstrative capability and discuss how it could be evaluated, or add a small quantitative study, to avoid overstating the claim.","section":"§4.3"},{"comment":"The supplementary states that text-to-motion quantitative metrics are not used because they are entangled with the HumanML3D representation. This is a reasonable caveat, but it should be mentioned in the main paper when discussing editable generation or the model's general motion-quality claims, rather than only in the supplementary.","section":"§7.5"},{"comment":"The temporal pre-training task is described as 'randomly masking off certain motion frames,' but the masking ratio, mask span length, and whether masking applies per body part or to the whole motion are not specified. Adding these details would make the pre-training section reproducible.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical systems contribution, but the missing speaker-exclusion disclosure is arguably the single most important issue for a fair assessment. The authors should be asked to either confirm that speaker-2 motion was excluded from all pre-training tasks or rerun the data-scarcity and SOTA comparisons with a clean holdout. In addition, the lack of error bars on the headline numbers and the small effect sizes should be addressed in revision. The emotion-prediction evaluation protocol also needs a clear train/test split. These are fixable within the manuscript's scope, so I would not reject the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about, but check the data split before believing the headline claims. The contribution is a single encoder-decoder LM (Flan-T5) that tokenizes speech (HuBERT), text (SentencePiece), and compositional body motion (four VQ-VAEs for face, hands, upper, lower) and is pre-trained on spatial body-part translation, temporal masking, and audio-text translation before instruction-tuning on downstream tasks. That integration is genuinely new: MotionGPT and EMAGE cover only text-motion or audio-motion, not arbitrary modality combinations with generative pre-training. The ablations in Table 2 are encouraging: removing spatial or temporal pre-training hurts FGD substantially (6.336 and 6.800 vs 5.301). The writing is clear and the authors acknowledge the discrete-tokenization limitation.\n\nThe soft spots are concentrated in the evaluation. The main one: Section 4.1 says pre-training uses BEATv2 and Librispeech, and that no audio-to-motion data is seen, but it never says the BEATv2 motion used for spatial/temporal pre-training excludes speaker-2 (the test speaker). Since the pre-training motion is unpaired only in the sense of not being paired with audio, speaker-2 clips could easily be in the pre-training set. If they are, Figure 5's data-efficiency curve and the \"novel speaker\" generalization claim are inflated: the model has already seen the target speaker's pose statistics and dynamics. This needs an explicit exclusion statement or a re-run on genuinely unseen speakers.\n\nTwo smaller issues. First, there is a tokenizer mismatch: the method section describes training four VQ-VAEs, but Table 1 says post-training \"uses their motion tokenizer\" (EMAGE's). Which tokenizer is used for pre-training? If it's EMAGE's, then the described tokenizer isn't the one evaluated. Second, no error bars anywhere; the headline gain over EMAGE is 5.301 vs 5.512 FGD, and only 0.107 over the no-multimodal-pre-training ablation, so variance matters.\n\nI'd still engage with this. The idea is useful and the pre-training tasks are sensible. The emotion-prediction and editable-gesture tasks are interesting, though the baselines are weak (MotionGPT, random) and not enough to support SOTA. The paper deserves a proper peer review, with the expectation that the authors clarify the split and ideally release code and checkpoints.","headline":"A promising unifier for speech, text, and motion, but the data-scarcity claim is at risk because pre-training may include the test speaker's motion.","tokens_in":18687,"tokens_out":3697,"would_cite":false,"duration_ms":31270,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that speech, text, and expressive body motion can be unified in one multimodal language model, and that pre-training on unpaired data is what makes the unification data-efficient.","keywords":["3D human motion generation","multimodal language model","co-speech gesture generation","motion tokenization","generative pre-training","emotion prediction from motion","text-to-motion","editable gesture generation"],"falsifier":"A decisive check is to rerun the data-scarcity experiment with the test speaker's motion explicitly withheld from the pre-training set; if the 1/32-data benefit over the no-pre-training ablation disappears, the generalization-from-unpaired-motion claim is unsupported.","tokens_in":17629,"feed_emoji":"🕺","tokens_out":8479,"duration_ms":72677,"temperature":0.7,"pith_summary":"Human communication couples what we say with how we move, but most motion-generation models accept only one input type: speech, text, or motion. This paper tries to establish that all three can be handled by one multimodal language model: it tokenizes face, hands, upper body, and lower body motion separately, merges those tokens with audio and text tokens in a single vocabulary, and trains an encoder-decoder language model to translate between any of the modalities. The payoff is on co-speech gestures: paired with a generative pre-training stage that uses only unpaired audio-text data and general body motion, the model reports state-of-the-art realism, beat correlation, and diversity, and keeps most of its quality with 1/32 of the usual paired training data. If the approach holds, the same architecture also performs editable gesture generation and predicts emotion from body motion, making one model serve several tasks that previously needed separate systems.","feed_headline":"One language model unifies speech, text, and 3D motion","feed_subtitle":"Pre-training on unpaired data lets one model generate co-speech gestures, follow text, and read emotion from motion.","key_machinery":"The carrying mechanism is a unified multimodal token vocabulary built from a compositional motion tokenizer. Four vector-quantized autoencoders discretize face, hands, upper body, and lower body motion into separate codebooks, so a full-body sequence becomes a sequence of part-specific token indices; audio and text are reduced to discrete tokens by standard self-supervised and subword tokenizers. All modalities share one vocabulary, so a pretrained encoder-decoder language model treats every input as text-like tokens and generates the target modality by next-token prediction. Before any paired speech-to-motion data is used, the model is pre-trained on two self-supervised objectives: compositional motion alignment, which translates between body parts and predicts masked motion frames, and audio-to-text alignment on large unpaired speech corpora. This pre-training is what lets the model learn body-part correlations, temporal dynamics, and semantic grounding before the downstream task.","core_discovery":"The paper's central claim is that a language model can learn a shared 'language of motion' in which speech, text, and expressive body motion are interchangeable token sequences. The authors show on the BEATv2 co-speech benchmark that their pre-trained model beats specialist per-task systems on FGD, beat correlation, and diversity, and that the advantage grows as paired data shrinks; with only 1/32 of the training data the pre-trained model still outperforms a randomly initialized version and a strong baseline. They further claim that post-training on instruction prompts yields joint audio-text conditioning for editable gesture generation and a motion-to-emotion task that existing motion-language models fail. The unifying premise is that pre-training on unpaired modalities supplies the motion priors and semantic grounding that paired downstream data alone does not.","pith_inferences":["The authors leave the scaling implication implicit: if pre-training on unpaired data is what drives the data-scarcity gains, even larger pools of motion from video or wearables should further reduce the need for motion-capture paired data.","Compositional tokens suggest a natural extension not evaluated here: reconstructing or completing full-body motion from a single observed body part, a test the spatial pre-training task is already primed for.","The motion-to-emotion evaluation uses text-similarity metrics rather than classification accuracy, so part of the reported score may reflect paraphrase skill; a held-out emotion-class accuracy probe would separate language fluency from true emotion recognition.","The authors note that discrete tokenization sometimes produces incoherent motion, implying an immediate experimental fork: replace the vector-quantized codebooks with a residual or continuous tokenizer and measure whether FGD and motion coherence improve under the same pre-training scheme."],"forward_implications":["A single model checkpoint can serve co-speech gesture generation, text-to-motion, editable body-part control, and motion-to-emotion understanding, replacing separate per-task pipelines.","New speakers can be added with a small fraction of the paired speech-to-motion data if the model has first been pre-trained on abundant unpaired motion and audio-text data.","Speech semantics are inherited from language-model pre-training, so gesture quality no longer depends on hand-crafted prosodic features or transcribed speech as auxiliary input.","Instruction-based post-training makes part-specific generation possible, such as generating lower-body locomotion from text while upper-body gestures follow the audio.","Because emotion can be read from motion as free text, motion-understanding tasks become answerable in language rather than restricted to fixed classification labels."],"supporting_citations":[{"why":"Supplies the BEATv2 co-speech benchmark, the compositional expressive motion representation, the speaker-2 evaluation protocol, the motion tokenizer, and the strongest baseline the paper must beat.","marker":"[43]"},{"why":"AMASS motion-capture archive is used together with BEATv2 to train the motion tokenizer and to provide broad motion priors for the text-to-motion post-training setting.","marker":"[46]"},{"why":"MotionGPT provides the prior language-model approach to text-driven motion, the instruction-formatting recipe the paper adapts, and the baseline for the motion-to-emotion comparison.","marker":"[23]"},{"why":"HumanML3D supplies text annotations for text-to-motion training and the skeletal H3D representation that the paper deliberately rejects in favor of its compositional expressive representation.","marker":"[16]"},{"why":"Librispeech is the large unpaired audio-text corpus used in the audio-to-text pre-training alignment objective.","marker":"[50]"},{"why":"The pre-trained encoder-decoder language model and its subword vocabulary anchor the unified multimodal token space and give the model its initial semantic understanding.","marker":"[55]"},{"why":"The self-supervised speech tokenizer converts audio into discrete tokens at a rate compatible with motion tokens, enabling speech to enter the language model as a token sequence.","marker":"[20]"}],"fun_headline_variants":["Language model unifies speech, text, and 3D motion in one token space","One pretrained model masters speech, text, and gesture generation","Pre-training on unpaired data makes one model handle all motion inputs","Editable gestures and emotion prediction from a single motion language","Multimodal model learns shared token language for speech, text, motion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that pre-training makes a new speaker's gestures learnable from very little paired data assumes the pre-training corpus did not already contain that speaker's motion.","fun_headline_variants_meta":{"raw":{"variants":["Language model unifies speech, text, and 3D motion in one token space","One pretrained model masters speech, text, and gesture generation","Pre-training on unpaired data makes one model handle all motion inputs","Editable gestures and emotion prediction from a single motion language","Multimodal model learns shared token language for speech, text, motion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4044,"prompt_tokens":914,"completion_tokens":3130,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":3037}},"tokens_in":530,"tokens_out":3130,"duration_ms":22304,"temperature":1.0,"reasoning_tokens":3037,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:53:09.680077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to rerun the data-scarcity experiment with the test speaker's motion explicitly withheld from the pre-training set; if the 1/32-data benefit over the no-pre-training ablation disappears, the generalization-from-unpaired-motion claim is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BEATv2 co-speech benchmark, the compositional expressive motion representation, the speaker-2 evaluation protocol, the motion tokenizer, and the strongest baseline the paper must beat."},{"cited_title":"Motiongpt: Human motion as a foreign language","cited_arxiv_id":null,"evidence_quote":"MotionGPT provides the prior language-model approach to text-driven motion, the instruction-formatting recipe the paper adapts, and the baseline for the motion-to-emotion comparison."},{"cited_title":"Librispeech: an asr corpus based on public do- main audio books","cited_arxiv_id":null,"evidence_quote":"Librispeech is the large unpaired audio-text corpus used in the audio-to-text pre-training alignment objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The pre-trained encoder-decoder language model and its subword vocabulary anchor the unified multimodal token space and give the model its initial semantic understanding."},{"cited_title":"Hubert: Self-supervised speech representation learning by masked prediction of hidden units","cited_arxiv_id":null,"evidence_quote":"The self-supervised speech tokenizer converts audio into discrete tokens at a rate compatible with motion tokens, enabling speech to enter the language model as a token sequence."}],"review_version":1}