{"id":"f9aaebb1-82e9-4b8b-a5cb-47cc36940abe","arxiv_id":"2505.23465","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An end-to-end audio-conditioned model generates 3D human motion from spoken instructions, with performance close to text-conditioned baselines on synthetic speech datasets.","lead":"Researchers built a model that turns spoken audio commands into 3D human motion, using a speech encoder and a memory-based attention module to compress the audio. They also created a spoken version of existing text-motion datasets and report that audio can drive motion generation almost as well as text, though their audio was mostly generated from the text itself.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The semantic-parity claim is conditional on Tortoise-synthesized audio derived from the same annotations used by text baselines; real-speech robustness is supported only by three qualitative examples, so the central comparison does not yet establish equivalence for actual voice input.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing premise: Tortoise-synthesized audio constructed from the original text annotations is the only quantitative testbed for the audio modality. My independent reading confirms this is where the central claim is least secure. The paper's stated contribution is a practical interaction paradigm, and Section 4 explicitly motivates the Oral Dataset as better aligned with real spoken language, yet the evaluation never measures how well the model handles genuine microphone input at scale. The three qualitative examples in Fig. 7 are not sufficient to support the strong claim that audio is 'equally effective' as text. I considered other possible concerns: the Table 6 column-labeling issue is real and worth fixing, but it does not by itself invalidate the central claim; the absence of code and data is an artifact limitation but not an argument against the method. The TTS-proxy concern, by contrast, directly undermines external validity. If the proposed real-audio test shows parity, the claim survives; if not, the claim must be scaled back. Since the reader already conditioned acceptance on real-audio evaluation and artifact release, my read does not change the verdict.","tokens_in":17223,"tokens_out":3261,"duration_ms":31176,"concrete_test":"Re-evaluate the released checkpoint on a real-speech test set: have at least five native speakers read 200 randomly sampled HumanML3D test descriptions (the same content as the synthetic audio) at 16 kHz, with and without 10 dB additive white noise, and compute R-Precision Top-1/2/3, FID, and MM Dist. Compare these numbers against the Table 3 Ours* row for synthetic audio. If Top-1 R-Precision falls by more than 0.05 or FID more than doubles relative to the synthetic condition, the near-parity conclusion does not transfer to real microphone input, and the paper should be revised to claim TTS-audio conditioning only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Sections 5.1 and 5.2, 'audio signals can serve as an effective alternative to text for semantic conditioning') is evaluated exclusively on audio synthesized by Tortoise from the exact text annotations used to condition all text baselines. Sections 4.1 and 4.2 construct both the Original and Oral datasets this way, and Tables 2 and 3 compare Ours* (audio) against MoMask et al. (text) on those same synthesized inputs. This design cannot separate two hypotheses: (H1) the model extracts motion semantics directly from acoustic signals, and (H2) WavLM plus Tortoise preserve the original text so faithfully that the model effectively solves text-to-motion from a noisy text channel. Either way, the headline is about real audio instructions, and the only real-audio evidence is Fig. 7: three qualitative samples, recorded at 12 kHz in a quiet room with one SNR condition. No quantitative R-Precision/FID is reported for real or noisy speech, and no word-error-rate check is given for how much Tortoise audio deviates from the source text or how much the model relies on recoverable words. A secondary internal inconsistency strengthens the need for artifact release: in Table 6, the Mem-Retr(Ours) row reports MM Dist 1.221, while Table 3 gives Ours* MM Dist 2.955 and MultiModality 1.221; the column appears mislabeled, so the reader cannot fully verify the reported numbers without code or data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end framework for generating 3D human motion from audio instructions. The method encodes raw speech with WavLM, compresses variable-length features through a memory-retrieval attention module, and uses a masked generative transformer with a residual transformer to produce motion tokens under classifier-free guidance. To obtain training data, the authors rewrite existing text annotations (HumanML3D, KIT-ML) into conversational style with ChatGPT and synthesize corresponding audio with the Tortoise TTS model, yielding 'Original' and 'Oral' audio-motion datasets. Experiments compare the audio-conditioned model against text-conditioned baselines on those datasets, reporting near-parity with MoMask (e.g., R-Precision Top-1 0.519 vs 0.521 on HumanML3D) and over 50% lower inference latency than a Whisper-based cascaded pipeline. The paper also includes ablations on audio encoders, compression methods, and a small qualitative robustness test with real and noisy recordings.","tokens_in":17545,"tokens_out":2024,"duration_ms":21244,"significance":"If the central claim holds, the paper introduces a useful new task and a practical interface for motion generation, and the efficiency gain over an ASR cascade is a concrete, reproducible advantage. The work is also constructive in creating augmented audio-motion datasets, which could support future research. The paper ships a clear architecture description, quantitative comparisons following standard text-to-motion metrics, and an explicit efficiency measurement. The main value is conditional on the evaluation design: the reported parity between audio and text is established only for synthesized speech derived from the same text descriptions, so the significance for real voice input is not yet demonstrated.","major_comments":[{"comment":"The central claim that 'audio signals can serve as an effective alternative to text for semantic conditioning' (Section 5.1) is evaluated exclusively on audio synthesized by Tortoise from the same text annotations that condition the text baselines. This design conflates direct audio-to-motion semantic extraction with the ability of WavLM/Tortoise to preserve text information in a noisy channel. The only real-audio evidence is the qualitative example in Fig. 7 (three recorded instructions, one SNR condition), which is not sufficient to support the headline conclusion for actual user voice input. The paper should either add quantitative evaluation on real recorded/perturbed speech (e.g., R-Precision, FID, or at least word-error-rate or ASR-fidelity statistics for the synthesized audio) or explicitly restrict the claim to synthesized audio instructions.","section":"§4.1, §4.2, Tables 2–3"},{"comment":"The Mem-Retr(Ours) row reports MM Dist 1.221, whereas Table 3 lists the same model (Ours*) with MM Dist 2.955 and MultiModality 1.221 on the same HumanML3D Original setting. The column in Table 6 appears mislabeled, because 1.221 matches the MultiModality value in Table 3. This discrepancy prevents the reader from verifying the claimed improvement of the memory-retrieval module over the Transformer encoder (MM Dist 3.259), and it should be corrected or clarified before the ablation can be assessed.","section":"Table 6"},{"comment":"The efficiency comparison against the cascaded method measures only speed, not generation quality of the cascade. The text says the cascaded approach is 'generally comparable' to text-to-motion, but no quantitative comparison is provided for the specific Whisper-to-MoMask pipeline used here. Since the paper's efficiency argument depends on the end-to-end method maintaining quality while being faster, reporting the cascade's R-Precision/FID on the same test split would make the comparison complete.","section":"§5.3, Table 4"}],"minor_comments":[{"comment":"Equation (2) contains a typo: 'finds the closet code' should read 'finds the closest code'.","section":"§3.3"},{"comment":"The text states that Tortoise does not support specifying speakers directly and then says 'we generate data using randomly selected voices'; this is slightly contradictory and should be rephrased to clarify how speaker variation is obtained.","section":"§4.1"},{"comment":"The caption says three audio instructions were tested under three conditions, but the figure shows only one representative example; the paper should state explicitly that the other two examples are omitted for space or include them in supplementary material.","section":"§5.1, Fig. 7"},{"comment":"The reference list includes placeholder entries (e.g., '[1] The frobnicatable foo filter', '[2] Frobnication tutorial') that appear to be template leftovers and are not cited in the text; these should be removed.","section":"References"},{"comment":"The ablation on audio feature compression is reported only on HumanML3D; adding the same ablation on KIT-ML would strengthen the generality of the conclusion, though this is not essential for the main claim.","section":"§5.3, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is the main risk: the paper's headline comparison is between text conditioning and audio synthesized from that same text, which makes the 'audio is as good as text' conclusion partly self-fulfilling. The fix is feasible within the paper's scope by adding quantitative real-audio evaluation or a fidelity analysis of the TTS channel. I also recommend the editor check the Table 6 labeling issue before publication, as it currently affects the credibility of the ablation results. The placeholder references in the bibliography should be cleaned up."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's central claim — that audio instructions can serve as an effective alternative to text for semantic motion conditioning — is plausible but not yet proven. Every quantitative comparison uses audio synthesized by Tortoise from the exact text annotations that the text baselines receive, so the comparison partly guarantees the outcome. That said, the paper does introduce a genuinely new task and a sensible first system.\n\nWhat's new: audio-instruction-to-motion as a task, a memory-retrieval compression module on top of WavLM and MoMask, and an oral dataset built by rewriting text with ChatGPT and synthesizing speech with Tortoise. The authors are honest that TTS introduces noise, and they report a real recorded audio test with noisy conditions, but only three qualitative examples. The efficiency gain over a Whisper-then-MoMask cascade (over 50% faster) is a real practical point and supports the direct-encoding design.\n\nThe main soft spot is the circularity. Because the test audio is synthesized from the source text, the results cannot separate two hypotheses: the model extracts motion semantics directly from acoustics, or it implicitly recovers the underlying text from TTS and then solves text-to-motion. A word-error-rate check on the synthesized audio and quantitative real-audio results (even a small set) would address this. The conclusion's stated limitations — multilingual instructions and long multi-step commands are untested — are appropriate and count in the paper's favor.\n\nTwo smaller issues. Table 6's last column is labeled MM Dist, but the value 1.221 matches the MultiModality column in Table 3; that is a mislabel. The reference list contains placeholder \"frobnication\" entries, which suggests the manuscript was assembled from a template too hastily. No code or data is released, so independent verification is currently impossible.\n\nBottom line: this is a legitimate new task, and the paper deserves a serious referee. I would accept it for review with a request for artifact release, a corrected table, and stronger real-audio evidence. I would not cite it in my own work yet, but I would bring it to a reading group.","headline":"The central claim that audio can replace text for semantic motion conditioning is plausible but not yet proven, because the quantitative comparisons use TTS audio synthesized from the same text annotations given to text baselines.","tokens_in":18048,"tokens_out":3122,"would_cite":false,"duration_ms":30581,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes that spoken audio instructions can replace text as the conditioning signal for 3D human motion generation, and reports near-parity with text-conditioned models while cutting inference latency by more than half.","keywords":["human motion generation","audio-conditioned generation","masked generative transformer","memory-retrieval attention","text-to-speech data augmentation","semantic audio encoding","multimodal learning"],"falsifier":"Run the same evaluation on a test set of real human recordings of the same instructions, with and without added noise, and compare R-precision and FID against the synthetic-audio test set; a substantial drop in R-precision or a large FID degradation would show that the equivalence claim holds only for synthesized speech rather than for real microphone input.","tokens_in":17043,"feed_emoji":"🎙️","tokens_out":4695,"duration_ms":40042,"temperature":0.7,"pith_summary":"The paper introduces the task of generating 3D human motion directly from spoken audio instructions, bypassing speech-to-text. It argues that audio can carry the same semantic content as text for motion generation, provided the audio is encoded into compact motion-relevant features rather than matched only to rhythm. To test this, the authors build an end-to-end masked-transformer model that conditions on WavLM audio features compressed by a memory-retrieval attention module, and they create oral-style versions of HumanML3D and KIT by rephrasing descriptions conversationally and synthesizing speech with varied voices. On standard benchmarks the audio-conditioned model lands close to the text-conditioned MoMask, and the direct pipeline generates motions over 50% faster than a speech-recognition-then-text pipeline.","feed_headline":"Audio instructions can drive 3D human motion nearly as well as text","feed_subtitle":"Direct audio conditioning skips speech-to-text and runs over 50% faster, with semantics close to text-based generation.","key_machinery":"The load-bearing component is a memory-retrieval based attention module: a set of learnable memory tokens that act as key-value pairs, so that each audio feature sequence generates a query vector and attends over the memory to produce a compact, fixed-size condition vector $y \\in \\mathbb{R}^{d_y}$ regardless of the original audio length. This addresses the sparse and lengthy nature of WavLM-extracted audio features. The condition feeds a Masked Transformer that iteratively generates base motion tokens and a Residual Transformer that generates residual tokens layer by layer, following the RVQ-VAE motion latent representation from MoMask and the masked generative paradigm from MaskGIT.","core_discovery":"The central claim is that audio signals can serve as an effective alternative to text for semantic conditioning in human motion generation. Built as an end-to-end masked generative transformer, the model takes raw audio instructions, compresses them into a fixed-size condition through a memory-retrieval attention module, and produces motion through base and residual quantization codes. On the HumanML3D original test set the audio-conditioned model reaches R-Precision Top-1 0.519 against MoMask's 0.521 with text, while FID is 0.121 versus 0.045; the paper reads this as audio conveying semantics comparable to text. It also shows that models trained on the newly synthesized oral-style data outperform those trained on the original text-style data when tested on spoken instructions, supporting the value of conversational audio training data.","pith_inferences":["If the TTS-to-real gap is closed by larger real-audio evaluation, this points toward voice-driven animation tools where a user narrates an action and sees motion nearly in real time; the latency result makes that plausible at interactive rates.","The same memory-retrieval conditioning could be applied to other long-horizon inputs where a fixed-size semantic bottleneck is needed, and a direct test would be to swap WavLM for a video or sensor encoder and measure whether the compression still preserves task-relevant semantics.","A testable extension is multilingual and multi-stage instructions, which the paper lists as untested limitations; the framework's performance there would reveal whether the semantic equivalence holds beyond short English single actions.","The finding that oral-trained models outperform original-trained models on oral-style tests suggests that collecting more natural spoken instruction data may contribute as much as architectural improvements to real-world usability."],"forward_implications":["Audio-conditioned motion generation can be built without a speech-to-text cascade, simplifying deployment and reducing inference latency by over 50%.","Spoken, conversational instructions become usable as conditioning inputs, and training on oral-style rephrased data better matches how users actually speak.","The memory-retrieval compression technique is a reusable pattern for conditioning transformers on long, sparse signals beyond audio, such as video or sensor streams.","Because audio and text conditions land close in quality, future systems can choose the input modality based on interface constraints rather than expected generation quality.","The near-parity result suggests that adding a speech recognizer before a text-to-motion model may be unnecessary when the audio encoder is trained end-to-end for the motion task."],"supporting_citations":[{"why":"Supplies the RVQ-VAE plus masked and residual transformer motion-generation architecture and the text-conditioned baseline the audio model is compared against.","marker":"[27]"},{"why":"Provides the pre-trained WavLM speech encoder that turns raw audio into the feature sequences the pipeline consumes.","marker":"[11]"},{"why":"Introduces the masked generative paradigm that the Masked Transformer uses to iteratively generate base motion tokens.","marker":"[10]"},{"why":"Provides the HumanML3D text-motion dataset used for augmentation and for the main benchmark comparisons.","marker":"[21]"},{"why":"Provides the KIT motion-language dataset used for the second benchmark and its oral-style augmentation.","marker":"[39]"},{"why":"The text-to-speech model that synthesizes the audio instructions from the rewritten texts, producing the training and test conditions.","marker":"[38]"},{"why":"The language model used to rewrite text descriptions into conversational oral style for the Oral Dataset.","marker":"[37]"},{"why":"The memory-retrieval attention idea that the proposed audio encoder adapts to compress lengthy audio features.","marker":"[41]"}],"fun_headline_variants":["Audio instructions match text for 3D human motion quality","Direct audio control of motion beats text-to-speech pipeline","Audio-driven motion synthesis rivals text-based methods","Semantic motion from audio: no need for speech-to-text","Audio conditioning generates human motion as well as text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that audio matches text depends on the assumption that the Tortoise-synthesized speech used for training and testing is a faithful stand-in for real spoken instructions; the only real-audio evidence in the paper is three qualitative examples.","fun_headline_variants_meta":{"raw":{"variants":["Audio instructions match text for 3D human motion quality","Direct audio control of motion beats text-to-speech pipeline","Audio-driven motion synthesis rivals text-based methods","Semantic motion from audio: no need for speech-to-text","Audio conditioning generates human motion as well as text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1275,"prompt_tokens":849,"completion_tokens":426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":349}},"tokens_in":465,"tokens_out":426,"duration_ms":4835,"temperature":1.0,"reasoning_tokens":349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:45:26.729738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same evaluation on a test set of real human recordings of the same instructions, with and without added noise, and compare R-precision and FID against the synthetic-audio test set; a substantial drop in R-precision or a large FID degradation would show that the equivalence claim holds only for synthesized speech rather than for real microphone input.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RVQ-VAE plus masked and residual transformer motion-generation architecture and the text-conditioned baseline the audio model is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained WavLM speech encoder that turns raw audio into the feature sequences the pipeline consumes."},{"cited_title":"Chang, H","cited_arxiv_id":null,"evidence_quote":"Introduces the masked generative paradigm that the Masked Transformer uses to iteratively generate base motion tokens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the HumanML3D text-motion dataset used for augmentation and for the main benchmark comparisons."},{"cited_title":"Plappert, C","cited_arxiv_id":null,"evidence_quote":"Provides the KIT motion-language dataset used for the second benchmark and its oral-style augmentation."}],"review_version":1}