{"id":"3b52c12f-ded9-4903-a414-f9eb90e5d994","arxiv_id":"2411.19786","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MoTe is a unified motion-text diffusion model that achieves strong text-to-motion generation and competitive motion captioning on HumanML3D and KIT by learning joint, conditional, and marginal distributions in one network.","lead":"Researchers built MoTe, a single AI model that generates human motion from text, writes text captions for motion, and produces random motion-text pairs. The same diffusion network handles all of these tasks by changing only the input, and it sets a new high score on several text-to-motion benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 6 is labeled 'conditional generation loss' but contains only unconditional terms; conditional inference feeds clean condition latents at timestep 0, a setting never seen in training, so the claimed conditional distribution learning is not directly optimized.","rationale":"The reader's weakest assumption about latent-space compatibility is plausible but not the most load-bearing issue: the paper's strong empirical results already show that the In-Context interaction can mix the two modalities well enough on the tested datasets. The sharper problem is an internal inconsistency between the training objective and the conditional inference protocol. Eq. 6 is explicitly called the 'conditional generation loss,' yet neither term conditions on a real modality; they are marginal losses. Conditional generation then uses clean condition latents at timestep 0, which never appear in Ljoint (t restricted to [1,T]) and only appear in Eq. 6 with the empty token. Therefore, the model is never directly trained to map a clean text latent to a noisy motion latent (or vice versa). This does not destroy the paper's empirical contribution—the joint loss can in principle approximate conditionals, and UniDiffuser-style methods often work in practice—but it means the stated mechanism for 'learning conditional distributions' is unsupported by the equations as written. A reviewer should ask the authors to either correct Eq. 6 to include the actual conditional losses used, or provide an analysis of why joint-only training supports timestep-0 conditioning. This is a concrete, testable concern that does not change the overall conditional verdict (which already flags overclaims and missing confidence intervals), but it must be resolved before the central claim is fully credible. The proposed retraining experiment would settle whether the missing explicit conditional terms materially affect performance, thereby determining whether this is a presentation flaw or a substantive training deficiency.","tokens_in":16084,"tokens_out":8026,"duration_ms":72156,"concrete_test":"Retrain MoTe on HumanML3D with Eq. 6 replaced (or augmented) by explicit conditional terms: E||ϵ_zm - ϵθ(zm_tm, tm, zs_0, 0)||^2 + E||ϵ_zs - ϵθ(zm_0, 0, zs_ts, ts)||^2, keeping all other hyperparameters identical. Compare text-to-motion (R-Precision, FID, MMDist) and motion-to-text (Bleu-4, CIDEr, R-Precision) against the published results. If metrics improve significantly, the published training omits direct conditional supervision; if they are unchanged, the joint loss alone suffices, making the Eq. 6 labeling a benign misnomer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MoTe learns marginal, conditional, and joint distributions in one diffusion network (Sec. III-B, III-C). However, the training objective in Eq. 5-6 does not include any term where one modality is clean and the other is noisy. Eq. 6 defines Lcond = E||ϵ_zm - ϵθ(zm_tm, tm, ∅, 0)||^2 + E||ϵ_zs - ϵθ(∅, 0, zs_ts, ts)||^2, which are unconditional single-modality denoising losses—there is no condition in either term. Yet Sec. III-C performs conditional generation by invoking the denoiser with a clean latent at timestep 0, e.g., ϵθ(zm_tm, tm, zs_0, 0). The diffusion timestep t is defined on [1,T] in Eq. 1, so timestep 0 is never sampled in Ljoint; the only place timestep 0 appears is paired with the empty token ∅ in Eq. 6. Thus the exact input configuration used for text-to-motion and motion-to-text inference (real clean condition + timestep 0) is out-of-distribution for the trained network. This does not necessarily invalidate the empirical results—the joint loss may implicitly support conditional generation through small-ts extrapolation—but it contradicts the paper's explicit claim that conditional distributions are optimized. Since this concern targets the mechanism behind the flagship 'unified model' claim and is not discussed in the paper, it is more load-bearing than the reader's latent-compatibility worry.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MoTe, a two-stage latent diffusion framework for human motion and text. In the first stage, a motion encoder-decoder (MED) and a text encoder-decoder (TED) compress motion sequences and language descriptions into latent embeddings; in the second stage, a Motion-Text Diffusion Model (MTDM) with dual-path diffusion blocks is trained to generate these latents. The paper claims that by learning marginal, conditional, and joint distributions of the motion and text latents in a single denoising network, MoTe can handle text-to-motion, motion-to-text, joint motion-text generation, and variation tasks by only changing the input context. Experiments on HumanML3D and KIT report state-of-the-art text-to-motion results and competitive motion captioning results, together with ablations on latent motion size, classifier-free guidance weights, and interaction module variants.","tokens_in":16582,"tokens_out":6306,"duration_ms":54810,"significance":"If the central claim holds, MoTe would be a useful unified baseline for motion-text generation, extending the UniDiffuser-style objective to human motion and language. The paper has several strengths: results are reported with 95% confidence intervals, the ablations cover latent size, CFG weights, and interaction modules, a user study is included, and the failure cases are honestly discussed. However, the stated training objective in Eq. (5)-(6) does not actually contain the conditional input configurations used at inference, which puts the paper's core claim of learning conditional distributions in question. Because this issue is load-bearing, the manuscript needs substantial clarification or a corrected objective before the results can be interpreted as the paper describes.","major_comments":[{"comment":"The training objective as written does not optimize the conditional distributions that the paper claims to learn. Eq. (6) is labeled 'conditional generation loss' but both of its terms are unconditional: εθ(zm_tm, tm, ∅, 0) and εθ(∅, 0, zs_ts, ts) contain the empty token as the other modality, so neither term conditions on a real clean condition. By contrast, Sec. III-B and Sec. III-C define conditional inference as εθ(zm_tm, tm, zs_0, 0) for text-to-motion and εθ(zm_0, 0, zs_ts, ts) for motion-to-text. These exact input configurations never appear in the stated losses. Moreover, Eq. (1) defines the diffusion timestep on t ∈ [1, T], so timestep 0 is never sampled in Ljoint; the only place timestep 0 appears in training is paired with the empty token in Eq. (6). Thus conditional inference is out-of-distribution for the trained network under the stated objective. This undermines the central claim of learning marginal, conditional, and joint distributions simultaneously. The authors should either correct Eq. (6) to include the missing conditional terms (and explain how timestep 0 is used in training) or substantially revise the claim that conditional distributions are directly optimized.","section":"Sec. III-B, Eq. (5)-(6); Sec. III-C"},{"comment":"The claim that MoTe achieves 'superior performance across all metrics except for MModality' is not supported by Table I. On HumanML3D, MoTe has FID 0.075±0.004, which is worse than MoMask (0.045±0.002) and DLP (0.071±0.002). The statement about MModality is also internally inconsistent: the text says MoTe has 'lower diversity with the given language descriptions,' but MoTe's MModality of 2.399 is higher than several strong baselines, including MoMask (1.241), DLP (0.452), and T2M-GPT (1.856), and only MDM reports a higher value. The summary of results should be corrected to state precisely which metrics are best, second-best, and worse than the best baseline.","section":"Sec. IV-D, Table I"},{"comment":"There is an inconsistency in how the missing-modality timestep is set for unconditional inference. In Sec. III-C, unconditional generation is described with the condition timestep equal to T, i.e., εθ(zm_tm, tm, ∅, T) and εθ(∅, T, zs_ts, ts). However, Eq. (6) and Eq. (7) use timestep 0 for the same empty-token inputs, i.e., εθ(zm_tm, tm, ∅, 0) and εθ(∅, 0, zs_ts, ts). Since the empty token is the unconditional-side input used in classifier-free guidance, the paper must specify which timestep is used for the empty condition during training and inference, and whether the network is trained with both conventions.","section":"Sec. III-C, Eq. (7)"},{"comment":"Eq. (5) writes the joint loss with separate timesteps tm and ts, while Sec. III-C states that joint generation uses empirically shared timesteps. Please clarify how (tm, ts) are sampled during training of Ljoint: are they always equal, sampled independently, or scheduled differently? This affects whether the joint-distribution claim in Sec. III-B is actually optimized as stated.","section":"Sec. III-B, Eq. (5)"}],"minor_comments":[{"comment":"The abstract contains the typo 'Moti-on-Text Diffusion Model' instead of 'Motion-Text Diffusion Model', and Sec. III-B uses 'reweighed' instead of 'reweighted'.","section":"Abstract and Sec. III"},{"comment":"The implementation details say the CLIP-GPT2 text autoencoder is 'introduced in Unidiffuser [11]'; the citation should be to UniDiffuser, reference [26], not to MotionGPT, reference [11].","section":"Sec. IV-C"},{"comment":"Table II labels the method rows as 'MTDM (l=2)' etc., but the model is called MoTe throughout the rest of the paper; please unify the naming. Also, Sec. IV-E contains the typo 'MDTM' instead of 'MTDM'.","section":"Table II and Sec. IV-E"},{"comment":"In the related work on T2M-GPT, 'VQVAE' is typeset as 'VQV AE'; this should be corrected.","section":"Sec. II"},{"comment":"The user study in Fig. 4(a) reports preference rates without error bars or a description of how many users and questions were used; please add this information in the supplement or main text.","section":"Sec. IV-E"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is potentially salvageable, but the training-objective mismatch in Eq. (6) is central to the paper's framing. If the actual implementation does include conditional terms with clean condition latents at timestep 0, the paper must state them explicitly and justify the t=0 convention relative to Eq. (1). If not, the 'unified distribution learning' claim is not established by the current write-up. The Table I summary also needs correction independent of the objective issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MoTe is a legitimate extension: it brings UniDiffuser's single-diffusion-fits-all-distributions idea to human motion and text, and it is the first diffusion-based unified motion-text model to post competitive text-to-motion numbers. The dual-path transformer with an interaction module is a sensible design, and the comparison of three interaction modules is useful. The ablations on latent size l are honest, including the observed trade-off between text-to-motion and motion-to-text. I also credit the failure-case analysis; the word-repetition and out-of-domain limitations are real and stated plainly.\n\nThe main text-to-motion results on HumanML3D are genuinely strong on R-Precision and MMDist. But the claim in Sec. IV-D that MoTe is 'superior across all metrics except for MModality' is contradicted by Table I: FID is worse than MoMask (0.075 vs 0.045) and DLP (0.071), and MModality is the highest in the table, not the exception. That is a simple overclaim and easy to fix.\n\nThe more substantive issue is Eq. 6. It is labeled 'conditional generation loss' but both terms are unconditional single-modality denoising losses with the empty token at timestep 0. Conditional inference then feeds a real clean condition at timestep 0, a configuration never sampled in training (Eq. 1 defines t on [1,T]). So the conditional distributions are not directly optimized; the model relies on extrapolation from small-t joint denoising. The empirical results suggest that extrapolation works, but the paper should say so. If the authors added t=0 with real conditions to Lcond, or at least discussed this gap, the mechanism would match the claim. This matters more than the latent-compatibility worry, though that one is also unexamined.\n\nOther soft spots: motion-to-text tables lack confidence intervals, and CFG weights and l are selected on the test set. Both are common in this literature, but a reviewer should ask for a proper validation split or a sensitivity discussion.\n\nThe writing is clear, the baselines are standard, and the citations are appropriate. The method is a straightforward extension of UniDiffuser, so the novelty is modest but real for the motion subfield.\n\nThis paper deserves a serious referee. With the overclaim corrected and Eq. 6 clarified, I would take it.","headline":"Solid UniDiffuser-style extension to motion-text with strong T2M results, but Eq. 6 is mislabeled and Sec. IV-D overclaims; fix those and it's a clear accept.","tokens_in":16975,"tokens_out":2931,"would_cite":true,"duration_ms":26134,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoTe proposes that a single diffusion denoiser can learn the marginal, conditional, and joint distributions of human motion and text, and demonstrates that changing only the input context lets that one model caption motion, generate…","keywords":["human motion synthesis","text-to-motion generation","motion captioning","multi-modal diffusion model","latent diffusion","motion-text alignment","unified generation","classifier-free guidance"],"falsifier":"Compute the per-dimension mean and variance of the motion latent $z^m$ and text latent $z^s$ on HumanML3D; if the two latent distributions differ by an order of magnitude in scale, the shared denoiser is only working by accident, and whitening or normalizing both latents before training should measurably change R-Precision and captioning quality. Alternatively, replace the In-Context interaction with a simple addition of the two latents; if multi-task capability persists, the interaction module is not doing the claimed work.","tokens_in":15865,"feed_emoji":"🏃","tokens_out":6698,"duration_ms":53541,"temperature":0.7,"pith_summary":"MoTe is a single diffusion-based model that tries to learn the marginal, conditional, and joint distributions of human motion and natural-language text at once. The authors show that one denoiser, fed both a motion latent and a text latent with their own noise timesteps, can handle text-to-motion generation, motion captioning, unconditional generation, and variation tasks by only changing the input context. On HumanML3D the model reports the best text-to-motion R-Precision (0.548 Top-1) among the compared methods, and its motion captions are competitive with MotionGPT and ground truth in a user study. The central design question is how the two modalities interact inside the denoiser, and the paper argues that a simple concatenation-based In-Context interaction beats more complex cross-attention and adaptive-normalization modules on this data.","feed_headline":"One diffusion model now handles motion and text in both directions","feed_subtitle":"The same network switches tasks by changing only the input context, and it beats prior text-to-motion systems.","key_machinery":"The load-bearing object is the Motion-Text Diffusion Model (MTDM), a denoiser $\\epsilon_\\theta(z^m_{t_m}, t_m, z^s_{t_s}, t_s)$ that predicts noise for both modalities; by fixing one latent at timestep 0 and leaving the other at its diffusion step, the same network realizes marginal, conditional, and joint objectives. The interaction inside each dual-path diffusion block is the mechanism that lets the two modalities influence each other: MoTe adopts the In-Context interaction, where motion embeddings, text embeddings, and timestep embeddings are concatenated and processed together by a vanilla transformer block, with timestep tokens dropped at the final block. Around this core sit two pretrained autoencoders — the MLD-style motion encoder-decoder (MED) and the CLIP-GPT2 text encoder-decoder (TED) — that convert raw motion and text into the latent embeddings the denoiser consumes.","core_discovery":"The paper's central claim is that a multi-modal diffusion model can fit all three relevant data distributions — $q(z^m_0)$, $q(z^s_0)$, and $q(z^m_0,z^s_0)$ — in one denoising network $\\epsilon_\\theta(z^m_{t_m}, t_m, z^s_{t_s}, t_s)$, so that marginal, conditional, and joint generation all become special cases of input context. To build it, MoTe first trains a motion encoder-decoder and a text encoder-decoder separately, then trains the Motion-Text Diffusion Model (MTDM) with a joint noise-prediction loss plus a conditional loss. In each dual-path diffusion block, motion and text tokens pass through unimodal transformers and then an interaction module; the paper compares three interaction designs and adopts the In-Context variant, which simply concatenates all motion, text, and timestep tokens into one vanilla transformer block. With classifier-free guidance, this single model reports state-of-the-art text-to-motion results on HumanML3D (R-Precision Top-1 0.548, FID 0.075) and competitive motion-to-text results on both HumanML3D and KIT, while also supporting variations and joint generation. The paper also documents a trade-off: a longer latent motion size improves motion-to-text but degrades text-to-motion, so it selects an intermediate size.","pith_inferences":["A natural extension not explored in the paper would be to add a contrastive or reconstruction alignment loss between the motion and text latents before diffusion training; this would test whether the In-Context module alone is sufficient for cross-modal understanding.","The same two-stage recipe — separate unimodal autoencoders plus a joint conditional diffusion denoiser — should transplant to other paired modalities, such as audio and motion or video and text, whenever compatible latent autoencoders are available.","The word-repetition failure the paper attributes to GPT2 suggests that swapping the text decoder for a more modern language model, or training it with a diverse negative-prompt set, could close the captioning gap without changing the diffusion framework.","The reported trade-off between latent motion size and task performance implies that a per-task or multi-resolution latent motion representation might remove the need to choose one intermediate size."],"forward_implications":["A single MoTe model can switch between text-to-motion, motion-to-text, unconditional motion/text generation, joint motion-text generation, and variation tasks by changing only which tokens are fixed and which are denoised.","The In-Context interaction module outperforms Cross-Attention and AdaLN interaction on most evaluated settings, and does so with fewer parameters and FLOPs.","On HumanML3D, MoTe reports the lowest FID and highest R-Precision among the compared text-to-motion methods, and its motion captioning is competitive with MotionGPT under the MotionGPT protocol.","The choice of latent motion length $l$ creates a trade-off: smaller $l$ favors text-to-motion, larger $l$ tends to favor motion-to-text.","Classifier-free guidance is required for strong conditional generation: setting the guidance weight to zero sharply degrades both text-to-motion and motion-to-text metrics."],"supporting_citations":[{"why":"Supplies the motion encoder-decoder (MLD-style Motion VAE) and the latent diffusion setup MoTe builds on; also the source of the In-Context interaction idea.","marker":"[8]"},{"why":"Supplies the unified 'one transformer fits all distributions' diffusion framework and the CLIP-GPT2 text autoencoder used as TED.","marker":"[26]"},{"why":"Main multi-modal motion-language baseline whose motion-to-text evaluation protocol MoTe follows and whose text-to-motion results MoTe aims to beat.","marker":"[11]"},{"why":"CLIP text encoder is the first half of TED, producing the latent text embeddings the denoiser consumes.","marker":"[5]"},{"why":"Classifier-free guidance is used at inference to trade diversity for condition fidelity in both generation tasks.","marker":"[35]"},{"why":"Provides the T2M evaluation protocol and the pretrained feature extractor used to compute R-Precision and FID.","marker":"[13]"},{"why":"Supplies the transformer-block and AdaLN design compared against the In-Context interaction module.","marker":"[32]"},{"why":"Provides the pretrained motion-text feature extractor for evaluation and a bidirectional generation baseline (TM2T).","marker":"[10]"}],"fun_headline_variants":["Same diffusion net flips between text2motion and motion2text","Diffusion model unifies text-to-motion and motion captioning","Single denoiser fits all motion-text distributions at once","In-context tokens let one model do both motion and text tasks","MoTe: One diffusion model for joint, conditional, and marginal tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the latent spaces produced by the motion autoencoder and the text autoencoder are compatible enough for one denoiser to treat them interchangeably; if their scales or information densities differ substantially, the joint and conditional training objectives will be poorly conditioned and the multi-task behavior will degrade.","fun_headline_variants_meta":{"raw":{"variants":["Same diffusion net flips between text2motion and motion2text","Diffusion model unifies text-to-motion and motion captioning","Single denoiser fits all motion-text distributions at once","In-context tokens let one model do both motion and text tasks","MoTe: One diffusion model for joint, conditional, and marginal tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3334,"prompt_tokens":1041,"completion_tokens":2293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":2202}},"tokens_in":657,"tokens_out":2293,"duration_ms":14058,"temperature":1.0,"reasoning_tokens":2202,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:47:55.538703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the per-dimension mean and variance of the motion latent $z^m$ and text latent $z^s$ on HumanML3D; if the two latent distributions differ by an order of magnitude in scale, the shared denoiser is only working by accident, and whitening or normalizing both latents before training should measurably change R-Precision and captioning quality. Alternatively, replace the In-Context interaction with a simple addition of the two latents; if multi-task capability persists, the interaction module is not doing the claimed work.","supporting_citations":[{"cited_title":"Executing your commands via motion diffusion in latent space,","cited_arxiv_id":null,"evidence_quote":"Supplies the motion encoder-decoder (MLD-style Motion VAE) and the latent diffusion setup MoTe builds on; also the source of the In-Context interaction idea."},{"cited_title":"One transformer fits all distributions in multi-modal diffusion at scale,","cited_arxiv_id":null,"evidence_quote":"Supplies the unified 'one transformer fits all distributions' diffusion framework and the CLIP-GPT2 text autoencoder used as TED."},{"cited_title":"Motiongpt: Human motion as a foreign language,","cited_arxiv_id":null,"evidence_quote":"Main multi-modal motion-language baseline whose motion-to-text evaluation protocol MoTe follows and whose text-to-motion results MoTe aims to beat."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"CLIP text encoder is the first half of TED, producing the latent text embeddings the denoiser consumes."},{"cited_title":"Generating diverse and natural 3d human motions from text,","cited_arxiv_id":null,"evidence_quote":"Provides the T2M evaluation protocol and the pretrained feature extractor used to compute R-Precision and FID."},{"cited_title":"Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts,","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained motion-text feature extractor for evaluation and a bidirectional generation baseline (TM2T)."}],"review_version":1}