Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

MoTe: Learning Motion-Text Diffusion Model for Multiple Generation Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MoTe proposes that a single diffusion denoiser can learn the marginal, conditional, and joint distributions of human motion and text, and demonstrates that changing only the input context lets that one model caption motion, generate…

desk verdict Solid UniDiffuser-style extension to motion-text with strong T2M results, but Eq. 6 is mislabeled and Sec. IV-D overclaims; fix those and it's a clear accept. read the letter →

arxiv 2411.19786 v1 pith:5IJBCQQO submitted 2024-11-29 cs.CV cs.CLcs.LG

Yiming Wu , Wei Ji , Kecheng Zheng , Zicheng Wang , Dong Xu This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords humanmotionsynthesistext-to-motiongenerationcaptioningmulti-modaldiffusionmodellatentmotion-textalignmentunifiedclassifier-freeguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MoTe is a single diffusion-based model that tries to learn the marginal, conditional, and joint distributions of human motion and natural-language text at once. The authors show that one denoiser, fed both a motion latent and a text latent with their own noise timesteps, can handle text-to-motion generation, motion captioning, unconditional generation, and variation tasks by only changing the input context. On HumanML3D the model reports the best text-to-motion R-Precision (0.548 Top-1) among the compared methods, and its motion captions are competitive with MotionGPT and ground truth in a user study. The central design question is how the two modalities interact inside the denoiser, and the paper argues that a simple concatenation-based In-Context interaction beats more complex cross-attention and adaptive-normalization modules on this data.

What carries the argument

The load-bearing object is the Motion-Text Diffusion Model (MTDM), a denoiser $\epsilon_\theta(z^m_{t_m}, t_m, z^s_{t_s}, t_s)$ that predicts noise for both modalities; by fixing one latent at timestep 0 and leaving the other at its diffusion step, the same network realizes marginal, conditional, and joint objectives. The interaction inside each dual-path diffusion block is the mechanism that lets the two modalities influence each other: MoTe adopts the In-Context interaction, where motion embeddings, text embeddings, and timestep embeddings are concatenated and processed together by a vanilla transformer block, with timestep tokens dropped at the final block. Around this core sit two pretrained autoencoders — the MLD-style motion encoder-decoder (MED) and the CLIP-GPT2 text encoder-decoder (TED) — that convert raw motion and text into the latent embeddings the denoiser consumes.

What would settle it

Compute the per-dimension mean and variance of the motion latent $z^m$ and text latent $z^s$ on HumanML3D; if the two latent distributions differ by an order of magnitude in scale, the shared denoiser is only working by accident, and whitening or normalizing both latents before training should measurably change R-Precision and captioning quality. Alternatively, replace the In-Context interaction with a simple addition of the two latents; if multi-task capability persists, the interaction module is not doing the claimed work.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multi-modal diffusion model can fit all three relevant data distributions — $q(z^m_0)$, $q(z^s_0)$, and $q(z^m_0,z^s_0)$ — in one denoising network $\epsilon_\theta(z^m_{t_m}, t_m, z^s_{t_s}, t_s)$, so that marginal, conditional, and joint generation all become special cases of input context. To build it, MoTe first trains a motion encoder-decoder and a text encoder-decoder separately, then trains the Motion-Text Diffusion Model (MTDM) with a joint noise-prediction loss plus a conditional loss. In each dual-path diffusion block, motion and text tokens pass through unimodal transformers and then an interaction module; the paper compares three interaction designs and adopts the In-Context variant, which simply concatenates all motion, text, and timestep tokens into one vanilla transformer block. With classifier-free guidance, this single model reports state-of-the-art text-to-motion results on HumanML3D (R-Precision Top-1 0.548, FID 0.075) and competitive motion-to-text results on both HumanML3D and KIT, while also supporting variations and joint generation. The paper also documents a trade-off: a longer latent motion size improves motion-to-text but degrades text-to-motion, so it selects an intermediate size.

Load-bearing premise

The load-bearing premise is that the latent spaces produced by the motion autoencoder and the text autoencoder are compatible enough for one denoiser to treat them interchangeably; if their scales or information densities differ substantially, the joint and conditional training objectives will be poorly conditioned and the multi-task behavior will degrade.

Editorial extensions

If this is right

  • A single MoTe model can switch between text-to-motion, motion-to-text, unconditional motion/text generation, joint motion-text generation, and variation tasks by changing only which tokens are fixed and which are denoised.
  • The In-Context interaction module outperforms Cross-Attention and AdaLN interaction on most evaluated settings, and does so with fewer parameters and FLOPs.
  • On HumanML3D, MoTe reports the lowest FID and highest R-Precision among the compared text-to-motion methods, and its motion captioning is competitive with MotionGPT under the MotionGPT protocol.
  • The choice of latent motion length $l$ creates a trade-off: smaller $l$ favors text-to-motion, larger $l$ tends to favor motion-to-text.
  • Classifier-free guidance is required for strong conditional generation: setting the guidance weight to zero sharply degrades both text-to-motion and motion-to-text metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not explored in the paper would be to add a contrastive or reconstruction alignment loss between the motion and text latents before diffusion training; this would test whether the In-Context module alone is sufficient for cross-modal understanding.
  • The same two-stage recipe — separate unimodal autoencoders plus a joint conditional diffusion denoiser — should transplant to other paired modalities, such as audio and motion or video and text, whenever compatible latent autoencoders are available.
  • The word-repetition failure the paper attributes to GPT2 suggests that swapping the text decoder for a more modern language model, or training it with a diverse negative-prompt set, could close the captioning gap without changing the diffusion framework.
  • The reported trade-off between latent motion size and task performance implies that a per-task or multi-resolution latent motion representation might remove the need to choose one intermediate size.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MoTe, a two-stage latent diffusion framework for human motion and text. In the first stage, a motion encoder-decoder (MED) and a text encoder-decoder (TED) compress motion sequences and language descriptions into latent embeddings; in the second stage, a Motion-Text Diffusion Model (MTDM) with dual-path diffusion blocks is trained to generate these latents. The paper claims that by learning marginal, conditional, and joint distributions of the motion and text latents in a single denoising network, MoTe can handle text-to-motion, motion-to-text, joint motion-text generation, and variation tasks by only changing the input context. Experiments on HumanML3D and KIT report state-of-the-art text-to-motion results and competitive motion captioning results, together with ablations on latent motion size, classifier-free guidance weights, and interaction module variants.

Significance. If the central claim holds, MoTe would be a useful unified baseline for motion-text generation, extending the UniDiffuser-style objective to human motion and language. The paper has several strengths: results are reported with 95% confidence intervals, the ablations cover latent size, CFG weights, and interaction modules, a user study is included, and the failure cases are honestly discussed. However, the stated training objective in Eq. (5)-(6) does not actually contain the conditional input configurations used at inference, which puts the paper's core claim of learning conditional distributions in question. Because this issue is load-bearing, the manuscript needs substantial clarification or a corrected objective before the results can be interpreted as the paper describes.

major comments (4)
  1. [Sec. III-B, Eq. (5)-(6); Sec. III-C] The training objective as written does not optimize the conditional distributions that the paper claims to learn. Eq. (6) is labeled 'conditional generation loss' but both of its terms are unconditional: εθ(zm_tm, tm, ∅, 0) and εθ(∅, 0, zs_ts, ts) contain the empty token as the other modality, so neither term conditions on a real clean condition. By contrast, Sec. III-B and Sec. III-C define conditional inference as εθ(zm_tm, tm, zs_0, 0) for text-to-motion and εθ(zm_0, 0, zs_ts, ts) for motion-to-text. These exact input configurations never appear in the stated losses. Moreover, Eq. (1) defines the diffusion timestep on t ∈ [1, T], so timestep 0 is never sampled in Ljoint; the only place timestep 0 appears in training is paired with the empty token in Eq. (6). Thus conditional inference is out-of-distribution for the trained network under the stated objective. This undermines the central claim of learning marginal, conditional, and joint distributions simultaneously. The authors should either correct Eq. (6) to include the missing conditional terms (and explain how timestep 0 is used in training) or substantially revise the claim that conditional distributions are directly optimized.
  2. [Sec. IV-D, Table I] The claim that MoTe achieves 'superior performance across all metrics except for MModality' is not supported by Table I. On HumanML3D, MoTe has FID 0.075±0.004, which is worse than MoMask (0.045±0.002) and DLP (0.071±0.002). The statement about MModality is also internally inconsistent: the text says MoTe has 'lower diversity with the given language descriptions,' but MoTe's MModality of 2.399 is higher than several strong baselines, including MoMask (1.241), DLP (0.452), and T2M-GPT (1.856), and only MDM reports a higher value. The summary of results should be corrected to state precisely which metrics are best, second-best, and worse than the best baseline.
  3. [Sec. III-C, Eq. (7)] There is an inconsistency in how the missing-modality timestep is set for unconditional inference. In Sec. III-C, unconditional generation is described with the condition timestep equal to T, i.e., εθ(zm_tm, tm, ∅, T) and εθ(∅, T, zs_ts, ts). However, Eq. (6) and Eq. (7) use timestep 0 for the same empty-token inputs, i.e., εθ(zm_tm, tm, ∅, 0) and εθ(∅, 0, zs_ts, ts). Since the empty token is the unconditional-side input used in classifier-free guidance, the paper must specify which timestep is used for the empty condition during training and inference, and whether the network is trained with both conventions.
  4. [Sec. III-B, Eq. (5)] Eq. (5) writes the joint loss with separate timesteps tm and ts, while Sec. III-C states that joint generation uses empirically shared timesteps. Please clarify how (tm, ts) are sampled during training of Ljoint: are they always equal, sampled independently, or scheduled differently? This affects whether the joint-distribution claim in Sec. III-B is actually optimized as stated.
minor comments (5)
  1. [Abstract and Sec. III] The abstract contains the typo 'Moti-on-Text Diffusion Model' instead of 'Motion-Text Diffusion Model', and Sec. III-B uses 'reweighed' instead of 'reweighted'.
  2. [Sec. IV-C] The implementation details say the CLIP-GPT2 text autoencoder is 'introduced in Unidiffuser [11]'; the citation should be to UniDiffuser, reference [26], not to MotionGPT, reference [11].
  3. [Table II and Sec. IV-E] Table II labels the method rows as 'MTDM (l=2)' etc., but the model is called MoTe throughout the rest of the paper; please unify the naming. Also, Sec. IV-E contains the typo 'MDTM' instead of 'MTDM'.
  4. [Sec. II] In the related work on T2M-GPT, 'VQVAE' is typeset as 'VQV AE'; this should be corrected.
  5. [Sec. IV-E] The user study in Fig. 4(a) reports preference rates without error bars or a description of how many users and questions were used; please add this information in the supplement or main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: MoTe's results are empirical comparisons on held-out splits with transparent ablations; the Eq. 5-6 objective is a standard joint/marginal diffusion loss, and no prediction reduces to its fitted inputs.

full rationale

I walked the derivation chain. The method is a two-stage latent diffusion model: MED/TED are pretrained autoencoders (MLD motion VAE and CLIP-GPT2), and MTDM is trained on real paired HumanML3D/KIT latents with L = Ljoint + Lcond (Eq. 5-6). The unconditional, conditional, and joint generation protocols in Sec. III-C follow directly from UniDiffuser-style input-context manipulation; the model is evaluated with the standard T2M and MotionGPT protocols on held-out test sets, so the reported numbers are not forced by the training objective. The only tuned hyperparameters (wm, ws, l) are presented in ablation tables and adopted with explicit statements ('we adopt wm = 7.5 and ws = 7.0 finally'; 'we finally select l = 4'), not relabeled as predictions. Self-citations are limited to the evaluation protocol of [13] and are not load-bearing for the model's design; the cited encoders/decoders and interaction modules come from external prior work (MLD, UniDiffuser, DiT). The skeptic observation that Eq. 6 contains unconditional single-modality terms while conditional inference uses a clean condition at timestep 0 is a potential training-inference mismatch worth a correctness check, but it is not a case of a result being equivalent to its input by construction, so it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a small number of manually chosen hyperparameters (latent length l, CFG weights, model scale) and on the compatibility assumption between independently trained latents. No new physical entities are introduced.

free parameters (3)
  • latent motion length l = 4 for HumanML3D, 2 for KIT
    Ablated over {2,4,6,8} and selected as a compromise between text-to-motion and motion-to-text performance based on test-set results (Sec. IV-E, Tables III and IV).
  • classifier-free guidance weights = wm=7.5, ws=7.0
    Selected from ablation on HumanML3D test set (Sec. IV-E, Table V) for text-to-motion and motion-to-text respectively.
  • model scale (DPD blocks / feedforward dim) = not stated in main text (variants MoTe-S/M/H with 7/9/11 blocks)
    The main results do not specify which variant from Table VI is used, weakening reproducibility of the exact architecture.
assumptions (3)
  • standard math The forward and reverse diffusion processes for both modalities follow the standard DDPM formulation with a shared noise scheduler.
    Used in Sec. III-A equations (1)-(3), inherited from Ho et al. and Song & Ermon.
  • domain assumption The independently trained motion VAE (MLD) and text autoencoder (CLIP-GPT2) produce latent embeddings that are sufficiently compatible for a single diffusion network to denoise both modalities together without an explicit cross-modal alignment loss.
    The MTDM in Sec. III-B receives zm and zs interchangeably; no alignment loss is used beyond the interaction module.
  • ad hoc to paper In joint generation, the timesteps for motion and text are shared ('empirically shared'), implying a synchronized noise schedule across modalities.
    Sec. III-C, Joint Motion-Text Generation: 'the timesteps for motion and language empirically shared.' This is a manual choice without justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoTe: Learning Motion-Text Diffusion Model for Multiple Generation Tasks." pith.science (2026). https://pith.science/paper/5IJBCQQO

@misc{pith2026241119786,
  author       = {Pith},
  title        = {Pith review of: MoTe: Learning Motion-Text Diffusion Model for Multiple Generation Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5IJBCQQO}},
  note         = {Machine review of arXiv:2411.19786}
}
read the original abstract

Recently, human motion analysis has experienced great improvement due to inspiring generative models such as the denoising diffusion model and large language model. While the existing approaches mainly focus on generating motions with textual descriptions and overlook the reciprocal task. In this paper, we present~\textbf{MoTe}, a unified multi-modal model that could handle diverse tasks by learning the marginal, conditional, and joint distributions of motion and text simultaneously. MoTe enables us to handle the paired text-motion generation, motion captioning, and text-driven motion generation by simply modifying the input context. Specifically, MoTe is composed of three components: Motion Encoder-Decoder (MED), Text Encoder-Decoder (TED), and Moti-on-Text Diffusion Model (MTDM). In particular, MED and TED are trained for extracting latent embeddings, and subsequently reconstructing the motion sequences and textual descriptions from the extracted embeddings, respectively. MTDM, on the other hand, performs an iterative denoising process on the input context to handle diverse tasks. Experimental results on the benchmark datasets demonstrate the superior performance of our proposed method on text-to-motion generation and competitive performance on motion captioning.

Figures

Figures reproduced from arXiv: 2411.19786 by the authors.

Figure 1
Figure 1. Illustration of diverse tasks: (a) The first row presents the unconditional and joint generation results. Motion, text, and text-motion pairs are generated [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of our proposed MoTe. (1) Motion encoder-decoder (MED) and text encoder-decoder (TED) compress the motion sequences and language descriptions into two latent representations, which are reconstructed by the corresponding decoders. We adopt Motion Transformer in MED and CLIP-GPT2 in TED. (2) Motion-text diffusion model (MTDM) maps Gaussian noise through stacked dual path diffusion (DPD) blocks, where each… view at source ↗
Figure 3
Figure 3. Three variants of the interaction module: 1) In-Context Interaction, all [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a) Statistics of our user study for evaluating the text-to-motion and motion-to-text tasks. (b) Qualitative comparison on the HumanML3D dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: (a) Comparison of different interaction modules at FID (lower is better) and Bleu@4 (higher is better) on the HumanML3D dataset. (b) Failure cases: [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MRBench: A Comprehensive Benchmark for Human Motion-Text Retrieval

    cs.CV 2026-08 conditional novelty 6.0 of 10

    MRBench is a multi-source, balanced, multi-granular human motion-text retrieval benchmark, and the proposed granularity-aware adapters improve mixed-granularity retrieval without degrading standard-caption retrieval.

  2. MUGEN: A Unified Framework for Efficient Motion Understanding and Generation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MUGEN lets a language model generate and read motion through a few continuous latent slots, achieving competitive retrieval and captioning with one draw and K language-model steps.

  3. IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Interleaving motion generation with text-motion assessment and refinement improves alignment between generated human motion and goal text.

  4. AI-Generated Fall Data: Assessing LLMs and Diffusion Model for Wearable Fall Detection

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Synthetic fall data from LLMs helps LSTM fall detectors on low-frequency waist datasets (UMAFall +56.83%) but hurts on high-frequency or wrist datasets; diffusion data matches real data best yet does not reliably impr...

Reference graph

Works this paper leans on

44 extracted references · 30 canonical work pages · cited by 4 Pith papers

  1. [1]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in CVPR. IEEE, 2022, pp. 10 674–10 685

    2022

  2. [2]

    Dreamfusion: Text- to-3d using 2d diffusion,

    B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “Dreamfusion: Text- to-3d using 2d diffusion,” in ICLR, 2023

    work page 2023

  3. [3]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in NeurIPS, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020

    work page 2020

  4. [4]

    Generative modeling by estimating gradients of the data distribution,

    Y . Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” in NeurIPS, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alch´e-Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 11 895–11 907

    work page 2019

  5. [5]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in ICML, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 2021, pp. 8748–8763

    work page 2021

  6. [6]

    Human motion diffusion model,

    G. Tevet, S. Raab, B. Gordon, Y . Shafir, D. Cohen-or, and A. H. Bermano, “Human motion diffusion model,” in ICLR, 2022

    work page 2022

  7. [7]

    Motiondiffuse: Text-driven human motion generation with diffusion model,

    M. Zhang, Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Z. Liu, “Motiondiffuse: Text-driven human motion generation with diffusion model,” arXiv preprint arXiv:2208.15001 , 2022

    arXiv 2022

  8. [8]

    Executing your commands via motion diffusion in latent space,

    X. Chen, B. Jiang, W. Liu, Z. Huang, B. Fu, T. Chen, and G. Yu, “Executing your commands via motion diffusion in latent space,” in CVPR, 2023, pp. 18 000–18 010

    work page 2023

Show all 44 references
  1. [9]

    T2m-gpt: Generating human motion from textual descriptions with discrete representations,

    J. Zhang, Y . Zhang, X. Cun, S. Huang, Y . Zhang, H. Zhao, H. Lu, and X. Shen, “T2m-gpt: Generating human motion from textual descriptions with discrete representations,” CVPR, 2023

    2023

  2. [10]

    Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts,

    C. Guo, X. Zuo, S. Wang, and L. Cheng, “Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts,” in ECCV. Springer, 2022, pp. 580–597

    2022

  3. [11]

    Motiongpt: Human motion as a foreign language,

    B. Jiang, X. Chen, W. Liu, J. Yu, G. Yu, and T. Chen, “Motiongpt: Human motion as a foreign language,” NeurIPS, 2023

    2023

  4. [12]

    Action-conditioned 3d human motion synthesis with transformer V AE,

    M. Petrovich, M. J. Black, and G. Varol, “Action-conditioned 3d human motion synthesis with transformer V AE,” in ICCV. IEEE, 2021, pp. 10 965–10 975

    2021

  5. [13]

    Generating diverse and natural 3d human motions from text,

    C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng, “Generating diverse and natural 3d human motions from text,” in CVPR, 2022, pp. 5152–5161

    2022

  6. [14]

    Ai choreographer: Music conditioned 3d dance generation with aist++,

    R. Li, S. Yang, D. A. Ross, and A. Kanazawa, “Ai choreographer: Music conditioned 3d dance generation with aist++,” in ICCV, 2021, pp. 13 401–13 412

    2021

  7. [15]

    Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks,

    M. Plappert, C. Mandery, and T. Asfour, “Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks,” Robotics and Autonomous Systems , vol. 109, pp. 13–26, 2018

    2018

  8. [16]

    Temos: Generating diverse human motions from textual descriptions,

    M. Petrovich, M. J. Black, and G. Varol, “Temos: Generating diverse human motions from textual descriptions,” in ECCV. Springer, 2022, pp. 480–497

    2022

  9. [17]

    Linguistic descriptions of human motion with generative adversarial seq2seq learning,

    Y . Goutsu and T. Inamura, “Linguistic descriptions of human motion with generative adversarial seq2seq learning,” in IEEE International Conference on Robotics and Automation, ICRA 2021, Xi’an, China, May 30 - June 5, 2021 . IEEE, 2021, pp. 4281–4287

    2021

  10. [18]

    Fg-t2m: Fine- grained text-driven human motion generation via diffusion model,

    Y . Wang, Z. Leng, F. W. Li, S.-C. Wu, and X. Liang, “Fg-t2m: Fine- grained text-driven human motion generation via diffusion model,” in ICCV, 2023, pp. 22 035–22 044

    2023

  11. [19]

    Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descrip- tions,

    T. Yamada, H. Matsunaga, and T. Ogata, “Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descrip- tions,” IEEE Robotics Autom. Lett. , vol. 3, no. 4, pp. 3441–3448, 2018

    2018

  12. [20]

    Scaling instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma et al., “Scaling instruction-finetuned language models,” arXiv preprint arXiv:2210.11416 , 2022

    2022 arXiv

  13. [21]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  14. [22]

    Smpler-x: Scaling up expressive human pose and shape estimation,

    Z. Cai, W. Yin, A. Zeng, C. Wei, Q. Sun, Y . Wang, H. E. Pang, H. Mei, M. Zhang, L. Zhang et al. , “Smpler-x: Scaling up expressive human pose and shape estimation,” arXiv preprint arXiv:2309.17448 , 2023

    2023 arXiv

  15. [23]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks,

    W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som et al. , “Image as a foreign language: Beit pretraining for all vision and vision-language tasks,” arXiv preprint arXiv:2208.10442 , 2022

    2022 arXiv

  16. [24]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in CVPR, 2023, pp. 15 180–15 190

    2023

  17. [25]

    Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation,

    L. Ruan, Y . Ma, H. Yang, H. He, B. Liu, J. Fu, N. J. Yuan, Q. Jin, and B. Guo, “Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation,” in CVPR. IEEE, 2023, pp. 10 219–10 228

    2023

  18. [26]

    One transformer fits all distributions in multi-modal diffusion at scale,

    F. Bao, S. Nie, K. Xue, C. Li, S. Pu, Y . Wang, G. Yue, Y . Cao, H. Su, and J. Zhu, “One transformer fits all distributions in multi-modal diffusion at scale,” in ICML, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, an...

    2023

  19. [27]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” in NeurIPS, vol. 35, 2022, pp. 23 716–23 736. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 10

    2022

  20. [28]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597 , 2023

    2023 arXiv

  21. [29]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  22. [30]

    Improved denoising diffusion proba- bilistic models,

    A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion proba- bilistic models,” in ICML. PMLR, 2021, pp. 8162–8171

    2021

  23. [31]

    Motionclip: Exposing human motion generation to CLIP space,

    G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, and D. Cohen-Or, “Motionclip: Exposing human motion generation to CLIP space,” in ECCV, ser. Lecture Notes in Computer Science, S. Avidan, G. J. Brostow, M. Ciss ´e, G. M. Farinella, and T. Hassner, Eds., vol. 13682. Springer, 2022...

    2022

  24. [32]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in ICCV, 2023, pp. 4195–4205

    2023

  25. [33]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V . N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 5998–6008

    2017

  26. [34]

    A style-based generator architecture for generative adversarial networks,

    T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” IEEE TPAMI, vol. 43, no. 12, pp. 4217–4228, 2021

    2021

  27. [35]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applica- tions, 2021

    2021

  28. [36]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Association for Computational Linguistics. , 2002, pp. 311–318

    2002

  29. [37]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  30. [38]

    Cider: Consensus-based image description evaluation,

    R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in CVPR. IEEE Computer Society, 2015, pp. 4566–4575

    2015

  31. [39]

    Bertscore: Evaluating text generation with BERT,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with BERT,” in ICLR, 2020

    2020

  32. [40]

    SGDR: stochastic gradient descent with warm restarts,

    I. Loshchilov and F. Hutter, “SGDR: stochastic gradient descent with warm restarts,” in ICLR, 2017

    2017

  33. [41]

    Hutumotion: Human-tuned navigation of latent motion diffusion models with minimal feedback,

    G. Han, S. Huang, M. Gong, and J. Tang, “Hutumotion: Human-tuned navigation of latent motion diffusion models with minimal feedback,” in AAAI, 2024

    2024

  34. [42]

    Remodiffuse: Retrieval-augmented motion diffusion model,

    M. Zhang, X. Guo, L. Pan, Z. Cai, F. Hong, H. Li, L. Yang, and Z. Liu, “Remodiffuse: Retrieval-augmented motion diffusion model,” in ICCV. Los Alamitos, CA, USA: IEEE Computer Society, oct 2023, pp. 364–373. [Online]. Available: https://doi.ieeecomputersociety.org/ 10.1109/ICC...

    2023

  35. [43]

    Digital life project: Autonomous 3d characters with social intelligence,

    Z. Cai, J. Jiang, Z. Qing, X. Guo, M. Zhang, Z. Lin, H. Mei, C. Wei, R. Wang, W. Yin et al., “Digital life project: Autonomous 3d characters with social intelligence,” in CVPR, 2024, pp. 582–592

    2024

  36. [44]

    Momask: Generative masked modeling of 3d human motions,

    C. Guo, Y . Mu, M. G. Javed, S. Wang, and L. Cheng, “Momask: Generative masked modeling of 3d human motions,” in CVPR, 2024

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.