Pith. sign in

REVIEW 5 major objections 5 minor 63 references

PlanMoGPT: Flow-Enhanced Progressive Planning for Text to Motion Synthesis

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PlanMoGPT claims that LLM-based text-to-motion generation becomes state-of-the-art when motion tokens are planned coarse-to-fine and decoded with flow matching.

desk verdict A well-engineered LLM text-to-motion system with real gains on HumanML3D, but the headline long-sequence SOTA rests on self-built benchmarks that need stronger validation. read the letter →

arxiv 2506.17912 v1 pith:5CJUFD42 submitted 2025-06-22 cs.CV cs.MM

Chuhao Jin , Haosen Li , Bingzi Zhang , Che Liu , Xiting Wang , Ruihua Song , Wenbing Huang , Ying Qin
show 2 more authors
Fuzheng Zhang Di Zhang
This is my paper
classification cs.CVcs.MM
keywords text-to-motionsynthesislargelanguagemodelsmotiontokenizationVQ-VAEflowmatchingprogressiveplanninglong-sequencegenerationdiversity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that LLMs can beat non-LLM methods in text-to-motion generation if motion is represented as fine-grained discrete tokens that are generated in a coarse-to-fine plan, then refined by a flow-matching decoder. It identifies token granularity as the bottleneck: too-fine tokens make LLMs copy nearby tokens and drift from the text, while too-coarse tokens lose detail. The proposed PlanMoGPT first generates a sparse skeleton plan, then fills in half-resolution tokens, then the full sequence, while a 4096-entry codebook at half downsampling with flow-based decoding preserves motion nuance. On long-sequence benchmarks built by stitching 2-5 motion clips, the paper reports FID improving from 0.380 to 0.141 and diversity rising 49.9% relative to a leading baseline. If right, this would make LLM-based motion generation both higher quality and more diverse than current diffusion approaches.

What carries the argument

The central machinery is a two-part pipeline. First, a fine-grained VQ-VAE tokenizer with downsampling rate 2 (rather than 4) and codebook size 4096 (rather than 512) converts motion into discrete tokens with less quantization loss; a flow-matching decoder then treats the token-decoded motion as time-zero of an ODE and integrates a learned vector field to add back detail. Second, the LLM generates tokens in three hierarchy levels: every-4-frames plan tokens, every-2-frames plan tokens, and the full token sequence, with special transition tokens separating levels. The sparse upper levels remove the strong local autocorrelation that makes fine-grained token prediction drift from the text, while lower levels supply detail; the ablation shows interval-4 plans improve semantic alignment, interval-2 plans improve FID, and combining them is cumulative.

What would settle it

Retrain MoMask, BAMM, MotionDiffuse, and T2M-GPT on HumanML3D++ and KIT-ML++ with matched compute and hyperparameter tuning, and evaluate all methods on an independently human-validated long-sequence test set built without language-model text merging. If the best retrained baseline comes within noise of FID 0.141 on HumanML3D++, or if removing progressive planning from PlanMoGPT leaves FID unchanged, then the paper's central attribution fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the apparent ceiling on LLM-based text-to-motion generation is set by motion token granularity, not by LLM capacity. Fine-grained discrete motion tokens make next-token prediction too locally predictable, so the model leans on adjacent tokens and drifts from the text; coarse tokens erase detail. PlanMoGPT attacks both sides: a tokenizer that runs at half the downsampling rate with an eight-times larger codebook (4096 entries) keeps detail, and a flow-matching decoder refines the coarse reconstruction back toward real motion; on the language side, an LLM first emits a sparse every-4-frames plan, then every-2-frames, then the full token sequence, so each level anchors the next. On the new long-sequence benchmarks, the method reports FID 0.141 versus 0.380 for MoMask on HumanML3D++, and MModality 2.538 versus 1.693, which the paper reads as resolving the diversity-quality dilemma. The authors also report suboptimal results on KIT-ML, attributing the shortfall to that dataset's smaller scale and lower temporal resolution.

Load-bearing premise

The load-bearing assumption is that HumanML3D++ and KIT-ML++, built by stitching 2-5 motion clips and merging their texts with an LLM, fairly represent long text-to-motion generation; only a 100-sample subset was manually checked (86% reliable) and only two baselines were retrained on these splits.

Editorial extensions

If this is right

  • If the reported numbers hold, LLM-based text-to-motion generators can match or beat diffusion-based methods on quality while generating markedly more diverse motions, so the previous gap between LLM and non-LLM routes is not a hard limit.
  • Long-sequence generation becomes a relative strength rather than a failure mode: the biggest FID gains appear on HumanML3D++ and KIT-ML++, where sequences reach roughly 50 seconds.
  • The diversity-quality trade-off observed in non-LLM methods is not intrinsic: PlanMoGPT keeps FID and R-precision nearly flat as multimodality rises, according to the paper's diversity analysis.
  • The flow-enhanced decoder, not just the larger codebook, drives the FID gain: ablations show the flow version beats residual VQ-VAE at the same codebook and downsampling settings, and the chosen 4096/2 configuration gives the best generation FID.
  • Training is comparatively practical: the paper reports convergence in 25 epochs versus 500 for MoMask-style training, which would make the LLM route more accessible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the coarse-to-fine plan-then-fill recipe is a general strategy for autoregressive generation over any locally autocorrelated discrete sequence, so it could transfer to video token prediction, gesture generation, or long-form audio.
  • Editorial inference: the ablation evidence suggests the 4096-entry codebook helps only when paired with a decoder that can repair quantization loss; a fair external test would vary codebook size and decoder type independently across more LLM backbones.
  • Editorial inference: because HumanML3D++ and KIT-ML++ were made by stitching clips and merging text with an LLM, independent human annotation of a larger sample would clarify whether the long-sequence gains reflect genuine long-motion understanding or distributional artifacts of the new benchmarks.
  • Editorial inference: the method's flat FID-versus-diversity curve predicts that pushing non-LLM baselines to comparable MModality by raising sampling temperature should degrade their FID; if it does not, the claimed resolution of the dilemma would need revision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes PlanMoGPT, an LLM-based text-to-motion framework that combines progressive multi-interval planning (generating tokens at intervals T=4, T=2, then T=1) with a flow-enhanced motion tokenizer (downsampling rate 2, codebook size 4096, and a flow-matching decoder). The authors claim state-of-the-art results on HumanML3D and introduce two new long-sequence benchmarks, HumanML3D++ and KIT-ML++, built by concatenating 2-5 motion clips and merging texts with GPT-4. The headline claim is a 63.8% FID improvement on long-sequence generation (0.380 to 0.141) and a 49.9% MModality improvement, which they interpret as resolving a diversity-quality trade-off in non-LLM methods.

Significance. If the long-sequence results hold, the paper would make a useful contribution by addressing the local-dependency problem in token-based motion generation and by demonstrating a concrete mechanism for coarse-to-fine LLM planning. The paper's strengths include a well-motivated and clearly ablated progressive planning design, competitive short-sequence results on HumanML3D against strong baselines, and detailed ablation studies covering codebook size, downsampling rate, plan intervals, and flow-matching steps. However, the long-sequence state-of-the-art claim currently rests on self-built benchmarks with only two retrained baselines, and the headline percentage improvement is arithmetically incorrect. These issues are load-bearing for the paper's central claim, so the manuscript needs substantial revision.

major comments (5)
  1. [§4.1, Table 2] The long-sequence SOTA claim is supported by only two retrained baselines, T2M-GPT and MoMask, on the self-built HumanML3D++ and KIT-ML++ datasets. No comparisons are provided against BAMM, MotionDiffuse, MotionGPT, or long-motion-specific methods such as Infinite Motion [25], which is cited for the dataset construction. Consequently, the abstract's claim of state-of-the-art performance on long-sequence generation is not established against the broader field; the authors should either add these baselines or temper the claim to a comparison against the two retrained methods.
  2. [§4.1, evaluation protocol] The paper does not specify how FID and MM-Dist are computed for sequences up to 50 seconds. The standard T2M feature extractor used in these metrics was trained on short HumanML3D motions, and it is not stated whether it is applied to full long sequences or to chunks, nor whether its validity for long inputs was verified. Additionally, the flow-matching decoder is described as splitting motions into 64-frame clips and re-stitching them after inference; the same detail is not given for evaluation. This is load-bearing because all long-sequence quality numbers depend on an evaluation protocol that is currently underspecified.
  3. [§4.1, dataset quality] The new benchmarks are validated with only 100 manually checked motion-text pairs, of which 86% were deemed reliable. No description is provided of how clip boundaries are blended, how the GPT-4 merged texts are aligned to the transitions, or whether the 14% unreliable pairs are excluded from the test set. At this validation scale, the benchmark noise could materially affect the reported FID values; the authors should provide a more systematic validation protocol or release the datasets so other groups can independently assess them.
  4. [Abstract, Table 2] The claimed improvement "from 0.380 to 0.141" is a relative FID reduction of (0.380 - 0.141)/0.380 = 0.629, i.e., 62.9%, not 63.8%. The abstract and contributions section should be corrected to state 62.9%.
  5. [§4.1, baseline retraining] T2M-GPT and MoMask are retrained using their source code on the new long-sequence datasets, but the paper does not describe any adaptations for sequences up to 50 seconds, such as extended positional encodings, longer training schedules, or adjusted batch sizes. Since these methods were originally designed for motions of at most about 10 seconds, the comparison may systematically disadvantage them. The authors should state what adaptations were made and provide evidence that the retrained baselines are not under-tuned.
minor comments (5)
  1. [§3.1, Eq. (2)] The interpolation y_t is not explicitly defined between t=0 and t=1; please specify that y_t = (1-t) y_0 + t y_1 (or otherwise) so that the conditional flow-matching objective is unambiguous.
  2. [§3.2] The offset b is sampled from {1,...,Tmax}, but Tmax is not defined; if it is the largest plan interval, please state this explicitly.
  3. [§4.5, Table 4] The checkmark layout in Table 4 makes it difficult to determine which interval combination each row corresponds to; please align the checkmarks explicitly with the T=8, T=4, and T=2 columns.
  4. [§4.2] There is a typo in the text: "an MMdoality of 2.524" should read "an MModality of 2.524".
  5. [§4.3] Table 3 states that all compared models are base versions, but the surrounding text refers to "MoMask and BAMM" without this qualification; please make the distinction between base and full models consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are held-out measurements, not quantities forced by the paper's definitions, fits, or self-citations.

full rationale

I walked the claimed derivation chain: the method's components are a finer-grained VQ-VAE tokenizer (r=2, codebook 4096), a flow-matching decoder, and an LLM with progressive multi-interval planning. None of these components is defined in terms of the headline metrics, and no fitted parameter is renamed as a prediction. The headline FID improvement (0.380 to 0.141 on HumanML3D++) is a measured comparison against retrained baselines on a held-out test split, not a quantity forced by construction. The paper does not invoke a load-bearing self-citation: its architectural antecedents (MoMask, T2M-GPT, flow matching) are external prior work, and the proposed mechanism is described and ablated directly. The self-built long-sequence benchmarks (HumanML3D++, KIT-ML++) and the limited baseline retraining raise external-validity concerns—the 100-sample manual check with 86% reliability and the absence of long-motion-specific baselines are genuine limitations—but these are correctness/benchmark-validity issues, not circularity. The dataset is not used to define the method, and the test-set numbers are not derived from the training procedure. No equation reduces to another equation by construction, and no prediction is equivalent to an input fit. Therefore the paper is self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method depends on a small number of design hyperparameters selected by ablation, plus the conceptual assumption that coarse-to-fine planning helps LLMs attend to global semantics. No new physical or formal entities are introduced.

free parameters (4)
  • codebook_size = 4096
    Chosen by ablation in Table 5a; larger codebooks improve generation FID up to 4096, which is picked for the final model.
  • downsampling_rate = 2
    Chosen by ablation in Table 5a; rate 2 with codebook 4096 gives best generation FID, so it is selected.
  • plan_intervals = T=4 and T=2
    Selected from the ablation in Table 4; T=4 improves semantic alignment, T=2 improves FID, and the pair is chosen as cumulative and sufficient.
  • flow_matching_inference_steps = 30
    Chosen from Figure 5b, where FID converges around 30 ODE steps.
assumptions (3)
  • domain assumption LLM autoregressive planning improves global semantic alignment in motion generation.
    Invoked in Section 1 and 3.2 to justify predicting sparse global plans first; it is a modeling hypothesis, not proven independently.
  • domain assumption Fine-grained motion tokenization causes a local dependency problem in LLMs.
    This is the paper's core diagnostic claim, supported only by a case study and an illustrative figure, not by a controlled experiment.
  • standard math Conditional flow matching (Eq. 2) provides a valid way to refine coarse motion into natural motion.
    Flow matching is an established generative technique; the paper applies it as in prior work [28].

how reviews work

0 comments
Cite this review

Pith. "Pith review of PlanMoGPT: Flow-Enhanced Progressive Planning for Text to Motion Synthesis." pith.science (2026). https://pith.science/paper/5CJUFD42

@misc{pith2026250617912,
  author       = {Pith},
  title        = {Pith review of: PlanMoGPT: Flow-Enhanced Progressive Planning for Text to Motion Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5CJUFD42}},
  note         = {Machine review of arXiv:2506.17912}
}
read the original abstract

Recent advances in large language models (LLMs) have enabled breakthroughs in many multimodal generation tasks, but a significant performance gap still exists in text-to-motion generation, where LLM-based methods lag far behind non-LLM methods. We identify the granularity of motion tokenization as a critical bottleneck: fine-grained tokenization induces local dependency issues, where LLMs overemphasize short-term coherence at the expense of global semantic alignment, while coarse-grained tokenization sacrifices motion details. To resolve this issue, we propose PlanMoGPT, an LLM-based framework integrating progressive planning and flow-enhanced fine-grained motion tokenization. First, our progressive planning mechanism leverages LLMs' autoregressive capabilities to hierarchically generate motion tokens by starting from sparse global plans and iteratively refining them into full sequences. Second, our flow-enhanced tokenizer doubles the downsampling resolution and expands the codebook size by eight times, minimizing detail loss during discretization, while a flow-enhanced decoder recovers motion nuances. Extensive experiments on text-to-motion benchmarks demonstrate that it achieves state-of-the-art performance, improving FID scores by 63.8% (from 0.380 to 0.141) on long-sequence generation while enhancing motion diversity by 49.9% compared to existing methods. The proposed framework successfully resolves the diversity-quality trade-off that plagues current non-LLM approaches, establishing new standards for text-to-motion generation.

Figures

Figures reproduced from arXiv: 2506.17912 by the authors.

Figure 1
Figure 1. Generating complex and long-sequence motion by our PlanMoGPT, T2M-GPT [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. PlanMoGPT consists of two components: (a). A flow-enhanced motion tokenizer converts [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Different methods repeatedly generate 30 motions based on the same text. Similar motions [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Case study on HumanML3D and HumanML3D++ datasets. The score range in Figure (a) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: (a). Comparisons of average inference time cost. (b). Exploring the inference step of the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 30 canonical work pages

  1. [25]

    Infinite Motion: Extended Motion Generation via Long Text Instructions

    Mengtian Li, Chengshuo Zhai, Shengxiang Yao, Zhifeng Xie, Keyu Chen, and Yu-Gang Jiang. Infinite motion: Extended motion generation via long text instructions.arXiv preprint arXiv:2407.08443, 2024

    work page Pith review arXiv 2024

  2. [1]

    Motion flow matching for human motion synthesis and editing.arXiv preprint arXiv:2312.08895, 2023

    arXiv 2023

  3. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [3]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  5. [4]

    Gesturediffuclip: Gesture diffusion model with clip latents.ACM Transactions on Graphics (TOG), 42(4):1–18, 2023

    Tenglong Ao, Zeyi Zhang, and Libin Liu. Gesturediffuclip: Gesture diffusion model with clip latents.ACM Transactions on Graphics (TOG), 42(4):1–18, 2023

    work page 2023

  6. [5]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  7. [6]

    Long- term human motion prediction with scene context

    Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh V o, and Jitendra Malik. Long- term human motion prediction with scene context. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 387–404. Springer, 2020

    work page 2020

  8. [7]

    Cmu graphics lab motion capture database, 2003

    Carnegie Mellon University. Cmu graphics lab motion capture database, 2003

    work page 2003

Show all 63 references
  1. [8]

    Matterport3d: Learning from rgb-d data in indoor environments.arXiv preprint arXiv:1709.06158, 2017

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments.arXiv preprint arXiv:1709.06158, 2017

    2017 arXiv

  2. [9]

    Executing your commands via motion diffusion in latent space

    Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18000–18010, 2023

    2023

  3. [10]

    Generative adversarial graph convolutional networks for human action synthesis

    Bruno Degardin, Joao Neves, Vasco Lopes, Joao Brito, Ehsan Yaghoubi, and Hugo Proença. Generative adversarial graph convolutional networks for human action synthesis. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1150–1159, 2022

    2022

  4. [11]

    Learning individual styles of conversational gesture

    Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Ma- lik. Learning individual styles of conversational gesture. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3497–3506, 2019

    2019

  5. [12]

    Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    2014

  6. [13]

    Momask: Generative masked modeling of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900–1910, 2024

    1900

  7. [14]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152–5161, 2022. 10

    2022

  8. [15]

    Action2motion: Conditioned generation of 3d human motions

    Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. InProceedings of the 28th ACM International Conference on Multimedia, pages 2021–2029, 2020

    2021

  9. [16]

    Stochastic scene-aware motion prediction

    Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11374–11384, 2021

    2021

  10. [17]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  11. [18]

    Avatarclip: zero-shot text-driven generation and animation of 3d avatars.ACM Transactions on Graphics (TOG), 41(4):1–19, 2022

    Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: zero-shot text-driven generation and animation of 3d avatars.ACM Transactions on Graphics (TOG), 41(4):1–19, 2022

    2022

  12. [19]

    Motiongpt: Human motion as a foreign language.Advances in Neural Information Processing Systems, 36:20067–20079, 2023

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language.Advances in Neural Information Processing Systems, 36:20067–20079, 2023

    2023

  13. [20]

    Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013

    Diederik P Kingma. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  14. [21]

    Generating images with multimodal language models.Advances in Neural Information Processing Systems, 36:21487–21506, 2023

    Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models.Advances in Neural Information Processing Systems, 36:21487–21506, 2023

    2023

  15. [22]

    Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    2022

  16. [23]

    Analyzing input and output representations for speech-driven gesture generation

    Taras Kucherenko, Dai Hasegawa, Gustav Eje Henter, Naoshi Kaneko, and Hedvig Kjellström. Analyzing input and output representations for speech-driven gesture generation. InProceedings of the 19th ACM International Conference on Intelligent Virtual Agents, pages 97–104, 2019

    2019

  17. [24]

    Au- dio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders

    Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Au- dio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11293–1...

    2021

  18. [26]

    Ai choreographer: Music conditioned 3d dance generation with aist++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 13401–13412, 2021

    2021

  19. [27]

    A survey of multimodel large language models

    Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. InProceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pages 405–409, 2024

    2024

  20. [28]

    Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  21. [29]

    Beat: A large-scale semantic and emotional multi-modal dataset for conversa- tional gestures synthesis

    Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. Beat: A large-scale semantic and emotional multi-modal dataset for conversa- tional gestures synthesis. InEuropean conference on computer vision, pages 612–630. Springer, 2022

    2022

  22. [30]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  23. [31]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. InProceedings of the IEEE/CVF international conference on computer vision, pages 5442–5451, 2019. 11

    2019

  24. [32]

    The kit whole-body human motion database

    Christian Mandery, Ömer Terlemez, Martin Do, Nikolaus Vahrenkamp, and Tamim Asfour. The kit whole-body human motion database. In2015 International Conference on Advanced Robotics (ICAR), pages 329–336. IEEE, 2015

    2015

  25. [33]

    Gan-based reactive motion synthesis with class-aware discriminators for human–human interaction.Computers & Graphics, 102:634–645, 2022

    Qianhui Men, Hubert PH Shum, Edmond SL Ho, and Howard Leung. Gan-based reactive motion synthesis with class-aware discriminators for human–human interaction.Computers & Graphics, 102:634–645, 2022

    2022

  26. [34]

    Action-conditioned 3d human motion synthesis with transformer vae

    Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10985–10995, 2021

    2021

  27. [35]

    Bamm: Bidirectional autoregressive motion model

    Ekkasit Pinyoanuntapong, Muhammad Usama Saleem, Pu Wang, Minwoo Lee, Srijan Das, and Chen Chen. Bamm: Bidirectional autoregressive motion model. InComputer Vision – ECCV 2024, 2024

    2024

  28. [36]

    Babel: Bodies, action and behavior with english labels

    Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J Black. Babel: Bodies, action and behavior with english labels. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 722–731, 2021

    2021

  29. [37]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  30. [38]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMedical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part I...

    2015

  31. [39]

    Human motion diffusion as a generative prior

    Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion as a generative prior. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  32. [40]

    Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  33. [41]

    Goal: Generating 4d whole-body motion for hand-object grasping

    Omid Taheri, Vasileios Choutas, Michael J Black, and Dimitrios Tzionas. Goal: Generating 4d whole-body motion for hand-object grasping. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13263–13273, 2022

    2022

  34. [42]

    Motionclip: Exposing human motion generation to clip space

    Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. InEuropean Conference on Computer Vision, pages 358–374. Springer, 2022

    2022

  35. [43]

    Human motion diffusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. InThe Eleventh International Conference on Learning Repre- sentations, 2023

    2023

  36. [44]

    Neural discrete representation learning.Advances in neural information processing systems, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advances in neural information processing systems, 30, 2017

    2017

  37. [45]

    Scene-aware generative network for human motion synthesis

    Jingbo Wang, Sijie Yan, Bo Dai, and Dahua Lin. Scene-aware generative network for human motion synthesis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12206–12215, 2021

    2021

  38. [46]

    Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasonin...

    2024 arXiv

  39. [47]

    Motiongpt-2: A general-purpose motion-language model for motion generation and understanding.arXiv preprint arXiv:2410.21747, 2024

    Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. Motiongpt-2: A general-purpose motion-language model for motion generation and understanding.arXiv preprint arXiv:2410.21747, 2024. 12

    arXiv

  40. [48]

    Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  41. [49]

    What are diffusion models?lilianweng

    Lilian Weng. What are diffusion models?lilianweng. github. io, page 21, 2021

    2021

  42. [50]

    Motion- agent: A conversational framework for human motion generation with LLMs

    Qi Wu, Yubo Zhao, Yifan Wang, Xinhang Liu, Yu-Wing Tai, and Chi-Keung Tang. Motion- agent: A conversational framework for human motion generation with LLMs. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  43. [51]

    Saga: Stochastic whole-body grasping with contact

    Yan Wu, Jiahao Wang, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, and Siyu Tang. Saga: Stochastic whole-body grasping with contact. InEuropean Conference on Computer Vision, pages 257–274. Springer, 2022

    2022

  44. [52]

    Actformer: A gan-based transformer towards general action-conditioned 3d human motion generation

    Liang Xu, Ziyang Song, Dongliang Wang, Jing Su, Zhicheng Fang, Chenjing Ding, Weihao Gan, Yichao Yan, Xin Jin, Xiaokang Yang, et al. Actformer: A gan-based transformer towards general action-conditioned 3d human motion generation. InProceedings of the IEEE/CVF International Co...

    2023

  45. [53]

    Qpgesture: Quantization-based and phase-guided motion matching for natural speech- driven gesture generation

    Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, and Haolin Zhuang. Qpgesture: Quantization-based and phase-guided motion matching for natural speech- driven gesture generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2023

  46. [54]

    Structure-aware human- action generation

    Ping Yu, Yang Zhao, Chunyuan Li, Junsong Yuan, and Changyou Chen. Structure-aware human- action generation. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, pages 18–34. Springer, 2020

    2020

  47. [55]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.arXiv preprint arXiv:2305.11000, 2023

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.arXiv preprint arXiv:2305.11000, 2023

    2023 arXiv

  48. [56]

    T2m-gpt: Generating human motion from textual descriptions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2023

  49. [57]

    Motiondiffuse: Text-driven human motion generation with diffusion model.arXiv preprint arXiv:2208.15001, 2022

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model.arXiv preprint arXiv:2208.15001, 2022

    2022 arXiv

  50. [58]

    Remodiffuse: Retrieval-augmented motion diffusion model

    Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmented motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 364–373, 2023

    2023

  51. [59]

    Finemogen: Fine-grained spatio-temporal motion generation and editing.NeurIPS, 2023

    Mingyuan Zhang, Huirong Li, Zhongang Cai, Jiawei Ren, Lei Yang, and Ziwei Liu. Finemogen: Fine-grained spatio-temporal motion generation and editing.NeurIPS, 2023

    2023

  52. [60]

    Tinyllama: An open-source small language model, 2024

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024

    2024

  53. [61]

    Large language models as commonsense knowledge for large-scale task planning.Advances in Neural Information Processing Systems, 36:31967– 31987, 2023

    Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning.Advances in Neural Information Processing Systems, 36:31967– 31987, 2023

    2023

  54. [62]

    Llamafactory: Unified efficient fine-tuning of 100+ language models.arXiv preprint arXiv:2403.13372, 2024

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models.arXiv preprint arXiv:2403.13372, 2024

    2024 arXiv

  55. [63]

    base” for “residual

    Wentao Zhu, Xiaoxuan Ma, Dongwoo Ro, Hai Ci, Jinlu Zhang, Jiaxin Shi, Feng Gao, Qi Tian, and Yizhou Wang. Human motion generation: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 13 Datasets Methods R-Precision↑ FID↓MM-Dist↓MModality↑Top-1 Top-2 ...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.