Pith. sign in

REVIEW 9 cited by

LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.07093 v2 pith:VFS5I4UD submitted 2024-10-09 cs.CV

Zhe Li , Weihao Yuan , Yisheng He , Lingteng Qiu , Shenhao Zhu , Xiaodong Gu , Weichao Shen , Yuan Dong
show 2 more authors
Zilong Dong Laurence T. Yang
This is my paper · ORCID
classification cs.CV
keywords motionlamptextcaptioninggenerationlanguage-motionclipfeatures
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Language plays a vital role in the realm of human motion. Existing methods have largely depended on CLIP text embeddings for motion generation, yet they fall short in effectively aligning language and motion due to CLIP's pretraining on static image-text pairs. This work introduces LaMP, a novel Language-Motion Pretraining model, which transitions from a language-vision to a more suitable language-motion latent space. It addresses key limitations by generating motion-informative text embeddings, significantly enhancing the relevance and semantics of generated motion sequences. With LaMP, we advance three key tasks: text-to-motion generation, motion-text retrieval, and motion captioning through aligned language-motion representation learning. For generation, we utilize LaMP to provide the text condition instead of CLIP, and an autoregressive masked prediction is designed to achieve mask modeling without rank collapse in transformers. For retrieval, motion features from LaMP's motion transformer interact with query tokens to retrieve text features from the text transformer, and vice versa. For captioning, we finetune a large language model with the language-informative motion features to develop a strong motion captioning model. In addition, we introduce the LaMP-BertScore metric to assess the alignment of generated motions with textual descriptions. Extensive experimental results on multiple datasets demonstrate substantial improvements over previous methods across all three tasks. The code of our method will be made public.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

    cs.CV 2026-03 conditional novelty 7.0 of 10

    UniMotion unifies continuous human-motion, text, and RGB understanding/generation/editing in one LLM backbone via CMA-VAE, Dual-Posterior KL Alignment, and Latent Reconstruction Alignment, reporting SOTA on seven tri-...

  2. Absolute Coordinates Make Motion Generation Easy

    cs.CV 2025-05 conditional novelty 7.0 of 10

    Using absolute 3D joint coordinates with a plain Transformer and velocity-prediction diffusion outperforms the standard local-relative motion representation, improving fidelity and enabling direct control.

  3. Language-Guided Transformer Tokenizer for Human Motion Generation

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Injecting language into the motion tokenizer yields more compact semantic tokens and state-of-the-art generation scores on HumanML3D and Motion-X.

  4. IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Interleaving motion generation with text-motion assessment and refinement improves alignment between generated human motion and goal text.

  5. Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A dexterous VLA pretrained on a 2.5M-instance human hand motion dataset transfers skills to a real robot hand, outperforming baselines in manipulation tasks.

  6. Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A system-first taxonomy and literature synthesis of multimodal unlearning across vision, language, video, and audio, with datasets, benchmarks, metrics, applications, and open challenges.

  7. Hierarchical Motion Captioning Utilizing External Text Data Source

    cs.LG 2025-09 conditional novelty 5.0 of 10

    This paper introduces a hierarchical motion captioning system that generates low-level descriptions with an LLM and retrieves high-level captions from a database, reporting large gains over prior methods on three datasets.

  8. Toward Rich Video Human-Motion2D Generation

    cs.CV 2025-06 reject novelty 4.0 of 10

    A new 150K-video 2D skeleton dataset with text captions and a diffusion model for single- and double-character motion generation, though the claimed FID-rewarded RL training is misrepresented.

  9. Motion Generation: A Survey of Generative Approaches and Benchmarks

    cs.CV 2025-07 unverdicted novelty 3.0 of 10

    A structured survey that categorizes recent motion generation methods by underlying generative approach and compiles datasets, metrics, and statistical trends.

Pith tools