REVIEW 8 cited by
MotionLLM: Understanding Human Behaviors from Human Motions and Videos
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This study delves into the realm of multi-modality (i.e., video and motion modalities) human behavior understanding by leveraging the powerful capabilities of Large Language Models (LLMs). Diverging from recent LLMs designed for video-only or motion-only understanding, we argue that understanding human behavior necessitates joint modeling from both videos and motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics and semantics effectively. In light of this, we present MotionLLM, a straightforward yet effective framework for human motion understanding, captioning, and reasoning. Specifically, MotionLLM adopts a unified video-motion training strategy that leverages the complementary advantages of existing coarse video-text data and fine-grained motion-text data to glean rich spatial-temporal insights. Furthermore, we collect a substantial dataset, MoVid, comprising diverse videos, motions, captions, and instructions. Additionally, we propose the MoVid-Bench, with carefully manual annotations, for better evaluation of human behavior understanding on video and motion. Extensive experiments show the superiority of MotionLLM in the caption, spatial-temporal comprehension, and reasoning ability.
Forward citations
Cited by 8 Pith papers
-
UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation
UniMotion unifies continuous human-motion, text, and RGB understanding/generation/editing in one LLM backbone via CMA-VAE, Dual-Posterior KL Alignment, and Latent Reconstruction Alignment, reporting SOTA on seven tri-...
-
Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation
A single MLLM trained with a vision-guided hybrid VQ-VAE tokenizer reports state-of-the-art or competitive results for 3D pose estimation, motion prediction, and motion in-betweening on Human3.6M and 3DPW.
-
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.
-
Being-M0.5: A Real-Time Controllable Vision-Language-Motion Model
Being-M0.5 combines part-aware residual quantization with a 5M-sequence web-video dataset to reach real-time, part-controllable 3D motion generation, though its state-of-the-art claim does not hold on every standard b...
-
Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos
A dexterous VLA pretrained on a 2.5M-instance human hand motion dataset transfers skills to a real robot hand, outperforming baselines in manipulation tasks.
-
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.
-
Hierarchical Motion Captioning Utilizing External Text Data Source
This paper introduces a hierarchical motion captioning system that generates low-level descriptions with an LLM and retrieves high-level captions from a database, reporting large gains over prior methods on three datasets.
-
KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model
KptLLM++ unifies keypoint semantic understanding, visual-prompt detection, and text-prompt detection in a single multimodal LLM, reporting SOTA accuracy on COCO, AP-10K, Human-Art, and other benchmarks.
Discussion (0). Sign in to comment.