REVIEW 27 cited by
MotionLLM: Understanding Human Behaviors from Human Motions and Videos
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
This study delves into the realm of multi-modality (i.e., video and motion modalities) human behavior understanding by leveraging the powerful capabilities of Large Language Models (LLMs). Diverging from recent LLMs designed for video-only or motion-only understanding, we argue that understanding human behavior necessitates joint modeling from both videos and motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics and semantics effectively. In light of this, we present MotionLLM, a straightforward yet effective framework for human motion understanding, captioning, and reasoning. Specifically, MotionLLM adopts a unified video-motion training strategy that leverages the complementary advantages of existing coarse video-text data and fine-grained motion-text data to glean rich spatial-temporal insights. Furthermore, we collect a substantial dataset, MoVid, comprising diverse videos, motions, captions, and instructions. Additionally, we propose the MoVid-Bench, with carefully manual annotations, for better evaluation of human behavior understanding on video and motion. Extensive experiments show the superiority of MotionLLM in the caption, spatial-temporal comprehension, and reasoning ability.
Forward citations
Cited by 27 Pith papers
-
UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation
UniMotion unifies continuous human-motion, text, and RGB understanding/generation/editing in one LLM backbone via CMA-VAE, Dual-Posterior KL Alignment, and Latent Reconstruction Alignment, reporting SOTA on seven tri-...
-
Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation
A single MLLM trained with a vision-guided hybrid VQ-VAE tokenizer reports state-of-the-art or competitive results for 3D pose estimation, motion prediction, and motion in-betweening on Human3.6M and 3DPW.
-
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.
-
Being-M0.5: A Real-Time Controllable Vision-Language-Motion Model
Being-M0.5 combines part-aware residual quantization with a 5M-sequence web-video dataset to reach real-time, part-controllable 3D motion generation, though its state-of-the-art claim does not hold on every standard b...
-
Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos
A dexterous VLA pretrained on a 2.5M-instance human hand motion dataset transfers skills to a real robot hand, outperforming baselines in manipulation tasks.
-
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.
-
IKMo: Image-Keyframed Motion Generation with Trajectory-Pose Conditioned Motion Diffusion Model
A motion diffusion model with decoupled trajectory and keyframe-pose control, wrapped in an MLLM agent system, produces more controllable 3D human motion from images and text.
-
HuMoCon: Concept Discovery for Human Motion Understanding
A framework that combines explicit video-motion feature alignment with velocity-aware masked autoencoding to improve LLM-based human motion and video question answering.
-
FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding
A new figure skating dataset and benchmark, FSAnno and FSBench, evaluates and improves multimodal large language models on technical and artistic understanding of figure skating.
-
CASIM: Composite Aware Semantic Injection for Text to Motion Generation
CASIM replaces fixed-length text embeddings with token-level cross-attention in text-to-motion models, improving alignment and quality for both diffusion and autoregressive generators.
-
Motion-X++: A Large-Scale Multimodal 3D Whole-body Human Motion Dataset
Motion-X++ provides 19.5M 3D whole-body pose annotations across 120.5K sequences with text, audio, video, and motion modalities.
-
CigTime: Corrective Instruction Generation Through Inverse Motion Editing
CigTime generates corrective text instructions from a source-target pair of 3D human motions by training an LLM on triplets synthesized with a text-conditioned motion editor.
-
Fleximo: Towards Flexible Text-to-Human Motion Video Generation
Fleximo generates a video of a person from one reference image and a text description of the motion, by chaining a text-to-3D-motion model, a skeleton rescaling step, and a trained skeleton adapter that fills in hand ...
-
VersatileMotion: A Unified Framework for Motion Synthesis and Comprehension
A unified multimodal motion LLM that handles nine generation and comprehension tasks across text, audio, and single/multi-agent motion, backed by a new dataset and tokenizer.
-
KinMo: Kinematic-aware Human Motion Understanding and Generation
KinMo adds hierarchical body-part-level text annotations to HumanML3D and shows that progressive text-motion alignment improves retrieval, generation, editing, and trajectory control.
-
Is this Generated Person Existed in Real-world? Fine-grained Detecting and Calibrating Abnormal Human-body
HumanCalibrator detects absent and redundant body parts in AI-generated human photos and repairs them while preserving the rest of the image.
-
Hierarchical Motion Captioning Utilizing External Text Data Source
This paper introduces a hierarchical motion captioning system that generates low-level descriptions with an LLM and retrieves high-level captions from a database, reporting large gains over prior methods on three datasets.
-
MAPF-World: Action World Model for Multi-Agent Path Finding
MAPF-World, an autoregressive action world model that predicts future states and actions, is claimed to beat state-of-the-art learnable MAPF solvers while using 96.5% fewer parameters and 92% less data.
-
ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understanding
ActionArt provides a fine-grained human-motion video benchmark and shows that proxy tasks trained on auto-generated captions lift a video MLLM from 55.8% to 63.0% average accuracy, still short of 67.6% with manual ann...
-
MONA: Moving Object Detection from Videos Shot by Dynamic Camera
MONA detects moving objects by filtering YOLO boxes with LEAP-VO dynamic point probabilities and optical flow, then segments them with SAM; integrated with LEAP-VO it reports state-of-the-art trajectory accuracy on MP...
-
RMD: A Simple Baseline for More General Human Motion Generation via Training-free Retrieval-Augmented Motion Diffuse
RMD improves text-to-motion generation by decomposing prompts with an LLM, retrieving and recombining body-part motions, and refining them with a pretrained diffusion model, all without additional training.
-
Human Motion Instruction Tuning
A language model trained on continuous motion features, video, and text with text-guided frame selection improves scores on motion understanding benchmarks.
-
KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model
KptLLM++ unifies keypoint semantic understanding, visual-prompt detection, and text-prompt detection in a single multimodal LLM, reporting SOTA accuracy on COCO, AP-10K, Human-Art, and other benchmarks.
-
HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding
A human-centric vision-speech language model with three instruction-weighted visual branches and audio input reports strong emotion, facial expression, and action results, but its evaluation is under-specified and not...
-
Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward
A survey paper reviews multimodal generative AI and autoregressive LLMs for text-driven human motion generation, with comparative tables of models, datasets, and metrics.
-
Text-driven Motion Generation: Overview, Challenges and Directions
A comprehensive taxonomy of text-to-motion generation methods along architectural and representation axes, with datasets and metrics.
-
Do Language Models Understand Time?
A survey arguing that video-LLMs rely on pretrained encoders and short-biased datasets, leaving them weak at long-term temporal reasoning such as causality and event progression.
Discussion (0). Continue with ORCID to comment.