Pith. sign in

REVIEW 3 major objections 5 minor 53 references

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

T0 review · 3 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single continuous model can understand, generate, and edit human motion together with text and RGB images.

desk verdict Solid first continuous Motion–Text–RGB any-to-any system; ablations support the mechanisms, but the multi-stage curriculum is the softest link on the SOTA claim. read the letter →

arxiv 2603.22282 v2 pith:HMUZ6LX5 submitted 2026-03-23 cs.CV cs.AI

Ziyi Wang , Xinshun Wang , Shuang Chen , Yang Cong , Mengyuan Liu This is my paper
classification cs.CVcs.AI
keywords humanmotiongenerationunifiedmultimodalmodelscontinuousrepresentationVAEcross-modalalignmentflowmatchingMLLMsediting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UniMotion argues that human motion can finally sit inside one multimodal language-model backbone on equal footing with text and RGB, supporting understanding, generation, and editing in both directions among all three modalities. Earlier systems covered only motion–text or static pose–image pairs and mostly quantized motion into discrete tokens, which injects jitter and breaks temporal continuity. The authors replace that with continuous motion latents, a Cross-Modal Aligned Motion VAE that absorbs visual-semantic priors during training, and a self-supervised latent-reconstruction warm-up that calibrates the new motion pathway before sparse text supervision is applied. The resulting model covers seven any-to-any tasks and leads prior partial solutions, with the largest gains on compositional cross-modal work such as motion-guided image editing and text-conditioned motion editing. A reader who builds animation, embodied agents, or rehab tools would care because one continuous pathway can replace a stack of specialized discrete pipelines.

What carries the argument

CMA-VAE with Dual-Posterior KL Alignment (DPA) and Latent Reconstruction Alignment (LRA): continuous motion latents are produced by a motion encoder whose posterior is distilled toward a vision-fused teacher (DPA), then the embedder, backbone, and flow head are co-calibrated by reconstructing those dense latents from noise (LRA) before multi-task fine-tuning.

What would settle it

Retrain the identical architecture and schedule with DPA and LRA both ablated; if Text-to-Motion R@3, motion-editing R@3, and Vision-to-Motion MPJPE remain within a few points of the full model on the same splits, the two alignment mechanisms are not load-bearing as claimed.

Watch

Extended reading notes

Core claim

The paper claims that treating motion as a first-class continuous modality—encoded by a Cross-Modal Aligned Motion VAE, aligned by Dual-Posterior KL distillation from a vision-fused teacher, and cold-started by Latent Reconstruction Alignment—yields a single shared LLM backbone that simultaneously supports understanding, generation, and editing across motion, text, and RGB, and that this continuous design reaches state-of-the-art on all seven evaluated tri-modal tasks.

Load-bearing premise

The result depends on the premise that reverse-KL distillation from a vision-fused encoder trained mainly on indoor paired frames, together with a multi-stage self-reconstruction warm-up, is enough to give useful visual geometry to unpaired motion–text data and to stabilize the whole unified model.

Editorial extensions

If this is right

  • One continuous backbone can replace separate motion–text and pose–vision systems for the seven listed understanding, generation, and editing tasks.
  • Continuous latents remove quantization-induced temporal jitter and improve fine-grained prediction and joint-level editing.
  • Visual-semantic priors learned on paired-image datasets transfer to unpaired motion–text data through shared encoder parameters.
  • Dual-path (semantic plus generation) embedders are required for motion-conditioned synthesis where structure and detail must be preserved together.
  • Self-supervised reconstruction of dense motion latents is a practical cold-start when text alone is too sparse to calibrate a new continuous pathway.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same continuous dual-path recipe could absorb multi-person or hand–object motion without redesigning a discrete codebook.
  • Reverse-KL posterior distillation from a sensor-paired teacher may be reusable for other sparse modalities such as audio or force inside an LLM backbone.
  • The remaining gap to specialist mesh-recovery numbers suggests unified continuous models may still need explicit geometric auxiliary losses for high-precision body recovery.
  • Zero-shot transfer to in-the-wild 3DPW implies that indoor paired-frame priors already encode transferable body-structure geometry.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. UniMotion claims to be the first single architecture that unifies understanding, generation, and editing across continuous human motion, natural language, and RGB images. It encodes motion with a Cross-Modal Aligned Motion VAE (CMA-VAE), injects visual-semantic priors via Dual-Posterior KL Alignment (DPA; reverse KL of the motion-only posterior to a vision-fused teacher, Eqs. 3–7), and cold-starts the continuous pathway with Latent Reconstruction Alignment (LRA / M2M self-reconstruction, Eq. 9). Symmetric dual-path embedders, hybrid attention, modality-routed LoRA, and modality-specific flow heads sit on a Show-o2 backbone. The model is evaluated on seven tasks (T2M, M2T, prediction, editing, V2M, V2T, MGIE) and reports SOTA or best-in-class numbers on most semantic metrics, with ablations isolating CMA-VAE, DPA, LRA, dual-path, hybrid attention, and routed LoRA.

Significance. If the results hold under fairer controls, the paper would be a genuine advance: the first continuous Motion–Text–RGB MLLM that supports any-to-any understanding, generation, and editing in one model, with clear gains on compositional tasks (editing, MGIE, prediction) that discrete Motion-Text or static Pose-Image systems cannot address. Strengths include systematic ablations (Tables 6, 9, 11–13), a unified 269-dim motion representation, zero-shot 3DPW transfer, and detailed multi-stage training disclosure (Table 15). The continuous-first design and DPA/LRA mechanisms are concrete, reusable contributions even if some absolute FID numbers remain behind single-task discrete specialists.

major comments (3)
  1. Sec. 3.2–3.4 and Table 9: The central claim that DPA + LRA are the load-bearing mechanisms for continuous tri-modal SOTA rests on ablations that remove each component from the full multi-stage pipeline (CMA-VAE 210k + LRA 80k + Stages 1a–3 ≈790k steps, Table 15). There are no compute-matched single-stage or order-permuted controls. The observed drops (T2M R@3 0.841→0.818/0.801; Edit R@3 84.94→80.35/78.72) could partly reflect curriculum length or progressive unfreezing rather than the reverse-KL or M2M objectives themselves. A matched-budget ablation (or at least Stage-0-only vs. Stage-0-shuffled) is needed to substantiate that the continuous pathway + DPA/LRA, not the elaborate schedule, drive the gains.
  2. Sec. 3.2, Eqs. 6–7 and the claim of transfer to unpaired HumanML3D: DPA is applied only on H3.6M-style paired frames; benefit on T2M/M2T is attributed to shared encoder parameters. Table 9 shows a drop without DPA, but there is no direct measurement of how much of the vision-fused posterior actually transfers (e.g., posterior-distance or representation-similarity diagnostics on HumanML3D latents). Without that, the assertion that reverse KL injects useful visual-semantic priors into unpaired motion remains an inference rather than a demonstrated mechanism.
  3. Table 3 vs. Tables 2/5/8: On pure T2M, UniMotion’s FID (0.194) is worse than single-task discrete methods (MoMask 0.045) while R-Precision/MMDist are better. The paper correctly emphasizes semantic alignment, yet the abstract and Fig. 1 frame the result as consistent superiority across seven tasks. The manuscript should more carefully qualify where continuous unification trades reconstruction fidelity for cross-modal compositionality, so the SOTA claim is not overstated on distribution-matching metrics.
minor comments (5)
  1. Fig. 1 radar and Table 2: Several baseline cells are N/A by design; the visual comparison can be misread as head-to-head superiority. Clarify that partial methods are scored only on supported tasks.
  2. Sec. 3.3 / hybrid attention mask (Eq. 14 in supplement): The formal mask is clear, but a short main-text sentence on why image tokens receive the same span-full pattern as motion would help readers who skip the supplement.
  3. Notation: z, z_fused, z_motion, and Embed_fused appear with slight variations across Sec. 3.2–3.4; a single consistent symbol table would reduce friction.
  4. Supplementary Sec. F limitations: The indoor-centric visual alignment and 1.5B compute cost are acknowledged; a one-sentence pointer in the main conclusion would be useful for readers who do not open the supplement.
  5. Typos / polish: “squate”/“squatting” in Fig. 1 demo text; occasional “MotionGPT3” labels in reconstruction figures; ensure all arXiv citations have consistent venue years.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: UniMotion is an empirical architecture/training paper whose SOTA claims are measured on external public benchmarks with independent metrics, not forced by definition or self-citation.

full rationale

The paper’s load-bearing claims are (i) a continuous Motion–Text–RGB architecture (CMA-VAE + dual-path embedders + hybrid attention + modality-routed LoRA) and (ii) two training objectives (DPA reverse-KL posterior alignment; LRA M2M latent reconstruction) that improve seven any-to-any tasks. Performance is reported against external datasets and protocols (HumanML3D, MotionFix, Human3.6M, MoVid, 3DPW) and external baselines (MotionGPT, MG-MotionLLM, UniPose, Show-o2, MoMask, etc.) using standard metrics (R@k, FID, BertScore, MPJPE, BLEU, Mot.Acc). DPA (Eqs. 3–7) and LRA (Eq. 9) are optimization losses; they do not redefine those evaluation metrics, so ablations (Table 9) and SOTA tables are not tautological. LRA’s self-reconstruction is ordinary self-supervised pretraining with explicit non-identity design (embedder compression, flow-from-noise, information bottlenecks) and is validated by transfer to text/vision tasks that never use M2M. There is no uniqueness theorem imported from the authors, no fitted parameter renamed as a prediction, and no self-citation that forces the central results. Minor author self-citations on related motion topics are non-load-bearing. Score 0 is appropriate.

Assumptions & free parameters 5 free parameters · 4 assumptions · 3 invented entities

As an empirical systems paper the central claim rests on standard deep-learning practice plus a handful of design choices and hyper-parameters that are not derived from first principles. The invented components (CMA-VAE, DPA, LRA) are engineering constructs whose value is demonstrated only by the reported ablations; they have no independent theoretical necessity.

free parameters (5)
  • DPA alignment weight λ_align and linear warm-up length
    Chosen to keep reconstruction stable early in CMA-VAE training; value not derived.
  • LRA information-bottleneck rates (temporal keep 20–50 %, feature dropout 0.15, Gaussian σ=0.02)
    Hand-tuned to prevent trivial identity mapping; no theoretical optimum given.
  • Modality-routed LoRA rank (32) and flow-matching λ_flow=0.8
    Standard hyper-parameters selected for the 1.5 B Show-o2 backbone.
  • Multi-stage learning rates, step counts, and partial LLM unfreeze schedule
    Six-stage curriculum (CMA-VAE → LRA → T2M warmup → motion-text → vision extension → full FT) is carefully engineered; order and lengths affect final numbers.
  • CFG scale s=3.0 and Euler steps N=50
    Inference hyper-parameters that influence reported generation quality.
assumptions (4)
  • domain assumption Continuous VAE latents preserve temporal continuity and structural fidelity better than VQ-VAE codebooks for motion sequences.
    Stated as core design philosophy (Sec. 1, 3.1); supported by reconstruction ablations but treated as given for the architecture.
  • ad hoc to paper Reverse KL (student motion posterior || teacher vision-fused posterior) yields mode-seeking, high-confidence motion representations that filter view-specific noise.
    Justified by knowledge-distillation intuition (Sec. 3.2); the opposite KL direction is argued against but not exhaustively ablated.
  • domain assumption Text captions alone are too sparse to calibrate a newly introduced continuous motion pathway (the “cold-start” problem).
    Motivates LRA (Sec. 3.4); ablation shows drop without LRA, yet the claim is taken as premise for the whole pre-training stage.
  • domain assumption Flow matching with AdaLN-conditioned heads and hybrid (causal + intra-motion full) attention is a suitable generative objective for both motion and image latents inside an autoregressive LLM.
    Inherited from Show-o2 and flow-matching literature; not re-derived.
invented entities (3)
  • Cross-Modal Aligned Motion VAE (CMA-VAE) with Dual-Posterior KL Alignment
    purpose: Encode variable-length motion into continuous latents that absorb visual-semantic supervision at training time without requiring images at inference.
    New architectural component; independent evidence is only the paper’s own reconstruction and downstream ablations.
  • Latent Reconstruction Alignment (LRA / M2M pre-training)
    purpose: Self-supervised co-calibration of embedder, backbone LoRA, and flow head using dense motion latents as unambiguous conditions.
    New training stage introduced to solve the claimed cold-start problem; value shown only by ablation within this paper.
  • Symmetric dual-path (semantic + generation) motion embedder with pose-aware vision backbone fusion
    purpose: Provide parallel continuous pathways for motion and RGB that decouple high-level semantics from fine kinematic detail.
    Architectural invention mirroring SigLIP/PatchEmbed; necessity demonstrated by branch-ablation table.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation." pith.science (2026). https://pith.science/paper/HMUZ6LX5

@misc{pith2026260322282,
  author       = {Pith},
  title        = {Pith review of: UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HMUZ6LX5}},
  note         = {Machine review of arXiv:2603.22282}
}
read the original abstract

We present UniMotion, to our knowledge the first unified framework for simultaneous understanding and generation of human motion, natural language, and RGB images within a single architecture. Existing unified models handle only restricted modality subsets (e.g., Motion-Text or static Pose-Image) and predominantly rely on discrete tokenization, which introduces quantization errors and disrupts temporal continuity. UniMotion overcomes both limitations through a core principle: treating motion as a first-class continuous modality on equal footing with RGB. A novel Cross-Modal Aligned Motion VAE (CMA-VAE) and symmetric dual-path embedders construct parallel continuous pathways for Motion and RGB within a shared LLM backbone. To inject visual-semantic priors into motion representations without requiring images at inference, we propose Dual-Posterior KL Alignment (DPA), which distills a vision-fused encoder's richer posterior into the motion-only encoder. To address the cold-start problem -- where text supervision alone is too sparse to calibrate the newly introduced motion pathway -- we further propose Latent Reconstruction Alignment (LRA), a self-supervised pre-training strategy that uses dense motion latents as unambiguous conditions to co-calibrate the embedder, backbone, and flow head, establishing a stable motion-aware foundation for all downstream tasks. UniMotion achieves state-of-the-art performance across seven tasks spanning any-to-any understanding, generation, and editing among the three modalities, with especially strong advantages on cross-modal compositional tasks.

Figures

Figures reproduced from arXiv: 2603.22282 by the authors.

Figure 1
Figure 1. Left: Overview and performance comparison of UniMotion, a unified framework for any-to-any Motion, Text, and Vision under￾standing, generation, and editing. UniMotion is the first model to support all seven tri-modal tasks and achieves consistent superiority over existing methods. Right: Representative task demonstrations. Abstract. We present UniMotion, to our knowledge the first unified framework for simultaneous … view at source ↗
Figure 2
Figure 2. Overview of UniMotion. (Left) UniMotion unifies motion, text, and RGB through symmetric continuous pathways: motion and images are encoded into continuous latents (via CMA-VAE and a vision VAE), mapped by a dual-path embedder that separates semantic abstraction from detail-preserving generation, and processed by a shared backbone for both multimodal understanding and modality-specific flow-based synthesis. (Right) L… view at source ↗
Figure 3
Figure 3. CMA-VAE with DPA. CMA-VAE learns a continuous motion latent space using a motion-only encoder for inference and a vision-fused encoder for training-time visual supervision. When paired images are available, motion-guided visual features are fused with motion and distilled via DPA, enabling the shared decoder to learn visually informed motion latents without requiring images at inference. CMA-VAE encodes variable￾len… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on T2M and M2T. Text-to-Motion Gen￾eration [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison of text-driven motion generation on HumanML3D [12]. Each column corresponds to one text prompt, and rows show outputs from Mo￾tionGPT [17], MoMask [11], and UniMotion. Red text highlights the key prompt constraints, while red dashed boxes mark pr…
Figure 6
Figure 6. Figure 6: Qualitative comparison of motion captioning (Motion-to-Text). Each column shows the input motion, the ground-truth caption, the MotionGPT prediction, and the UniMotion prediction. Red phrases in the ground-truth and UniMotion captions highlight the key motion semantics…
Figure 7
Figure 7. Figure 7: Qualitative visualization of motion prediction. Each column shows the observed input prefix (top) and the predicted continuation overlaid on the input motion (bottom), where the input segment is shown in yellow and the forecasted future is shown in purple. UniMotion ex…
Figure 8
Figure 8. Figure 8: presents qualitative results on MotionFix-style editing instructions. UniMotion performs targeted motion modification while retaining the unedited content of the source sequence. The edits are more localized and semantically precise, particularly for instructions invol…
Figure 9
Figure 9. Figure 9: Qualitative Vision-to-Motion results. For each example, we show a sequence of sampled visual inputs together with the corresponding motion outputs recovered by UniMotion. The predicted body structure remains consistent across observations, with accurate limb orientatio…
Figure 10
Figure 10. Figure 10: Qualitative Vision-to-Text results on H3.6M. Each example shows the visual input together with the ground-truth description and the caption generated by UniMo￾tion. Red phrases highlight the key pose-aware semantics shared by the reference and the prediction. UniMotio…
Figure 11
Figure 11. Figure 11: Qualitative Vision-to-Text results on MoVid (video inputs). Each example shows multiple sampled video frames followed by the description generated by UniMotion, with red phrases highlighting the key temporal action components. Given sparse frame observations, UniMotio…
Figure 12
Figure 12. Figure 12: Cumulative distribution of per-sequence reconstruction MPJPE for CMA-VAE, VQ-VAE, and MLD-VAE. Left: zoomed view (0–100 mm) with reference thresholds at 40, 60, and 80 mm. Right: full-range view (0–3000 mm), where the green dashed box indicates the region enlarged in …
Figure 13
Figure 13. Figure 13: Joint-wise spatial error heatmap (dataset average, mm). Each column shows one model, and rows correspond to the 22 HumanML3D joints, with the numerical error annotated in each cell. Color encodes error magnitude (blue = low, red = high), and end￾effector joints (⋆) ar…
Figure 14
Figure 14. Figure 14: Residual frequency spectrum analysis. The main plot shows the mean residual spectral energy (log scale) versus frequency (Hz), with background shading denoting low (0–2 Hz), mid (2–6 Hz), and high (6–10 Hz) bands. The inset provides a magnified view of the high-freque…
Figure 15
Figure 15. Figure 15: Acceleration-based jitter analysis on a high-dynamic sample (M009968). Top: right wrist acceleration (mm/frame2 ). Bottom: right ankle acceleration. Each panel overlays GT (gray), CMA-VAE (blue), MLD-VAE (orange), and VQ-VAE (red), and the inset box reports the accele…
Figure 16
Figure 16. Figure 16: Multi-stage training pipeline of UniMotion. (a) Stage 0 – LRA Pre-training: the LLM is frozen; motion components are trained via M2M self-reconstruction with information bottlenecks to calibrate the motion pathway. (b) Stage 1a – Motion-Text Warmup: basic T2M generati…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 12 linked inside Pith

  1. [1]

    In: SIGGRAPH Asia 2024 Conference Papers

    Athanasiou, N., Cseke, A., Diomataris, M., Black, M.J., Varol, G.: Motionfix: Text- driven 3d human motion editing. In: SIGGRAPH Asia 2024 Conference Papers. pp. 1–11 (2024)

    2024

  2. [2]

    arXiv (2023)

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv (2023)

    2023

  3. [3]

    Advances in neural information processing systems33, 1877–1901 (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020)

    1901

  4. [4]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2019)

    Cao, Z., Hidalgo Martinez, G., Simon, T., Wei, S., Sheikh, Y.A.: Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019)

    2019

  5. [5]

    arXiv preprint arXiv:2405.20340 (2024)

    Chen, L.H., Lu, S., Zeng, A., Zhang, H., Wang, B., Zhang, R., Zhang, L.: Motionllm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.20340 (2024)

    arXiv 2024

  6. [6]

    arXiv preprint arXiv:2501.17811 (2025)

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 (2025)

    arXiv 2025

  7. [7]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Dwivedi, S.K., Sun, Y., Patel, P., Feng, Y., Black, M.J.: Tokenhmr: Advancing human mesh recovery with a tokenized pose representation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1323–1333 (2024)

    2024

  8. [8]

    In: CVPR

    Feng, Y., Lin, J., Dwivedi, S.K., Sun, Y., Patel, P., Black, M.J.: Chatpose: Chatting about 3d human pose. In: CVPR. pp. 2093–2103 (2024)

Show all 53 references
  1. [9]

    In: CVPR

    Goel, S., Pavlakos, G., Rajasegaran, J., Kanazawa, A., Malik, J.: Humans in 4d: Reconstructing and tracking humans with transformers. In: CVPR. pp. 14783–14794 (2023)

    2023

  2. [10]

    arXiv preprint arXiv:1706.02677 (2017)

    Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He, K.: Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017)

    2017 arXiv

  3. [11]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Guo, C., Mu, Y., Javed, M.G., Wang, S., Cheng, L.: Momask: Generative masked modeling of 3d human motions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1900–1910 (2024)

    1900

  4. [12]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., Cheng, L.: Generating diverse and natural 3d human motions from text. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5152–5161 (June 2022)

    2022

  5. [13]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., Cheng, L.: Generating diverse and natural 3d human motions from text. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5152–5161 (2022)

    2022

  6. [14]

    In: ECCV (2022)

    Guo, C., Zuo, X., Wang, S., Cheng, L.: Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In: ECCV (2022)

    2022

  7. [15]

    arXiv preprint arXiv:2511.01463 (2025)

    Hu, L., Ye, Y., Xia, S.: Hmvlm: Human motion-vision-lanuage model via moe lora. arXiv preprint arXiv:2511.01463 (2025)

    2025

  8. [16]

    6m: Large scale datasets and predictive methods for 3d human sensing in natural environments

    Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI36(7), 1325–1339 (2013)

    2013

  9. [17]

    Advances in Neural Information Processing Systems36, 20067–20079 (2023) 16 Z

    Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., Chen, T.: Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems36, 20067–20079 (2023) 16 Z. Wang et al

    2023

  10. [18]

    In: CVPR

    Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: CVPR. pp. 7122–7131 (2018)

    2018

  11. [19]

    arXiv preprint arXiv:1312.6114 (2013)

    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

    2013 arXiv

  12. [20]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Li, P., Wang, Z., Yuan, Y., Liu, H., Meng, X., Yuan, J., Liu, M.: Ust-ssm: Unified spatio-temporal state space models for point cloud video modeling. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6738–6747 (2025)

    2025

  13. [21]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Li, Y., Hou, R., Chang, H., Shan, S., Chen, X.: Unipose: A unified multimodal framework for human pose comprehension, generation and editing. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 27805–27815 (2025)

    2025

  14. [22]

    arXiv preprint arXiv:2410.07093 (2024)

    Li, Z., Yuan, W., He, Y., Qiu, L., Zhu, S., Gu, X., Shen, W., Dong, Y., Dong, Z., Yang, L.T.: Lamp: Language-motion pretraining for motion generation, retrieval, and captioning. arXiv preprint arXiv:2410.07093 (2024)

    2024 arXiv

  15. [23]

    In: The Eleventh International Conference on Learning Representations (2023),https://openreview.net/forum?id=PqvMRDCJT9t

    Lipman, Y., Chen, R.T.Q., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. In: The Eleventh International Conference on Learning Representations (2023),https://openreview.net/forum?id=PqvMRDCJT9t

    2023

  16. [24]

    In: NeurIPS (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: NeurIPS (2023)

    2023

  17. [25]

    arXiv preprint arXiv:2309.01372 (2023)

    Lou, Y., Zhu, L., Wang, Y., Wang, X., Yang, Y.: Diversemotion: Towards diverse human motion generation via discrete diffusion. arXiv preprint arXiv:2309.01372 (2023)

    2023 arXiv

  18. [26]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5693–5703 (2019)

    2019

  19. [27]

    arXiv preprint arXiv:2405.09818 (2024)

    Team, C.: Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 (2024)

    2024 arXiv

  20. [28]

    arXiv preprint arXiv:2209.14916 (2022)

    Tevet, G., Raab, S., Gordon, B., Shafir, Y., Bermano, A.H., Cohen-Or, D.: Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022)

    2022 arXiv

  21. [29]

    Advances in neural information processing systems30(2017)

    Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. Advances in neural information processing systems30(2017)

    2017

  22. [30]

    In: Proceedings of the European conference on computer vision (ECCV)

    Von Marcard, T., Henschel, R., Black, M.J., Rosenhahn, B., Pons-Moll, G.: Recov- ering accurate 3d human pose in the wild using imus and a moving camera. In: Proceedings of the European conference on computer vision (ECCV). pp. 601–617 (2018)

    2018

  23. [31]

    arXiv preprint arXiv:2503.20314 (2025)

    Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.W., Chen, D., Yu, F., Zhao, H., Yang, J., et al.: Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 (2025)

    2025 arXiv

  24. [32]

    In: ICCV

    Wang, W., Ge, Y., Mei, H., Cai, Z., Sun, Q., Wang, Y., Shen, C., Yang, L., Komura, T.: Zolly: Zoom focal length correctly for perspective-distorted human mesh reconstruction. In: ICCV. pp. 3925–3935 (2023)

    2023

  25. [33]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, Z., Li, P., Liu, H., Deng, Z., Wang, C., Liu, J., Yuan, J., Liu, M.: Recognizing actions from robotic view for natural human-robot interaction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14218–14227 (2025)

    2025

  26. [34]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Wu, B., Xie, J., Shen, K., Kong, Z., Ren, J., Bai, R., Qu, R., Shen, L.: Mg-motionllm: A unified framework for motion comprehension and generation across multiple granularities. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 27849–27858 (2025) U...

    2025

  27. [35]

    arXiv preprint arXiv:2410.13848 (2024)

    Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al.: Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848 (2024)

    2024 arXiv

  28. [36]

    In: Forty-first International Conference on Machine Learning (2024)

    Wu, S., Fei, H., Qu, L., Ji, W., Chua, T.S.: Next-gpt: Any-to-any multimodal llm. In: Forty-first International Conference on Machine Learning (2024)

    2024

  29. [37]

    arXiv (2024)

    Xie, J., Mao, W., Bai, Z., Zhang, D.J., Wang, W., Lin, K.Q., Gu, Y., Chen, Z., Yang, Z., Shou, M.Z.: Show-o: One single transformer to unify multimodal understanding and generation. arXiv (2024)

    2024

  30. [38]

    arXiv preprint arXiv:2506.15564 (2025)

    Xie, J., Yang, Z., Shou, M.Z.: Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506.15564 (2025)

    2025 arXiv

  31. [39]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2023)

    Xin, C., Jiang, B., Liu, W., Huang, Z., Fu, B., Chen, T., Yu, J., Yu, G.: Execut- ing your commands via motion diffusion in latent space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2023)

    2023

  32. [40]

    TPAMI (2023)

    Xu, X., Liu, L., Yan, S.: Smpler: Taming transformers for monocular 3d human shape and pose estimation. TPAMI (2023)

    2023

  33. [41]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre- training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975–11986 (2023)

    2023

  34. [42]

    In: ICCV (2021)

    Zhang, H., Tian, Y., Zhou, X., Ouyang, W., Liu, Y., Wang, L., Sun, Z.: Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In: ICCV (2021)

    2021

  35. [43]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)

    Zhang, J., Zhang, Y., Cun, X., Huang, S., Zhang, Y., Zhao, H., Lu, H., Shen, X.: T2m-gpt: Generating human motion from textual descriptions with discrete representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)

    2023

  36. [44]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3836–3847 (2023)

    2023

  37. [45]

    IEEE transactions on pattern analysis and machine intelligence46(6), 4115–4128 (2024)

    Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., Liu, Z.: Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE transactions on pattern analysis and machine intelligence46(6), 4115–4128 (2024)

    2024

  38. [46]

    with his hands out

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., Levy, O.: Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039 (2024) 18 Z. Wang et al. In this supplementary ma...

    2024 arXiv

  39. [47]

    Root motion increment: 3-dim (1-dim rotation increment + 2-dim planar translation increment)

  40. [48]

    Relative joint positions: 63-dim (21joints×3)

  41. [49]

    Local joint rotations (continuous 6D form): 126-dim (21joints×6)

  42. [50]

    Local joint velocities: 66-dim (22joints×3)

  43. [51]

    7.Global orientation (continuous 6D form): 6-dim

    Foot contact states: 4-dim. 7.Global orientation (continuous 6D form): 6-dim. The first six components sum to 263-dim, consistent with mainstream motion generation representations. The newly added 6-dim component maintains global orientation consistency in visual-motion tasks....

  44. [52]

    Dataset-providedimage-spaceannotations( camera_params[‘joint3d_image’]): pixel-space joint coordinates from dataset annotations, used when precom- puted files are unavailable

  45. [53]

    Describe the motion

    Runtime forward-kinematics recovery(fallback): 3D joint positions are recovered from the 269-dim motion representation viarecover_from_ric and projected using weak-perspective camera parameters. All three paths share the key property that the grid-sampling positions are struct...

    2000

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.