Pith. sign in

REVIEW 5 major objections 6 minor 45 references

Human Motion Instruction Tuning

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that keeping human motion in its native continuous form, rather than converting it to language tokens, lets large language models understand complex human behavior better, and it supports this with a new framework called…

desk verdict Plausible architecture, unproven attribution: LLaMo's native-motion design shows consistent gains but the experiments don't isolate the cause. read the letter →

arxiv 2411.16805 v4 pith:EHHV4BHT submitted 2024-11-25 cs.AI cs.CV

Lei Li , Sen Jia , Jianhao Wang , Zhongyu Jiang , Feng Zhou , Ju Dai , Tianfang Zhang , Zongkai Wu
show 1 more author
Jenq-Neng Hwang
This is my paper · ORCID
classification cs.AIcs.CV
keywords humanmotioninstructiontuningmultimodallargelanguagemodelunderstandingtext-guidedframeselectionmotion-videofusioncontinuousmodalitypreservationsportsanalyticsBABEL-QA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLaMo, a framework for human motion instruction tuning, is built on the claim that encoding motion as tokens or translating it into text discards fine-grained spatial-temporal detail needed for understanding complex behavior. Instead of translation, LLaMo feeds raw motion (and, when needed, motion estimated from video) into a large language model, using a Cross Talker module to select the motion frames most relevant to the query text and fuse them with it. The paper reports state-of-the-art results on MoVid-Bench-Motion, BABEL-QA, and a professional baseball/golf swing dataset, arguing that keeping motion native makes the model better at body-part awareness, reasoning, and fine-grained detail such as repetition counts. If correct, this would establish a design principle for multimodal assistants in motion-intensive domains like sports analytics and behavioral monitoring.

What carries the argument

The load-bearing component is the Cross Talker module, a text-guided attention mechanism that performs language-guided frame selection, adaptive contextual aggregation, and bidirectional motion–text fusion. Given enhanced motion features and text embeddings, it computes a cross-attention matrix, max-pools it over text tokens to score each motion frame, picks the top $K$ frames, enriches each selected frame with local window attention and segment-level global attention, and then applies cross-attention in both directions before concatenating the features. This mechanism carries the paper's central claim: it keeps motion native while making it computationally efficient and semantically aligned with the query.

What would settle it

A controlled experiment that trains LLaMo and a tokenized equivalent (e.g., MotionGPT-style) on the exact same data with the same backbone and size, then compares them on MoVid-Bench-Motion or BABEL-QA; if the tokenized variant matches or beats LLaMo, the native-motion claim is refuted. A simpler version is ablating the Cross Talker module and seeing whether performance collapses.

Watch

Extended reading notes

Core claim

The central discovery is that a large language model can process human motion as a continuous, independent modality without any tokenization or textification step, and that this preserves information that discrete encodings lose. LLaMo does this with three components: a motion estimator and enhancer that can derive motion features from video and enrich them with video context, a Cross Talker module that selects $K$ 'viewpoint frames' from the motion sequence by max-pooled cross-attention with text embeddings and then aggregates local and global context around each, and a behavior generation module that produces text from the fused motion-text representation. Evaluations show LLaMo exceeding GPT-3.5, MotionGPT, and MotionLLM on most motion-understanding metrics, scoring 0.458 overall on BABEL-QA (versus 0.436 for MotionLLM), and outperforming baselines on the professional swing dataset. The paper attributes these gains specifically to native motion processing and to text-guided frame selection, which also cuts the LLM self-attention cost from $O((L_T + T)^2)$ to $O((L_T + K)^2)$.

Load-bearing premise

The empirical comparisons assume the performance gap is caused by LLaMo's native-motion design rather than by its larger or different training set, since LLaMo trains on additional custom data while several baselines were trained without any added data.

Editorial extensions

If this is right

  • Instruction-tuned language models can accept continuous non-linguistic modalities like motion without a tokenizer, preserving detail that discrete codebooks lose.
  • Video-only inputs can be handled by estimating motion first, so the same model spans both motion-capture data and ordinary video.
  • Text-guided frame selection makes long motion sequences tractable for transformer LLMs by reducing per-layer complexity from quadratic in total sequence length to quadratic in text length plus a small number of selected frames.
  • The approach generalizes to specialized domains such as professional sports coaching, where expert-level motion nuance matters, and to detail-oriented tasks such as repetition counting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same native-continuous principle likely extends to other high-rate non-linguistic modalities—audio, physiological signals, inertial sensors—where quantization to text is lossy; a testable extension is applying Cross Talker-style selection to those streams.
  • Because the reported comparisons vary in training data and backbone, the paper does not isolate whether the gains come from native motion or from training on the custom Swing set; a controlled same-data ablation separating these factors would either confirm or undercut the stated attribution.
  • The Cross Talker's frame-selection step is a generic attention-distillation idea that could be reused for long video or point-cloud sequences in any LLM-based multimodal model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript presents LLaMo, a large language and human motion assistant that processes raw motion sequences and video without converting motion into language tokens. The framework includes a motion estimator, motion and video encoders, a feature enhancement module that distills video information into motion features, a Cross Talker module that performs text-guided frame selection and bidirectional cross-modal fusion, and a behavior generation module. Training uses MoVid, a custom Swing dataset, HumanML3D, KIT-ML, and Mo-RepCount. Evaluations on MoVid-Bench, BABEL-QA, the Swing set, and Mo-RepCount report improvements over MotionGPT and MotionLLM, leading the authors to claim that preserving native motion representation yields state-of-the-art motion understanding.

Significance. The idea of keeping motion as a continuous modality rather than tokenizing or translating it into language is timely and plausible; if substantiated, LLaMo would provide a useful design point for human-centric multimodal LLMs. The paper also addresses efficiency through text-guided frame selection, a practical concern. However, the current evidence does not establish the central attribution claim: comparisons are confounded by training data, architecture, and backbone differences, and there are no ablations or error bars. The manuscript is therefore a promising proposal rather than a validated system. On the positive side, the paper is clearly written and the proposed pipeline is well-motivated, with the main open questions being experimental rather than conceptual.

major comments (5)
  1. [Section 4.1–4.2, Tables 1–2] The claim in Section 4.2 that LLaMo's performance is 'primarily attributed to its preservation of critical motion-specific nuances' is not supported by the comparisons shown. LLaMo is trained on MoVid, the custom Swing set, HumanML3D, KIT-ML, and Mo-RepCount, while the MotionGPT and MotionLLM baselines are previously published results that were not retrained on the same mixture. A same-data, same-backbone comparison that varies only the motion representation (native versus tokenized or translated) is necessary to justify the attribution; without it, the observed gap could be due to data scale, the video feature enhancer, the Cross Talker, or the LLM backbone.
  2. [Sections 3.1–3.2] No ablation isolates the proposed modules. The reader cannot tell whether Cross Talker's text-guided frame selection, the receptive-field regression, the adaptive aggregation, or the feature enhancer contributes to the results. For example, comparing against a variant that feeds all motion frames without selection, or one that removes the enhancer, would identify which component matters. Since these components are described as core contributions and Section 4.2 attributes performance to them, this omission is load-bearing.
  3. [Section 4.2, Table 3] The Swing evaluation rests on an author-created dataset and GPT-4 scoring, but the dataset construction, exact questions, and scoring protocol are deferred to a missing appendix. The manuscript reports LLaMo's accuracy relative to 'GT' (100/5) without providing any measure of agreement with professional coaches or ablating the evaluation prompt. This makes the 24.8 overall accuracy difficult to interpret and hard to reproduce.
  4. [Section 4.2, Table 4] The Mo-RepCount comparison is not controlled. The text states that all state-of-the-art models were trained 'without using any additional data,' while LLaMo was trained on the same Mo-RepCount plus MoVid, HumanML3D, KIT-ML, and 20,000 Swing videos. Additionally, reference [44] 'RepNet' cites a 3D human pose estimation paper rather than the repetition-counting RepNet used in the table, so it is unclear which baseline was actually evaluated. The comparison should be repeated with all methods trained on identical data, and the correct baseline should be cited.
  5. [Tables 1–4] No error bars, confidence intervals, or significance tests are reported for any benchmark. Several gaps are small (e.g., BABEL-QA overall 0.458 versus 0.436 for MotionLLM, and some MoVid-Bench score differences are below 0.1). Without variance estimates, the claimed state-of-the-art results are not statistically supported.
minor comments (6)
  1. [Section 3.1, first paragraph] The sentence 'which makes LLaMo is a general human motion assistant' should read 'which makes LLaMo a general human motion assistant.'
  2. [Table 1, bottom half] The column header 'Hull.' appears to be a typo for 'Hall.' (the text refers to hallucination).
  3. [Section 4.2, Professional Sports Analysis] The sentence 'Specifically, LLaMo achieves an overall accuracy of 24.80' is missing a period before 'The superior performance...'.
  4. [Equation (2)] The max-pooling aggregation of attention weights should specify how ties are broken and whether the selection is deterministic under re-runs.
  5. [Tables 2 and 3] Several citations are inconsistent, e.g., MotionCLIP-M and MotionCLIP-R are cited as [41] (OpenAI GPT-3.5), and reference [44] does not match the RepNet baseline; these need to be corrected.
  6. [Section 3.1] The description of the feature enhancement module references an Appendix for detailed design, but no appendix is present; the relevant equations and hyperparameters (K, N, S_n, r_k) should be in the main text or a provided appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical comparisons, not derived predictions.

full rationale

LLaMo is an empirical systems paper. The method section defines an architecture (multimodal encoders, a Cross Talker module, and a behavior generation module) and a standard negative-log-likelihood training objective (Eq. 11); the results section reports benchmark accuracy and score tables. There is no claimed derivation chain in which an output quantity is constructed from its inputs by definition, so the core self-definitional and fitted-input-as-prediction patterns do not arise. The central claim—that keeping motion in its native form improves instruction-tuned motion understanding—is tested against external baselines such as MotionGPT, MotionLLM, GPT-3.5, and Video-LLaVA on external benchmarks such as MoVid-Bench and BABEL-QA. Those comparisons provide independent empirical content. The self-citations in the reference list (e.g., refs. 7, 10, 23, 27, 29, 36) are incidental related-work citations and are not load-bearing. Some validity concerns exist but are not circularity: the Swing benchmark is author-constructed and GPT-4-scored, Mo-RepCount is used both as a training source and as an evaluation set without an explicit held-out split in the text, and no same-data, same-backbone ablation isolates the native-motion representation. These are confounding, reproducibility, or evaluation-fairness issues, not reductions of the claimed result to its own inputs. Accordingly, no circular step is identified and the score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities are postulated. The Motion Enhancer, Cross Talker, and viewpoint frames are neural components, not independently verifiable entities. The model's load-bearing assumptions are empirical data and evaluation assumptions rather than new physics or formal axioms.

free parameters (4)
  • K (number of viewpoint frames) = not stated
    Cross Talker selects top K motion frames; K is a hyperparameter controlling sequence length, not derived from first principles (Section 3.2).
  • N (number of motion segments) = not stated
    Global context uses N segments via average pooling; N is chosen by hand (Section 3.2).
  • S_n (segment size) = not stated
    Segment size for global pooling is a hyperparameter, not justified from data (Section 3.2).
  • r_k (receptive field size) = predicted by learned module
    A per-frame receptive field is estimated by a regression module with sigmoid activation, introducing learned parameters that set local window sizes (Section 3.2).
assumptions (4)
  • domain assumption Motion data (skeleton sequences) is an appearance-invariant representation sufficient for behavior understanding.
    Invoked in Section 1 to justify using motion as the core modality.
  • domain assumption The motion estimator recovers accurate motion from video, so video-only inputs benefit from native motion.
    Section 3.1 states the estimator addresses missing motion data; video results in Section 4.2 depend on this.
  • ad hoc to paper GPT-4 scoring on the custom Swing dataset reflects professional coaching quality.
    Section 4.1 uses GPT-4 evaluations from 0 to 5 without human agreement metrics; authors created the Swing dataset and its expert Q&A.
  • domain assumption Mo-RepCount, a filtered subset of RepCount, is a fair high-quality test for repetition counting.
    Section 4.1 defines Mo-RepCount as a filtered subset; no filtering details are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human Motion Instruction Tuning." pith.science (2026). https://pith.science/paper/EHHV4BHT

@misc{pith2026241116805,
  author       = {Pith},
  title        = {Pith review of: Human Motion Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EHHV4BHT}},
  note         = {Machine review of arXiv:2411.16805}
}
read the original abstract

This paper presents LLaMo (Large Language and Human Motion Assistant), a multimodal framework for human motion instruction tuning. In contrast to conventional instruction-tuning approaches that convert non-linguistic inputs, such as video or motion sequences, into language tokens, LLaMo retains motion in its native form for instruction tuning. This method preserves motion-specific details that are often diminished in tokenization, thereby improving the model's ability to interpret complex human behaviors. By processing both video and motion data alongside textual inputs, LLaMo enables a flexible, human-centric analysis. Experimental evaluations across high-complexity domains, including human behaviors and professional activities, indicate that LLaMo effectively captures domain-specific knowledge, enhancing comprehension and prediction in motion-intensive scenarios. We hope LLaMo offers a foundation for future multimodal AI systems with broad applications, from sports analytics to behavioral prediction. Our code and models are available on the project website: https://github.com/ILGLJ/LLaMo.

Figures

Figures reproduced from arXiv: 2411.16805 by the authors.

Figure 1
Figure 1. A comparison of MotionLLM [1], MotionGPT [2], and LLaMo highlights LLaMo’s motion-specific capabilities. Equipped with a Motion Enhancer and Cross Talker module to align motion and text, LLaMo supports both video and motion in￾puts, enabling text-aware, fine-grained motion analysis. applications continue to expand, developing models that can leverage the unique characteristics of motion data becomes increasingly cru… view at source ↗
Figure 2
Figure 2. Overview of the LLaMo framework. It includes three main modules: (1) Multimodal Feature Extraction for encoding video [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of the Cross Talker Module, which selects key [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example outputs from LLaMo across human activities and professional sports, showcasing its reasoning capabilities and domain [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 33 canonical work pages

  1. [41]

    Gpt-3.5: Generative pre-trained transformer 3.5,

    OpenAI. Gpt-3.5: Generative pre-trained transformer 3.5,

    work page

  2. [1]

    Motionllm: Understanding human behaviors from human motions and videos

    Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.20340, 2024. 1, 2, 6, 7, 8

    arXiv 2024

  3. [44]

    Repnet: Weakly super- vised training of an adversarial reprojection network for 3d human pose estimation

    Bastian Wandt and Bodo Rosenhahn. Repnet: Weakly super- vised training of an adversarial reprojection network for 3d human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7782–7791, 2019. 8

    work page 2019

  4. [2]

    Motiongpt: Human motion as a foreign language

    Ling-Hao Chen, Jiawei Zhang, Wen Liu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. In Proceedings of the Conference on Neural Information Pro- cessing Systems (NeurIPS), 2024. 1, 2, 6, 7, 8

    work page 2024

  5. [3]

    The kit motion-language dataset

    Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 4(4):236–252,

  6. [4]

    We are more than our joints: Predicting how 3d bodies move

    Yan Zhang, Michael J Black, and Siyu Tang. We are more than our joints: Predicting how 3d bodies move. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    work page 2021

  7. [5]

    Versatile multi-modal pre-training for human-centric percep- tion

    Fangzhou Hong, Liang Pan, Zhongang Cai, and Ziwei Liu. Versatile multi-modal pre-training for human-centric percep- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 16156– 16166, 2022. 2

    work page 2022

  8. [6]

    Llms are good action recognizers

    Haoxuan Qu, Yujun Cai, and Jun Liu. Llms are good action recognizers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18395– 18406, 2024. 1

    work page 2024

Show all 45 references
  1. [7]

    Dense point clouds mat- ter: Dust-gs for scene reconstruction from sparse viewpoints

    Shan Chen, Jiale Zhou, and Lei Li. Dense point clouds mat- ter: Dust-gs for scene reconstruction from sparse viewpoints. arXiv preprint arXiv:2409.08613, 2024. 1

    2024 arXiv

  2. [8]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Ger- ard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 1, 2

    2019

  3. [9]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 2

    2015

  4. [10]

    Chatmotion: A multimodal multi-agent for human motion analysis

    Li Lei, Jia Sen, Wang Jianhao, An Zhaochong, Li Jiaang, Hwang Jenq-Neng, and Belongie Serge. Chatmotion: A multimodal multi-agent for human motion analysis. arXiv preprint arXiv:2502.18180, 2025. 1

    2025 arXiv

  5. [11]

    Motion question answering via modular motion programs

    Mark Endo, Joy Hsu, Jiaman Li, and Jiajun Wu. Motion question answering via modular motion programs. In Pro- ceedings of the International Conference on Machine Learn- ing (ICML), 2023. 1, 2

    2023

  6. [12]

    Motiongpt: Human motion as a foreign lan- guage

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign lan- guage. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2023. 1

    2023

  7. [13]

    Learning video-text aligned repre- sentations for video captioning

    Yaya Shi, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, and Zheng-Jun Zha. Learning video-text aligned repre- sentations for video captioning. IEEE Transactions on Mul- timedia (TMM), 2023. 2

    2023

  8. [14]

    Tm2t: Stochastic and tokenized modeling for the reciprocal gener- ation of 3d human motions and texts

    Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal gener- ation of 3d human motions and texts. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 580–597, 2022. 2

    2022

  9. [15]

    Mvbench: A com- prehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, et al. Mvbench: A com- prehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2024. 2

    2024

  10. [16]

    Multimodality helps few-shot 3d point cloud semantic seg- mentation

    Zhaochong An, Guolei Sun, Yun Liu, Runjia Li, Min Wu, Ming-Ming Cheng, Ender Konukoglu, and Serge Belongie. Multimodality helps few-shot 3d point cloud semantic seg- mentation. arXiv preprint arXiv:2410.22489, 2024. 2

    2024 arXiv

  11. [17]

    The process of motion capture: Dealing with the data

    Bobby Bodenheimer, Chuck Rose, Seth Rosenthal, and John Pella. The process of motion capture: Dealing with the data. In Proceedings of the Eurographics Workshop on Com- puter Animation and Simulation (EG Workshop) , pages 3– 18, 1997. 2

    1997

  12. [18]

    Text-driven human motion generation with diffusion models

    Xiangyu Peng, Yuxuan Zhang, Xiaolong Wang, and Ziwei Liu. Text-driven human motion generation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  13. [19]

    Motion question answering via modular motion programs

    Mark Endo, Joy Hsu, Jiaman Li, and Jiajun Wu. Motion question answering via modular motion programs. In In- ternational Conference on Machine Learning , pages 9312–

  14. [20]

    Videollm: Lan- guage models for video understanding

    Jie Lin, Wei Zhang, and Zhe Chen. Videollm: Lan- guage models for video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2405–2415, 2023. 2

    2023

  15. [21]

    Videobench: A bench- mark for large-scale video understanding

    Xu Ning, Hongwei Li, and Yu Zhao. Videobench: A bench- mark for large-scale video understanding. arXiv preprint arXiv:2304.12345, 2023. 2

    2023 arXiv

  16. [22]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2, 6

    2023 arXiv

  17. [23]

    Segment any building

    Lei Li. Segment any building. In Computer Graphics Inter- national Conference, pages 155–166. Springer, 2023. 2

    2023

  18. [24]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 2

    2023 arXiv

  19. [25]

    Adaptive multimodal learning for behavior analysis

    Enxin Song, Guanhong Zhang, and Haoyang Zhou. Adaptive multimodal learning for behavior analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

    2023

  20. [26]

    Recognizing human behaviors with skeletal data in llm-based frameworks

    Yunhua Yang, Ziwang Zhao, and Yiming Xie. Recognizing human behaviors with skeletal data in llm-based frameworks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    2023

  21. [27]

    Mask-fpan: Semi-supervised face parsing in the wild with de-occlusion and uv gan

    Lei Li, Tianfang Zhang, Zhongfeng Kang, and Xikun Jiang. Mask-fpan: Semi-supervised face parsing in the wild with de-occlusion and uv gan. Computers & Graphics, 116:185– 193, 2023. 2

    2023

  22. [28]

    Learning a bidirectional mapping between human whole- body motion and natural language using deep recurrent neu- ral networks

    Matthias Plappert, Christian Mandery, and Tamim Asfour. Learning a bidirectional mapping between human whole- body motion and natural language using deep recurrent neu- ral networks. Robotics and Autonomous Systems (RAS), 109: 13–26, 2018. 2

    2018

  23. [29]

    Cpseg: Finer-grained image semantic segmentation via chain-of-thought language prompting

    Lei Li. Cpseg: Finer-grained image semantic segmentation via chain-of-thought language prompting. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 513–522, 2024

    2024

  24. [30]

    Understanding hu- man behaviors from skeletal data: A review of datasets and methods

    Yunhua Yang, Liang Zhang, and Hui Li. Understanding hu- man behaviors from skeletal data: A review of datasets and methods. arXiv preprint arXiv:2310.12998, 2023. 2

    2023 arXiv

  25. [31]

    Two- stream adaptive graph convolutional networks for skeleton- based action recognition

    Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two- stream adaptive graph convolutional networks for skeleton- based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12026–12035, 2019. 2

    2019

  26. [32]

    Fine- grained spatial-temporal motion understanding in complex video environments

    Enxin Song, Wenhao Chai, and Yucheng Zhang. Fine- grained spatial-temporal motion understanding in complex video environments. arXiv preprint arXiv:2310.08639 ,

  27. [33]

    Temporal-aware hierarchical mask classi- fication for video semantic segmentation

    Zhaochong An, Guolei Sun, Zongwei Wu, Hao Tang, and Luc Van Gool. Temporal-aware hierarchical mask classi- fication for video semantic segmentation. arXiv preprint arXiv:2309.08020, 2023. 2

    2023 arXiv

  28. [34]

    Category-agnostic pose estimation for point clouds

    Xianghong Liu, Haoxuan Wang, Zhiwei Zhang, Huan Wu, Baoquan Chen, and Xiaoguang Han. Category-agnostic pose estimation for point clouds. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  29. [35]

    Activitynet++: A large-scale benchmark for video under- standing

    Xiaolong He, Yi Zhang, Chen Chen, and Kyoung Mu Lee. Activitynet++: A large-scale benchmark for video under- standing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  30. [36]

    Hierarchical edge aware learning for 3d point cloud

    Lei Li. Hierarchical edge aware learning for 3d point cloud. In Computer Graphics International Conference, pages 81–

  31. [37]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5152–5161, 2022. 6

    2022

  32. [38]

    Transrac: Encoding multi- scale temporal correlation with transformers for repetitive action counting

    Huazhang Hu, Sixun Dong, Yiqun Zhao, Dongze Lian, Zhengxin Li, and Shenghua Gao. Transrac: Encoding multi- scale temporal correlation with transformers for repetitive action counting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages ...

    2022

  33. [39]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 6, 7

    2023 arXiv

  34. [40]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 137...

    2024

  35. [42]

    Ev- ery shot counts: Using exemplars for repetition counting in videos

    Saptarshi Sinha, Alexandros Stergiou, and Dima Damen. Ev- ery shot counts: Using exemplars for repetition counting in videos. arXiv preprint arXiv:2403.18074, 2024. 8

    2024 arXiv

  36. [43]

    Poserac: Pose saliency transformer for repetitive action counting

    Ziyu Yao, Xuxin Cheng, and Yuexian Zou. Poserac: Pose saliency transformer for repetitive action counting. arXiv preprint arXiv:2303.08450, 2023. 8

    2023 arXiv

  37. [2023]

    https://platform.openai.com/docs/ models/gpt-3-5. 6, 7, 8

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.