pith. sign in

arxiv: 2606.20888 · v1 · pith:BBU6DL2Dnew · submitted 2026-06-18 · 💻 cs.CV

Fine-grained Human Motion Understanding with Language Models

Thomas Markhorst , Zhi-Yi Lin , Jouh Yeong Chew , Jan van Gemert , Xucong Zhang This is my paper

Pith reviewed 2026-06-26 17:53 UTC · model grok-4.3

classification 💻 cs.CV
keywords human motion understandinglanguage modelsskeletal posestemporal encodingpose captioningmotion question answering2D skeletal inputmulti-task training
0
0 comments X

The pith

Explicit timestamps on skeletal poses let language models achieve state-of-the-art fine-grained human motion understanding.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes representing human motion as sequences of skeletal poses each paired with timestamp tokens so that an LLM can track order, duration, and rhythm. It constructs a training mixture covering pose captioning, pose QA, motion captioning, and motion QA to determine what supervision is required for good motion-language reasoning. Experiments on five benchmarks show state-of-the-art results, with the diversity of supervision providing the largest gains and the model succeeding even with only 2D input where it beats previous 3D methods. This matters for building motion understanding systems that work from ordinary video without specialized 3D hardware. The ablations indicate that explicit time encoding and task variety are the key ingredients.

Core claim

Representing motion as a sequence of skeletal poses with explicit timestamps for each pose, combined with training on a diverse mixture of pose- and motion-level captioning and question answering tasks, enables an LLM-based model to achieve state-of-the-art performance on fine-grained human motion understanding benchmarks, supporting both 2D and 3D inputs.

What carries the argument

Motion as timestamped skeletal pose sequences fed to an LLM, with a unified pose encoder and a four-task training mixture of pose captioning, pose QA, motion captioning, and motion QA.

If this is right

  • Explicit timestamp tokens allow reasoning about motion order, duration, and rhythm.
  • Diverse supervision from pose and motion tasks drives most of the performance gains.
  • Staged training adds only a small extra benefit.
  • The unified encoder allows the same model to handle 2D or 3D skeletal data, optionally with video context.
  • The method exceeds prior 3D-based approaches on multiple benchmarks even when restricted to 2D skeletal input.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This representation could enable motion analysis from standard 2D pose detectors in everyday settings like fitness apps or security cameras.
  • It opens the possibility of training effective models with less reliance on expensive 3D motion capture data.
  • The approach might be tested on multi-person or interactive motions to see if timestamps scale to complex scenarios.
  • Integrating this with other LLM capabilities like planning could lead to systems that describe and suggest corrections for observed movements.

Load-bearing premise

The premise that timestamp tokens in the input sequence combined with the constructed training mixture suffice to let the LLM effectively reason about motion timing and details.

What would settle it

An ablation removing the timestamp tokens and measuring the drop in performance on questions involving motion duration or sequence order would directly test the contribution of explicit temporal encoding.

Figures

Figures reproduced from arXiv: 2606.20888 by Jan van Gemert, Jouh Yeong Chew, Thomas Markhorst, Xucong Zhang, Zhi-Yi Lin.

Figure 1
Figure 1. Figure 1: Given a motion sequence (top left), our method answers questions about fine-grained [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pipeline of our FiGMo, comprising (i) pose encoder pre-training, (ii) encoding high [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training data examples. Pose-related datasets are shown in the top row and motion-related [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison of FiGMo with DEMO [ [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative examples showing fine-grained temporal motion reasoning (left), action de [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt template used for GPT-based semantic evaluation on BABEL-QA. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt template used for GPT-based semantic evaluation on ActivityNet-QA. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt templates used to construct PS-Short. Top: generation prompt used by Qwen2.5-VL [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt templates used for creating PS-Fine. Top: fine-grained generation prompt used to [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt templates used for creating PF-Fine. Top: generation prompt used to extract [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt template used to generate FTM-QA. The LLM receives sequence-level and [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt template used to generate HML3D-QA. The LLM is given multiple natural [PITH_FULL_IMAGE:figures/full_fig_p028_12.png] view at source ↗
read the original abstract

In this work, we propose \methodname, an LLM-based model for fine-grained human motion understanding that represents motion as a sequence of skeletal poses with explicit timestamps for each pose. Each pose encodes body joint positions and is temporally grounded with timestamp tokens, allowing the model to reason about motion order, duration, and rhythm. To study what supervision is needed for motion-language reasoning, we construct a diverse training mixture spanning pose captioning, pose question answering, motion captioning, and motion question answering. Our ablations show that the primary gains come from the diversity of pose- and motion-level supervision, while staged training provides a smaller additional benefit. Different from previous works that rely on ground-truth 3D motion capture, our approach supports both 2D and 3D skeletal motion representations through a unified pose encoder, and can optionally incorporate video to provide contextual information. Extensive experiments on BABEL-QA, HuMMan-QA, CompMo, NTU-RGB+D, and QEVD-Coach demonstrate that our method achieves state-of-the-art performance across multiple benchmarks, highlighting the effectiveness of explicit temporal encoding and diverse pose- and motion-level supervision for fine-grained human motion understanding. Notably, even when using only 2D skeletal input, our approach surpasses previous 3D-based methods.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes exttt{\\methodname}, an LLM-based model for fine-grained human motion understanding. Human motion is encoded as a sequence of skeletal poses, each augmented with explicit timestamp tokens, and processed by a unified pose encoder that supports both 2D and 3D inputs (optionally with video context). Training uses a diverse mixture of pose captioning, pose QA, motion captioning, and motion QA tasks. The central empirical claim is that this yields state-of-the-art results on BABEL-QA, HuMMan-QA, CompMo, NTU-RGB+D, and QEVD-Coach, with 2D-only input surpassing prior 3D-based methods; ablations attribute primary gains to supervision diversity and a smaller benefit to staged training.

Significance. If the reported results and ablations hold under scrutiny, the work would establish that explicit timestamp-based temporal grounding plus a broad pose/motion supervision mixture enables LLMs to perform fine-grained motion reasoning at a level competitive with or superior to specialized 3D pipelines. This would be notable for lowering the barrier to high-quality motion understanding (no ground-truth 3D mocap required) and for providing concrete evidence on the value of supervision diversity in motion-language models.

major comments (2)
  1. [Abstract] Abstract: The central claim credits 'explicit temporal encoding' (timestamp tokens) with enabling the LLM to reason about motion order, duration, and rhythm, yet the described ablations only vary supervision diversity and staged training. No direct comparison of the identical architecture with versus without timestamp tokens is reported, leaving open whether the sequence of poses alone suffices for the claimed SOTA gains on BABEL-QA, HuMMan-QA, etc.
  2. [Abstract] Abstract: The manuscript asserts 'state-of-the-art performance across multiple benchmarks' and that 'even when using only 2D skeletal input, our approach surpasses previous 3D-based methods,' but supplies no quantitative metrics, baseline comparisons, error bars, or ablation tables. Without these data the magnitude, statistical reliability, and attribution of gains to the proposed components cannot be assessed.
minor comments (1)
  1. [Abstract] The method name is written as \methodname in the abstract; a concrete name or acronym should be introduced at first use for readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address the two major comments point by point below, providing clarifications and committing to revisions where they strengthen the work.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim credits 'explicit temporal encoding' (timestamp tokens) with enabling the LLM to reason about motion order, duration, and rhythm, yet the described ablations only vary supervision diversity and staged training. No direct comparison of the identical architecture with versus without timestamp tokens is reported, leaving open whether the sequence of poses alone suffices for the claimed SOTA gains on BABEL-QA, HuMMan-QA, etc.

    Authors: We acknowledge that our reported ablations focus on supervision diversity and staged training rather than isolating the timestamp tokens. The timestamps form an integral part of the input representation to explicitly support reasoning over duration and rhythm; however, to directly substantiate their contribution, we will add a controlled ablation comparing the full model against an identical architecture without timestamp tokens in the revised manuscript. revision: yes

  2. Referee: [Abstract] Abstract: The manuscript asserts 'state-of-the-art performance across multiple benchmarks' and that 'even when using only 2D skeletal input, our approach surpasses previous 3D-based methods,' but supplies no quantitative metrics, baseline comparisons, error bars, or ablation tables. Without these data the magnitude, statistical reliability, and attribution of gains to the proposed components cannot be assessed.

    Authors: The abstract is a concise summary and does not contain numerical results, which follows standard practice for brevity. All quantitative metrics, SOTA comparisons on BABEL-QA, HuMMan-QA, CompMo, NTU-RGB+D and QEVD-Coach, baseline tables, ablation results, and any error bars are provided in full in the Experiments section. We will verify that cross-references from the abstract to these tables are explicit in the revision. revision: no

Circularity Check

0 steps flagged

No circularity; empirical SOTA claims rest on benchmark results, not self-referential definitions or fits

full rationale

The paper presents an LLM-based architecture with timestamp-augmented pose sequences and a mixed supervision regime, then reports performance on external benchmarks (BABEL-QA, HuMMan-QA, etc.). No derivation chain, uniqueness theorem, or parameter fit is invoked whose output is definitionally identical to its input. Ablation descriptions vary supervision diversity and staging but do not reduce any claimed gain to a tautology. Self-citations, if present, are not load-bearing for the central empirical result. The work is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review limits visibility into parameters or entities; the approach rests on the domain assumption that LLMs can leverage the described pose sequence format for temporal reasoning.

axioms (1)
  • domain assumption LLMs can effectively reason about motion order, duration, and rhythm when motion is represented as timestamped skeletal pose sequences.
    Invoked in the abstract to justify the representation choice.

pith-pipeline@v0.9.1-grok · 5770 in / 1411 out tokens · 37508 ms · 2026-06-26T17:53:30.137477+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

84 extracted references · 7 canonical work pages · 2 internal anchors

  1. [1]

    Posetrack: A benchmark for human pose estimation and tracking, 2018

    Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A benchmark for human pose estimation and tracking, 2018

    2018

  2. [2]

    Edge computing and cloud computing for internet of things: A review.Informatics, 11(4), 2024

    Francesco Cosimo Andriulo, Marco Fiore, Marina Mongiello, Emanuele Traversa, and Vera Zizzo. Edge computing and cloud computing for internet of things: A review.Informatics, 11(4), 2024

    2024

  3. [3]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare V oss, editors,Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan,...

    2005

  4. [4]

    Beit: Bert pre-training of image trans- formers, 2022

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers, 2022

    2022

  5. [5]

    B˘alt˘are¸ tu, P

    A. B˘alt˘are¸ tu, P. Benschop, and Jan C Van Gemert. Are we smply biased: Identifying ethical biases in action recognition. Master’s thesis, Delft University of Technology, Delft, The Netherlands, 2025

    2025

  6. [6]

    Gender shades: Intersectional accuracy disparities in commercial gender classification

    Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson, editors,Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 ofProceedings of Machine Learning Research, pages 77–91. PMLR, 23–24 Feb 2018

    2018

  7. [7]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. InCVPR, pages 6299–6308, 2017

    2017

  8. [8]

    Human and llm-based voice assistant interaction: An analytical framework for user verbal and nonverbal behaviors.arXiv preprint arXiv:2408.16465, 2024

    Szeyi Chan, Shihan Fu, Jiachen Li, Bingsheng Yao, Smit Desai, Mirjana Prpa, and Dakuo Wang. Human and llm-based voice assistant interaction: An analytical framework for user verbal and nonverbal behaviors.arXiv preprint arXiv:2408.16465, 2024

    work page arXiv 2024

  9. [9]

    Poselifter: Absolute 3d human pose lifting network from a single noisy 2d human pose, 2020

    Ju Yong Chang, Gyeongsik Moon, and Kyoung Mu Lee. Poselifter: Absolute 3d human pose lifting network from a single noisy 2d human pose, 2020

    2020

  10. [10]

    Motionllm: Understanding human behaviors from human motions and videos.arXiv preprint arXiv:2405.20340, 2024

    Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos.arXiv preprint arXiv:2405.20340, 2024

    work page arXiv 2024

  11. [11]

    Sportscap: Monocular 3d human motion capture and fine-grained understanding in challenging sports videos.IJCV, 129(10):2846–2864, 2021

    Xin Chen, Anqi Pang, Wei Yang, Yuexin Ma, Lan Xu, and Jingyi Yu. Sportscap: Monocular 3d human motion capture and fine-grained understanding in challenging sports videos.IJCV, 129(10):2846–2864, 2021

    2021

  12. [12]

    Openmmlab pose estimation toolbox and benchmark, 2020

    MMPose Contributors. Openmmlab pose estimation toolbox and benchmark, 2020

    2020

  13. [13]

    Posescript: Linking 3d human poses and natural language.IEEE TPAMI, 2024

    Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. Posescript: Linking 3d human poses and natural language.IEEE TPAMI, 2024

    2024

  14. [14]

    Posefix: Correcting 3d human poses with natural language, 2024

    Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, and Grégory Rogez. Posefix: Correcting 3d human poses with natural language, 2024

    2024

  15. [15]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    2019

  16. [16]

    Skeletr: Towards skeleton-based action recognition in the wild

    Haodong Duan, Mingze Xu, Bing Shuai, Davide Modolo, Zhuowen Tu, Joseph Tighe, and Alessandro Bergamo. Skeletr: Towards skeleton-based action recognition in the wild. InICCV, pages 13634–13644, 2023

    2023

  17. [17]

    Revisiting skeleton-based action recognition

    Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recognition. InCVPR, pages 2969–2978, 2022. 10

    2022

  18. [18]

    Motion question answering via modular motion programs

    Mark Endo, Joy Hsu, Jiaman Li, and Jiajun Wu. Motion question answering via modular motion programs. InInternational Conference on Machine Learning, pages 9312–9328. PMLR, 2023

    2023

  19. [19]

    Regulation (EU) 2016/679 of the European Parliament and of the Council

    European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council

    2016

  20. [20]

    Humocon: Concept discovery for human motion understanding

    Qihang Fang, Chengcheng Tang, Bugra Tekin, Shugao Ma, and Yanchao Yang. Humocon: Concept discovery for human motion understanding. InCVPR, pages 7179–7190, 2025

    2025

  21. [21]

    Chatpose: Chatting about 3d human pose

    Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Chatpose: Chatting about 3d human pose. InCVPR, pages 2093–2103, 2024

    2093

  22. [22]

    Privacy- preserving 3-d skeleton-based video action recognition via graph convolution network.IEEE Transactions on Consumer Electronics, 71(2):6627–6641, 2025

    Xuesong Gao, Keqiu Li, Xiulong Liu, Jie Nie, Weiqiang Chen, and Yonghong Tian. Privacy- preserving 3-d skeleton-based video action recognition via graph convolution network.IEEE Transactions on Consumer Electronics, 71(2):6627–6641, 2025

    2025

  23. [23]

    Momask: Generative masked modeling of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. InCVPR, pages 1900–1910, 2024

    1900

  24. [24]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, June 2022

    2022

  25. [25]

    Masked Autoencoders Are Scalable Vision Learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners.CoRR, abs/2111.06377, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  26. [26]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  27. [27]

    Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, 2014

    2014

  28. [28]

    Privacy-Preserving Human Activity Recognition System for Assisted Living Environments .IEEE Transactions on Artificial Intelligence, 5(05):2342–2357, May 2024

    Ankit Jain, Rajendra Akerkar, and Abhishek Srivastava. Privacy-Preserving Human Activity Recognition System for Assisted Living Environments .IEEE Transactions on Artificial Intelligence, 5(05):2342–2357, May 2024

    2024

  29. [29]

    Motiongpt: Human motion as a foreign language.NeurIPS, 36:20067–20079, 2023

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language.NeurIPS, 36:20067–20079, 2023

    2023

  30. [30]

    Zhang, Panna Felsen, and Jitendra Malik

    Angjoo Kanazawa, Jason Y . Zhang, Panna Felsen, and Jitendra Malik. Learning 3d human dynamics from video, 2019

    2019

  31. [31]

    Katrina Karkazis and Jennifer R. Fishman. Tracking u.s. professional athletes: The ethics of biometric technologies.The American Journal of Bioethics, 17(1):45–60, 2017. PMID: 27996918

    2017

  32. [32]

    Kuehne, H

    H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In2011 International Conference on Computer Vision, pages 2556–2563, 2011

    2011

  33. [33]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024

    2024

  34. [34]

    Imore: Implicit program-guided reasoning for human motion q&a, 2025

    Chen Li, Chinthani Sugandhika, Yeo Keat Ee, Eric Peh, Hao Zhang, Hong Yang, Deepu Rajan, and Basura Fernando. Imore: Implicit program-guided reasoning for human motion q&a, 2025

    2025

  35. [35]

    Unimotion: Unifying 3d human motion synthesis and understanding, 2024

    Chuqiao Li, Julian Chibane, Yannan He, Naama Pearl, Andreas Geiger, and Gerard Pons-moll. Unimotion: Unifying 3d human motion synthesis and understanding, 2024

    2024

  36. [36]

    Videochat: Chat-centric video understanding, 2024

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024. 11

    2024

  37. [37]

    Chatmotion: A multimodal multi-agent for human motion analysis.arXiv preprint arXiv:2502.18180, 2025

    Lei Li, Sen Jia, Jianhao Wang, Zhaochong An, Jiaang Li, Jenq-Neng Hwang, and Serge Belongie. Chatmotion: A multimodal multi-agent for human motion analysis.arXiv preprint arXiv:2502.18180, 2025

    work page arXiv 2025

  38. [38]

    Human motion instruction tuning

    Lei Li, Sen Jia, Jianhao Wang, Zhongyu Jiang, Feng Zhou, Ju Dai, Tianfang Zhang, Zongkai Wu, and Jenq-Neng Hwang. Human motion instruction tuning. InCVPR, pages 17582–17591, 2025

    2025

  39. [39]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971–5984, 2024

    2024

  40. [40]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summariza- tion Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics

    2004

  41. [41]

    Tsm: Temporal shift module for efficient video understand- ing

    Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understand- ing. InICCV, pages 7083–7093, 2019

    2019

  42. [42]

    Motion-x: A large-scale 3d expressive whole-body human motion dataset.Advances in Neural Information Processing Systems, 2023

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset.Advances in Neural Information Processing Systems, 2023

    2023

  43. [43]

    Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want, 2025

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want, 2025

    2025

  44. [44]

    Polyslgen: Online multimodal speaking-listening reaction generation in polyadic interaction

    Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, and Xucong Zhang. Polyslgen: Online multimodal speaking-listening reaction generation in polyadic interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 29379–29390, June 2026

    2026

  45. [45]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  46. [46]

    Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding.IEEE TPAMI, 42(10):2684–2701, 2019

    Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding.IEEE TPAMI, 42(10):2684–2701, 2019

    2019

  47. [47]

    Videogpt+: Integrating image and video encoders for enhanced video understanding, 2024

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding, 2024

    2024

  48. [48]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. InICCV, pages 5442–5451, 2019

    2019

  49. [49]

    Muppet: Multi-person 2d-to-3d pose lifting

    Thomas Markhorst, Zhi-Yi Lin, Jouh Yeong Chew, Jan Van Gemert, and Xucong Zhang. Muppet: Multi-person 2d-to-3d pose lifting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 5320–5330, June 2026

    2026

  50. [50]

    A survey on bias and fairness in machine learning, 2022

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning, 2022

    2022

  51. [51]

    Skeleton-dml: Deep metric learning for skeleton-based one-shot action recognition

    Raphael Memmesheimer, Simon Häring, Nick Theisen, and Dietrich Paulus. Skeleton-dml: Deep metric learning for skeleton-based one-shot action recognition. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3702–3710, 2022

    2022

  52. [52]

    Sl-dml: Signal level deep metric learning for multimodal one-shot action recognition

    Raphael Memmesheimer, Nick Theisen, and Dietrich Paulus. Sl-dml: Signal level deep metric learning for multimodal one-shot action recognition. In2020 25th International conference on pattern recognition (ICPR), pages 4573–4580. IEEE, 2021

    2021

  53. [53]

    What to say and when to say it: Live fitness coaching as a testbed for situated interaction, 2024

    Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Bohm, Florian Dietrichkeit, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Lee, Mark Todorovich, Ingo Bax, and Roland Memisevic. What to say and when to say it: Live fitness coaching as a testbed for situated interaction, 2024. 12

    2024

  54. [54]

    Babel: Bodies, action and behavior with english labels

    Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J Black. Babel: Bodies, action and behavior with english labels. InCVPR, pages 722–731, 2021

    2021

  55. [55]

    Llms are good action recognizers, 2024

    Haoxuan Qu, Yujun Cai, and Jun Liu. Llms are good action recognizers, 2024

    2024

  56. [56]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    2025

  57. [57]

    One-shot action recognition in challenging therapy scenarios

    Alberto Sabater, Laura Santos, Jose Santos-Victor, Alexandre Bernardino, Luis Montesano, and Ana C Murillo. One-shot action recognition in challenging therapy scenarios. InCVPR, pages 2777–2785, 2021

    2021

  58. [58]

    Camera-based monitoring of neck move- ments for cervical rehabilitation mobile applications.Sensors, 21(6), 2021

    Iosune Salinas-Bueno, Maria Francesca Roig-Maimó, Pau Martínez-Bueso, Katia San-Sebastián- Fernández, Javier Varona, and Ramon Mas-Sansó. Camera-based monitoring of neck move- ments for cervical rehabilitation mobile applications.Sensors, 21(6), 2021

    2021

  59. [59]

    Humanizing human- robot interaction: On the importance of mutual understanding.IEEE Technology and Society Magazine, 37(1):22–29, 2018

    Alessandra Sciutti, Martina Mara, Vincenzo Tagliasco, and Giulio Sandini. Humanizing human- robot interaction: On the importance of mutual understanding.IEEE Technology and Society Magazine, 37(1):22–29, 2018

    2018

  60. [60]

    Finegym: A hierarchical video dataset for fine-grained action understanding

    Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine-grained action understanding. InCVPR, pages 2616–2625, 2020

    2020

  61. [61]

    Optimize cloud computations using edge computing

    Sachchidanand Singh. Optimize cloud computations using edge computing. In2017 Interna- tional Conference on Big Data, IoT and Data Science (BID), pages 49–53, 2017

    2017

  62. [62]

    Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012

    2012

  63. [63]

    Yolo Y . Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali V osoughi, Chao Huang, Zeliang Zhang, Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, and Chenliang Xu. Video understanding with large language models: A survey, 2025

    2025

  64. [64]

    Motionclip: Exposing human motion generation to clip space

    Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. InECCV, pages 358–374. Springer, 2022

    2022

  65. [65]

    Learning spatiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. InICCV, pages 4489–4497, 2015

    2015

  66. [66]

    Fg-t2m: Fine-grained text-driven human motion generation via diffusion model

    Yin Wang, Zhiying Leng, Frederick WB Li, Shun-Cheng Wu, and Xiaohui Liang. Fg-t2m: Fine-grained text-driven human motion generation via diffusion model. InProceedings of the IEEE/CVF international conference on computer vision, pages 22035–22044, 2023

    2023

  67. [67]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions, 2023

    2023

  68. [68]

    MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding

    Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. Motiongpt-2: A general-purpose motion-language model for motion generation and understanding.arXiv preprint arXiv:2410.21747, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  69. [69]

    Multimodal dialog act classification for digital character conversations

    Philine Witzig, Rares Constantin, Nikola Kovacevic, and Rafael Wampfler. Multimodal dialog act classification for digital character conversations. InProceedings of the 6th ACM conference on conversational user interfaces, pages 1–14, 2024

    2024

  70. [70]

    Motion- agent: A conversational framework for human motion generation with llms

    Qi Wu, Yubo Zhao, Yifan Wang, Xinhang Liu, Yu-Wing Tai, and Chi-Keung Tang. Motion- agent: A conversational framework for human motion generation with llms. 2025. 13

    2025

  71. [71]

    Dense motion captioning, 2025

    Shiyao Xu, Benedetta Liberatori, Gül Varol, and Paolo Rota. Dense motion captioning, 2025

    2025

  72. [72]

    Improving fine-grained understanding for retrieval in human motion and text.IEEE Signal Processing Letters, 2024

    Sheng Yan, Yong Wang, Xin Du, Hongchang Jin, and Mengyuan Liu. Improving fine-grained understanding for retrieval in human motion and text.IEEE Signal Processing Letters, 2024

    2024

  73. [73]

    Spatial temporal graph convolutional networks for skeleton-based action recognition

    Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. InAAAI, volume 32, 2018

    2018

  74. [74]

    Vid2seq: Large-scale pretraining of a visual language model for dense video captioning, 2023

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning, 2023

    2023

  75. [75]

    Osprey: Pixel understanding with visual instruction tuning, 2025

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning, 2025

    2025

  76. [76]

    Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025

    2025

  77. [77]

    Finemogen: Fine-grained spatio-temporal motion generation and editing, 2023

    Mingyuan Zhang, Huirong Li, Zhongang Cai, Jiawei Ren, Lei Yang, and Ziwei Liu. Finemogen: Fine-grained spatio-temporal motion generation and editing, 2023

    2023

  78. [78]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert, 2020

    2020

  79. [79]

    Motion-X++: A large-scale multimodal 3D whole-body human motion dataset,

    Yuhong Zhang, Jing Lin, Ailing Zeng, Guanlin Wu, Shunlin Lu, Yurong Fu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x++: A large-scale multimodal 3d whole-body human motion dataset.arXiv preprint arXiv:2501.05098, 2025

    work page arXiv 2025

  80. [80]

    Temporal relational reasoning in videos

    Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. InECCV, pages 803–818, 2018

    2018

Showing first 80 references.