Pith. sign in

REVIEW 5 major objections 5 minor 91 references

HuMoCon: Concept Discovery for Human Motion Understanding

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HuMoCon claims that explicit cross-modal alignment and velocity reconstruction in masked autoencoders let a frozen LLM answer fine-grained questions about human motion and video far better than prior models.

desk verdict Plausible recipe, uncontrolled comparison: explicit alignment and velocity reconstruction for motion LLMs, but the headline gains over prior work are not yet attributable to the method. read the letter →

arxiv 2505.20920 v1 pith:LGYRNP3I submitted 2025-05-27 cs.CV

Qihang Fang , Chengcheng Tang , Bugra Tekin , Shugao Ma , Yanchao Yang This is my paper
classification cs.CV
keywords humanmotionunderstandingquestionansweringvideocross-modalfeaturealignmentmaskedautoencodervelocityreconstructionVQ-VAEconceptdiscoverymultimodalLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HuMoCon is a framework for making large language models understand human motion and video by first discovering discrete 'concepts' in the motion itself. The paper argues that two specific gaps explain why earlier systems fall short: explicit alignment of video and motion features is missing, and masked autoencoding washes out high-frequency temporal details. HuMoCon adds a feature-alignment loss that pulls paired video and motion representations together, and a velocity-reconstruction objective that forces the encoder to preserve frame-to-frame changes. On the BABEL-QA motion benchmark it reports an overall score of 0.711 versus 0.436 for the strongest prior generative method, and on ActivityNet-QA it reports 54.2 accuracy versus 53.3. A sympathetic reader would take the central claim to be that encoder-level concept discovery, not just LLM fine-tuning, is what unlocks fine-grained motion understanding.

What carries the argument

The machine is a VQ-VAE-style autoencoder whose codebook of 512 discrete vectors is treated as the discovered set of human motion concepts. Three auxiliary heads hang off it: a masked reconstruction decoder; a hypernetwork that generates a per-code classifier (discriminative informativeness) and whose gradients feed a velocity decoder that predicts per-frame differences (actionable informativeness); and two projection layers with a temperature-scaled normalized alignment loss computed only on paired video-motion data. The masked decoder keeps low-frequency structure, the velocity path keeps high-frequency dynamics, and the alignment loss ties the two modalities together before the LLM ever sees the features.

What would settle it

Retrain the strongest baseline, or an identical-backbone control, on HuMoCon's exact 200K Motion-XQA pairs, Video-ChatGPT data, and BABEL-QA training set, then evaluate with the same judge prompt; if it still trails by roughly 0.28 on BABEL-QA, the margin is not caused by the alignment and velocity objectives.

Watch

Extended reading notes

Core claim

The central claim is that human motion concepts, treated as discrete codes in a VQ-VAE codebook, become substantially more useful for downstream reasoning when the encoder is trained with three objectives rather than masked reconstruction alone. Explicit cross-modal alignment projects video and motion features into a shared space using frame-aligned pairs from a large motion dataset, so video supplies context and motion supplies human-centric detail. Velocity reconstruction, implemented through a hypernetwork that generates a classifier and uses its gradients to predict optical flow or delta motion, restores the high-frequency information that masking destroys. Together these objectives let a frozen Llama-3.1-8B, trained only through a small projection layer and LoRA, answer relational and body-part questions that prior models miss. The reported numbers are 0.711 overall on BABEL-QA and 54.2 accuracy on ActivityNet-QA, with ablations showing each loss contributes and velocity reconstruction contributing the most.

Load-bearing premise

The leaderboard comparisons rest on the assumption that the baseline scores quoted from earlier papers were produced under the same training data, instruction-tuning recipe, and LLM-judge evaluation prompts as HuMoCon; if those differ, the reported margins may reflect the recipe rather than the proposed losses.

Editorial extensions

If this is right

  • Explicit encoder-level alignment can replace implicit alignment done during LLM fine-tuning, making the pretraining stage the place where cross-modal semantics are decided.
  • Velocity reconstruction can be added to any masked autoencoder for temporal data, not only human motion, whenever high-frequency dynamics matter downstream.
  • Because the encoders are frozen during instruction tuning, the discovered concepts and pretrained features can be reused across different LLMs and tasks without retraining the perception stack.
  • The discrete codebook gives a compact, inspectable representation of human motion that could be probed or edited for interpretability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the margin is real, it suggests video-only models are bottlenecked by missing human-centric signal; adding motion alignment during pretraining may improve video QA even when only video is available at test time.
  • The velocity-reconstruction trick could transfer to other sequential modalities such as audio, hand-object interaction, and robot trajectories, where masked autoencoders currently over-smooth.
  • The dependence on LLM-judge prompts suggests the field needs a fixed, open evaluation protocol before cross-paper comparisons can be read literally; the paper itself notes baselines were not retrained.
  • One testable extension is to probe whether the learned codebook entries correspond to semantically nameable motion primitives, such as 'bend knees' or 'lean forward', by aligning them with text in the manner the paper's qualitative examples suggest.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces HuMoCon, a framework for human motion and video understanding built around a VQ-VAE-based concept-discovery pretraining stage. Two encoders (video and motion) are trained with masked reconstruction, a velocity-reconstruction objective composed of discriminative and actionable informativeness terms, and a cross-modal feature-alignment loss. The encoders' outputs are then mapped into an LLM (Llama3.1-8B) through modality-translation layers and multi-modality instruction tuning with LoRA. The method is evaluated on BABEL-QA and ActivityNet-QA, where it reports large gains over MotionLLM and other baselines, and on ablations that remove individual loss components. The central claim is that explicit feature alignment and velocity reconstruction enable effective motion concept discovery and significantly improve large-model performance on human-motion understanding.

Significance. If the reported gains are real, the paper makes a useful contribution: it identifies a concrete failure mode of masked autoencoding for motion (loss of high-frequency detail) and proposes two plausible remedies, namely velocity reconstruction and explicit video-motion encoder alignment. The authors also provide unusually detailed implementation information, including network architectures, dataset counts, and the LLM-judge prompts, which aids reproducibility. The conceptual framing of discrete codebooks as 'discovered concepts' is interesting, and the qualitative examples suggest the model can answer fine-grained kinematic questions. However, the evaluation as presented does not currently support the strong claim of significant state-of-the-art performance, because the head-to-head comparisons are not controlled and the central metric is an unvalidated LLM judge. The method itself is defensible; what is missing is a matched comparison and additional validation of the concept-discovery mechanism.

major comments (5)
  1. [Section 4.1 and Tables 1-2] The headline comparisons are not controlled. The MotionLLM numbers in Tables 1 and 2 are quoted from the original paper, whereas HuMoCon is trained with a different base LLM (Llama3.1-8B), a different instruction-tuning dataset recipe (200K Motion-XQA pairs, Video-ChatGPT, VATEX, BABEL-QA), and a specific LLM-judge prompt. The ablations in Table 3 make this issue concrete: even 'Ours-w/o Ldis&Lact' (0.637) and 'Ours-w/o Lalign' (0.675) are far above MotionLLM's 0.436, indicating that most of the gap may come from the data recipe, base LLM, or judge, not from the proposed losses. To attribute the gain to the proposed components, the authors need a baseline trained with the same LLM, data, and judge but without the proposed losses, and ideally a MotionLLM model retrained under the HuMoCon recipe.
  2. [Table 1 and Section 4.1 (Evaluation metrics)] Table 1 mixes incompatible metric types. The classification baselines (2s-AGCN, MotionCLIP) are scored by direct accuracy, while the generative baselines (MotionLLM, Ours) are scored by an LLM judge on a 0-to-1 scale. The 'Overall' column therefore does not place all models on a common scale, and the reported margin (e.g., 0.711 vs. 0.355) is not a like-for-like comparison. The LLM judge in Fig. 7 has no human validation, and the ActivityNet prompt in Fig. 8 is internally inconsistent (it asks for an integer score but gives '4.8' as an example). The authors should either evaluate all methods with the same metric or clearly separate the two evaluations, and they should report human agreement on a sample of LLM-judge scores.
  3. [Table 3 vs. Table 7 (Supplementary)] The ablation results are internally inconsistent. Table 3 reports that removing both L_dis and L_act together gives 0.637, while Table 7 reports that removing L_act alone gives 0.563 and removing L_dis alone gives 0.629. A model with both losses removed should not outperform a model with only one of them removed. The note in Sec. 9.4 says the computation of L_dis is adjusted in the absence of L_act, but this makes the two ablations non-comparable and undermines the conclusion that removing velocity reconstruction has the greatest impact. The authors should rerun a consistent set of ablations and reconcile Tables 3 and 7.
  4. [Section 3.2, Eq. (7)] The discriminative informativeness loss is self-referential. In Eq. (7), the labels d_i^u are the VQ-VAE codebook assignments produced by the same encoder being trained, so minimizing this loss can reinforce trivial or degenerate assignments rather than learn semantically meaningful concepts. The paper does not report codebook utilization, per-code frequency, or any external validation that the codebook entries correspond to interpretable motion concepts. Since the central claim is 'concept discovery,' the authors should provide evidence that the codebook is not collapsing and that the discovered concepts are meaningful, for example by visualizing nearest codes, measuring codebook entropy, or conducting a small human study on code interpretability.
  5. [Tables 1-3 and Section 4.2] No error bars, multiple seeds, or significance tests are reported. On ActivityNet-QA the gain over MotionLLM is only 0.9 accuracy points (54.2 vs. 53.3) and 0.1 in score (3.6 vs. 3.5), which may be within run-to-run noise. On BABEL-QA the differences are large, but given the uncontrolled comparison and single-run evaluation, the word 'significantly' in the abstract and Section 4.2 is not justified. The authors should run at least three seeds for the main results and ablations and report mean and standard deviation.
minor comments (5)
  1. [Eq. (11)] The alignment loss as written is not a standard contrastive loss: the numerator and denominator both contain sums over indices, and the notation is ambiguous about which terms are inside the outer summation. Please clarify the indexing and state whether a logarithm is intended.
  2. [Section 3.1 vs. Section 3.2] Section 3.1 lists four key learning objectives, while Section 3.2 says 'we design three distinct learning objectives.' Please make the counts consistent.
  3. [Acknowledgment] There is a typo: 'the the the University of Hong Kong' should be 'the University of Hong Kong.'
  4. [Table 6 (Supplementary)] The table uses 'CasualTransformerLayer' in several places; this appears to be a typo for 'CausalTransformerLayer.'
  5. [Section 4.2, Ablation study] The text contains a typo: 'pexrformance' should be 'performance.'

Circularity Check

2 steps flagged · score 4.0 of 10

Self-referential concept-discovery objective and a self-cited velocity-reconstruction mechanism introduce partial circularity, but the central QA claim rests on external benchmarks.

  1. self definitional [Section 3.2, 'Discriminative informativeness' (Eq. 7)]
    "we employ a codebook C^u, modeled with VQ-VAE to quantize f^u into a discrete feature representation d^u ... and C^u is the codebook used for discretization, also serving as the discovered human motion concepts. The discriminative loss for video is defined as: L_dis^video = ... CE(H^u(c^u_k, u_i), d^u_i), where |C^u| is the size of the codebook C^u, CE(·) is the cross-entropy loss for classification, d^u_i represents the 'ground-truth label' for the input state u_i determined by the VQ-VAE, and T is the number of frames."

    The 'discovered human motion concepts' are defined as the VQ-VAE codebook entries (Eq. 2), and the discriminative-informativeness loss (Eq. 7) trains a hyper-network to predict the very codebook assignments d^u_i that the same VQ-VAE encoder and codebook produced. The objective's target is therefore a deterministic function of the model being trained, with no external ground truth for concept quality. The paper presents this self-prediction as 'increasing the correlation between the discovered concepts and the corresponding state features,' which is true by construction and can be trivially satisfied if the codebook collapses.

  2. self citation load bearing [Section 3.2, 'Actionable informativeness'; Introduction]
    "Following InfoCon [46], we design a VQ-VAE-based loss function that helps retain the fine-grained features of the original masked autoencoder while further enhancing high-frequency feature expression ... Inspired by the idea that gradients of a discriminator function can inform the state changes, as observed in [46], this objective ensures that the model captures not only static state information, but also the dynamic changes in the sequence."

    The core velocity-reconstruction mechanism is motivated by an 'observation' reported in InfoCon [46], which is co-authored by HuMoCon's senior/corresponding author (Yanchao Yang). The paper imports the design premise—that discriminator gradients inform state changes—from this self-citation rather than re-deriving or independently validating it. This is load-bearing because velocity reconstruction is one of the two main claimed contributions of the paper. However, the ablation in Table 3 and the external QA benchmarks provide independent empirical evidence for the loss's effect, so the circularity is partial rather than total.

full rationale

The central downstream claim—that HuMoCon significantly improves motion-video question answering—is grounded in external benchmarks (BABEL-QA and ActivityNet-QA) and is therefore not itself circular. The reported gains could be affected by uncontrolled differences in data recipe, base LLM, or LLM-judge prompts, but that is a validity concern, not a circularity concern. The clearest circularity is in the concept-discovery framing: the 'discovered concepts' are the VQ-VAE codebook, and the discriminative-informativeness loss uses the same VQ-VAE assignments as ground-truth labels, making the objective self-referential by construction. Additionally, the velocity-reconstruction mechanism, one of the paper's two main contributions, is motivated almost entirely by a self-citation to InfoCon [46] by a co-author; the paper does not independently establish the cited premise. Because the external QA results and ablations provide independent content, the paper is not wholly circular, but these two steps prevent it from being fully self-contained.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several domain assumptions about data quality, velocity definitions, codebook behavior, and the validity of LLM-based evaluation, plus a set of hand-chosen hyperparameters. None of these are derived from first principles, and several are not stress-tested with sensitivity analyses.

free parameters (6)
  • Mask ratio sigma = 0.75
    Chosen by hand for masked reconstruction in encoder pre-training (supplementary Implementation Details).
  • Loss weight lambda_dis = 0.3
    Weight for the discriminative informativeness objective, chosen by hand.
  • Loss weight lambda_act = 0.1
    Weight for the actionable informativeness (velocity reconstruction) objective, chosen by hand.
  • Loss weight lambda_align = 0.1
    Weight for the cross-modal feature alignment loss, chosen by hand.
  • Alignment temperature epsilon = not reported
    Temperature in Eq. 11 controls the sharpness of the alignment distribution; no value or sensitivity analysis is given.
  • Video keyframe count = 8
    Set due to GPU memory; affects temporal resolution and alignment quality.
assumptions (5)
  • domain assumption Motion-X video-motion pairs are correctly synchronized and the pose estimates are accurate enough for frame-level alignment.
    The feature alignment loss L_align (Eq. 11) and velocity reconstruction rely on align(i) mapping video frames to motion poses; if pairs are noisy, the alignment objective injects errors.
  • domain assumption Optical flow and delta motion are faithful velocity signals for video and motion.
    The actionable informativeness objective treats optical flow and delta pose as ground-truth velocity for reconstruction; if these are noisy or misaligned, high-frequency reconstruction is trained on wrong targets.
  • ad hoc to paper The VQ-VAE codebook with EMA updates does not collapse and its entries are semantically meaningful concepts.
    The paper's central notion of 'concept discovery' is not externally evaluated; it assumes codebook entries correspond to interpretable motion concepts rather than arbitrary quantization artifacts.
  • domain assumption LLM-as-judge scoring is a valid and unbiased measure of answer correctness.
    All generative scores on BABEL-QA and ActivityNet-QA come from an LLM judge; if the judge rewards style or refuses to penalize errors consistently, the reported margins may not reflect true understanding.
  • domain assumption The gradient signal from the hypernetwork (InfoCon-style) is informative for reconstructing velocity.
    Actionable informativeness assumes gradients of the discriminator around a state encode state changes; this is carried over from robot manipulation [46] and may transfer imperfectly to human motion.
invented entities (1)
  • Discrete human motion concepts (codebook entries)
    purpose: Serve as quantized semantic units for video and motion features, the discovered concepts claimed by the title.
    The paper provides no external measure that codebook entries are interpretable or correspond to real motion concepts; the only evidence is downstream QA performance, which does not isolate the concept semantics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HuMoCon: Concept Discovery for Human Motion Understanding." pith.science (2026). https://pith.science/paper/LGYRNP3I

@misc{pith2026250520920,
  author       = {Pith},
  title        = {Pith review of: HuMoCon: Concept Discovery for Human Motion Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LGYRNP3I}},
  note         = {Machine review of arXiv:2505.20920}
}
read the original abstract

We present HuMoCon, a novel motion-video understanding framework designed for advanced human behavior analysis. The core of our method is a human motion concept discovery framework that efficiently trains multi-modal encoders to extract semantically meaningful and generalizable features. HuMoCon addresses key challenges in motion concept discovery for understanding and reasoning, including the lack of explicit multi-modality feature alignment and the loss of high-frequency information in masked autoencoding frameworks. Our approach integrates a feature alignment strategy that leverages video for contextual understanding and motion for fine-grained interaction modeling, further with a velocity reconstruction mechanism to enhance high-frequency feature expression and mitigate temporal over-smoothing. Comprehensive experiments on standard benchmarks demonstrate that HuMoCon enables effective motion concept discovery and significantly outperforms state-of-the-art methods in training large models for human motion understanding. We will open-source the associated code with our paper.

Figures

Figures reproduced from arXiv: 2505.20920 by the authors.

Figure 1
Figure 1. We present HuMoCon for concept discovery and human motion understanding with video or motion input. To address the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. System overview of our method. (a) The encoder pre-training process for learning and aligning video and motion features and enhancing high-frequency details through velocity reconstruction. We utilize a VQ-VAE-based structure, and we design effective learning objectives to enhance the encoder to extract semantic meaningful and fine-grained features. (b) The fine-tuning of the large language model (LLM) for video and… view at source ↗
Figure 3
Figure 3. Overview of the velocity reconstruction components. We build similar network structures for both video and motion, and we present the video part in this figure. This module is composed of two learning objectives. 1) Discriminative informativeness (left) aims to improve the distinctiveness of encoded features by reducing representational ambiguity. 2) Actionable informativeness (right) focuses on reconstructing the v… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Example of motion understanding. We validate the proposed model’s ability to understand motion from multiple aspects. Q1 on the left and Q1–3 on the right evaluate its comprehension of motion sequences, while Q2 and Q3 on the left push it to analyze kinematic, kinesthe…
Figure 5
Figure 5. Figure 5: The structure of the hyper-network [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Comparison between different video understanding methods. We compare our method with Video-Llava and MotionLLM. We highlight the correct and wrong answers in blue and red, respectively. Input: question, answer, prediction LLM evaluation prompts: Question | Ground truth…
Figure 7
Figure 7. Figure 7: Evaluation prompt for BABEL-QA benchmark. 8. Evaluation Details 8.1. Evaluation on BABEL-QA Benchmark For the BABEL-QA benchmark, we extend the evalua￾tion protocol used in previous multi-modality LLMs evalu￾ations [80]. The evaluation involves scoring the correctness …
Figure 8
Figure 8. Figure 8: Evaluation prompt for ActivityNet-QA benchmark. puts. The results showed a loss of 0.88 for motion and 0.91 for video. The higher loss with video input is likely due to the fact that we are limited to using only 8 key frames (restricted by GPU memory; same as MotionLLM…
Figure 9
Figure 9. Figure 9: Visualizations for motion understanding [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Visualizations for motion understanding [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

91 extracted references · 50 canonical work pages

  1. [1]

    Teach: Temporal action composition for 3d hu- mans

    Nikos Athanasiou, Mathis Petrovich, Michael J Black, and G¨ul Varol. Teach: Temporal action composition for 3d hu- mans. In2022 International Conference on 3D Vision (3DV), pages 414–423. IEEE, 2022. 2

    2022

  2. [2]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 3

    arXiv 2023

  3. [3]

    Ac- tion quality assessment with temporal parsing transformer

    Yang Bai, Desen Zhou, Songyang Zhang, Jian Wang, Er- rui Ding, Yu Guan, Yang Long, and Jingdong Wang. Ac- tion quality assessment with temporal parsing transformer. InEuropean conference on computer vision, pages 422–438. Springer, 2022. 1, 3

    2022

  4. [4]

    Implicit neural representations for variable length human motion generation

    Pablo Cervantes, Yusuke Sekikawa, Ikuro Sato, and Koichi Shinoda. Implicit neural representations for variable length human motion generation. InEuropean Conference on Com- puter Vision, pages 356–372. Springer, 2022. 1, 2

    2022

  5. [5]

    Motionllm: Understanding human behaviors from human motions and videos.arXiv preprint arXiv:2405.20340, 2024

    Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos.arXiv preprint arXiv:2405.20340, 2024. 1, 2, 3, 5, 6, 7, 4

    arXiv 2024

  6. [6]

    Pose trainer: correct- ing exercise posture using pose estimation.arXiv preprint arXiv:2006.11718, 2020

    Steven Chen and Richard R Yang. Pose trainer: correct- ing exercise posture using pose estimation.arXiv preprint arXiv:2006.11718, 2020. 1, 3

    arXiv 2006

  7. [7]

    Valor: Vision-audio- language omni-perception pretraining model and dataset

    Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Wein- ing Wang, Jinhui Tang, and Jing Liu. Valor: Vision-audio- language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345, 2023. 3

    arXiv 2023

  8. [8]

    Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.Advances in Neural Information Processing Sys- tems, 36:72842–72866, 2023

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.Advances in Neural Information Processing Sys- tems, 36:72842–72866, 2023. 3

    2023

Show all 91 references
  1. [9]

    Posefix: correcting 3d hu- man poses with natural language

    Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno- Noguer, and Gr ´egory Rogez. Posefix: correcting 3d hu- man poses with natural language. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15018–15028, 2023. 2

    2023

  2. [10]

    Behavior recognition via sparse spatio-temporal features

    Piotr Doll ´ar, Vincent Rabaud, Garrison Cottrell, and Serge Belongie. Behavior recognition via sparse spatio-temporal features. In2005 IEEE international workshop on visual surveillance and performance evaluation of tracking and surveillance, pages 65–72. IEEE, 2005. 3

    2005

  3. [11]

    Hierarchical recur- rent neural network for skeleton based action recognition

    Yong Du, Wei Wang, and Liang Wang. Hierarchical recur- rent neural network for skeleton based action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1110–1118, 2015. 1, 2

    2015

  4. [12]

    Clap learning audio concepts from nat- ural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from nat- ural language supervision. InICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 1–5. IEEE, 2023. 3

    2023

  5. [13]

    Motion question answering via modular motion programs

    Mark Endo, Joy Hsu, Jiaman Li, and Jiajun Wu. Motion question answering via modular motion programs. InIn- ternational Conference on Machine Learning, pages 9312–

  6. [14]

    Towards accurate active camera localization

    Qihang Fang, Yingda Yin, Qingnan Fan, Fei Xia, Siyan Dong, Sheng Wang, Jue Wang, Leonidas J Guibas, and Bao- quan Chen. Towards accurate active camera localization. In European Conference on Computer Vision, pages 122–139. Springer, 2022. 1

    2022

  7. [15]

    Cigtime: Corrective instruction generation through inverse motion editing

    Qihang Fang, Chengcheng Tang, Bugra Tekin, and Yanchao Yang. Cigtime: Corrective instruction generation through inverse motion editing. InThe Thirty-Eighth Annual Con- ference on Neural Information Processing Systems, 2024. 1, 2

    2024

  8. [16]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 2, 3

    2019

  9. [17]

    Chatpose: Chatting about 3d human pose

    Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Chatpose: Chatting about 3d human pose. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2093– 2103, 2024. 2

    2024

  10. [18]

    Aifit: Automatic 3d human-interpretable feedback models for fitness training

    Mihai Fieraru, Mihai Zanfir, Silviu Cristian Pirlea, Vlad Olaru, and Cristian Sminchisescu. Aifit: Automatic 3d human-interpretable feedback models for fitness training. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 9919–9928, 2021. 1, 3

    2021

  11. [19]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 3

    2023

  12. [20]

    Ac- tion2motion: Conditioned generation of 3d human motions

    Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Ac- tion2motion: Conditioned generation of 3d human motions. InProceedings of the 28th ACM International Conference on Multimedia, pages 2021–2029, 2020. 1, 2

    2021

  13. [21]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, 2022. 6

    2022

  14. [22]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152–5161, 2022. 2, 1

    2022

  15. [23]

    Tm2t: Stochastic and tokenized modeling for the reciprocal genera- tion of 3d human motions and texts

    Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal genera- tion of 3d human motions and texts. InEuropean Conference on Computer Vision, pages 580–597. Springer, 2022. 2

    2022

  16. [24]

    Contrastive learning from ex- tremely augmented skeleton sequences for self-supervised action recognition

    Tianyu Guo, Hong Liu, Zhan Chen, Mengyuan Liu, Tao Wang, and Runwei Ding. Contrastive learning from ex- tremely augmented skeleton sequences for self-supervised action recognition. InProceedings of the AAAI Conference on Artificial Intelligence, pages 762–770, 2022. 1, 2

    2022

  17. [25]

    Autoad ii: The sequel-who, when, and what in movie audio description

    Tengda Han, Max Bain, Arsha Nagrani, Gul Varol, Weidi Xie, and Andrew Zisserman. Autoad ii: The sequel-who, when, and what in movie audio description. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 13645–13655, 2023. 3

    2023

  18. [26]

    Autoad iii: The prequel-back to the pixels

    Tengda Han, Max Bain, Arsha Nagrani, G ¨ul Varol, Weidi Xie, and Andrew Zisserman. Autoad iii: The prequel-back to the pixels. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18164– 18174, 2024. 1, 3

    2024

  19. [27]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022. 2, 4

    2022

  20. [28]

    Phase- functioned neural networks for character control.ACM Transactions on Graphics (TOG), 36(4):1–13, 2017

    Daniel Holden, Taku Komura, and Jun Saito. Phase- functioned neural networks for character control.ACM Transactions on Graphics (TOG), 36(4):1–13, 2017. 2

    2017

  21. [29]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 6

    2021 arXiv

  22. [30]

    Motiongpt: Human motion as a foreign lan- guage.Advances in Neural Information Processing Systems, 36:20067–20079, 2023

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign lan- guage.Advances in Neural Information Processing Systems, 36:20067–20079, 2023. 2

    2023

  23. [31]

    Hand-object contact consistency reasoning for human grasps generation

    Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. InProceedings of the IEEE/CVF interna- tional conference on computer vision, pages 11107–11116,

  24. [32]

    Smplx-lite: A realistic and driv- able avatar benchmark with rich geometry and texture anno- tations.arXiv preprint arXiv:2405.19609, 2024

    Yujiao Jiang, Qingmin Liao, Zhaolong Wang, Xiangru Lin, Zongqing Lu, Yuxi Zhao, Hanqing Wei, Jingrui Ye, Yu Zhang, and Zhijing Shao. Smplx-lite: A realistic and driv- able avatar benchmark with rich geometry and texture anno- tations.arXiv preprint arXiv:2405.19609, 2024. 1

    2024 arXiv

  25. [33]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1370...

    2024

  26. [34]

    A new representation of skele- ton sequences for 3d action recognition

    Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, and Farid Boussaid. A new representation of skele- ton sequences for 3d action recognition. InProceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3288–3297, 2017. 1, 2

    2017

  27. [35]

    Flame: Free- form language-based motion synthesis & editing

    Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free- form language-based motion synthesis & editing. InPro- ceedings of the AAAI Conference on Artificial Intelligence, pages 8255–8263, 2023. 2

    2023

  28. [36]

    Danceconv: Dance motion genera- tion with convolutional networks.IEEE Access, 10:44982– 45000, 2022

    Kosmas Kritsis, Aggelos Gkiokas, Aggelos Pikrakis, and Vassilis Katsouros. Danceconv: Dance motion genera- tion with convolutional networks.IEEE Access, 10:44982– 45000, 2022. 2

    2022

  29. [37]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 2, 3, 6, 7

    2024 arXiv

  30. [38]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023. 7

    2023 arXiv

  31. [39]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195– ...

    2024

  32. [40]

    Video-llava: Learning united visual rep- resentation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023. 1, 2, 3, 6, 7, 4

    2023 arXiv

  33. [41]

    Motion-x: A large- scale 3d expressive whole-body human motion dataset.Ad- vances in Neural Information Processing Systems, 36, 2024

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large- scale 3d expressive whole-body human motion dataset.Ad- vances in Neural Information Processing Systems, 36, 2024. 6

    2024

  34. [42]

    Actionlet- dependent contrastive learning for unsupervised skeleton- based action recognition

    Lilang Lin, Jiahang Zhang, and Jiaying Liu. Actionlet- dependent contrastive learning for unsupervised skeleton- based action recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2363–2372, 2023. 1, 2

    2023

  35. [43]

    Towards unified sur- gical skill assessment

    Daochang Liu, Qiyue Li, Tingting Jiang, Yizhou Wang, Rulin Miao, Fei Shan, and Ziyu Li. Towards unified sur- gical skill assessment. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 9522–9531, 2021. 1, 3

    2021

  36. [44]

    Spatio-temporal lstm with trust gates for 3d human action recognition

    Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang. Spatio-temporal lstm with trust gates for 3d human action recognition. InComputer Vision–ECCV 2016: 14th Euro- pean Conference, Amsterdam, The Netherlands, October 11- 14, 2016, Proceedings, Part III 14, pages 816–833. Springer,

    2016

  37. [45]

    Enhanced skele- ton visualization for view invariant human action recogni- tion.Pattern Recognition, 68:346–362, 2017

    Mengyuan Liu, Hong Liu, and Chen Chen. Enhanced skele- ton visualization for view invariant human action recogni- tion.Pattern Recognition, 68:346–362, 2017. 1, 2

    2017

  38. [46]

    Infocon: Con- cept discovery with generative and discriminative informa- tiveness.arXiv preprint arXiv:2404.10606, 2024

    Ruizhe Liu, Qian Luo, and Yanchao Yang. Infocon: Con- cept discovery with generative and discriminative informa- tiveness.arXiv preprint arXiv:2404.10606, 2024. 2, 4, 5

    2024 arXiv

  39. [47]

    Posegpt: Quantization-based 3d human mo- tion generation and forecasting

    Thomas Lucas, Fabien Baradel, Philippe Weinzaepfel, and Gr´egory Rogez. Posegpt: Quantization-based 3d human mo- tion generation and forecasting. InEuropean Conference on Computer Vision, pages 417–435. Springer, 2022. 2

    2022

  40. [48]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.Neu- rocomputing, 508:293–304, 2022

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.Neu- rocomputing, 508:293–304, 2022. 3

    2022

  41. [49]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023. 7

    2023 arXiv

  42. [50]

    Pg-video-llava: Pixel grounding large video- language models.arXiv preprint arXiv:2311.13435, 2023

    Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. Pg-video-llava: Pixel grounding large video- language models.arXiv preprint arXiv:2311.13435, 2023. 1, 3

    2023 arXiv

  43. [51]

    What and how well you performed? a multitask learning approach to action quality assessment

    Paritosh Parmar and Brendan Tran Morris. What and how well you performed? a multitask learning approach to action quality assessment. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 304–313, 2019. 1, 3

    2019

  44. [52]

    Action- conditioned 3d human motion synthesis with transformer vae

    Mathis Petrovich, Michael J Black, and G ¨ul Varol. Action- conditioned 3d human motion synthesis with transformer vae. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 10985–10995, 2021. 1, 2

    2021

  45. [53]

    Temos: Generating diverse human motions from textual descriptions

    Mathis Petrovich, Michael J Black, and G ¨ul Varol. Temos: Generating diverse human motions from textual descriptions. InEuropean Conference on Computer Vision, pages 480–

  46. [54]

    Babel: Bodies, action and behavior with english la- bels

    Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J Black. Babel: Bodies, action and behavior with english la- bels. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 722–731, 2021. 2

    2021

  47. [55]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  48. [56]

    On the benefits of 3d pose and tracking for human action recognition

    Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, Christoph Feichtenhofer, and Jitendra Malik. On the benefits of 3d pose and tracking for human action recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 640–649,

  49. [57]

    Zero-shot audio captioning with audio-language model guidance and audio context keywords.arXiv preprint arXiv:2311.08396, 2023

    Leonard Salewski, Stefan Fauth, A Koepke, and Zeynep Akata. Zero-shot audio captioning with audio-language model guidance and audio context keywords.arXiv preprint arXiv:2311.08396, 2023. 3, 7

    2023 arXiv

  50. [58]

    Two- stream adaptive graph convolutional networks for skeleton- based action recognition

    Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two- stream adaptive graph convolutional networks for skeleton- based action recognition. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12026–12035, 2019. 1, 2, 7

    2019

  51. [59]

    Neural state machine for character-scene interactions.ACM Transactions on Graphics, 38(6):178, 2019

    Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions.ACM Transactions on Graphics, 38(6):178, 2019. 1, 2

    2019

  52. [60]

    Local motion phases for learning multi-contact charac- ter movements.ACM Transactions on Graphics (TOG), 39 (4):54–1, 2020

    Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Za- man. Local motion phases for learning multi-contact charac- ter movements.ACM Transactions on Graphics (TOG), 39 (4):54–1, 2020. 2

    2020

  53. [61]

    Deepphase: Periodic autoencoders for learning motion phase manifolds

    Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG), 41(4):1–13, 2022. 1, 2

    2022

  54. [62]

    Convolutional learning of spatio-temporal features

    Graham W Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler. Convolutional learning of spatio-temporal features. InComputer Vision–ECCV 2010: 11th European Confer- ence on Computer Vision, Heraklion, Crete, Greece, Septem- ber 5-11, 2010, Proceedings, Part VI 11, pages 140–...

    2010

  55. [63]

    Motionclip: Exposing human motion generation to clip space

    Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. InEuropean Conference on Com- puter Vision, pages 358–374. Springer, 2022. 2, 3, 7

    2022

  56. [64]

    Human motion dif- fusion model.arXiv preprint arXiv:2209.14916, 2022

    Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion dif- fusion model.arXiv preprint arXiv:2209.14916, 2022. 2

    2022 arXiv

  57. [65]

    Learning spatiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. InProceedings of the IEEE inter- national conference on computer vision, pages 4489–4497,

  58. [66]

    Human action recognition by representing 3d skeletons as points in a lie group

    Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 588–595,

  59. [67]

    Action recognition with improved trajectories

    Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. InProceedings of the IEEE inter- national conference on computer vision, pages 3551–3558,

  60. [68]

    Learn- ing human dynamics in autonomous driving scenarios

    Jingbo Wang, Ye Yuan, Zhengyi Luo, Kevin Xie, Dahua Lin, Umar Iqbal, Sanja Fidler, and Sameh Khamis. Learn- ing human dynamics in autonomous driving scenarios. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20796–20806, 2023. 1

    2023

  61. [69]

    Vatex: A large-scale, high- quality multilingual dataset for video-and-language research

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high- quality multilingual dataset for video-and-language research. InProceedings of the IEEE/CVF international conference on computer vision, pages 4581–4591, 2019. 6

    2019

  62. [70]

    Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191, 2022

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191, 2022. 2, 3

    2022 arXiv

  63. [71]

    Internvideo2: Scaling video foundation mod- els for multimodal video understanding.arXiv preprint arXiv:2403.15377, 2024

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation mod- els for multimodal video understanding.arXiv preprint arXiv:2403.15377, 2024. 2, 3

    2024 arXiv

  64. [72]

    Unified human-scene interaction via prompted chain-of-contacts

    Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-contacts. arXiv preprint arXiv:2309.07918, 2023. 1

    2023 arXiv

  65. [73]

    Autoad-zero: A training-free framework for zero-shot audio description

    Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, G ¨ul Varol, Weidi Xie, and Andrew Zisserman. Autoad-zero: A training-free framework for zero-shot audio description. arXiv preprint arXiv:2407.15850, 2024. 3

    2024 arXiv

  66. [74]

    Dexterous grasp transformer

    Guo-Hao Xu, Yi-Lin Wei, Dian Zheng, Xiao-Ming Wu, and Wei-Shi Zheng. Dexterous grasp transformer. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17933–17942, 2024. 2

    2024

  67. [75]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning.arXiv preprint arXiv:2404.16994, 2024

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning.arXiv preprint arXiv:2404.16994, 2024. 1, 3

    2024 arXiv

  68. [76]

    Spatial tempo- ral graph convolutional networks for skeleton-based action recognition

    Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial tempo- ral graph convolutional networks for skeleton-based action recognition. InProceedings of the AAAI conference on arti- ficial intelligence, 2018. 1, 2

    2018

  69. [77]

    Learning to use chopsticks in diverse gripping styles

    Zeshi Yang, Kangkang Yin, and Libin Liu. Learning to use chopsticks in diverse gripping styles. 2022. 2

    2022

  70. [78]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. InAAAI, pages 9127–9134, 2019. 2, 7

    2019

  71. [79]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 6

    2023

  72. [80]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding.arXiv preprint arXiv:2306.02858, 2023. 3, 7, 1, 2

    2023 arXiv

  73. [81]

    T2m-gpt: Generating human motion from textual descriptions with discrete representations.arXiv preprint arXiv:2301.06052, 2023

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations.arXiv preprint arXiv:2301.06052, 2023. 2

    2023 arXiv

  74. [82]

    Motiondif- fuse: Text-driven human motion generation with diffusion model.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 2

    2024

  75. [83]

    Pointclip: Point cloud understanding by clip

    Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xu- peng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8552–8562, 2022. 3

    2022

  76. [84]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023. 7

    2023 arXiv

  77. [85]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 18243–18252, 2024. 3

    2024

  78. [86]

    Avatargpt: All- in-one framework for motion understanding planning gener- ation and beyond

    Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All- in-one framework for motion understanding planning gener- ation and beyond. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1357–1366, 2024. 2

    2024

  79. [87]

    Languagebind: Extending video-language pretrain- ing to n-modality by language-based semantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretrain- ing to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023. 2, 3, 5 Hu...

    arXiv

  80. [88]

    Limited by GPU memory, we only take 8 key frames for each video following MotionLLM [5]

    Implementation Details We resize all video frames to a resolution of 320 × 256. Limited by GPU memory, we only take 8 key frames for each video following MotionLLM [5]. And we represent the motion following Humanml3D [22]. During the encoder pre-training stage, we set the mask...

  81. [89]

    Visualizations for Motion Understanding Additional visualization results for motion understanding are provided in Fig

    Additional Visualizations 7.1. Visualizations for Motion Understanding Additional visualization results for motion understanding are provided in Fig. 9 and Fig. 10. These examples highlight our method’s ability to capture detailed motion information and perform accurate motion...

  82. [90]

    Evaluation on BABEL-QA Benchmark For the BABEL-QA benchmark, we extend the evalua- tion protocol used in previous multi-modality LLMs evalu- ations [80]

    Evaluation Details 8.1. Evaluation on BABEL-QA Benchmark For the BABEL-QA benchmark, we extend the evalua- tion protocol used in previous multi-modality LLMs evalu- ations [80]. The evaluation involves scoring the correctness of each Q&A pair generated by the LLMs. Detailed in...

  83. [91]

    Structure of Hyper-Networks We utilize hyper-networksH u andH m, for velocity recon- struction, following InfoCon [46]

    Network Structure for Encoder Pretraining 9.1. Structure of Hyper-Networks We utilize hyper-networksH u andH m, for velocity recon- struction, following InfoCon [46]. We split the discrete fea- turesD u andD m intoKparts. We generateK−1layer- network weights using the firstK−1...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.