REVIEW 4 major objections 7 minor 29 references
DeepHuMS: Deep Human Motion Signature for 3D Skeletal Sequences
T0 review · 4 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A learned motion signature beats recognition nets at 3D motion retrieval
desk verdict Plausible architecture and useful robustness tests, but the NTU top-1/top-10 numbers are internally inconsistent, so the SOTA claim is unverifiable as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a Siamese pair of RNNs whose contrastive loss (Eq. 1) enforces small Euclidean distance $D_w$ for similar pairs and margin $m$ for dissimilar pairs, optionally combined with cross-entropy (Eq. 2). Similarity between two sequences is defined by the joint-wise motion field, $MF[i,j]=F[i]-F[j]$, and the joint-wise motion distance, $MD[j]=\sum_{i=1}^{N-1}\|F[i+1][j]-F[i][j]\|$: the field encodes direction and which joints matter, while the distance keeps the metric non-degenerate when the field vanishes. To handle speed variation, the paper synthesizes faster and slower versions of each sequence and trains the network to embed them together, since the full-sequence motion field and distance are the same across resampling. This trajectory metric carries the entire self-supervised training signal, and it is what distinguishes the embedding from a recognition-only objective.
What would settle it
Take two sequences of different action classes whose smoothed joint trajectories are nearly identical except for a semantic detail, for example a hand raising to the head versus to the face. If nearest-neighbor retrieval in the DeepHuMS embedding ranks these two cross-class sequences above same-class sequences, the trajectory-similarity premise is falsified. A quantitative version is to compute the proposed motion-field and motion-distance score over held-out pairs and compare it with class agreement; zero correlation would mean the self-supervised signal carries no semantic information.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a deep recurrent network can learn a discriminative 3D motion signature from trajectory cues alone, without action labels, by treating the motion field $MF[i, j] = F[i] - F[j]$ and the per-joint motion distance $MD[j] = \sum_{i=1}^{N-1} \|F[i+1][j] - F[i][j]\|$ as the notion of similarity between sequences. The learned embedding is what does the retrieval work: a query sequence is embedded once and compared with stored embeddings by nearest neighbor, which makes retrieval fast and lets the same representation handle noisy or missing joints, different execution speeds, and partial sub-motions via a second network. The reported consequence is that this trajectory-supervised embedding outperforms embeddings trained purely for recognition, whose class-boundary geometry transfers poorly to ranking.
Load-bearing premise
The load-bearing premise is that two sequences are semantically similar exactly when their joint-wise motion field and motion distance are similar; since this hand-defined metric is the only supervision in the self-supervised setup, any class whose members are not closest under this metric will be embedded wrongly.
Editorial extensions
If this is right
- Retrieval reduces to a single nearest-neighbor pass over a 512-dimensional embedding, giving measured query times of 18 ms on NTU RGB+D and 0.8 ms on HDM05.
- Self-supervision alone yields inter-class matches that are closer in DTW frame error (31 mm) than the supervised variant (34 mm), so label-free training is not just a fallback.
- The descriptor inherits robustness to missing data: training with 20 percent of joints randomly removed still produces working precision-recall curves.
- Speed variations of the same motion collapse to nearby embeddings, so sequences sampled at different frame rates can be compared directly.
- A second RNN trained with L2 loss can map a sub-motion to the embedding of the full sequence, enabling retrieval of the longer sequence that contains the query.
Reading between the lines
- The paper does not test whether its trajectory metric matches semantic similarity in general; I infer that the method's ceiling is set by that metric, so a useful stress test is to search for two different actions whose joint trajectories nearly coincide and check whether the embedding ranks them as close.
- Because the contrastive signal uses whole-sequence motion field and distance, the descriptor likely blurs temporal order within a sequence; repetitive actions such as multiple identical cycles may map to similar embeddings even when the phase or count differs.
- The embedding's success at matching human-judged motion similarity, if confirmed, suggests the same trajectory objective could serve as pretraining for few-shot action recognition, where labels are scarce.
- The sub-motion network currently regresses to a full-sequence embedding with L2 loss; a direct extension would be to predict the temporal offset of the sub-motion within the sequence, turning retrieval into localization.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DeepHuMS, a deep RNN-based Siamese architecture that learns a discriminative 3D human motion descriptor for retrieval. The model is trained with a contrastive loss on hand-defined trajectory cues (motion field and motion distance), optionally combined with a classification loss, and is evaluated on NTU RGB+D and HDM05 for retrieval accuracy, precision-recall behavior, robustness to speed and missing joints, and sub-motion retrieval. The authors claim state-of-the-art retrieval performance over both hand-crafted retrieval methods and recognition-based embedding methods, and they provide a code link and supplementary video.
Significance. If the reported results are reproducible, DeepHuMS is a simple and potentially useful contribution to 3D motion retrieval, demonstrating that a contrastive loss based on trajectory similarity can produce an embedding that generalizes to large datasets and supports subsequence search. The self-supervised aspect is valuable given the limited availability of class labels, and the robustness experiments with speed variation and missing data address practically relevant issues. The paper also provides a public code repository link, which aids reproducibility. However, the headline quantitative claims are currently undermined by an ambiguous and internally inconsistent evaluation metric, the HDM05 precision-recall comparison lacks numeric values, and a directly relevant learned-retrieval baseline (Deep Signatures [27]) is not compared. The significance of the claimed state-of-the-art results therefore cannot be fully assessed without a corrected evaluation.
major comments (4)
- [§4.3, Table 1] The definition of "top-n retrieval accuracy" as "how many of the n retrieved results belong to the same class as the query motion" is ambiguous, and Table 1 reports top-1 accuracy (0.78) greater than top-10 accuracy (0.753) for DeepHuMS, with the same pattern for the baselines. Under any standard top-n retrieval metric that counts queries with a correct hit among the first n retrieved results, top-10 accuracy must be at least top-1 accuracy because the top-1 result is included in the top-10 list. If "top-10" instead denotes a different quantity such as precision@10 or the mean number of correct retrievals in the top-10, then it is not commensurate with the top-1 numbers reported for the baselines, and the comparison is invalid. The authors must define the metric precisely, recompute all numbers under that single definition, and report results that are internally consistent.
- [§4.4, Figure 4a] The HDM05 comparison asserts that "the area under the PR curve is far larger" for DeepHuMS than for AFMS, Feature Ranking, and UDFS, but the figure reports no numeric area or average precision values and no error bars or statistical significance tests. This makes the claimed superiority over existing 3D motion retrieval methods unverifiable. Please report numeric AP/mAP values with variance and describe how the PR curves are averaged over queries.
- [§4.4, Related Work] The paper cites Deep Signatures [27] as a learned auto-encoder-based method for motion retrieval but never compares against it in the experiments. Since the central claim is that DeepHuMS is a superior learned representation for retrieval, a direct comparison with [27] on HDM05 (or a concrete explanation of why such a comparison is not possible) is necessary to support that claim.
- [§3, Eq. (4)] The hand-defined trajectory similarity (motion distance and motion field) is the only training signal in the self-supervised setup, yet the paper provides only qualitative evidence (Figure 3) that this metric is a reliable proxy for semantic action similarity. A quantitative validation, such as k-NN purity under the raw metric or a correlation with class labels, is needed to support the claim that the self-supervised embedding is semantically meaningful.
minor comments (7)
- [Eq. (1)] The notation in Eq. (1) is hard to parse, and the sentence "Y is if the label value" is grammatically incomplete; please rewrite this section for clarity.
- [Figure 5] The t-SNE comparison in Figure 5 has no legend or color coding for classes, making it impossible to assess the claim that DeepHuMS produces a more clustered and discriminative space.
- [§4.5] The statement that the self-supervised setup improves per-frame DTW error from 34mm to 31mm is reported without describing the comparison protocol, the number of query sequences, or the variance across queries.
- [Figure 8] The captions "before training" and "after training" in Figure 8 are ambiguous; it should be clarified whether the curves show the raw similarity metric or the learned embedding.
- [Introduction, References] References [28] and [29] are pose-reconstruction papers that are not clearly related to the retrieval motivation; either connect them to the discussion or remove them.
- [Abstract] The abstract claims "State of the art results on all these fronts," which is stronger than the evidence supports given the metric inconsistencies and missing numeric PR results.
- [§4.5, Retrieval Time] The retrieval times of 18ms and 0.8ms are reported without specifying the search implementation (e.g., brute-force k-NN versus index-based) or the hardware, so the numbers are not reproducible.
Circularity Check
No significant circularity: the learned embedding is trained with a hand-defined trajectory similarity and evaluated on held-out test splits, so the retrieval claim does not reduce to its inputs.
full rationale
The derivation chain in DeepHuMS is self-contained and empirically grounded. The training signal is a contrastive loss (Eq. 1) applied to pairs deemed similar by the hand-defined motion-field and motion-distance similarity (Eqs. 3-4). This similarity is an input design choice, not a quantity fitted to retrieval performance. The retrieval evaluation is then performed with a nearest-neighbour search in the learned embedding space on test splits: NTU RGB+D uses a performer-wise split and HDM05 uses the first four performers for training and the last for testing (Sec. 4.1). Class-wise top-1/top-10 retrieval accuracy and DTW distance are external, class-based benchmarks rather than re-statements of the trajectory metric. The paper's self-citations [28,29] concern pose estimation/reconstruction and are not load-bearing for the retrieval claim. The reported top-1 (0.78) exceeding top-10 (0.753) in Table 1 is a metric-consistency or reporting concern, but under the review rules that is a correctness issue, not evidence that any prediction is equivalent to its inputs by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- Margin m in contrastive loss
- Embedding dimension =
512
- Speed augmentation factor =
2x
- Siamese pair-sampling distribution
- Supervised loss weight
assumptions (4)
- domain assumption Skeletal joint positions and their frame-to-frame differences are sufficient to characterize the semantic content of a human motion for retrieval.
- domain assumption The proposed similarity metric (joint-wise motion distance and motion field aggregated over the full sequence) captures the perceptual similarity of motions.
- domain assumption Uniformly subsampling or interpolating a sequence by 2x preserves the action category, so faster or slower versions can be treated as similar motions.
- domain assumption The RNN used (IndRNN, [19]) can learn a fixed-length embedding from variable-length skeleton sequences.
Cite this review
Pith. "Pith review of DeepHuMS: Deep Human Motion Signature for 3D Skeletal Sequences." pith.science (2026). https://pith.science/paper/LLTCLNYZ
@misc{pith2026190805750,
author = {Pith},
title = {Pith review of: DeepHuMS: Deep Human Motion Signature for 3D Skeletal Sequences},
year = {2026},
howpublished = {\url{https://pith.science/paper/LLTCLNYZ}},
note = {Machine review of arXiv:1908.05750}
}
read the original abstract
3D Human Motion Indexing and Retrieval is an interesting problem due to the rise of several data-driven applications aimed at analyzing and/or re-utilizing 3D human skeletal data, such as data-driven animation, analysis of sports bio-mechanics, human surveillance etc. Spatio-temporal articulations of humans, noisy/missing data, different speeds of the same motion etc. make it challenging and several of the existing state of the art methods use hand-craft features along with optimization based or histogram based comparison in order to perform retrieval. Further, they demonstrate it only for very small datasets and few classes. We make a case for using a learned representation that should recognize the motion as well as enforce a discriminative ranking. To that end, we propose, a 3D human motion descriptor learned using a deep network. Our learned embedding is generalizable and applicable to real-world data - addressing the aforementioned challenges and further enables sub-motion searching in its embedding space using another network. Our model exploits the inter-class similarity using trajectory cues, and performs far superior in a self-supervised setting. State of the art results on all these fronts is shown on two large scale 3D human motion datasets - NTU RGB+D and HDM05.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[27]
Wang, Yingying, and Michael Neff. ”Deep signatures for indexing and retrieval in large motion databases.” Conference on Motion in Games, pp. 37-45, 2015
work page 2015
-
[1]
M¨ uller, M., R¨ oder, T., Clausen, M., Eberhardt, B., Kr¨ uger, B. & Weber, A. ”Doc- umentation mocap database hdm05.” Technical report, No. CG-2007-2, ISSN 1610- 8892, Universitt Bonn, June 2007
work page 2007
-
[2]
Shahroudy, Amir, Jun Liu, Tian-Tsong Ng, and Gang Wang. ”Ntu rgb+ d: A large scale dataset for 3d human activity analysis.” Computer Vision and Pattern Recog- nition, pp. 1010-1019. 2016
work page 2016
-
[3]
Junejo, I., Dexter, E., Laptev, I. & Perez, P. ”View-independent action recognition from temporal self-similarities.” Transactions On Pattern Analysis And Machine Intelligence. 33, 172–185, 2010
work page 2010
-
[4]
”Efficient and robust anno- tation of motion capture data.” Eurogr
M¨ uller, Meinard, Andreas Baak, and Hans-Peter Seidel. ”Efficient and robust anno- tation of motion capture data.” Eurogr. Symp. on Computer Animation, pp. 17-26. 2009
work page 2009
- [5]
-
[6]
Ramezani, M. & Yaghmaee, F. ”Motion pattern based representation for improving human action retrieval.”Multimedia Tools And Applications. 77, 26009–26032, 2018
work page 2018
-
[7]
”Retrieval and visualization of human motion data via stick figures.” Computer Graphics Forum, vol
Choi, Myung Geol, Kyungyong Yang, Takeo Igarashi, Jun Mitani, and Jehee Lee. ”Retrieval and visualization of human motion data via stick figures.” Computer Graphics Forum, vol. 31, no. 7, pp. 2057-2065. 2012
work page 2012
Show all 29 references
-
[8]
”Motion retrieval using probability graph model.” International Symposium on Computational Intel- ligence and Design, vol
Xiao, Qinkun, Junfang Li, Yi Wang, Zhao Li, and Haiyun Wang. ”Motion retrieval using probability graph model.” International Symposium on Computational Intel- ligence and Design, vol. 2, pp. 150-153. 2013
2013
-
[9]
& Feng, Y
Qi, T., Xiao, J., Zhuang, Y., Zhang, H., Yang, X., Zhang, J. & Feng, Y. ”Real-time Motion Data Annotation via Action String.” Comput. Animat. Virtual Worlds . 25, 293–302, 2014
2014
-
[10]
& Xiao, J
Chen, C., Zhuang, Y., Nie, F., Yang, Y., Wu, F. & Xiao, J. ”Learning a 3D human pose distance metric from geometric pose descriptor.” Transactions On Visualiza- tion And Computer Graphics . 17, 1676–1689, 2010. 14 N Battan, A Venkat et al
2010
-
[11]
”Motion templates for automatic classification and retrieval of motion capture data.” In Proceedings of the 2006 Eurogr
M¨ uller, Meinard, and Tido Rder. ”Motion templates for automatic classification and retrieval of motion capture data.” In Proceedings of the 2006 Eurogr. Symp. on Computer animation, pp. 137-146. Eurographics Association, 2006
2006
-
[12]
& Zhang, J
Wang, Z., Feng, Y., Qi, T., Yang, X. & Zhang, J. ”Adaptive multi-view feature selection for human motion retrieval.” Signal Processing. 120 pp. 691–701, 2016
2016
-
[13]
Aggarwal
Xia, Lu, Chia-Chih Chen, and Jake K. Aggarwal. ”View invariant human action recognition using histograms of 3d joints.” Computer Vision and Pattern Recogni- tion Workshops, pp. 20-27. IEEE, 2012
2012
-
[14]
”Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition.” Computer Vision and Pattern Recognition, pp
Li, Maosen, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. ”Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition.” Computer Vision and Pattern Recognition, pp. 3595-3603, 2019
2019
-
[15]
”Skeleton-Based Action Recognition with Directed Graph Neural Networks.” Conference on Computer Vi- sion and Pattern Recognition, pp
Shi, Lei, Yifan Zhang, Jian Cheng, and Hanqing Lu. ”Skeleton-Based Action Recognition with Directed Graph Neural Networks.” Conference on Computer Vi- sion and Pattern Recognition, pp. 7912-7921. 2019
2019
-
[16]
Yan, Sijie, Yuanjun Xiong, and Dahua Lin.” Spatial temporal graph convolutional networks for skeleton-based action recognition.” In Thirty-Second AAAI Conference on Artificial Intelligence. 2018
2018
-
[17]
& Yuan, J
Wang, J., Liu, Z., Wu, Y. & Yuan, J. ”Learning actionlet ensemble for 3D human action recognition.” Transactions On Pattern Analysis And Machine Intelligence . 36, 914–927, 2013
2013
-
[18]
& Nikolaidis, N
Kapsouras, I. & Nikolaidis, N. ”Action recognition on motion capture data using a dynemes and forward differences representation.” Journal Of Visual Commun. And Image Rep.. 25, 1432–1445, 2014
2014
-
[19]
& Gao, Y
Li, S., Li, W., Cook, C., Zhu, C. & Gao, Y. Independently Recurrent Neural Net- work (IndRNN): Building A Longer and Deeper RNN. 2018 Conference On Com- puter Vision And Pattern Recognition , 2018
2018
-
[20]
& Luo, J
Li, Q., Qiu, Z., Yao, T., Mei, T., Rui, Y. & Luo, J. Action Recognition by Learning Deep Multi-Granular Spatio-Temporal Video Representation, 2016
2016
-
[21]
”Deep progressive reinforcement learning for skeleton-based action recognition.” Computer Vision and Pattern Recognition, pp
Tang, Yansong, Yi Tian, Jiwen Lu, Peiyang Li, and Jie Zhou. ”Deep progressive reinforcement learning for skeleton-based action recognition.” Computer Vision and Pattern Recognition, pp. 5323-5332. 2018
2018
-
[22]
& Lee, T
Chao, M., Lin, C., Assa, J. & Lee, T. Human motion retrieval from hand-drawn sketch. Transactions On Visualization And Computer Graphics . 18, 729–740, 2011
2011
-
[23]
Li, C., Zhong, Q., Xie, D. & Pu, S. ”Co-occurrence Feature Learning from Skele- ton Data for Action Recognition and Detection with Hierarchical Aggregation”. International Joint Conference On Artificial Intelligence . 2018
2018
-
[24]
& Bajcsy, R
Ofli, F., Chaudhry, R., Kurillo, G., Vidal, R. & Bajcsy, R. ”Sequence of the most informative joints (smij): A new representation for human skeletal action recogni- tion.” Journal Of Visual Commun. And Image Rep. . 25, 24–38, 2014
2014
-
[25]
& El-saban, M
Gowayyed, M., Torki, M., Hussein, M. & El-saban, M. ”Histogram of oriented dis- placements (HOD): Describing trajectories of human joints for action recognition”, International Joint Conference on Artificial Intelligence, pp. 1351–1357, 2013
2013
-
[26]
& Zezula, P
Carrara, F., Elias, P., Sedmidubsky, J. & Zezula, P. ”LSTM-based real-time ac- tion detection and prediction in human motion streams.” Multimedia Tools And Applications. pp. 1–23, 2019
2019
-
[28]
”HumanMeshNet: Polygonal Mesh Recovery of Humans.” arXiv preprint arXiv:1908.06544, 2019
Venkat, Abbhinav, et al. ”HumanMeshNet: Polygonal Mesh Recovery of Humans.” arXiv preprint arXiv:1908.06544, 2019
1908 arXiv
-
[29]
”Deep textured 3d re- construction of human bodies.” arXiv preprint arXiv:1809.06547 2018
Venkat, Abbhinav, Sai Sagar Jinka, and Avinash Sharma. ”Deep textured 3d re- construction of human bodies.” arXiv preprint arXiv:1809.06547 2018
2018 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.