Pith. sign in

REVIEW 4 major objections 5 minor 69 references

DeSPITE: Exploring Contrastive Deep Skeleton-Pointcloud-IMU-Text Embeddings for Advanced Point Cloud Human Activity Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that LiDAR point clouds can replace RGB as the binding visual modality in multi-modal contrastive learning, enabling cross-modal matching, temporal moment retrieval, and state-of-the-art point cloud human activity…

desk verdict DeSPITE delivers a genuinely novel LiDAR-centric multi-modal embedding and a useful new dataset, but the SOTA claims are overstated and the text-binding rests on a qualitative alignment check. read the letter →

arxiv 2506.13897 v3 pith:YVJCDENC submitted 2025-06-16 cs.CV

Thomas Kreutz , Max Mühlhäuser , Alejandro Sanchez Guinea This is my paper
classification cs.CV
keywords multi-modalcontrastivelearningLiDARpointcloudshumanactivityrecognitioncross-modalretrievaltemporalmomentskeletonposeIMUsignalsLIPD-Babel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DeSPITE is an attempt to put LiDAR point clouds on equal footing with RGB video in multi-modal contrastive learning. The paper argues that point cloud sequences, skeleton poses, IMU signals, and natural-language activity descriptions of the same motions can be pulled into one embedding space using a symmetric InfoNCE objective, with text used as a partial binding signal. To train and test this, it synchronizes the LIPD dataset's point clouds, skeletons, and IMU readings with Babel's frame-level text labels, creating LIPD-Babel. The authors claim this shared space unlocks tasks that were previously unavailable for point cloud activity understanding, including cross-modal matching, temporal moment retrieval, and cross-modal database retrieval, and that pre-training on it improves point cloud HAR beyond current state of the art on HMPEAR and MSR-Action3D. If true, LiDAR becomes a privacy-preserving visual anchor for human activity understanding rather than a sensing modality that needs RGB to be interpreted.

What carries the argument

The load-bearing object is the multi-modal contrastive training loop over LIPD-Babel. Four encoders, a point spatio-temporal transformer for point clouds, an LSTM for IMU, a transformer-based skeleton encoder, and a frozen language-image text encoder, produce embeddings for 24-frame windows. A symmetric InfoNCE loss is applied to all modality pairs among point cloud, skeleton, and IMU, with a masked text-binding term that only acts on batch elements that carry Babel labels. The dataset itself is the second piece of machinery: LIPD-Babel aligns LIPD's 10 FPS point cloud, skeleton, and IMU sequences to Babel's 30 FPS text labels by downsampling to 10 FPS, with sequence IDs from AMASS used to make the correspondence.

What would settle it

Take a random sample of LIPD-Babel sequences, compute a pose-similarity metric between the LIPD skeleton frames and the corresponding downsampled Babel skeleton frames at every time index, and measure the offset that maximizes agreement; if the median optimal offset is more than one frame from zero, the synchronization is wrong and the retrieval and HAR results would not be reproducible.

Watch

Extended reading notes

Core claim

The paper's central claim is that the natural correspondence between LiDAR point clouds, human skeletons, and IMU readings, which are recordings of the same body motion from different sensors, is enough to learn a joint embedding space without RGB images, and that adding text labels makes that space more semantically organized. Trained contrastively on paired 24-frame windows, the encoders map each modality to a shared 512-dimensional space where matched motion snippets have high cosine similarity and mismatched ones are pushed apart. The result is that a query in any one modality can retrieve the same moment or the same person in any other modality, and the learned representations transfer to single-modality HAR. The authors also report that text guidance helps downstream HAR but slightly hurts matching and temporal moment retrieval compared to text-free variants.

Load-bearing premise

The load-bearing premise is that LIPD's 10 FPS point cloud, skeleton, and IMU frames line up with Babel's text labels after downsampling the text to 10 FPS, a synchronization the paper verifies only by visually inspecting skeleton poses.

Editorial extensions

If this is right

  • Cross-modal matching: an IMU snippet can identify the correct person in a multi-person LiDAR scene, and skeleton snippets can do the same, which the authors frame as a privacy-preserving re-identification primitive.
  • Temporal moment retrieval: a short IMU, skeleton, or point cloud query can locate the matching moment inside a long point cloud sequence, with recall measured within about 1.5 seconds of ground truth.
  • Pre-training for HAR: fine-tuning a point cloud HAR model from DeSPITE-style embeddings outperforms prior uni-modal pre-training and supervised methods on MSR-Action3D and HMPEAR, despite domain shift in point density and sensor type.
  • Text as a partial binder: text labels improve downstream HAR fine-tuning, while models trained without text give better matching and moment retrieval, so the two objectives are not equally served by the same loss weighting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper leaves implicit: the same contrastive recipe should be testable with real LiDAR human activity data beyond LIPD, such as HMPEAR or future outdoor captures, to show whether the synthetic-to-real domain shift is handled by the embedding itself or only by downstream fine-tuning.
  • Because text-free variants win on matching and moment retrieval, one can expect a two-stage design, train a text-free alignment first and then distill text semantics only into the downstream classifier, to get both properties at once; the paper does not explore this.
  • The masking of unlabeled batch elements means the method can scale to large unlabeled LiDAR plus IMU plus skeleton corpora with only sparse text annotations, so extending LIPD-Babel with additional unlabeled sequences should continue improving the sensor-side alignment even without new text.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. DeSPITE combines the LIPD LiDAR point-cloud/IMU/skeleton dataset with BABEL text annotations to construct 'LIPD-Babel' (v1 for matching/retrieval, v2 for HAR), and trains modality encoders (PST-Transformer for point clouds, LSTM for IMU, ACTOR for skeletons, frozen CLIP for text) with an InfoNCE-style contrastive loss, masking text when unavailable. The paper evaluates many modality subsets (DeSPITE, DeSPIE, DePITE, etc.) on three tasks: (1) multi-person cross-modal matching among point cloud, IMU, and skeleton, (2) temporal moment retrieval across modalities, and (3) pre-training for point cloud HAR on MSR-Action3D, HMPEAR, and LIPD-Babel-v2. The central findings are that matching and retrieval are learned effectively on held-out LIPD, TC, and DIP test sets; text-trained variants perform worse than text-free variants on matching and retrieval; and contrastive pre-training improves HAR fine-tuning, with claims of new state-of-the-art results on HMPEAR and MSR-Action3D. The paper also reports text-to-motion retrieval compared with TMR++ and qualitative retrieval examples from AMASS and LIPD.

Significance. If the results hold, the paper's core contribution is substantial: it establishes LiDAR point clouds as a viable privacy-preserving visual anchor for multi-modal contrastive pre-training, demonstrates novel cross-modal tasks (LiDAR-IMU and LiDAR-skeleton matching and retrieval), and provides a reusable evaluation suite based on LIPD-Babel. The matching and retrieval claims rest on externally held-out test sets (TC, DIP, and LIPD test), so the central embedding claim is empirically grounded and not circular. Strengths include the unusually honest finding that text alignment hurts matching/retrieval, the extensive per-modality ablations in Tables 6-8, and the clearly specified artificial multi-person scene protocol. However, the text-binding results depend on a dataset alignment that is verified only qualitatively, the SOTA claims overstate what Table 1 and the HMPEAR protocol support, and the self-authored LIPD-Babel-v2 split needs clarification regarding possible train/test leakage. The paper does not yet ship code or data, so reproducibility rests on the promised release.

major comments (4)
  1. [Supplementary Section 6; Eq. (4); Table 5] The temporal alignment between LIPD (10 FPS) and BABEL (30 FPS, downsampled) is verified only "qualitatively ... by plotting them next to each other" (Supplementary Section 6). Sub-frame or phase misalignment would corrupt the text-binding loss in Eq. (4), the text-to-motion retrieval results in Table 5, and the HAR gains attributed to text in Table 3, and the paper itself asserts this limitation. Please add a quantitative alignment check, e.g., per-sequence pose similarity between LIPD skeletons and downsampled BABEL skeletons, or a frame-offset search whose offset distribution is reported; a constant per-sequence offset is enough to invalidate the text pathway. Note that the point-cloud/skeleton/IMU matching and retrieval results do not depend on text alignment and would survive such a failure, so the fix is local to the text-related claims.
  2. [§1 (Contributions) and §5; Table 1] The manuscript claims "new state-of-the-art performance" on MSR-Action3D, but Table 1 reports KAN-HyperpointNet at 95.59, above DeSPITE's 95.47, and §4.6 itself concedes the method "nearly matches" that number. Please restrict the claim to "state of the art among pre-training methods," correct the statements in §1 and §5, and add the missing comparison to the table or explain why supervised-only methods are excluded from the SOTA claim.
  3. [§4.6; Table 2] The HMPEAR comparison is not apples-to-apples: the authors acknowledge in §4.6 that their setup "uses twice the frames of previous methods," yet Table 2 still reports the comparison as new SOTA against methods evaluated with shorter windows. Please report a 12-frame variant (or otherwise control for window length) so that the improvement over PEAR-Proj and PST-Transformer is not an artifact of longer temporal context.
  4. [§4.1; Supplementary Section 6; Table 3] For LIPD-Babel-v2, the test set is BABEL's validation split, and the paper states that "all the remaining data of LIPD" is added to the training set. Since LIPD contains the same AMASS sequences (ACCAD, BML-Movi, CMU, TC) that BABEL annotates, it must be clarified whether the LIPD copies of BABEL validation sequences were excluded from training; as written, the test set may leak into the training set, which would inflate the fine-tuning numbers in Table 3. Please specify the exclusion rule explicitly and re-split if necessary.
minor comments (5)
  1. [§4.1 vs. Table 4] The main text states that 85,551 training windows have text annotations in LIPD-Babel-v1, but Table 4 reports #Text Train = 187,641; please reconcile these numbers.
  2. [Tables 1-3; Figures 2-3] Results are reported without error bars or standard deviations; given the small margins (e.g., 95.47 vs. 95.59 on MSR-Action3D), reporting variance over at least a few seeds or scene-sampling runs would strengthen the claims.
  3. [Tables 1-2 captions] The dagger (†) used to mark PST-Transformer entries is never defined in the captions; please state what it denotes.
  4. [§4.8; Supplementary Section 6] There are minor typos, including "T o better assess" in §4.8 and "combning" in Supplementary Section 6; a proofreading pass is recommended.
  5. [§1 (Contributions)] The contribution statement promises release of pre-trained encoders, code, and data; a repository URL or release footnote would help reviewers and future users verify the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: matching, retrieval, and HAR results are evaluated on held-out and external benchmarks, and no prediction reduces to a fitted input by construction.

full rationale

The paper's central claim is empirical: a joint embedding space is learned with InfoNCE-style contrastive losses (Eqs. 2-6) over paired LIPD-Babel sequences, and the claimed outcomes are matching accuracies, recall@k scores, and HAR accuracies computed after training. Matching and temporal moment retrieval are evaluated on the held-out LIPD test set, TotalCapture, and DIP (Sections 4.4-4.5), none of which supply fitted values to the reported numbers. HAR claims are tested on HMPEAR and MSR-Action3D (Tables 1-2), which are external benchmarks with domain shifts, and on the LIPD-Babel-v2 validation split used only for fine-tuning. No fitted parameter is renamed as a prediction: the CLIP text encoder is frozen, and the text loss is masked on batch elements without text pairings (Eq. 4), so text-to-modality retrieval is not forced by construction. The closest same-source element is that LIPD-Babel-v2 HAR labels and the text pre-training signal both derive from Babel annotations; however, the text signal is vocabulary-level, the HAR evaluation uses a disjoint validation split, and the main SOTA claims rest on external datasets, so this is a transfer consideration rather than a circular reduction. The supplementary's qualitative-only check of LIPD-Babel temporal alignment (Supp. Section 6) is a correctness risk, not circularity: a misalignment would invalidate the data, but no equation in the paper equates the claimed output to its input. There are no load-bearing self-citations and no imported uniqueness theorems; the derivation is self-contained against external benchmarks.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The central claims are empirical, so the ledger contains no analytical free constants that masquerade as predictions. The main burdens are dataset-level assumptions: temporal alignment of Babel text with LIPD, the semantics of LIPD's synthetic sensors, and the validity of simulated multi-person scenes. The loss weights, temperature, window length, and point sampling are modeling choices that also shape the results.

free parameters (4)
  • Temperature tau in InfoNCE = learned, final value not reported
    Scales cosine similarity in Eq. (2) and controls the sharpness of the contrastive distribution; it is optimized during training.
  • Loss weights alpha and beta = 0.5 and 0.5
    Hand-chosen equal weighting of text-binding and modality-alignment losses in Eq. (6); no ablation is reported for these weights.
  • Sliding window length = 24 frames at 10 FPS
    All modalities are encoded as 24-frame windows; every downstream result depends on this temporal resolution choice.
  • Point subsampling counts = 256 (pretrain), 1024 (HMPEAR), 2048 (MSR-Action3D)
    The point cloud encoder input density is chosen per dataset and affects what the model can represent.
assumptions (5)
  • standard math InfoNCE with cosine similarity is a valid objective for aligning semantically paired sequences across modalities.
    Equations (1) and (2); this is the standard CLIP-style foundation the paper builds on.
  • domain assumption A frozen CLIP text encoder defines a suitable semantic anchor for human activity language.
    Section 3.1; the text branch is never trained, so the quality of the whole space depends on CLIP's text geometry.
  • domain assumption Babel's 30 FPS activity annotations remain correctly synchronized with LIPD after downsampling to 10 FPS.
    Supplementary Section 6; the mapping is sequence-ID based and was verified only by visual inspection of a few sequences.
  • domain assumption Synthetic point clouds and IMU signals generated by LIPD from AMASS SMPL poses inherit the activity semantics of the AMASS-derived Babel labels.
    Section 4.1; the synthetic modality generation is taken from LIPD and is not independently validated here.
  • domain assumption Artificial multi-person scenes formed by sampling single-person sequences are a valid proxy for real multi-person matching.
    Section 4.2; matching is measured on artificially assembled scenes, not on native multi-person LiDAR captures.
invented entities (1)
  • LIPD-Babel dataset
    purpose: Provides synchronized point cloud, skeleton, IMU, and text training and test data for all four modalities.
    Newly constructed by mapping LIPD sequence IDs to Babel annotations and downsampling; it is not released, so its validity cannot be checked externally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeSPITE: Exploring Contrastive Deep Skeleton-Pointcloud-IMU-Text Embeddings for Advanced Point Cloud Human Activity Understanding." pith.science (2026). https://pith.science/paper/YVJCDENC

@misc{pith2026250613897,
  author       = {Pith},
  title        = {Pith review of: DeSPITE: Exploring Contrastive Deep Skeleton-Pointcloud-IMU-Text Embeddings for Advanced Point Cloud Human Activity Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YVJCDENC}},
  note         = {Machine review of arXiv:2506.13897}
}
read the original abstract

Despite LiDAR (Light Detection and Ranging) being an effective privacy-preserving alternative to RGB cameras to perceive human activities, it remains largely underexplored in the context of multi-modal contrastive pre-training for human activity understanding (e.g., human activity recognition (HAR), retrieval, or person re-identification (RE-ID)). To close this gap, our work explores learning the correspondence between LiDAR point clouds, human skeleton poses, IMU data, and text in a joint embedding space. More specifically, we present DeSPITE, a Deep Skeleton-Pointcloud-IMU-Text Embedding model, which effectively learns a joint embedding space across these four modalities. At the heart of our empirical exploration, we have combined the existing LIPD and Babel datasets, which enabled us to synchronize data of all four modalities, allowing us to explore the learning of a new joint embedding space. Our experiments demonstrate novel human activity understanding tasks for point cloud sequences enabled through DeSPITE, including Skeleton<->Pointcloud<->IMU matching, retrieval, and temporal moment retrieval. Furthermore, we show that DeSPITE is an effective pre-training strategy for point cloud HAR through experiments in MSR-Action3D and HMPEAR.

Figures

Figures reproduced from arXiv: 2506.13897 by the authors.

Figure 1
Figure 1. DeSPITE links different data modalities that represent human activities and, therefore, have a natural correspondence into a joint [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Matching performance between all modality pairs [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 6
Figure 6. Point cloud→Skeleton and Skeleton→Point cloud Re￾trieval from AMASS and LIPD database, respectively [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figures from the paper (11 more)
Figure 5
Figure 5. Figure 5: IMU→Skeleton and IMU→Point cloud Retrieval from AMASS and LIPD database, respectively. visualize cosine similarity across a 1400-frame sequence containing diverse activities, with peaks aligning precisely with the ground truth timestamps. Despite no explicit train￾ing …
Figure 8
Figure 8. Figure 8: Heatmap to visualize the respective matching results on average across all modalities and datasets at a glance [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Heatmap to visualize the respective temporal moment retrieval results on average across all modalities and datasets at a glance [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Matching, subjects=2 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Matching, subjects=4 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Matching, subjects=8 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 16
Figure 16. Figure 16: Matching, subjects=24 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Matching, subjects=28 [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Matching, subjects=32 [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 22
Figure 22. Figure 22: Temporal Moment Retrieval, topk=50 10 [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]
Figure 23
Figure 23. Figure 23: Performance of computing matching scores based on 1,2, or 4 consecutive windows. The matching scores are presented [PITH_FULL_IMAGE:figures/full_fig_p022_23.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 56 canonical work pages

  1. [1]

    Simultaneous real-time human fall detection and reidentification based on multisensors data

    Matteo Bastico, Ver ´onica Ruiz Bejerano, and Alberto Belmonte-Hern´andez. Simultaneous real-time human fall detection and reidentification based on multisensors data. In Proceedings of the 15th International Conference on PErva- sive Technologies Related to Assistive Environments , pages 365–370, 2022. 1, 3

    work page 2022

  2. [2]

    A cross- dataset study for text-based 3d human motion retrieval

    L ´eore Bensabath, Mathis Petrovich, and Gul Varol. A cross- dataset study for text-based 3d human motion retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 1932–1940, 2024. 2

    work page 1932

  3. [3]

    Is space-time attention all you need for video understanding? In ICML, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, page 4, 2021. 7

    2021

  4. [4]

    Vitag: Online wifi fine time measurements aided vision-motion identity association in multi-person environ- ments

    Bryan Bo Cao, Abrar Alali, Hansi Liu, Nicholas Meegan, Marco Gruteser, Kristin Dana, Ashwin Ashok, and Shub- ham Jain. Vitag: Online wifi fine time measurements aided vision-motion identity association in multi-person environ- ments. In 2022 19th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON) , pages 19–27. IEEE, 2022. 1, 3

    work page 2022

  5. [5]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017. 7

    2017

  6. [6]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on ma- chine learning, pages 1597–1607. PmLR, 2020. 5

    2020

  7. [7]

    KAN-HyperpointNet for Point Cloud Sequence-Based 3D Human Action Recognition

    Zhaoyu Chen, Xing Li, Qian Huang, Qiang Geng, Tianjin Yang, and Shihao Han. Kan-hyperpointnet for point cloud sequence-based 3d human action recognition. arXiv preprint arXiv:2409.09444, 2024. 6, 7

    work page Pith review arXiv 2024

  8. [8]

    Advancing multi-modal sens- ing through expandable modality alignment

    Shenghong Dai, Shiqi Jiang, Yifan Yang, Ting Cao, Mo Li, Suman Banerjee, and Lili Qiu. Advancing multi-modal sens- ing through expandable modality alignment. arXiv preprint arXiv:2407.17777, 2024. 1, 2, 3

    arXiv 2024

Show all 69 references
  1. [9]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the asso- ciation for computational linguistics: human l...

    2019

  2. [10]

    Point 4d trans- former networks for spatio-temporal modeling in point cloud videos

    Hehe Fan, Yi Yang, and Mohan Kankanhalli. Point 4d trans- former networks for spatio-temporal modeling in point cloud videos. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 14204–14213,

  3. [11]

    Pstnet: Point spatio-temporal convolution on point cloud sequences

    Hehe Fan, Xin Yu, Yuhang Ding, Yi Yang, and Mohan Kankanhalli. Pstnet: Point spatio-temporal convolution on point cloud sequences. In International Conference on Learning Representations, 2021. 7

    2021

  4. [12]

    Deep hierarchical representation of point cloud videos via spatio-temporal decomposition

    Hehe Fan, Xin Yu, Yi Yang, and Mohan Kankanhalli. Deep hierarchical representation of point cloud videos via spatio-temporal decomposition. IEEE Transactions on Pat- tern Analysis and Machine Intelligence , 44(12):9918–9930,

  5. [13]

    Point spatio- temporal transformer networks for point cloud video mod- eling

    Hehe Fan, Yi Yang, and Mohan Kankanhalli. Point spatio- temporal transformer networks for point cloud video mod- eling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2181–2192, 2022. 5, 7

    2022

  6. [14]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 7

    2019

  7. [15]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 15180–15190, 2023....

    2023

  8. [16]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, 2022. 3

    2022

  9. [17]

    Separating the” chirp” from the” chat”: Self-supervised visual grounding of sound and language

    Mark Hamilton, Andrew Zisserman, John R Hershey, and William T Freeman. Separating the” chirp” from the” chat”: Self-supervised visual grounding of sound and language. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 13117–13127, 2024. 3

    2024

  10. [18]

    Masked motion prediction with semantic contrast for point cloud sequence learning

    Yuehui Han, Can Xu, Rui Xu, Jianjun Qian, and Jin Xie. Masked motion prediction with semantic contrast for point cloud sequence learning. In European Conference on Com- puter Vision, pages 414–431. Springer, 2024. 3, 7

    2024

  11. [19]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 5, 7

    1997

  12. [20]

    Wifi and vision-integrated fingerprint for smartphone-based self-localization in public indoor scenes

    Gang Huang, Zhaozheng Hu, Jie Wu, Hanbiao Xiao, and Fan Zhang. Wifi and vision-integrated fingerprint for smartphone-based self-localization in public indoor scenes. IEEE Internet of Things Journal, 7(8):6748–6761, 2020. 1

    2020

  13. [21]

    Deep iner- tial poser: Learning to reconstruct human pose from sparse inertial measurements in real time

    Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J Black, Otmar Hilliges, and Gerard Pons-Moll. Deep iner- tial poser: Learning to reconstruct human pose from sparse inertial measurements in real time. ACM Transactions on Graphics (TOG), 37(6):1–15, 2018. 4, 1

    2018

  14. [22]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  15. [23]

    Coopera- tive learning of audio and video models from self-supervised synchronization

    Bruno Korbar, Du Tran, and Lorenzo Torresani. Coopera- tive learning of audio and video models from self-supervised synchronization. Advances in Neural Information Process- ing Systems, 31, 2018. 3

    2018

  16. [24]

    Li- darcap: Long-range marker-less 3d human motion capture with lidar point clouds

    Jialian Li, Jingyi Zhang, Zhiyong Wang, Siqi Shen, Chenglu Wen, Yuexin Ma, Lan Xu, Jingyi Yu, and Cheng Wang. Li- darcap: Long-range marker-less 3d human motion capture with lidar point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognitio...

    2022

  17. [25]

    Uniformer: Unified transformer for efficient spatiotemporal representation learning

    Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint arXiv:2201.04676, 2022. 7 9

    2022 arXiv

  18. [26]

    Ai choreographer: Music conditioned 3d dance generation with aist++

    Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF international conference on computer vision , pages 13401– 13412, 2021. 1

    2021

  19. [27]

    Action recognition based on a bag of 3d points

    Wanqing Li, Zhengyou Zhang, and Zicheng Liu. Action recognition based on a bag of 3d points. In 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pages 9–14. IEEE, 2010. 2, 3, 4

    2010

  20. [28]

    En- hancing person identification for smart cities: Fusion of video surveillance and wearable device data based on ma- chine learning

    Jia-Ming Liang, Shashank Mishra, and Chun-Che Wu. En- hancing person identification for smart cities: Fusion of video surveillance and wearable device data based on ma- chine learning. IEEE Sensors Journal, 2024. 3

    2024

  21. [29]

    Hm- pear: A dataset for human pose estimation and action recog- nition

    Yitai Lin, Zhijie Wei, Wanfa Zhang, Xiping Lin, Yudi Dai, Chenglu Wen, Siqi Shen, Lan Xu, and Cheng Wang. Hm- pear: A dataset for human pose estimation and action recog- nition. In Proceedings of the 32nd ACM International Con- ference on Multimedia, pages 2069–2078, 2024. 1, ...

    2024

  22. [30]

    Ntu rgb+d 120: A large- scale benchmark for 3d human activity understanding

    Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C Kot. Ntu rgb+d 120: A large- scale benchmark for 3d human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(10):2684–2701, 2020. 3

    2020

  23. [31]

    Mamba4d: Efficient long-sequence point cloud video understanding with disentangled spatial-temporal state space models

    Jiuming Liu, Jinru Han, Lihao Liu, Angelica I Aviles-Rivero, Chaokang Jiang, Zhe Liu, and Hesheng Wang. Mamba4d: Efficient long-sequence point cloud video understanding with disentangled spatial-temporal state space models. arXiv preprint arXiv:2405.14338, 2024. 6, 7

    2024 arXiv

  24. [32]

    Meteor- net: Deep learning on dynamic 3d point cloud sequences

    Xingyu Liu, Mengyuan Yan, and Jeannette Bohg. Meteor- net: Deep learning on dynamic 3d point cloud sequences. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9246–9255, 2019. 7

    2019

  25. [33]

    Leaf: Learning frames for 4d point cloud sequence un- derstanding

    Yunze Liu, Junyu Chen, Zekai Zhang, Jingwei Huang, and Li Yi. Leaf: Learning frames for 4d point cloud sequence un- derstanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 604–613, 2023. 7

    2023

  26. [34]

    Smpl: A skinned multi- person linear model

    Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi- person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851–866. 2023. 1

    2023

  27. [35]

    Cross- modal contrastive pre-training for few-shot skeleton action recognition

    Mingqi Lu, Siyuan Yang, Xiaobo Lu, and Jun Liu. Cross- modal contrastive pre-training for few-shot skeleton action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2024. 3

    2024

  28. [36]

    Troje, Ger- ard Pons-Moll, and Michael J

    Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Ger- ard Pons-Moll, and Michael J. Black. AMASS: Archive of motion capture as surface shapes. In International Confer- ence on Computer Vision, pages 5442–5451, 2019. 3, 4, 1

    2019

  29. [37]

    Who goes there? exploiting silhouettes and wearable signals for subject identification in multi-person environments

    Alessandro Masullo, Tilo Burghardt, Dima Damen, Toby Perrett, and Majid Mirmehdi. Who goes there? exploiting silhouettes and wearable signals for subject identification in multi-person environments. InProceedings of the IEEE/CVF International Conference on Computer Vision Work...

    2019

  30. [38]

    Person re-id by fusion of video silhouettes and wearable signals for home monitoring appli- cations

    Alessandro Masullo, Tilo Burghardt, Dima Damen, Toby Perrett, and Majid Mirmehdi. Person re-id by fusion of video silhouettes and wearable signals for home monitoring appli- cations. Sensors, 20(9):2576, 2020. 1, 3

    2020

  31. [39]

    Imu2clip: Language-grounded motion sensor translation with multi- modal contrastive learning

    Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Apara- jita Saraf, Amy Bearman, and Babak Damavandi. Imu2clip: Language-grounded motion sensor translation with multi- modal contrastive learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 132...

    2023

  32. [40]

    Person tracking by fusing pos- ture data from uav video and wearable sensors.IEEE Sensors Journal, 22(24):24150–24160, 2022

    Alisher Mukashev, Lan-Da Van, Susanta Sharma, M Farhan Tandia, and Yu-Chee Tseng. Person tracking by fusing pos- ture data from uav video and wearable sensors.IEEE Sensors Journal, 22(24):24150–24160, 2022. 5

    2022

  33. [41]

    Repre- sentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 1, 3, 4

    2018 arXiv

  34. [42]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2

    2023 arXiv

  35. [43]

    Action- conditioned 3d human motion synthesis with transformer vae

    Mathis Petrovich, Michael J Black, and G ¨ul Varol. Action- conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 10985–10995, 2021. 5, 7

    2021

  36. [44]

    Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis

    Mathis Petrovich, Michael J Black, and G ¨ul Varol. Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9488–9497, 2023. 3, 5

    2023

  37. [45]

    Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J

    Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J. Black. BABEL: Bodies, action and behavior with english labels. In Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 722–731, 2021. 2, 3, 4, 1

    2021

  38. [46]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  39. [47]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

    2024 arXiv

  40. [48]

    Lidar-aid inertial poser: Large-scale human motion capture by sparse inertial and lidar sensors

    Yiming Ren, Chengfeng Zhao, Yannan He, Peishan Cong, Han Liang, Jingyi Yu, Lan Xu, and Yuexin Ma. Lidar-aid inertial poser: Large-scale human motion capture by sparse inertial and lidar sensors. IEEE Transactions on Visualiza- tion and Computer Graphics, 29(5):2337–2347, 2023....

    2023

  41. [49]

    An overview of gradient descent optimiza- tion algorithms

    Sebastian Ruder. An overview of gradient descent optimiza- tion algorithms. arXiv preprint arXiv:1609.04747, 2016. 5 10

    2016 arXiv

  42. [50]

    Ntu rgb+d: A large scale dataset for 3d human activity anal- ysis

    Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity anal- ysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010–1019, 2016. 3

    2016

  43. [51]

    Masked spatio-temporal structure prediction for self- supervised learning on point cloud videos

    Zhiqiang Shen, Xiaoxiao Sheng, Hehe Fan, Longguang Wang, Yulan Guo, Qiong Liu, Hao Wen, and Xi Zhou. Masked spatio-temporal structure prediction for self- supervised learning on point cloud videos. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, p...

    2023

  44. [52]

    Pointcmp: Contrastive mask prediction for self-supervised learning on point cloud videos

    Zhiqiang Shen, Xiaoxiao Sheng, Longguang Wang, Yulan Guo, Qiong Liu, and Xi Zhou. Pointcmp: Contrastive mask prediction for self-supervised learning on point cloud videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1212–1222, 2023. 3, 7

    2023

  45. [53]

    Point contrastive predic- tion with semantic clustering for self-supervised learning on point cloud videos

    Xiaoxiao Sheng, Zhiqiang Shen, Gang Xiao, Longguang Wang, Yulan Guo, and Hehe Fan. Point contrastive predic- tion with semantic clustering for self-supervised learning on point cloud videos. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 165...

  46. [54]

    Motionclip: Exposing human motion generation to clip space

    Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In European Conference on Com- puter Vision, pages 358–374. Springer, 2022. 1, 2, 3, 4

    2022

  47. [55]

    Total capture: 3d human pose estimation fusing video and inertial sensors

    Matthew Trumble, Andrew Gilbert, Charles Malleson, Adrian Hilton, and John Collomosse. Total capture: 3d human pose estimation fusing video and inertial sensors. In Proceedings of 28th British Machine Vision Conference , pages 1–13, 2017. 4, 1

    2017

  48. [56]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9 (11), 2008. 7

    2008

  49. [57]

    Recovering accurate 3d human pose in the wild using imus and a moving camera

    Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In European Conference on Computer Vision (ECCV), 2018. 1, 3

    2018

  50. [58]

    Self-supervised 4d spatio-temporal feature learning via order prediction of sequential point cloud clips

    Haiyan Wang, Liang Yang, Xuejian Rong, Jinglun Feng, and Yingli Tian. Self-supervised 4d spatio-temporal feature learning via order prediction of sequential point cloud clips. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 3762–377...

    2021

  51. [59]

    Pvnext: Rethinking network design and temporal motion for point cloud video recognition

    Jie Wang, Tingfa Xu, Lihe Ding, Xinjie Zhang, Long Bai, and Jianan Li. Pvnext: Rethinking network design and temporal motion for point cloud video recognition. In The Thirteenth International Conference on Learning Represen- tations. 6, 7

  52. [60]

    Point primitive transformer for long-term 4d point cloud video understanding

    Hao Wen, Yunze Liu, Jingwei Huang, Bo Duan, and Li Yi. Point primitive transformer for long-term 4d point cloud video understanding. In European Conference on Computer Vision, pages 19–35. Springer, 2022. 7

    2022

  53. [61]

    Human-centric scene understanding for 3d large-scale scenarios

    Yiteng Xu, Peishan Cong, Yichen Yao, Runnan Chen, Yue- nan Hou, Xinge Zhu, Xuming He, Jingyi Yu, and Yuexin Ma. Human-centric scene understanding for 3d large-scale scenarios. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20349–20359, 2023. 1, 3

    2023

  54. [62]

    Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing

    Jianfei Yang, He Huang, Yunjiao Zhou, Xinyan Chen, Yue- cong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing. Advances in Neural Information Processing Systems, 36:18756–18768, 2023. 3

    2023

  55. [63]

    Tri- modal motion retrieval by learning a joint embedding space

    Kangning Yin, Shihao Zou, Yuxuan Ge, and Zheng Tian. Tri- modal motion retrieval by learning a joint embedding space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1596–1605, 2024. 1, 2, 3

    2024

  56. [64]

    Complete-to-partial 4d distillation for self-supervised point cloud sequence representation learning

    Zhuoyang Zhang, Yuhao Dong, Yunze Liu, and Li Yi. Complete-to-partial 4d distillation for self-supervised point cloud sequence representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17661–17670, 2023. 3, 7

    2023

  57. [65]

    No pain, big gain: clas- sify dynamic point cloud sequences with static models by fitting feature-level space-time surfaces

    Jia-Xing Zhong, Kaichen Zhou, Qingyong Hu, Bing Wang, Niki Trigoni, and Andrew Markham. No pain, big gain: clas- sify dynamic point cloud sequences with static models by fitting feature-level space-time surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and...

    2022

  58. [66]

    dataset/sequencecategory/posesequence poses.npz

    More Information on LIPD-Babel LIPD [48] is a large-scale dataset combining LiDAR point clouds, IMU, and skeleton poses. It includes a mix of real and synthetic LiDAR point clouds and IMU measure- ments, taking advantage of the AMASS [36] motion cap- ture dataset. It combines ...

  59. [67]

    Specific Performance Scores for Matching and Temporal Moment Retrieval Figure 2 and Figure 3 in the main paper effectively visual- izes the differences in the performance of each model and parameters for matching and temporal moment retrieval, which makes a comparison at the s...

  60. [68]

    A Simple Improved Matching Algorithm to Associate Different Modalities in the Em- bedding Space In practice, we observe continuous streams of point cloud, skeleton, and IMU time series data. Therefore, a match- ing score can be computed not only on similarities between a singl...

  61. [69]

    We ablate linear/non-linear probing and freezing or fine-tuning each model when train- ing for HAR on LIPD-Babel-v2

    Ablation Study on LIPD-Babel-v2 for HAR We perform a large ablation study between all modalities for downstream classification. We ablate linear/non-linear probing and freezing or fine-tuning each model when train- ing for HAR on LIPD-Babel-v2. The results are presented in Tab...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.