Multimodal Foundation Model for Cross-Modal Retrieval and Activity Recognition Tasks

Koki Matsuishi; Kosuke Ukita; Tsuyoshi Okita

REVIEW 4 major objections 4 minor 23 references

Multimodal Foundation Model for Cross-Modal Retrieval and Activity Recognition Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AURA-MFM aligns IMU signals with third-person video, motion capture, and text in one CLIP-based space, and reports large gains over IMU2CLIP in zero-shot activity recognition.

desk verdict The four-modality extension and the internal Transformer-vs-RNN comparison are reasonably convincing, but the headline 'surpasses existing methods' rests on an unmatched baseline and a zero-shot protocol that is not zero-shot. read the letter →

arxiv 2506.03174 v1 pith:DOH3WMYG submitted 2025-05-29 cs.CV cs.AIcs.LG

Koki Matsuishi , Kosuke Ukita , Tsuyoshi Okita This is my paper

classification cs.CVcs.AIcs.LG

keywords multimodalfoundationmodelIMUcross-modalretrievalhumanactivityrecognitionzero-shotclassificationcontrastivelearningmotioncapturetransformerencoder

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes AURA-MFM, a multimodal foundation model that aligns inertial measurement unit (IMU) signals from wearable devices with third-person video, motion capture, and text in a single shared embedding space. It claims this is the first such model to include motion capture and third-person video for detailed full-body activity understanding, going beyond first-person-only approaches. Using cross-modal contrastive learning with a Transformer-based IMU encoder, AURA-MFM reports consistent improvements over the IMU2CLIP baseline across retrieval and activity recognition on Ego-Exo4D, with the largest gain in zero-shot action recognition: F1 0.6226 and accuracy 0.7320 versus IMU2CLIP's F1 0.0747 and accuracy 0.1961. If correct, the model would let wearable motion data be searched, described, and classified through natural language without task-specific labeled training.

What carries the argument

The load-bearing object is the shared latent embedding space constructed by symmetric InfoNCE contrastive loss (Equations 1-3), which pulls unit-normalized embeddings of IMU windows close to their corresponding text, video, or motion capture embeddings while pushing unrelated pairs apart. The text and video encoders are frozen pretrained CLIP encoders, and the IMU and mocap encoders are Transformer-based signal encoders with patch embedding, multi-head self-attention, layer normalization, and skip connections. Training proceeds progressively: the IMU encoder is first aligned to CLIP text or video, then the mocap encoder is trained against the IMU encoder. This machinery is what allows raw IMU waveforms to be compared directly with arbitrary text prompts and to retrieve matching video or mocap sequences.

What would settle it

Retrain IMU2CLIP's RNN encoder on the same Ego-Exo4D IMU-text and IMU-video pairs used for AURA-MFM and rerun the zero-shot, transfer, and fine-tuning evaluations; if the F1 gap shrinks from roughly 0.55 toward zero, the claimed superiority is largely dataset mismatch. Separately, evaluate zero-shot classification on a held-out activity vocabulary whose class names never appeared in the text-matching soft-label step.

Watch

Extended reading notes

Core claim

The paper's central claim is that integrating IMU data with third-person video, motion capture, and text—rather than only first-person video and text—yields a more complete representation of human activity, and that a Transformer-based IMU encoder learns better latent representations than the RNN-based encoder used in IMU2CLIP. AURA-MFM trains the IMU encoder with symmetric InfoNCE contrastive loss against fixed pretrained CLIP text and video encoders, then trains a motion capture encoder against the IMU encoder, so all four modalities share one CLIP-aligned embedding space. Reported results on Ego-Exo4D show AURA-MFM outperforming IMU2CLIP on all eight bidirectional retrieval tasks and on zero-shot, transfer-learning, and fine-tuning activity recognition settings. The paper also claims a qualitative ability to retrieve relevant text, mocap, and video from novel, freely written prompts not present in the test set.

Load-bearing premise

The central claim rests on treating the reported IMU2CLIP numbers as a fair baseline, but IMU2CLIP was trained on a different dataset and not retrained here, so part of the performance gap may come from the baseline being out of its original domain.

Editorial extensions

If this is right

Zero-shot activity recognition from IMU data becomes practical: a new activity can be recognized from its textual description without collecting labeled examples.
Third-person video and motion capture provide full-body movement detail that first-person video cannot, enabling finer-grained retrieval such as distinguishing a soccer trap from a pass.
The Transformer IMU encoder consistently outperforms the RNN-based encoder across most settings, suggesting that encoder architecture is a meaningful factor in cross-modal IMU alignment.
Cross-modal retrieval works in eight directional pairs, so a query from any one modality can retrieve matching content in any other, including novel text prompts.
Cross-domain transfer to PAMAP2 is much weaker than within-dataset performance, indicating that pretraining on a single synchronized multimodal dataset limits generalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension: Because IMU2CLIP was trained on Ego4D and evaluated on Ego-Exo4D without retraining, the true advantage of AURA-MFM may be smaller than the reported 8x zero-shot gap; a direct retrained baseline would settle it.
Extension: The zero-shot activity labels are generated by text matching using the same activity vocabulary that appears in the training text, so part of the zero-shot score may reflect vocabulary overlap rather than genuine generalization to unseen concepts.
Extension: The paper trains IMU against each other modality one pair at a time; training all four modalities jointly with a single objective could exploit interactions among video, mocap, and text that pairwise alignment misses.
Extension: If the IMU-text alignment holds beyond this dataset, it opens applications such as searching wearable sensor archives by prose descriptions or combining clinical notes with motion data for rehabilitation monitoring.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about arXiv:2506.03174 before deciding whether to engage. First, the paper does something genuinely new: it extends the IMU2CLIP contrastive recipe to four modalities (IMU, third-person video, motion capture, text) and shows that a Transformer IMU encoder beats an RNN encoder when trained on the same data. That internal comparison is controlled, and the gains are consistent across retrieval and classification. Second, the headline claim that AURA-MFM 'surpasses existing methods' is not supported by the experiments as reported. IMU2CLIP was pretrained on Ego4D and evaluated on Ego-Exo4D with no adaptation. The paper itself admits in Section 4.3 that 'the comparison is not entirely direct due to differences in datasets and experimental setups.' That is not a fair yardstick. The near-zero IMU2CLIP scores likely reflect domain shift, not architectural inferiority. The PAMAP2 results make this even clearer: on a truly different domain, their zero-shot expert-commentary Transformer gets F1 0.0293, worse than IMU2CLIP's 0.0404. So the claimed advantage does not generalize.

The zero-shot protocol is also misnamed. The soft labels come from text matching, and the eight Ego-Exo4D class names ('Soccer', 'Cooking', etc.) appear in the AAD and Expert Commentary text used for contrastive training. The IMU encoder has been trained on text containing the exact class vocabulary, so the evaluation is closer to a constrained vocabulary transfer than to zero-shot classification. This does not make the internal Transformer-vs-RNN comparison invalid, but it does mean the headline zero-shot numbers are not evidence of generalizable cross-modal alignment.

There are also smaller issues: the Transformer architecture is cited only as '[anonymous reference]', which is unverifiable, and the qualitative discussion in the Figure 5 heading lapses into first-person subjective evaluation. No code, error bars, or hyperparameter details are given, so the work is not reproducible as it stands.

The core idea—aligning IMU with third-person video and mocap in a CLIP-style space—is worth exploring, and the controlled encoder comparison is a useful data point. But the central comparative claim needs a real baseline retrained on Ego-Exo4D and a leakage-free zero-shot protocol. As submitted, I would reject it, though I would send it to referees rather than desk-reject it, because the underlying direction is reasonable and the internal comparison is fair. If the authors fix the baseline and the zero-shot evaluation, this could become a passable contribution to wearable-sensing activity recognition. I would not cite it in its current form, but I would bring it to a reading group to discuss the baseline and leakage issues.

Referee Report

4 major / 4 minor

Summary. The paper proposes AURA-MFM, a multimodal foundation model that aligns IMU, third-person video, motion capture, and text via contrastive learning with fixed CLIP encoders, using a Transformer-based encoder for IMU and mocap. The authors evaluate on cross-modal retrieval and human activity recognition using Ego-Exo4D and PAMAP2, reporting large gains over IMU2CLIP, particularly in zero-shot classification (F1 0.6226 vs 0.0747).

Significance. If the comparisons were properly controlled, the integration of four modalities and the Transformer encoder would be a meaningful contribution to multimodal activity understanding, and the paper provides a useful practical recipe (InfoNCE loss, fixed CLIP encoders, stepwise training). The authors are transparent about the dataset mismatch, and the inclusion of random-init baselines is a good practice. However, the central claim of surpassing existing methods is currently not supported by the evidence due to uncontrolled baselines and a leakage-prone zero-shot protocol.

major comments (4)

[Section 4.3, Tables 2 and 3] The comparison with IMU2CLIP is uncontrolled because IMU2CLIP was trained on Ego4D, not on the Ego-Exo4D data used here. The paper itself acknowledges this in Section 4.3: 'the comparison is not entirely direct due to differences in datasets and experimental setups.' The large gaps, e.g., zero-shot F1 0.6226 vs 0.0747, are confounded by domain shift in IMU distributions, sensor placement, and activity vocabularies. To support the claim of outperforming existing methods, the authors must retrain or fine-tune IMU2CLIP on Ego-Exo4D under the same preprocessing and evaluation protocol.
[Section 4.2, zero-shot classification] The 'zero-shot' activity labels are derived by text matching using the same eight activity names that appear in the AAD and Expert Commentary training texts. Because the IMU encoder is contrastively trained against text containing those exact class names, the model has seen the class vocabulary during training. This makes the term 'zero-shot' misleading and the comparison with IMU2CLIP, which did not train on this vocabulary, partly reflects training-data overlap. The authors should use held-out classes absent from all training text, or at least quantify the vocabulary leakage.
[Section 4.3, Tables 3 and 4] The 'transfer learning' setting is not defined. The paper does not specify what is transferred, how the classifier is trained, or which data are used for adaptation. Without this definition, the transfer-learning results cannot be interpreted or reproduced.
[Section 4.3, Table 3] The fine-tuning results show that a randomly initialized Transformer achieves F1 0.6251 and accuracy 0.7511, which is comparable to or better than some AURA-MFM variants (e.g., RNN with Expert Commentary: F1 0.6228). This suggests that part of the reported gain may be attributable to the architecture rather than the multimodal contrastive pretraining. The authors should analyze the contribution of the contrastive alignment separately, e.g., by comparing the same architecture with and without pretraining.

minor comments (4)

[Section 3.2, Figure 3] The Transformer architecture is cited as '[anonymous reference]' and as 'the signal-based Transformer architecture designed for signal processing.' In a journal submission, the reference should be identified; if the review is double-blind, the anonymization should be handled without leaving a placeholder in the final text.
[Table 1] The rows 'IMU → 8 classes' and 'IMU → 18 classes' are not modalities; they are classification tasks. The table header 'Data Modality' should be adjusted or the rows should be labeled accordingly.
[Section 5 and throughout] There are numerous typos and grammatical errors, e.g., 'ubuquitous' (should be 'ubiquitous'), 'T ransfer' in Table 3, 'Modarity' in Table 1, and 'differnt' in Section 5. The manuscript needs a thorough proofread.
[Section 4.3, Figures 5 and 6] The qualitative analysis contains informal first-person narrative ('I think it is necessary to evaluate this action a little earlier...'), which is inconsistent with journal style and obscures the assessment. Please rewrite in a neutral, objective tone.

Circularity Check

1 steps flagged · score 4.0 of 10

Zero-shot classification is partially self-referential: the text-matched class labels are drawn from the same activity vocabulary used in the contrastive training text, so the headline F1 gain is partly a retrieval of training vocabulary rather than an independent zero-shot prediction.

fitted input called prediction [Section 4.2 (Task 2: Human Activity Recognition Using IMU Data) and Section 4.3 (Result 2, Table 3)]
"The activity labels are derived through a soft annotation process via text matching, which assigns labels based on semantic similarity between motion patterns and textual descriptions. ... Since classification relies on text-derived labels, we exclude evaluations that use encoders trained solely between Mocap and IMU data. ... the best-performing AURA-MFM model, trained with the AAD↔IMU pairing and using a Transformer-based encoder, achieved a much higher F1 score of 0.6226 and accuracy of 0.7320."

The contrastive training aligns the IMU encoder with AAD and Expert Commentary text; Section 4.1 says Expert Commentary 'includes expert annotations, activity names, and the duration', so the eight class names used as zero-shot labels are part of the training text vocabulary. The zero-shot classifier then labels an IMU window by text-matching similarity to those same class names. Thus the 'prediction' is a retrieval to text embeddings whose vocabulary was part of the training signal, not an unseen-class generalization. The headline 0.6226 F1 is therefore partially forced by the training objective, and the gap over IMU2CLIP is inflated because the baseline was trained on Ego4D without this vocabulary.

full rationale

The core of AURA-MFM is standard cross-modal contrastive learning: a Transformer-based IMU encoder and a Mocap encoder are trained from scratch against fixed CLIP text/video encoders and evaluated on held-out retrieval and fine-tuning splits. That derivation is not circular; the RNN-vs-Transformer comparison is run internally and does not depend on self-citation. The anonymous reference for the signal-transformer architecture is a missing reference rather than a load-bearing circular step, because its value is tested empirically, not assumed. The main circularity concern is the 'zero-shot' HAR protocol: class labels are generated by text matching from the same activity vocabulary present in the AAD/Expert Commentary training text, so the zero-shot result is partly a retrieval of training vocabulary and is not a clean out-of-distribution prediction. The paper's own acknowledgement that the IMU2CLIP comparison 'is not entirely direct due to differences in datasets and experimental setups' is a validity limitation, not a circularity, but it compounds the zero-shot issue. Overall, the central training method and most evaluations remain independent, so the circularity is partial rather than total.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central contribution is empirical, so the ledger lists the preprocessing and architectural choices that shape results, plus the domain assumptions about dataset validity. No invented entities are introduced.

free parameters (5)

Contrastive training hyperparameters (batch size, learning rate, epochs, temperature) = not reported
These affect alignment quality and are not stated; no code or config is provided.
Transformer IMU/mocap encoder hyperparameters (patch size, depth, heads, dims) = not reported
The architecture is adapted from an anonymous reference with only two modifications noted; exact settings are absent.
Sliding window duration = 5 s
Chosen for all Ego-Exo4D pairings; window size affects temporal context and pair counts (Table 1).
Resampling rates = 200 Hz (Ego-Exo4D from 1000 Hz; PAMAP2 upsampled from 100 Hz)
Chosen by hand; interpolation for PAMAP2 affects signal fidelity.
Text truncation length = 77 tokens
Capped to CLIP's token limit; truncation may drop relevant context.

assumptions (5)

domain assumption CLIP's fixed text and video encoders define a semantic space appropriate for human activity descriptions and third-person video frames.
Section 3.1 fixes CLIP parameters and aligns IMU/mocap embeddings to this space; if CLIP lacks temporal or full-body action semantics, the alignment target is weak.
domain assumption Ego-Exo4D provides synchronized and correctly paired IMU, mocap, video, and text streams after the authors' custom 5-second window construction.
Section 4.1 says the original dataset did not provide 5-second pairs and these were built by the authors; synchronization errors would degrade all downstream results.
ad hoc to paper The signal-based Transformer architecture referenced as [anonymous reference] exists and is applicable to IMU and mocap signals.
Section 3.2 and Figure 3 rely on this anonymous architecture; no citation is given, so the reader cannot verify its properties.
ad hoc to paper Text-based soft annotation via semantic matching yields correct activity labels for the 8 Ego-Exo4D classes and 18 PAMAP2 classes.
Section 4.2: 'activity labels are derived through a soft annotation process via text matching'; if class names leak into training captions, zero-shot scores are inflated.
standard math InfoNCE is the best contrastive loss for this setting, adopted from IMU2CLIP's experiments.
Section 3.1 cites IMU2CLIP's comparison; no new loss comparison is run.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multimodal Foundation Model for Cross-Modal Retrieval and Activity Recognition Tasks." pith.science (2026). https://pith.science/paper/DOH3WMYG

@misc{pith2026250603174,
  author       = {Pith},
  title        = {Pith review of: Multimodal Foundation Model for Cross-Modal Retrieval and Activity Recognition Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DOH3WMYG}},
  note         = {Machine review of arXiv:2506.03174}
}

read the original abstract

In recent years, the widespread adoption of wearable devices has highlighted the growing importance of behavior analysis using IMU. While applications span diverse fields such as healthcare and robotics, recent studies have increasingly focused on multimodal analysis, in addition to unimodal analysis. Several studies have proposed multimodal foundation models that incorporate first-person video and text data; however, these models still fall short in providing a detailed analysis of full-body human activity. To address this limitation, we propose Activity Understanding and Representations Alignment - Multimodal Foundation Model (AURA-MFM), a foundational model integrating four modalities: third-person video, motion capture, IMU, and text. By incorporating third-person video and motion capture data, the model enables a detailed and multidimensional understanding of human activity, which first-person perspectives alone fail to capture. Additionally, a Transformer-based IMU encoder is employed to enhance the model's overall performance. Experimental evaluations on retrieval and activity recognition tasks demonstrate that our model surpasses existing methods. Notably, in the zero-shot classification for action recognition, our method achieved significantly higher performance, with an F1-score of 0.6226 and an accuracy of 0.7320, whereas the existing method recorded an F1-score of 0.0747 and an accuracy of 0.1961.

Figures

Figures reproduced from arXiv: 2506.03174 by the authors.

**Figure 1.** AURA-MFM: A multimodal foundational model that integrates four modalities—text, motion capture, third-person perspective videos, and IMU (accelerometer and gyroscope). Each modality is processed by a dedicated encoder, and their synchronized embeddings are mapped into a shared latent space. By minimizing the distance between embeddings from different modalities, AURA-MFM aims to enhance the understanding of human ac… view at source ↗

**Figure 2.** Cross-Modal Contrastive Learning: We perform cross-modal contrastive learning between IMU data and other modalities. The text encoder and video encoder use the publicly available pre-trained CLIP parameters, and their parameters are fixed during training. Although the figure specifically illustrates training between text and IMU, the model is trained using three types of contrastive learning: video and IMU, motion c… view at source ↗

**Figure 3.** Encoder architectures for IMU and Mocap (Left: [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Task 1: A retrieval task that measures recall from arbitrary queries [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

**Figure 5.** Figure 5: Retrieval results for IMU→ Expert Commentary: This figure shows the retrieved text (Expert Commentary) from the Ego-Exo4D test set in response to an IMU query. The ground-truth activity label for the IMU query is ”Soccer Drills - Outside Trap and Outside Play.” The top…

**Figure 6.** Figure 6: Retrieval results for IMU → Mocap: The figure shows motion capture sequences retrieved based on IMU signals. The IMU query is the same as the one shown in [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

**Figure 7.** Figure 7: Retrieval results of IMU from freely created prompts (Prompt → IMU): This figure shows the retrieval results for the IMU modality when given a free-form text query that is not included in the test set. For reference, video frames synchronized with the IMU waveforms are…

**Figure 8.** Figure 8: Retrieval results of Video from freely created prompts (Prompt → Video): This figure shows the retrieval results for the video modality when given a free-form text query (identical to [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 21 canonical work pages

[1]

Imu2doppler: Cross-modal domain adaptation for doppler-based activity recogni- tion using imu data

Sejal Bhalla, Mayank Goel, and Rushil Khurana. Imu2doppler: Cross-modal domain adaptation for doppler-based activity recogni- tion using imu data. In Proceedings of the ACM on Interactive and Mobile and Wearable and Ubiquitous Technologies, volume 5, pages 1–20, 2021

work page 2021
[2]

On the opportunities and risks of foundation models

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Sim- ran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

arXiv 2021
[3]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

work page 1901
[4]

A simple framework for contrastive learning of visual rep- resentations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual rep- resentations. In Proceedings of the 37th International Conference on Machine Learning, 2020

work page 2020
[5]

Cocoa: Cross modality contrastive learning for sensor data

Shohreh deldari, Hao Xue, Aaqib Saeed, Daniel Smith, and Flora Salim. Cocoa: Cross modality contrastive learning for sensor data. In Proceedings of the ACM on Interactive and Mobile and Wearable and Ubiquitous Technologies, volume 6, pages 1–28, 2022

work page 2022
[6]

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. pages 4171–4186, 2019

work page 2019
[7]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021

work page 2021
[8]

Unsupervised representation learning by predicting image rotations

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th Inter- national Conference on Learning Representations, 2018. Multimodal Foundation Model for Cross-Modal Retrieval and Activity Recognition Tasks IJABC: International Journal of Activity and Behavior Computing 24

work page 2018

Show all 23 references

[9]

Image- bind: One embedding space to bind them all

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Image- bind: One embedding space to bind them all. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15180–15190, 2023

2023
[10]

Ego4d: Around the World in 3,000 Hours of Egocentric Video

Kristen Grauman, Andrew Westbury, Zachary Chavis Eugene Byrne, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, and et al. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In IEEE/CVF Computer Vision and Pattern Recognition (CVPR)...

2022
[11]

Bikram Boote

Kristen Grauman, Andrew Westbury, Kris Kitani Lorenzo Torre- sani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, and et al. Bikram Boote. Ego-exo4d: Un- derstanding skilled human activity from first- and third-person per- spectives. In P...

2024
[12]

Momentum contrast for unsupervised visual representation learning

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726–9735, June 2020

2020
[13]

Abowd, Nicholas D

Hyeokhyen Kwon, Catherine Tong, Harish Haresamudram, Yan Gao, Gregory D. Abowd, Nicholas D. Lane, and Thomas Pl¨ otz. Imutube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition. In Proceedings of the ACM on Interactive and Mobile a...

2020
[14]

Imu2clip: Language- grounded motion sensor translation with multimodal contrastive learning

Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, and Babak Damavandi. Imu2clip: Language- grounded motion sensor translation with multimodal contrastive learning. In Findings of the Association for Computational Linguis- tics: EMNLP 2023, pages 1324...

2023
[15]

Unsupervised learning of visual representations by solving jigsaw puzzles

Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Computer Vision – ECCV 2016, pages 69–84, 2016

2016
[16]

Berkeley mhad: A comprehensive multimodal hu- man action database

Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, Ren´ e Vidal, and Ruzena Bajcsy. Berkeley mhad: A comprehensive multimodal hu- man action database. In IEEE Workshop on Applications of Com- puter Vision WACV, page 53—60, 2013

2013
[17]

Representation learning with contrastive predictive coding

Aaron Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

2018 arXiv
[18]

Learning transferable visual models from natural language supervi- sion

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In International c...
[19]

Introducing a new benchmarked dataset for activity monitoring

Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th International Sympo- sium on Wearable Computers, pages 108–109, 2012

2012
[20]

Facenet: A unified embedding for face recognition and clustering

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015

2015
[21]

Catherine Tong, Jinchen Ge, and Nicholas D. Lane. Zero-shot learn- ing for imu-based activity recognition using video embeddings. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 5(4), 2022

2022
[22]

Atten- tion is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Atten- tion is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neu...

2017
[8763]

Multimodal Foundation Model for Cross-Modal Retrieval and Activity Recognition Tasks IJABC: International Journal of Activity and Behavior Computing 25

PMLR, 2021. Multimodal Foundation Model for Cross-Modal Retrieval and Activity Recognition Tasks IJABC: International Journal of Activity and Behavior Computing 25

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Imu2doppler: Cross-modal domain adaptation for doppler-based activity recogni- tion using imu data

Sejal Bhalla, Mayank Goel, and Rushil Khurana. Imu2doppler: Cross-modal domain adaptation for doppler-based activity recogni- tion using imu data. In Proceedings of the ACM on Interactive and Mobile and Wearable and Ubiquitous Technologies, volume 5, pages 1–20, 2021

work page 2021

[2] [2]

On the opportunities and risks of foundation models

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Sim- ran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

arXiv 2021

[3] [3]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

work page 1901

[4] [4]

A simple framework for contrastive learning of visual rep- resentations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual rep- resentations. In Proceedings of the 37th International Conference on Machine Learning, 2020

work page 2020

[5] [5]

Cocoa: Cross modality contrastive learning for sensor data

Shohreh deldari, Hao Xue, Aaqib Saeed, Daniel Smith, and Flora Salim. Cocoa: Cross modality contrastive learning for sensor data. In Proceedings of the ACM on Interactive and Mobile and Wearable and Ubiquitous Technologies, volume 6, pages 1–28, 2022

work page 2022

[6] [6]

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. pages 4171–4186, 2019

work page 2019

[7] [7]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021

work page 2021

[8] [8]

Unsupervised representation learning by predicting image rotations

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th Inter- national Conference on Learning Representations, 2018. Multimodal Foundation Model for Cross-Modal Retrieval and Activity Recognition Tasks IJABC: International Journal of Activity and Behavior Computing 24

work page 2018