Pith. sign in

REVIEW 5 cited by

IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.14395 v1 pith:D4QI374D submitted 2022-10-26 cs.CV cs.CLcs.LG

Seungwhan Moon , Andrea Madotto , Zhaojiang Lin , Alireza Dirafzoon , Aparajita Saraf , Amy Bearman , Babak Damavandi This is my paper
classification cs.CVcs.CLcs.LG
keywords imu2clipmotionapproachcontrastivepre-trainingsensorstextvideos
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos -- while preserving the transitivity across these modalities. We explore several new IMU-based applications that IMU2CLIP enables, such as motion-based media retrieval and natural language reasoning tasks with motion data. In addition, we show that IMU2CLIP can significantly improve the downstream performance when fine-tuned for each application (e.g. activity recognition), demonstrating the universal usage of IMU2CLIP as a new pre-trained resource. Our code will be made publicly available.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BasketHAR: A Multimodal Dataset for Human Activity Recognition and Sport Analysis in Basketball Training Scenarios

    cs.CV 2026-04 conditional novelty 7.0 of 10

    BasketHAR is a publicly released multimodal dataset of professional basketball training activities captured with inertial sensors, physiological signals, and video, accompanied by a baseline alignment method.

  2. Modular Retrieval-Augmented Generalization for Human Action Recognition

    eess.SP 2026-04 unverdicted novelty 6.0 of 10

    MoRA is a new retrieval-augmented module for IMU-based human activity recognition that uses uncertainty-adaptive fusion of retrieved motion patterns to improve model performance.

  3. SImpHAR: Advancing impedance-based human activity recognition using 3D simulation and text-to-motion models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    SImpHAR simulates bio-impedance signals from 3D motion and text, then uses contrastive pretraining and fine-tuning to improve impedance-based human activity recognition on two of three datasets.

  4. QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

    cs.AI 2026-07 conditional novelty 5.0 of 10

    QLPO resamples GRPO training groups to favor short correct and long incorrect responses, cutting reasoning length substantially while keeping accuracy roughly unchanged.

  5. EgoSelf: From Memory to Personalized Egocentric Assistant

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    EgoSelf uses graph-based memory of user interactions to derive personalized profiles and predict future behaviors for egocentric assistants.

Pith tools