Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture

Feng Xu; Jijunnan Li; Qi Ma; Shaohua Pan; Weifeng Hu; Xingkang Zhou; Xinyu Yi; Xiong Wang

RECORD

Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture

Shaohua Pan, Qi Ma, Xinyu Yi, Weifeng Hu, Xiong Wang, Xingkang Zhou, Jijunnan Li, Feng Xu

Computer Vision and Pattern Recognition · Submitted 1 Sep 2023

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Quantitative and qualitative results demonstrate that by delicately designing the fusion method, our technique significantly outperforms the state-of-the-art vision, IMU, and combined methods on both global orientation and local pose estimation.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2309.00310 (version 1)

Pith IDLLOKZ6EC, Pith's permanent record of this paper and its review

CitationFeng Xu, Jijunnan Li, Qi Ma, Shaohua Pan, Weifeng Hu, Xingkang Zhou, Xinyu Yi, Xiong Wang. "Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture." arXiv:2309.00310 (2023). https://pith.science/paper/LLOKZ6EC

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,028 input tokens.

What the abstract itself states, as probabilities: claims first 0.30 · states limitation 0.12 · human participants 0.42 · controlled comparison 0.59 · randomized assignment 0.01 · states quantitative result 0.08 · compares with prior results 0.96 · mentions code or data availability 0.98.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

Either RGB images or inertial signals have been used for the task of motion capture (mocap), but combining them together is a new and interesting topic. We believe that the combination is complementary and able to solve the inherent difficulties of using one modality input, including occlusions, extreme lighting/texture, and out-of-view for visual mocap and global drifts for inertial mocap. To this end, we propose a method that fuses monocular images and sparse IMUs for real-time human motion capture. Our method contains a dual coordinate strategy to fully explore the IMU signals with different goals in motion capture. To be specific, besides one branch transforming the IMU signals to the camera coordinate system to combine with the image information, there is another branch to learn from the IMU signals in the body root coordinate system to better estimate body poses. Furthermore, a hidden state feedback mechanism is proposed for both two branches to compensate for their own drawbacks in extreme input cases. Thus our method can easily switch between the two kinds of signals or combine them in different cases to achieve a robust mocap. %The two divided parts can help each other for better mocap results under different conditions. Quantitative and qualitative results demonstrate that by delicately designing the fusion method, our technique significantly outperforms the state-of-the-art vision, IMU, and combined methods on both global orientation and local pose estimation. Our codes are available for research at https://shaohua-pan.github.io/robustcap-page/.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Either RGB images or inertial signals have been used for the task of motion capture (mocap), but combining them together is a new and interesting topic.	background 0.87	0.05	0.03	asserted 0.99
s1	We believe that the combination is complementary and able to solve the inherent difficulties of using one modality input, including occlusions, extreme lighting/texture, and out-of-view for visual mocap and global drifts for inertial mocap.	background 0.71	0.11	0.08	hedged possibility 0.47
s2	To this end, we propose a method that fuses monocular images and sparse IMUs for real-time human motion capture.	contribution 0.99	0.85	0.06	asserted 0.97
s3	Our method contains a dual coordinate strategy to fully explore the IMU signals with different goals in motion capture.	method 0.69	0.87	0.03	asserted 0.99
s4	To be specific, besides one branch transforming the IMU signals to the camera coordinate system to combine with the image information, there is another branch to learn from the IMU signals in the body root coordinate system to better estimate body poses.	method 0.99	0.90	0.03	asserted 0.99
s5	Furthermore, a hidden state feedback mechanism is proposed for both two branches to compensate for their own drawbacks in extreme input cases.	method 0.85	0.90	0.03	asserted 0.99
s6	Thus our method can easily switch between the two kinds of signals or combine them in different cases to achieve a robust mocap. %The two divided parts can help each other for better mocap results under different conditions.	method 0.72	0.67	0.11	asserted 0.85
s7	Quantitative and qualitative results demonstrate that by delicately designing the fusion method, our technique significantly outperforms the state-of-the-art vision, IMU, and combined methods on both global orientation and local pose estimation.	result 1.00	0.96	0.89	asserted 1.00
s8	Our codes are available for research at https://shaohua-pan.github.io/robustcap-page/.	contribution 0.51	0.52	0.02	asserted 0.94

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Either RGB images or inertial signals have been used for the task of motion capture (mocap), but combining them together is a new and interesting topic. We believe that the combination is complementary and able to solve the inherent difficulties of using one modality input, including occlusions, extreme lighting/texture, and out-of-view for visual mocap and global drifts for inertial mocap. To this end, we propose a method that fuses monocular images and sparse IMUs for real-time human motion capture. Our method contains a dual coordinate strategy to fully explore the IMU signals with different goals in motion capture. To be specific, besides one branch transforming the IMU signals to the camera coordinate system to combine with the image information, there is another branch to learn from the IMU signals in the body root coordinate system to better estimate body poses. Furthermore, a hidden state feedback mechanism is proposed for both two branches to compensate for their own drawbacks in extreme input cases. Thus our method can easily switch between the two kinds of signals or combine them in different cases to achieve a robust mocap. %The two divided parts can help each other for better mocap results under different conditions. Quantitative and qualitative results demonstrate that by delicately designing the fusion method, our technique significantly outperforms the state-of-the-art vision, IMU, and combined methods on both global orientation and local pose estimation. Our codes are available for research at https://shaohua-pan.github.io/robustcap-page/.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

2 arXiv papers cite this in 2 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it

Cites it as background 2 papers

The other category of methods [ref] uses six 3DOF IMUs (inertial measurement units) worn on the user's head, forearms, pelvis, and lower legs respectively for HMT.

HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations · Introduction · 0.96
To address the ongoing difficulties in achieving precise joint angle and position estimations, researchers have reconsidered the use of external [ref] or body-worn [ref] cameras for visual-inertial tracking.

Transformer IMU Calibrator: Dynamic On-body IMU Calibration for Inertial Motion Capture · Inertial Motion Capture · 0.84

What it cites · 1 sentence without a confident label

Uses the method or tool of 1 paper

We then reproject the 2D keypoints, detected by MediaPipe [ref] , onto the $Z$ = 1 plane using camera intrinsics to generalize to various camera settings.

MediaPipe: A Framework for Building Perception Pipelines · Local Pose Estimation · 1.00

Uses data or values from 1 paper

We use the AIST++ dataset [ref] and the AMASS dataset [ref] for training.

AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ · Implementation Details · 1.00

Compares with 1 paper

We perform the comparisons with the state-of-the-art vision-based methods ROMP [ref] , PARE [ref] , IMU-based methods TIP [ref] , PIP [ref] and combined method HybridCap [ref] , VIP [ref] .

HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions · Comparisons · 1.00

Cites as support 1 paper

The translation estimation differs from pose estimation because pure inertial-based methods suffer from large drifts due to accumulated errors, as demonstrated in [ref] .

EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors · Global Translation Estimation · 0.99

Cites as background 2 papers

Depending on the type of visual input, the vision-inertia fusion methods can be divided into three categories: fusing multi-view video with IMUs [ref] , fusing monocular RGBD video with IMUs [ref] , and fusing monocular RGB video with IMUs [ref] .

HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions · Vision-inertia Fusion Methods · 0.99
Other works [ref] also focus on solving occlusions, and they design unique mechanisms, such as a deep generative motion infiller or a hypothetical bird's eye view.

Occluded Human Body Capture with Self-Supervised Spatial-Temporal Motion Prior · Vision-based Mocap Methods · 0.95

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.