Estimating Body and Hand Motion in an Ego-sensed World
read the original abstract
We present EgoAllo, a system for human motion estimation from a head-mounted device. Using only egocentric SLAM poses and images, EgoAllo guides sampling from a conditional diffusion model to estimate 3D body pose, height, and hand parameters that capture a device wearer's actions in the allocentric coordinate frame of the scene. To achieve this, our key insight is in representation: we propose spatial and temporal invariance criteria for improving model performance, from which we derive a head motion conditioning parameterization that improves estimation by up to 18%. We also show how the bodies estimated by our system can improve hand estimation: the resulting kinematic and temporal constraints can reduce world-frame errors in single-frame estimates by 40%. Project page: https://egoallo.github.io/
This paper has not been read by Pith yet.
Forward citations
Cited by 3 Pith papers
-
OpenWatch: A Multimodal Benchmark for Hand Gesture Recognition on Smartwatches
OpenWatch provides the first open multimodal smartwatch gesture dataset and benchmark, with MixToken and NormWear-Lora methods reaching 90% F1-score using 223k parameters versus 66% for 136M-parameter foundation models.
-
Ultra Diffusion Poser: Diffusion-Based Human Motion Tracking From Sparse Inertial Sensors and Ranging-Based Between-Sensor Distances
Ultra Diffusion Poser improves sparse inertial human pose estimation by reconstructing 3D sensor layouts from UWB ranging measurements and using UWB-Diffusion Guidance in a diffusion model, claiming up to 22% lower jo...
-
ARAPDiffusion: ARAP Regularization for Diffusion-Based Deformable Shape Space Learning
ARAPDiffusion adds ARAP regularization losses to a latent diffusion model and alternates between improving the shape autoencoder and the diffusion model to learn continuous deformable shape spaces from limited data.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.