Pith. sign in

REVIEW 4 major objections 4 minor 58 references

XR-MBT: Multi-modal Full Body Tracking for XR through Self-Supervision with Learned Depth Point Cloud Registration

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that egocentric headset depth, combined with 3-point synthesis, can become self-supervised multi-modal full-body tracking that tracks legs in XR in real time.

desk verdict Plausible architecture and a solid synthetic ablation, but the real-data 'tracks legs' claim rests on a metric the paper itself calls too ambiguous. read the letter →

arxiv 2411.18377 v1 pith:GDM2IN5O submitted 2024-11-27 cs.CV cs.LG

Denys Rozumnyi , Nadine Bertsch , Othman Sbai , Filippo Arcadu , Yuhua Chen , Artsiom Sanakoyeu , Manoj Kumar , Catherine Herold
show 1 more author
Robin Kips
This is my paper · ORCID
classification cs.CVcs.LG
keywords XRbodytrackingself-supervisedlearningegocentricdepthpointcloudsemanticregistration3-pointmotionsynthesislower-bodymulti-modalposeestimationresidualrefinement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the unregistered, partial depth point cloud already available in XR headsets can be used, without labeled real data, to turn 3-point body synthesis (head and controllers) into genuine full-body tracking. Its central claim is that a semantic encoder-decoder that assigns each depth point a body-joint probability provides enough structure for a residual pose network to refine the synthesized pose and track the legs. On synthetic mocap data, lower-body MPJPE drops from 11.21 cm with the previous synthesis-only method to 9.27 cm; on real unlabeled headset depth, the point-to-mesh error drops from 3.02 to 2.58 cm. If true, it matters because social presence in VR/AR would no longer depend on imagined legs: the device's own depth sensor would drive them in real time.

What carries the argument

The load-bearing object is the Semantic Point Cloud (SPC) network, a PointNet encoder-decoder that produces per-point joint probabilities and a 128-dim embedding, together with the SPC-loss: for each joint with support above 0.05P, the loss penalizes only distances beyond 10 cm between the predicted joint and the probability-weighted point centroid. This machinery turns an unregistered point cloud into a semantically meaningful supervisory signal, and its embedding is what the Multi-modal Pose Estimation residual MLP conditions on to refine the synthesized pose.

What would settle it

Collect real egocentric depth sequences where lower-body ground truth is captured by an external mocap system; if XR-MBT's real-data self-supervision does not improve true lower-body MPJPE over the synthetic-only variant, or if the SPC network's per-point joint labels on real point clouds match ground-truth body-part labels no better than chance, the central claim of tracking legs from unregistered depth fails.

Watch

Extended reading notes

Core claim

XR-MBT restructures XR body tracking as a fusion problem: the 3-point signal from headset and controllers proposes a plausible pose via a diffusion synthesis model, while the headset's depth point cloud refines it. The discovered mechanism is that a PointNet-style encoder-decoder, trained with synthetic body point clouds, can register an egocentric point cloud by outputting joint-association probabilities; those probabilities define a semantic point cloud loss (SPC-loss) that gives self-supervision on real, unlabeled depth data. This lets the model learn sensor-specific noise and partial visibility without ground truth. The paper claims this is the first system to track legs for XR, switching between synthesis and tracking depending on what the sensor sees, with a full pipeline runtime of 41.41 ms per frame.

Load-bearing premise

The semantic point network, trained only on synthetic point clouds, must produce correct joint assignments on real, unregistered headset depth, because the self-supervision that trains the pose refiner on real data inherits any error in those assignments.

Editorial extensions

If this is right

  • If the central claim holds, future XR body tracking no longer needs dedicated leg sensors or labeled real poses: the headset's own depth stream plus a 3-point synthesis prior can drive the lower body.
  • The ablation results imply that a learned semantic registration matters more than raw point distance: replacing the SPC-loss with a naive point-cloud loss raises lower-body MPJPE on all motions from 9.27 to 10.25 cm.
  • Self-supervision on real unlabeled depth improves real-data point-to-mesh error from 3.02 to 2.58 cm, indicating that sensor-specific noise is learnable without ground truth.
  • The full pipeline runs at 41.41 ms per frame, so the proposed refinement does not break the real-time budget of XR devices.
  • The method still fails when the moving leg is entirely outside the depth sensor's field of view, which bounds the claim to motions that the headset can at least partially see.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the SPC decoder produces a joint-association probability for every point, the same machinery could be extended to estimate the user's body shape at runtime, which the paper lists as an open limitation.
  • Inference: a labeled subset of real depth frames would let future work measure registration accuracy directly; the paper evaluates real data only through a point-to-mesh proxy and qualitative visualizations.
  • Inference: the reported rise in lower-body jitter (predicted/ground-truth ratio from 6.19 to 9.47) suggests the next natural improvement is temporal smoothing or an inverse-kinematics pass, a direction the paper itself flags.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. XR-MBT is a full-body tracking method for XR devices that combines the conventional 3-point signal (head and hand controllers) with an egocentric depth point cloud. A frozen 3-point synthesis model (AGRoL) generates an initial pose; a Semantic Point Cloud (SPC) network predicts per-point joint-class probabilities and features; and a Multi-modal Pose Estimation (MPE) network predicts a residual pose correction. The SPC and MPE networks are trained jointly on synthetic point clouds from Mocap data and then fine-tuned on real unregistered depth data using a self-supervised Semantic Point Cloud loss that pulls predicted joints toward soft semantic centroids of the point cloud. On synthetic data the method lowers lower-body MPJPE from 11.21 cm (AGRoL) to 9.27 cm on all motions; on real test data it lowers point-to-mesh distance (PC-loss) from 3.74 cm (AGRoL) or 3.02 cm (synthetic-only XR-MBT) to 2.58 cm. The real-data evaluation uses no ground-truth body pose.

Significance. If the real-data claims were supported by ground-truth pose measurements, this would be a valuable contribution: it is the first XR body-tracking formulation to fuse egocentric depth with the 3-point signal, the SPC-loss is a principled way to use unregistered point clouds, the joint training is simple and reproducible in structure, and the reported 41 ms pipeline suggests practical feasibility. The synthetic ablation is controlled and internally consistent, and the authors are transparent about the marginal contribution of the SPC-loss in simulation and about failure cases. However, no code or dataset release is mentioned, and the central 'tracks legs in XR' claim currently rests on a real-data evaluation whose only quantitative metric overlaps with the training objective; more evidence is needed before the claim is established.

major comments (4)
  1. [4.3, Table 3; Section 3.3] Section 3.3 explicitly rejects PC-loss as a training signal because 'mapping a pose to an unlabeled point cloud is too ambiguous' and the pose could 'deform to collide with the center of the point cloud while minimizing the PC-loss.' Table 3, however, uses exactly this PC-loss as the sole quantitative metric on real data. Since the model is fine-tuned with the SPC-loss (Eq. 8), whose centroids are derived from the same point cloud, a reduced PC-loss may simply reflect better point-cloud fitting rather than more accurate body tracking. The paper should report real-data pose accuracy against ground truth (e.g., external motion capture or manual 3D annotations) or, at minimum, quantitative SPC registration accuracy; without this, the abstract's 'tracks legs in XR for the first time' exceeds the evidence.
  2. [4.3, Figs. 3, 7, 8] The transfer of the SPC network to real depth data is validated only qualitatively. The real-data fine-tuning assumes that SPC per-point joint probabilities are accurate enough for the centroids in Eq. (7) to be reliable pseudo-labels. No quantitative SPC accuracy on real test sequences is reported, and no real-pose ground truth is used. Given that the only quantitative real-data metric (Table 3) suffers from the training/evaluation overlap described above, the qualitative figures alone cannot support the claim that the SPC network learns a meaningful registration on real data. Please add a quantitative real-data SPC evaluation, such as agreement with manual point labels or joint hit rates against reference poses.
  3. [4.2, Table 1] The controlled synthetic ablation shows that the SPC-loss adds very little on its own: lower-body MPJPE for all motions goes from 9.30 cm to 9.27 cm, and for kicking from 7.97 cm to 7.78 cm, while the PC-loss baseline is clearly worse (10.25 cm). The authors acknowledge this, but the paper's main claim is that self-supervision on real data is what brings the benefit. Because the real-data evidence is the PC-loss comparison criticized above, the synthetic experiments do not by themselves demonstrate that the SPC-loss provides useful supervision. This is not a fatal flaw, but it increases the burden on the real-data evaluation.
  4. [Table 2; Section 3.4; Abstract] The abstract claims 'real time on XR devices,' but Table 2 reports 41.41 ms per frame on a single NVIDIA P100 GPU, and the SPC network alone uses 89 GFLOPs. No measurement on actual XR device hardware is reported; the suggestion that PointNeXt could replace PointNet is not benchmarked. Please either provide on-device runtime/energy results or qualify the real-time claim to desktop/server inference.
minor comments (4)
  1. [Section 3.4] Section 3.4: 'PSC and MPE models' should read 'SPC and MPE models'.
  2. [Section 4.2, paragraph after Table 1] Section 4.2, paragraph after Table 1: 'decreasing the lower body pose error for kicking motions from 8.91 cm to 7.79 cm' is inconsistent with Table 1, where the SPC-loss row reports 7.78 cm.
  3. [Table 4] Table 4, 'Elbow knee strikes' row: the MPJVE values for AGRoL (2.09 | 5.54) are an order of magnitude lower than the corresponding values in other motion rows; please verify that these are not transcription errors.
  4. [Section 4.1] Section 4.1: the Mocap dataset is said to be fitted with 'a skeleton of 89 body joints,' while the method's output is described as local rotations for J joints of a skinned model (e.g., SMPL); the relationship between the 89-joint skeleton and the model's output joints should be stated explicitly.

Circularity Check

1 steps flagged · score 5.0 of 10

The real-data tracking claim rests on a point-cloud fitting metric (PC-loss) that overlaps the same geometric objective used for self-supervised training on real depth data; the synthetic MPJPE evaluation remains independent.

  1. fitted input called prediction [Section 3.3 (PC-loss definition, Eq. 8, fine-tuning paragraph); Section 4.3 (Table 3 caption)]
    "The most straightforward Point Cloud loss (PC-loss) is defined as an average distance between the point cloud and the closest face of the predicted body pose mesh... However, mapping a pose to an unlabeled point cloud is too ambiguous... The predicted pose could simply deform to collide with the center of the point cloud while minimizing the PC-loss. ... Point cloud error (PC-loss, in cm) computed on real test data for different actions."

    Table 3 reports PC-loss as the real-data accuracy metric, but PC-loss is the average distance from the point cloud to the closest mesh face. The model compared as 'XR-MBT + real' is fine-tuned on unlabeled real depth with the SPC-loss (Eq. 8), which minimizes distances between predicted joint positions and point-cloud-derived centroids. Thus the reported 3.02-to-2.58 cm improvement is a measure of how well the predicted mesh aligns with the point cloud after training to align with the point cloud, not an independent measure of true body tracking.

full rationale

The synthetic experiments are self-contained and credible: Table 1 measures MPJPE against Mocap ground truth on synthetically generated point clouds and shows a lower-body improvement from 11.21 cm (AGRoL) to 9.27 cm (XR-MBT with SPC-loss). Those numbers do not reduce to the training objective. The circularity is confined to the real-data evaluation: the paper rejects raw point-to-mesh distance (PC-loss) as too ambiguous for supervision in Section 3.3, then uses exactly that quantity as the only quantitative real-data error metric in Table 3, after fine-tuning the model on real point clouds with the semantic point-cloud alignment loss (Eq. 8). Lowering PC-loss is therefore expected from the training procedure and cannot by itself establish that legs are tracked correctly in XR. The qualitative SPC visualizations and failure-case discussion are auxiliary. The use of the authors' earlier AGRoL method [11] as backbone and baseline is a self-citation, but AGRoL is publicly available and retrained here, so it is not load-bearing circularity. Overall score reflects partial circularity of the real-data claim, not of the synthetic derivation.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions (SPC transfer to real data, body point-cloud filtering, PC-loss as a quality proxy) and on hand-chosen thresholds and weights. No new physical or metaphysical entities are introduced; the SPC network and SPC-loss are model components, not postulated entities requiring independent evidence.

free parameters (6)
  • SPC-loss threshold θ = 10 cm
    Eq. 8: distances below θ are not penalized; chosen to tolerate the offset between body-surface centroids and true joint positions.
  • Loss weights = w_rot=1, w_pos=0.01, w_ce=0.1, w_spc=0.01
    Eq. 9: set empirically; no sensitivity analysis is provided.
  • Point cloud sample count P = 100
    Inputs section: points sampled proportional to depth value; chosen for computational reasons.
  • Support threshold for joints = 0.05P
    Eq. 6 context: only joints with support > 0.05P are used in the SPC-loss.
  • Synthetic point cloud noise parameters = Gaussian σ=2cm; outliers σ=20cm at 2%; background threshold 10cm
    Sec. 3.1: parameters for generating training point clouds; chosen by hand.
  • Sequence length N = 196
    Inputs: sequences of length 196 frames are used.
assumptions (4)
  • domain assumption AGRoL synthesis provides a plausible initial full-body pose from 3-point input.
    Sec. 3.2: the residual MPE network refines this pose; if the synthesis is far off, the refinement cannot recover accuracy.
  • domain assumption The SPC network trained on synthetic point clouds generalizes to real depth sensor noise and produces correct semantic registration.
    Sec. 3.3: the self-supervised SPC-loss on real data trusts the SPC predictions; no quantitative real-data validation is provided.
  • domain assumption The body point cloud can be extracted from the full scene using simple proximity and density assumptions.
    Sec. 4.1: the real dataset was filtered this way, which may bias the point cloud toward the predicted pose.
  • domain assumption Point-to-mesh distance (PC-loss) is a valid proxy for body tracking quality.
    Table 3: real-data evaluation uses PC-loss in the absence of ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of XR-MBT: Multi-modal Full Body Tracking for XR through Self-Supervision with Learned Depth Point Cloud Registration." pith.science (2026). https://pith.science/paper/GDM2IN5O

@misc{pith2026241118377,
  author       = {Pith},
  title        = {Pith review of: XR-MBT: Multi-modal Full Body Tracking for XR through Self-Supervision with Learned Depth Point Cloud Registration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDM2IN5O}},
  note         = {Machine review of arXiv:2411.18377}
}
read the original abstract

Tracking the full body motions of users in XR (AR/VR) devices is a fundamental challenge to bring a sense of authentic social presence. Due to the absence of dedicated leg sensors, currently available body tracking methods adopt a synthesis approach to generate plausible motions given a 3-point signal from the head and controller tracking. In order to enable mixed reality features, modern XR devices are capable of estimating depth information of the headset surroundings using available sensors combined with dedicated machine learning models. Such egocentric depth sensing cannot drive the body directly, as it is not registered and is incomplete due to limited field-of-view and body self-occlusions. For the first time, we propose to leverage the available depth sensing signal combined with self-supervision to learn a multi-modal pose estimation model capable of tracking full body motions in real time on XR devices. We demonstrate how current 3-point motion synthesis models can be extended to point cloud modalities using a semantic point cloud encoder network combined with a residual network for multi-modal pose estimation. These modules are trained jointly in a self-supervised way, leveraging a combination of real unregistered point clouds and simulated data obtained from motion capture. We compare our approach against several state-of-the-art systems for XR body tracking and show that our method accurately tracks a diverse range of body motions. XR-MBT tracks legs in XR for the first time, whereas traditional synthesis approaches based on partial body tracking are blind.

Figures

Figures reproduced from arXiv: 2411.18377 by the authors.

Figure 1
Figure 1. XR-MBT uses 3-points (head/wrists) and depth sensing [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. XR-MBT architecture. First, we use AGRoL [11] to synthesize the initial pose (yellow). Second, we process the point cloud by the Semantic Point Cloud (SPC) encoder (green) to generate point features. Third, the SPC decoder (blue) generates the probability of mapping of every point to a body joint, which is used for self-supervised learning. Last, the Multi-modal Pose Estimation (MPE) network (orange) estimates the f… view at source ↗
Figure 3
Figure 3. An example of real 3-Point and point cloud data accessi [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison of our XR-MBT method to other XR body tracking methods on real data. By leveraging multi-modal inputs in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: MPJPE for lower body computed on the Mocap test [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: A failure in case of limited field of view, [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Additional results on real data. cloud. We showed that this learned point cloud registra￾tion could be leveraged to formulate a self-supervised loss, which in turn enabled training a multi-modal body tracking model on real unlabeled depth data. As a result, our ar￾chit…
Figure 8
Figure 8. Figure 8: Additional point cloud labeling results. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 52 canonical work pages

  1. [1]

    https://www.meta.com/en-gb/blog/ quest / ai - powered - technologies - quest - 3 - pro-ray-ban-meta-smart-glasses/

    Meta blog. https://www.meta.com/en-gb/blog/ quest / ai - powered - technologies - quest - 3 - pro-ray-ban-meta-smart-glasses/ . Accessed: 2024-07-09. 3 Figure 8. Additional point cloud labeling results

    work page 2024

  2. [2]

    Un- realego: A new dataset for robust egocentric 3d human mo- tion capture

    Hiroyasu Akada, Jian Wang, Soshi Shimada, Masaki Taka- hashi, Christian Theobalt, and Vladislav Golyanik. Un- realego: A new dataset for robust egocentric 3d human mo- tion capture. In ECCV, 2022. 3

    work page 2022

  3. [3]

    Flag: Flow- based 3d avatar generation from sparse observations

    Sadegh Aliakbarian, Pashmina Cameron, Federica Bogo, Andrew Fitzgibbon, and Thomas Cashman. Flag: Flow- based 3d avatar generation from sparse observations. In CVPR, 2022. 3

    work page 2022

  4. [4]

    A stochastic con- ditioning scheme for diverse human motion prediction

    Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salz- mann, Lars Petersson, and Stephen Gould. A stochastic con- ditioning scheme for diverse human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 5223–5232, 2020. 2

    work page 2020

  5. [5]

    Fast on-board 3d torso pose recovery and forecasting

    Abhijat Biswas, Henny Admoni, and Aaron Steinfeld. Fast on-board 3d torso pose recovery and forecasting. In Pro- ceedings of the International Conference on Robotics and Automation (ICRA), pages 20–24, 2019. 3

    work page 2019

  6. [6]

    Young-Woon Cha, True Price, Zhen Wei, Xinran Lu, Nicholas Rewkowski, Rohan Chabra, Zihe Qin, Hyounghun Kim, Zhaoqi Su, and Yebin et al. Liu. Towards fully mo- bile 3d face, body, and environment capture using only head- worn cameras. In IEEE TVCG, 2018. 3

    work page 2018

  7. [7]

    Passthrough+ real-time stereoscopic view synthesis for mo- bile mixed reality

    Gaurav Chaurasia, Arthur Nieuwoudt, Alexandru-Eugen Ichim, Richard Szeliski, and Alexander Sorkine-Hornung. Passthrough+ real-time stereoscopic view synthesis for mo- bile mixed reality. ACM TOG, 3(1):1–17, 2020. 1, 3

    work page 2020

  8. [8]

    Refinet: 3d human pose re- finement with depth maps

    Andrea D’Eusanio, Stefano Pini, Guido Borghi, Roberto Vezzani, and Rita Cucchiara. Refinet: 3d human pose re- finement with depth maps. In 2020 25th International Con- ference on Pattern Recognition (ICPR) , pages 2320–2327. IEEE, 2021. 3

    work page 2020

Show all 58 references
  1. [9]

    Recent advances and perspectives in deep learning techniques for 3d point cloud data process- ing

    Zifeng Ding, Yuxuan Sun, Sijin Xu, Yan Pan, Yanhong Peng, and Zebing Mao. Recent advances and perspectives in deep learning techniques for 3d point cloud data process- ing. Robotics, 12(4):100, 2023. 3

    2023

  2. [10]

    Full-body motion from a single head-mounted device: Generating smpl poses from partial observations

    Andrea Dittadi, Sebastian Dziadzio, Darren Cosker, Ben Lundell, and Tom Cashman. Full-body motion from a single head-mounted device: Generating smpl poses from partial observations. In ICCV, 2021. 3

    2021

  3. [11]

    Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, and Artsiom Sanakoyeu. Avatars grow legs: Generating smooth human motion from sparse tracking in- Model MPJPE (cm) up | low MPJRE (radians) up | low MPJVE (m2/s3) up | low jitter (pred/gt) up | low PC-loss A...

  4. [12]

    Recurrent network models for human dynam- ics

    Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Ji- tendra Malik. Recurrent network models for human dynam- ics. In Proceedings of the IEEE international conference on computer vision, pages 4346–4354, 2015. 2

    2015

  5. [13]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5152–5161, 2022. 2

    2022

  6. [14]

    Human motion prediction via spatio-temporal in- painting

    Alejandro Hernandez, Jurgen Gall, and Francesc Moreno- Noguer. Human motion prediction via spatio-temporal in- painting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7134–7143, 2019. 2

    2019

  7. [15]

    A deep learning framework for character motion synthesis and editing

    Daniel Holden, Jun Saito, and Taku Komura. A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics (TOG), 35(4):1–11, 2016. 2

    2016

  8. [16]

    Black, Otmar Hilliges, and Gerard Pons-Moll

    Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J. Black, Otmar Hilliges, and Gerard Pons-Moll. Deep inertial poser: Learning to reconstruct human pose from sparse iner- tial measurements in real time. In ACM TOG, pages 1–15,

  9. [17]

    See- ing invisible poses: Estimating 3d body pose from egocentric video

    Hao Jiang, Hao Jiang Grauman, and Kristen Grauman. See- ing invisible poses: Estimating 3d body pose from egocentric video. In CVPR, 2017. 1, 3

    2017

  10. [18]

    Avatarposer: Ar- ticulated full-body pose tracking from sparse motion sens- ing

    Jiaxi Jiang, Paul Streli, Huajian Qiu, Andreas Fender, Larissa Laich, Patrick Snape, and Christian Holz. Avatarposer: Ar- ticulated full-body pose tracking from sparse motion sens- ing. In ECCV, 2022. 3

    2022

  11. [19]

    Transformer inertial poser: Real-time human motion reconstruction from sparse imus with simultaneous terrain generation

    Yifeng Jiang, Yuting Ye, Deepak Gopinath, Jungdam Won, Alexander Winkler, and Karen Liu. Transformer inertial poser: Real-time human motion reconstruction from sparse imus with simultaneous terrain generation. In CVPR, 2022. 3

    2022

  12. [20]

    Ego3dpose: Capturing 3d cues from binocular egocentric views

    Taeho Kang, Kyungjin Lee, Jinrui Zhang, and Youngki Lee. Ego3dpose: Capturing 3d cues from binocular egocentric views. 2023. 3

    2023

  13. [21]

    Human motion denoising using attention-based bidirectional recur- rent neural network

    Seong Uk Kim, Hanyoung Jang, and Jongmin Kim. Human motion denoising using attention-based bidirectional recur- rent neural network. InSIGGRAPH Asia 2019 Posters, pages 1–2. 2019. 2

    2019

  14. [22]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , San Diega, CA, USA,

  15. [23]

    Practical stereo matching via cascaded recurrent net- work with adaptive correlation

    Jiankun Li, Peisen Wang, Pengfei Xiong, Tao Cai, Ziwei Yan, Lei Yang, Jiangyu Liu, Haoqiang Fan, and Shuaicheng Liu. Practical stereo matching via cascaded recurrent net- work with adaptive correlation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2022

  16. [24]

    Matthew Loper, Naureen Mahmood, Javier Romero, Ger- ard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, Oct. 2015. 4

    2015

  17. [25]

    Naureen Mahmood, Nima Ghorbani, Nikolaus Troje, Gerard Pons-Moll, and Michael J. Black. Human motion prediction via spatio-temporal inpainting. In ICCV, pages 7134–7143,

  18. [26]

    On human motion prediction using recurrent neural networks

    Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2891–2900, 2017. 2

    2017

  19. [27]

    Create more natural movements using inside-out body tracking and generative legs, 2023

    Meta. Create more natural movements using inside-out body tracking and generative legs, 2023. Last accessed March 5th

    2023

  20. [28]

    You2me: Inferring body pose in egocentric video via first and second person interactions

    Evonne Ng, Donglai Xiang, Hanbyul Joo, and Kristen Grau- man. You2me: Inferring body pose in egocentric video via first and second person interactions. 2020. 3

    2020

  21. [29]

    Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J

    Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J. Black. BABEL: Bodies, action and behavior with english labels. In CVPR, pages 722–731, June 2021. 6

    2021

  22. [30]

    Pointnet: Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 652–660, 2017. 4

    2017

  23. [31]

    Point- net++: Deep hierarchical feature learning on point sets in a metric space

    Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Point- net++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413, 2017. 4

    2017 arXiv

  24. [32]

    Pointnext: Revisiting pointnet++ with improved training and scaling strategies

    Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In NeurIPS, 2022. 6

    2022

  25. [33]

    Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J. Guibas. Humor: 3d human motion model for robust pose estimation. In ICCV, pages 11468–11479, 2021. 8

    2021

  26. [34]

    Livehps: Lidar-based scene- level human pose and shape estimation in free environment

    Yiming Ren, Xiao Han, Chengfeng Zhao, Jingya Wang, Lan Xu, Jingyi Yu, and Yuexin Ma. Livehps: Lidar-based scene- level human pose and shape estimation in free environment. In CVPR, pages 1281–1291, June 2024. 3

    2024

  27. [35]

    Lidar-aid inertial poser: Large-scale human motion capture by sparse inertial and lidar sensors

    Yiming Ren, Chengfeng Zhao, Yannan He, Peishan Cong, Han Liang, Jingyi Yu, Lan Xu, and Yuexin Ma. Lidar-aid inertial poser: Large-scale human motion capture by sparse inertial and lidar sensors. IEEE TVCG , 29(5):2337–2347, may 2023. 3

    2023

  28. [36]

    Egocap: egocentric marker-less mo- tion capture with two fisheye cameras

    Helge Rhodin, Christian Richardt, Dan Casas, Eldar Insafut- dinov, Mohammad Shafiei, Hans-Peter Seidel, Bernt Schiele, and Christian Theobalt. Egocap: egocentric marker-less mo- tion capture with two fisheye cameras. In ACM TOG, 2016. 1, 3

    2016

  29. [37]

    Real-time 3d pose estimation from sin- gle depth images

    Thomas Schn ¨urer, Stefan Fuchs, Markus Eisenbach, and Horst-Michael Groß. Real-time 3d pose estimation from sin- gle depth images. In VISIGRAPP (5: VISAPP) , pages 716– 724, 2019. 3

    2019

  30. [38]

    Phasemp: Robust 3d pose estimation via phase-conditioned human motion prior

    Mingyi Shi, Sebastian Starke, Yuting Ye, Taku Komura, and Jungdam Won. Phasemp: Robust 3d pose estimation via phase-conditioned human motion prior. In ICCV, pages 14725–14737, 2023. 1

    2023

  31. [39]

    Human motion dif- fusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion dif- fusion model. arXiv preprint arXiv:2209.14916, 2022. 3

    2022 arXiv

  32. [40]

    D. Tome, T. Alldieck, P. Peluse, G. Pons-Moll, L. Agapito, H. Badino, and F. De la Torre. Selfpose: 3d egocentric pose estimation from a headset mounted camera. IEEE TPAMI, pages 1–1, 2020. 3

    2020

  33. [41]

    xr-egopose: Egocentric 3d human pose from an hmd camera

    Denis Tome, Patrick Peluse, Lourdes Agapito, and Hernan Badino. xr-egopose: Egocentric 3d human pose from an hmd camera. In ICCV, pages 7728–7738, 2019. 3

    2019

  34. [42]

    Multi-person 3d pose estimation from 3d cloud data using 3d convolutional neural networks

    Manolis Vasileiadis, Christos-Savvas Bouganis, and Dim- itrios Tzovaras. Multi-person 3d pose estimation from 3d cloud data using 3d convolutional neural networks. Com- puter Vision and Image Understanding, 185:12–23, 2019. 3

    2019

  35. [43]

    Egocentric whole-body motion capture with fisheyevit and diffusion-based motion refinement

    Jian Wang, Zhe Cao, Diogo Luvizon, Lingjie Liu, Kri- pasindhu Sarkar, Danhang Tang, Thabo Beeler, and Chris- tian Theobalt. Egocentric whole-body motion capture with fisheyevit and diffusion-based motion refinement. 2023. 3

    2023

  36. [44]

    Sequential 3d human pose and shape estimation from point clouds

    Kangkan Wang, Jin Xie, Guofeng Zhang, Lei Liu, and Jian Yang. Sequential 3d human pose and shape estimation from point clouds. In CVPR, pages 7275–7284, 2020. 3

    2020

  37. [45]

    Parametric model estimation for 3d clothed humans from point clouds

    Kangkan Wang, Huayu Zheng, Guofeng Zhang, and Jian Yang. Parametric model estimation for 3d clothed humans from point clouds. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) , pages 156–165. IEEE, 2021. 3

    2021

  38. [46]

    Gorban, Jingwei Ji, Mahyar Najibi, Yin Zhou, and Dragomir Anguelov

    Zhenzhen Weng, Alexander S. Gorban, Jingwei Ji, Mahyar Najibi, Yin Zhou, and Dragomir Anguelov. 3d human key- points estimation from point clouds in the wild without hu- man labels. In CVPR, pages 1158–1167, June 2023. 3

    2023

  39. [47]

    Quest- sim: Human motion tracking from sparse sensors with sim- ulated avatars

    Alexander Winkler, Yuting Ye, and Jungdam Won. Quest- sim: Human motion tracking from sparse sensors with sim- ulated avatars. In ACM TOG, pages 1–8, 2022. 1, 3

    2022

  40. [48]

    Neuralpassthrough: Learned real-time view synthesis for vr

    Lei Xiao, Salah Nouri, Joel Hegland, Alberto Garcia Garcia, and Douglas Lanman. Neuralpassthrough: Learned real-time view synthesis for vr. In ACM TOG, pages 1–9, 2022. 1

    2022

  41. [49]

    A re- view: Point cloud-based 3d human joints estimation

    Tianxu Xu, Dong An, Yuetong Jia, and Yang Yue. A re- view: Point cloud-based 3d human joints estimation. Sen- sors, 21(5):1684, 2021. 3

    2021

  42. [50]

    Lobstr: Real-time lower-body pose prediction from sparse upper- body tracking signals

    Dongseok Yang, Doyeon Kim, and Sung-Hee Lee. Lobstr: Real-time lower-body pose prediction from sparse upper- body tracking signals. 2021. 3

    2021

  43. [51]

    Neu- ral3points: Learning to generate physically realistic full- body motion for virtual reality users

    Yongjing Ye, Libin Liu, Lei Hu, and Shihong Xia. Neu- ral3points: Learning to generate physically realistic full- body motion for virtual reality users. pages 183–94, 2023. 3

    2023

  44. [52]

    3d ego-pose estimation via imita- tion learning

    Ye Yuan and Kris Kitani. 3d ego-pose estimation via imita- tion learning. In ECCV, 2018. 1, 3

    2018

  45. [53]

    Physdiff: Physics-guided human motion diffusion model

    Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 16010–16021, 2023. 3

    2023

  46. [54]

    Motiondif- fuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 2, 3

    2024

  47. [55]

    Weakly supervised adversarial learning for 3d human pose estimation from point clouds

    Zihao Zhang, Lei Hu, Xiaoming Deng, and Shihong Xia. Weakly supervised adversarial learning for 3d human pose estimation from point clouds. IEEE transactions on visual- ization and computer graphics, 26(5):1851–1859, 2020. 3

    2020

  48. [56]

    Egoglass: Egocentric-view human pose estimation from an eyeglass frame

    Dongxu Zhao, Zhen Wei, Jisan Mahmud, and Jan-Michael Frahm. Egoglass: Egocentric-view human pose estimation from an eyeglass frame. In Int. Conf. on 3D Vision, 2021. 3

    2021

  49. [57]

    Realistic full-body tracking from sparse observa- tions via joint-level modeling

    Xiaozheng Zheng, Zhuo Su, Chao Wen, Zhou Xue, and Xi- aojie Jin. Realistic full-body tracking from sparse observa- tions via joint-level modeling. In ICCV, 2023. 1

    2023

  50. [58]

    On the continuity of rotation representations in neural networks

    Yi Zhou, Connelly Barnes, Lu Jingwan, Yang Jimei, and Li Hao. On the continuity of rotation representations in neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 4

    2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.