Pith. sign in

REVIEW 3 major objections 6 minor 74 references

Egocentric Action-aware Inertial Localization in Point Clouds with Vision-Language Guidance

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A head-mounted IMU can localize a person inside a pre-scanned 3D point cloud by turning everyday actions into spatial anchors, learned through contrastive alignment guided by vision and language.

desk verdict A real contribution to inertial localization, but the seen-room numbers are contaminated by point-cloud leakage; the unseen-room results still support the core claim. read the letter →

arxiv 2505.14346 v2 pith:DVK6E3QS submitted 2025-05-20 cs.CV

Mingfang Zhang , Ryo Yonetani , Yifei Huang , Liangyang Ouyang , Ruicong Liu , Yoichi Sato This is my paper
classification cs.CV
keywords inertiallocalizationegocentricIMUpointcloudactioncuesmultimodalcontrastivelearningvision-languageguidancetrajectorydriftEgoExo4D
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a person wearing a head-mounted inertial sensor can be tracked inside a pre-scanned 3D map without the usual drift that plagues IMU navigation. Its key idea is that everyday actions—washing dishes, bending to check a stove, reaching into a cabinet—leave IMU signatures that reliably point to where those actions happen, so they can serve as spatial anchors. The proposed EAIL framework learns those anchors by contrastively aligning one-second IMU windows with local 1 m² patches of the point cloud, using concurrently recorded egocentric images and action captions as training-time guides from a pretrained vision-language model. At test time only IMU data and the point cloud are required, and the paper reports better localization than velocity-integration and scene-specific baselines, including in unseen kitchens and participants. If the claim holds, it makes drift-free indoor tracking possible with sensors that are energy-efficient and privacy-preserving, with action recognition coming along as a by-product.

What carries the argument

EAIL's load-bearing mechanism is a two-stage contrastive alignment. In Stage 1, an IMU encoder and a point-cloud encoder are trained with a pairwise contrastive loss across four synchronized modalities: the egocentric image frame, the text action caption, the one-second IMU window, and the local 1 m² point-cloud patch, with a pretrained vision-language model providing the shared semantic space. In Stage 2, the encoders are frozen; the full point cloud is partitioned into patches and per-second correspondence heatmaps are computed between IMU windows and patches, then refined by a temporal reasoning module built on 3D convolutions and a spatial reasoning module built on dilated 3D convolutions. The output is a per-second location classification over patches, and that predicted location heatmap is reused as spatial attention to fuse point-cloud features into the IMU stream for action recognition.

What would settle it

Take a trained model into a kitchen whose layout was changed after scanning, or ask people to perform the same actions at unusual locations, such as washing dishes at a dining table, and measure localization success at the 0.4 m threshold; if accuracy does not drop substantially, the action-location anchors are not the mechanism carrying the result.

Watch

Extended reading notes

Core claim

The central claim is that inertial localization can be recast as a retrieval problem: rather than integrating noisy velocity measurements into a path that drifts, the network compares each short window of head motion against every local patch of a known environment map and directly predicts which patch contains the person. Action semantics are the bridge between the two modalities: the training procedure forces IMU features and point-cloud features into a shared embedding space organized by what a person is doing, so 'washing dishes' naturally lands near the sink region. On the EgoExo4D cooking subset, the paper reports that this approach outperforms prior inertial localization and inertial action recognition baselines in both rooms seen during training and entirely unseen kitchens and participants. It also claims the same encoders recognize the performed action sequence as a by-product, and that using the predicted location as spatial attention over point-cloud features improves action recognition accuracy.

Load-bearing premise

The method depends on the premise that the motions people make during everyday actions reliably indicate where those actions happen, such as washing dishes near a sink rather than in the middle of a room, and that this regularity transfers across kitchens and people, together with the requirement that the supplied 3D point cloud is complete and up to date.

Editorial extensions

If this is right

  • A user can be tracked without a known starting position or velocity integration; direct prediction keeps the error roughly stable with elapsed time, while velocity-accumulation baselines drift out of the point cloud within minutes.
  • A single model trained on many kitchens transfers to unseen kitchens and participants, instead of requiring one model trained per scene.
  • The action-recognition head benefits from the localization: location-aware attention over point-cloud features improves top-1 and top-5 accuracy over IMU-only action recognition.
  • Vision and language are training-only crutches; inference needs only the head-mounted IMU and the pre-scanned point cloud, so the system remains privacy-preserving at runtime.
  • The intermediate correspondence heatmaps provide interpretable evidence of where the model locates the person at each second, and the spatiotemporal modules resolve ambiguous multi-peak heatmaps into a single predicted location.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the action-location anchor assumption generalizes, the same contrastive recipe could be transferred to other domains with stereotyped affordances, such as workshops, hospitals, or retail stores, with text captions naming new anchors without retraining per scene.
  • A natural extension the paper leaves implicit is coupling this with incremental mapping, since the stated limitation of requiring an up-to-date point cloud could be addressed by feeding online SLAM updates into the same patch encoder.
  • The paper notes that idle or low-motion segments break the anchor signal; a practical system would likely fuse the action-location predictions with pedestrian dead reckoning or a Kalman filter during those gaps.
  • Because action recognition emerges from the same encoders, the framework could in principle bootstrap action labels for new environments from IMU-plus-map data alone, avoiding manual annotation; the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes EAIL, a two-stage framework for inertial localization inside a pre-scanned 3D point cloud using head-mounted IMU signals. Stage 1 trains IMU and point-cloud encoders by contrastively aligning them with synchronized egocentric images and text captions, so that short-term action cues become associated with local environmental structures. Stage 2 freezes these encoders and learns spatiotemporal reasoning modules that convert a sequence of IMU features and a partitioned point cloud into per-second location heatmaps, with an auxiliary location-aware action classifier. Experiments on the EgoExo4D cooking subset compare EAIL with RoNIN, IMUNet, NILoc+, and action-recognition baselines, reporting higher success rates at 0.2/0.4/0.6 m thresholds and higher action-recognition accuracy on both seen and unseen rooms. The authors acknowledge the need for an up-to-date point cloud and discuss limitations such as idle-user ambiguity and sensor-placement dependence.

Significance. If the results hold, EAIL provides a credible new mechanism for drift-free inertial localization: using action-environment correlations as spatial anchors, with no vision or language input at inference time. The framework is clearly specified, the public EgoExo4D dataset is appropriate, and the component ablations in Tables 3 and 5 are informative; the two-stage design and the use of frozen Stage 1 encoders for efficient inference are sensible engineering choices. The unseen-room results, in particular, support the claim that action-location alignment can transfer to new kitchens and participants, which is the scientifically strongest contribution. However, the headline state-of-the-art claim is currently tied to seen-room numbers that may be inflated by environment memorization, the main baseline is a modified variant introduced by the authors, and no uncertainty quantification is reported. These issues are fixable within the manuscript's scope.

major comments (3)
  1. [Table 1 and Section 5.2] The seen-room evaluation is contaminated by the fact that the same static point clouds appear in Stage 1 and Stage 2 training and in the test-seen split. Because the point cloud for a given kitchen is identical between training and evaluation, the model can memorize environment-specific segment features rather than learning transferable action-location alignment. The much larger seen-to-unseen drop for EAIL (43.86 to 26.86 at 0.2 m) than for NILoc+ (17.03 to 13.32) is consistent with this memorization concern. The claim of state-of-the-art performance in 'diverse settings' should be based on the unseen-room columns, or the paper should report leave-one-kitchen-out results and explicitly separate in-map localization from cross-environment generalization.
  2. [Table 1 and Section 5.2] No error bars, confidence intervals, or statistical significance tests are reported for any localization or action-recognition metric. This matters because the unseen-room margin over NILoc+ is large at 0.2 m but modest for the Relative Score (89.55 vs. 84.08), and the action-recognition margins in Table 2 are small (e.g., 15.03 vs. 12.27 top-1 on unseen rooms). The authors should report variance across participants or kitchens and provide a significance test, such as a paired bootstrap, to support the state-of-the-art claim.
  3. [Section 5.2, 'Direct Localization in Point Clouds'] The primary baseline NILoc+ is a modified multi-scene variant introduced by the authors rather than the original NILoc method, which is scene-specific and trained separately for each environment. It is unclear whether training a single NILoc+ model on multi-scene data is a fair or well-tuned baseline, since multi-scene training may disadvantage a method that is designed for per-scene fitting. The paper should either include original per-scene NILoc results, validate that NILoc+ was tuned appropriately, or clearly frame the comparison as against a multi-scene adaptation rather than against NILoc itself.
minor comments (6)
  1. [Section 5.4, 'Modalities Engagement'] The sentence claiming that 'even in scenarios where action caption annotations are unavailable in the training set, our method does not fail' is unsupported by any experiment reported in the paper or supplementary material. Either add an ablation without text captions or soften this claim.
  2. [Section 4.1, Eq. (1)] The loss weights (alpha = 0.1; beta, theta, delta, gamma = 1) are chosen without reported sensitivity analysis, and the supplementary ablations in Table 5 do not vary these weights. A brief sensitivity study would strengthen the claim that the alignment coefficients are not critical.
  3. [Section 5.3 and Table 2] The EVIMAE baseline is described as a 'modified version' trained on a single IMU device with a ViT-Base backbone, but the modifications beyond the modal input and backbone are not specified. Please state what changes were made relative to the original EVIMAE and how this affects the comparison.
  4. [Figure 4] The localization-error-over-time curves do not specify whether they are averaged over seen rooms, unseen rooms, or both, and no confidence intervals are shown. Please clarify the averaging procedure and the number of test sequences used.
  5. [Section 5.1 and Supplementary Table 5] In Table 5, InternVL obtains a higher Relative Score than SLIP on unseen rooms (90.18 vs. 89.55) but much lower action top-1 (9.84 vs. 15.03). The choice of SLIP as the default vision-language encoder would be better justified if the trade-off between localization and action metrics were discussed explicitly.
  6. [Abstract and Section 1] The abstract and introduction state that the method is evaluated 'in diverse settings,' but the evaluation is limited to a cooking subset of EgoExo4D. Please either qualify this claim or add experiments on non-cooking activities to support it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the localization and action recognition results are learned from held-out-supervised training and evaluated against external baselines, not derived from the method's own inputs by construction.

full rationale

The paper's derivation chain is a standard supervised learning pipeline: Stage 1 trains IMU and point-cloud encoders with a contrastive loss (Eq. 1) using synchronized image, text, IMU, and point-cloud samples; Stage 2 freezes those encoders and trains spatiotemporal reasoning modules supervised by ground-truth trajectory labels and action labels (Eqs. 2 and 3). The evaluation is performed on EgoExo4D with held-out 'unseen rooms' that contain neither the same participants nor the same environments as the training set, and all baselines are trained on the same data for fair comparison. No parameter is fitted to the reported success-rate metric and then renamed as a prediction; the loss weights and temporal length are fixed by ablations, not optimized against the test set. The paper does cite several prior works by the same authors (e.g., EVIMAE as a baseline, and other egocentric works in related work), but none of these citations is load-bearing for the central claim: the action-environment correlation premise is argued from examples in Section 1, not imported from a self-citation, and no uniqueness theorem or ansatz is smuggled in via the authors' prior work. The concern that 'seen rooms' results may reflect memorization of identical point clouds is a generalization and evaluation-protocol risk, not a circularity, and the paper separately reports unseen-room numbers that support the method's transferability. Therefore, the core results are self-contained with respect to their inputs and no circular step is present.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical entities. The main assumptions are domain-specific premises about action-scene correlations and data availability. The free parameters are standard training hyperparameters and loss weights, not fitted constants derived from the target metric.

free parameters (5)
  • Loss coefficients alpha, beta, theta, delta, gamma = alpha=0.1, beta=theta=delta=gamma=1.0
    Chosen by hand to weight modality pairs; affects Stage 1 alignment.
  • Temporal sequence length T = 10 seconds
    Selected via ablation in supplementary; balances seen/unseen performance.
  • Spatial grid size S = 20x20=400 segments
    Uniform partition of point cloud; affects classification resolution.
  • Point cloud sub-sampling = 8192 points per local segment
    Fixed input size for PointNet++.
  • IMU downsampling rate = 400 Hz
    Preprocessing choice from RoNIN [22].
assumptions (6)
  • domain assumption Action cues in head-mounted IMU signals correlate with spatial environmental structures (e.g., washing dishes near sink).
    Key premise motivating the method, introduced in Section 1 and used throughout.
  • domain assumption The 3D point cloud of the environment is available and accurate at test time.
    Assumed in Section 3 and acknowledged as a limitation in Section 6.
  • domain assumption Egocentric images and action captions are synchronized with IMU signals in the training data.
    Needed for Stage 1 contrastive alignment; Section 4.1.
  • domain assumption Pretrained vision-language encoders (SLIP) provide semantically meaningful features for action concepts.
    Used as guidance in Stage 1; Section 4.1 and supplementary.
  • standard math Standard contrastive learning with pairwise losses aligns the four modalities as intended.
    The alignment relies on standard contrastive objectives; Section 4.1 Eq. 1.
  • domain assumption The EgoExo4D dataset provides accurate ground truth trajectories and action labels.
    Used for supervision in both stages; Section 5.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Egocentric Action-aware Inertial Localization in Point Clouds with Vision-Language Guidance." pith.science (2026). https://pith.science/paper/DVK6E3QS

@misc{pith2026250514346,
  author       = {Pith},
  title        = {Pith review of: Egocentric Action-aware Inertial Localization in Point Clouds with Vision-Language Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DVK6E3QS}},
  note         = {Machine review of arXiv:2505.14346}
}
read the original abstract

This paper presents a novel inertial localization framework named Egocentric Action-aware Inertial Localization (EAIL), which leverages egocentric action cues from head-mounted IMU signals to localize the target individual within a 3D point cloud. Human inertial localization is challenging due to IMU sensor noise that causes trajectory drift over time. The diversity of human actions further complicates IMU signal processing by introducing various motion patterns. Nevertheless, we observe that some actions captured by the head-mounted IMU correlate with spatial environmental structures (e.g., bending down to look inside an oven, washing dishes next to a sink), thereby serving as spatial anchors to compensate for the localization drift. The proposed EAIL framework learns such correlations via hierarchical multi-modal alignment with vision-language guidance. By assuming that the 3D point cloud of the environment is available, it contrastively learns modality encoders that align short-term egocentric action cues in IMU signals with local environmental features in the point cloud. The learning process is enhanced using concurrently collected vision and language signals to improve multimodal alignment. The learned encoders are then used in reasoning the IMU data and the point cloud over time and space to perform inertial localization. Interestingly, these encoders can further be utilized to recognize the corresponding sequence of actions as a by-product. Extensive experiments demonstrate the effectiveness of the proposed framework over state-of-the-art inertial localization and inertial action recognition baselines.

Figures

Figures reproduced from arXiv: 2505.14346 by the authors.

Figure 1
Figure 1. Egocentric Action-aware Inertial Localization (EAIL). Our framework leverages egocentric action cues obtained from the head-mounted IMU to perform inertial localization in the environ￾mental 3D point cloud. The corresponding sequence of actions can also be recognized as a by-product. to vision-based localization methods [28, 39], inertial lo￾calization enables user tracking in an energy-efficient and privacy-preserv… view at source ↗
Figure 2
Figure 2. Short-Term Action-Location Alignment. In this first stage, our objective is to train a point cloud encoder and an IMU encoder using contrastive learning. A positive sample set consists of simultaneous multimodal data samples at the human’s location. Negative samples are generated by randomly sampling data from other times and locations. by works like EgoVLP [33, 47] and HierVL [2]. In addi￾tion to the human language… view at source ↗
Figure 3
Figure 3. Sequential Motion Localization. In this second stage, we generate a sequence of the user’s locations and actions over T seconds using a series of IMU signals ({Mt} T t=1) alongside the point cloud of the entire scene. This point cloud is divided into S local segments ({Ps} S s=1). The IMU encoder and the point cloud encoder are frozen networks pre-trained in Stage 1 for efficient spatial-temporal reasoning in large-… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Inertial Localization Error Over Time Elapsed. While velocity accumulation-based methods experience signifi￾cant trajectory drift, our approach remains accurate over time. Velocity Cumulation versus Direct Location Prediction Previous approaches can be divided into two…
Figure 5
Figure 5. Figure 5: Visualization of Heatmaps in Each Stage. Quantitative Evaluation As shown in Tab. 2, DeepCon￾vLSTM performs relatively poorly due to its architecture’s difficulty in capturing complex action patterns effectively. Although EVIMAE gains from the advanced reasoning ca￾pab…
Figure 6
Figure 6. Figure 6: Qualitative Comparison with Previous Methods. 6. Limitations and Future Directions While our method can robustly exploit head-mounted IMU signals for human localization within pre-built point clouds, it does hinge on several factors that present avenues for fu￾ture res…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

74 extracted references · 64 canonical work pages

  1. [1]

    Accessed on March 7th, 2025

    Scaniverse. Accessed on March 7th, 2025. 3

    work page 2025

  2. [2]

    Hiervl: Learning hierarchical video- language embeddings

    Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hierarchical video- language embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23066–23078, 2023. 3

    work page 2023

  3. [3]

    Walk detection and step counting on unconstrained smartphones

    Agata Brajdic and Robert Harle. Walk detection and step counting on unconstrained smartphones. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 225–234, 2013. 1, 2

    work page 2013

  4. [4]

    Ionet: Learning to cure the curse of drift in iner- tial odometry

    Changhao Chen, Xiaoxuan Lu, Andrew Markham, and Niki Trigoni. Ionet: Learning to cure the curse of drift in iner- tial odometry. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. 2

    work page 2018

  5. [5]

    OxIOD: The Dataset for Deep Inertial Odometry

    Changhao Chen, Peijun Zhao, Chris Xiaoxuan Lu, Wei Wang, Andrew Markham, and Niki Trigoni. Oxiod: The dataset for deep inertial odometry. arXiv preprint arXiv:1809.07491, 2018. 2

    work page Pith review arXiv 2018

  6. [6]

    Soundingactions: Learning how actions sound from narrated egocentric videos

    Changan Chen, Kumar Ashutosh, Rohit Girdhar, David Har- wath, and Kristen Grauman. Soundingactions: Learning how actions sound from narrated egocentric videos. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27252–27262, 2024. 3

    work page 2024

  7. [7]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24185–24198, 2024. 4, 1

    work page 2024

  8. [8]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 720–736, 2018. 2, 4

    work page 2018

Show all 74 references
  1. [9]

    Scenefun3d: fine-grained functionality and affordance un- derstanding in 3d scenes

    Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, and Francis Engelmann. Scenefun3d: fine-grained functionality and affordance un- derstanding in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (C...

    2024

  2. [10]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  3. [11]

    Un- locking exocentric video-language data for egocentric video representation learning

    Zi-Yi Dou, Xitong Yang, Tushar Nagarajan, Huiyu Wang, Jing Huang, Nanyun Peng, Kris Kitani, and Fu-Jen Chu. Un- locking exocentric video-language data for egocentric video representation learning. arXiv preprint arXiv:2408.03567 ,

    arXiv

  4. [12]

    Project aria: A new tool for egocentric multi-modal ai research

    Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023. 1, 2, 5

    2023 arXiv

  5. [13]

    Actionslam: Using location-related actions as landmarks in pedestrian slam

    Hardegger et al. Actionslam: Using location-related actions as landmarks in pedestrian slam. In IPIN, 2012. 2

    2012

  6. [14]

    Positioning method of pedestrian dead reckon- ing based on human activity recognition

    Zhang et al. Positioning method of pedestrian dead reckon- ing based on human activity recognition. In IPIN, 2022. 2

    2022

  7. [15]

    Egovqa-an egocentric video question answer- ing benchmark dataset

    Chenyou Fan. Egovqa-an egocentric video question answer- ing benchmark dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pages 0–0, 2019. 3

    2019

  8. [16]

    Mmg-ego4d: Multimodal generalization in egocentric action recognition

    Xinyu Gong, Sreyas Mohan, Naina Dhingra, Jean-Charles Bazin, Yilei Li, Zhangyang Wang, and Rakesh Ranjan. Mmg-ego4d: Multimodal generalization in egocentric action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6...

    2023

  9. [17]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InPro- ceedings of the IEEE/CVF Conference on Computer Vision...

    2022

  10. [18]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings...

    2024

  11. [19]

    Conceptgraphs: Open-vocabulary 3d scene graphs for per- ception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for per- ception and planning. In IEEE International Conference o...

  12. [20]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 5

    2016

  13. [21]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, 2022. 6

    2022

  14. [22]

    Ronin: Robust neural inertial navigation in the wild: Benchmark, evaluations, & new methods

    Sachini Herath, Hang Yan, and Yasutaka Furukawa. Ronin: Robust neural inertial navigation in the wild: Benchmark, evaluations, & new methods. In IEEE International Confer- ence on Robotics and Automation (ICRA), pages 3146–3152. IEEE, 2020. 1, 2, 3, 5, 6, 8 9

    2020

  15. [23]

    Fusion-dhl: Wifi, imu, and floorplan fusion for dense history of loca- tions in indoor environments

    Sachini Herath, Saghar Irandoust, Bowen Chen, Yiming Qian, Pyojin Kim, and Yasutaka Furukawa. Fusion-dhl: Wifi, imu, and floorplan fusion for dense history of loca- tions in indoor environments. In IEEE International Confer- ence on Robotics and Automation (ICRA), pages 5677–5...

    2021

  16. [24]

    Neural inertial localization

    Sachini Herath, David Caruso, Chen Liu, Yufan Chen, and Yasutaka Furukawa. Neural inertial localization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6604–6613, 2022. 2, 3, 5, 6, 8

    2022

  17. [25]

    Egoexolearn: A dataset for bridging asyn- chronous ego-and exo-centric view of procedural activities in real world

    Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Li- jin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, et al. Egoexolearn: A dataset for bridging asyn- chronous ego-and exo-centric view of procedural activities in real world. In Proceedings of the IEEE/CVF Co...

    2024

  18. [26]

    Vinci: A real-time embodied smart assistant based on egocentric vision-language model

    Yifei Huang, Jilan Xu, Baoqi Pei, Yuping He, Guo Chen, Li- jin Yang, Xinyuan Chen, Yaohui Wang, Zheng Nie, Jinyao Liu, et al. Vinci: A real-time embodied smart assistant based on egocentric vision-language model. arXiv preprint arXiv:2412.21080, 2024. 3

    2024 arXiv

  19. [27]

    Video re- cap: Recursive captioning of hour-long videos

    Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Na- garajan, Lorenzo Torresani, and Gedas Bertasius. Video re- cap: Recursive captioning of hour-long videos. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18198–18208, 2024. 3

    2024

  20. [28]

    Posenet: A convolutional network for real-time 6-dof cam- era relocalization

    Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof cam- era relocalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2938–2946,

  21. [29]

    Deep learning-based pdr scheme that fuses smartphone sensors and gps location changes

    Kwan-Soo Kim and Yoan Shin. Deep learning-based pdr scheme that fuses smartphone sensors and gps location changes. IEEE Access, 9:158616–158631, 2021. 2

    2021

  22. [30]

    Rtab-map as an open- source lidar and visual simultaneous localization and map- ping library for large-scale and long-term online operation

    Mathieu Labb ´e and Franc ¸ois Michaud. Rtab-map as an open- source lidar and visual simultaneous localization and map- ping library for large-scale and long-term online operation. Journal of Field Robotics, 36(2):416–446, 2019. 3

    2019

  23. [31]

    Keyframe-based visual–inertial odometry using nonlinear optimization

    Stefan Leutenegger, Simon Lynen, Michael Bosse, Roland Siegwart, and Paul Furgale. Keyframe-based visual–inertial odometry using nonlinear optimization. The International Journal of Robotics Research, 34(3):314–334, 2015. 2

    2015

  24. [32]

    A bluetooth/pdr inte- gration algorithm for an indoor positioning system

    Xin Li, Jian Wang, and Chunyan Liu. A bluetooth/pdr inte- gration algorithm for an indoor positioning system. Sensors, 15(10):24862–24885, 2015. 2

    2015

  25. [33]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Sol- dan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. arXiv preprint arXiv:2206.01670, 2022. 3

    2022 arXiv

  26. [34]

    Sim- hand: Mining similar hands for large-scale 3d hand pose pre- training

    Nie Lin, Takehiko Ohkawa, Yifei Huang, Mingfang Zhang, Minjie Cai, Ming Li, Ryosuke Furuta, and Yoichi Sato. Sim- hand: Mining similar hands for large-scale 3d hand pose pre- training. arXiv preprint arXiv:2502.15251, 2025. 2

    2025 arXiv

  27. [35]

    Egohdm: An online egocentric-inertial human motion capture, localization, and dense mapping system

    Bonan Liu, Handi Yin, Manuel Kaufmann, Jinhao He, Sammy Christen, Jie Song, and Pan Hui. Egohdm: An online egocentric-inertial human motion capture, localization, and dense mapping system. arXiv preprint arXiv:2409.00343 ,

    arXiv

  28. [36]

    Single-to-dual-view adaptation for egocentric 3d hand pose estimation

    Ruicong Liu, Takehiko Ohkawa, Mingfang Zhang, and Yoichi Sato. Single-to-dual-view adaptation for egocentric 3d hand pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 677–686, 2024. 2

    2024

  29. [37]

    Tlio: Tight learned inertial odometry

    Wenxin Liu, David Caruso, Eddy Ilg, Jing Dong, Anasta- sios I Mourikis, Kostas Daniilidis, Vijay Kumar, and Jakob Engel. Tlio: Tight learned inertial odometry. IEEE Robotics and Automation Letters, 5(4):5653–5660, 2020. 1

    2020

  30. [38]

    Vsrd: Instance-aware volumetric silhouette rendering for weakly supervised 3d object detection

    Zihua Liu, Hiroki Sakuma, and Masatoshi Okutomi. Vsrd: Instance-aware volumetric silhouette rendering for weakly supervised 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17354–17363, 2024. 3

    2024

  31. [39]

    Distinctive image features from scale- invariant keypoints

    David G Lowe. Distinctive image features from scale- invariant keypoints. International Journal of Computer Vi- sion, 60:91–110, 2004. 1

    2004

  32. [40]

    Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data

    Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boular- ias, and Kostas Bekris. Ovir-3d: Open-vocabulary 3d in- stance retrieval without training on 3d data. In Conference on Robot Learning, pages 1610–1620. PMLR, 2023. 3

    2023

  33. [41]

    Imu2clip: language-grounded motion sensor translation with multi- modal contrastive learning

    Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Apara- jita Saraf, Amy Bearman, and Babak Damavandi. Imu2clip: language-grounded motion sensor translation with multi- modal contrastive learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 132...

    2023

  34. [42]

    A multi- state constraint kalman filter for vision-aided inertial navi- gation

    Anastasios I Mourikis and Stergios I Roumeliotis. A multi- state constraint kalman filter for vision-aided inertial navi- gation. In IEEE International Conference on Robotics and Automation (ICRA), pages 3565–3572. IEEE, 2007. 2

    2007

  35. [43]

    Slip: Self-supervision meets language-image pre- training

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 529–544. Springer, 2022. 4, 5, 6, 7, 1

    2022

  36. [44]

    Deep convo- lutional and lstm recurrent neural networks for multimodal wearable activity recognition

    Francisco Javier Ord ´o˜nez and Daniel Roggen. Deep convo- lutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1):115, 2016. 6

    2016

  37. [45]

    Re- view of microsoft hololens applications over the past five years

    Sebeom Park, Shokhrukh Bokijonov, and Yosoon Choi. Re- view of microsoft hololens applications over the past five years. Applied sciences, 11(16):7259, 2021. 1

    2021

  38. [46]

    Modeling fine-grained hand-object dynamics for egocentric video representation learning

    Baoqi Pei, Yifei Huang, Jilan Xu, Guo Chen, Yuping He, Lijin Yang, Yali Wang, Weidi Xie, Yu Qiao, Fei Wu, and Limin Wang. Modeling fine-grained hand-object dynamics for egocentric video representation learning. InInternational Conference on Learning Representations, 2025. 3

    2025

  39. [47]

    Egovlpv2: Egocentric video-language pre-training with fusion in the backbone

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2023

  40. [48]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems, 30, 2017. 5

    2017

  41. [49]

    Vins-mono: A robust and versatile monocular visual-inertial state estimator

    Tong Qin, Peiliang Li, and Shaojie Shen. Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, 34(4):1004–1020, 2018. 2

    2018

  42. [50]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  43. [51]

    Lamar: Benchmarking localization and mapping for augmented reality

    Paul-Edouard Sarlin, Mihai Dusmanu, Johannes L Sch¨onberger, Pablo Speciale, Lukas Gruber, Viktor Larsson, Ondrej Miksik, and Marc Pollefeys. Lamar: Benchmarking localization and mapping for augmented reality. In Proceedings of the European Conference on Computer Vision (ECCV)...

    2022

  44. [52]

    Learning to segment referred objects from narrated egocen- tric videos

    Yuhan Shen, Huiyu Wang, Xitong Yang, Matt Feiszli, Ehsan Elhamifar, Lorenzo Torresani, and Effrosyni Mavroudi. Learning to segment referred objects from narrated egocen- tric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p...

    2024

  45. [53]

    Last-mile navigation using smartphones

    Yuanchao Shu, Kang G Shin, Tian He, and Jiming Chen. Last-mile navigation using smartphones. In Proceedings of the 21st annual international conference on mobile comput- ing and networking, pages 512–524, 2015. 2

    2015

  46. [54]

    Idol: Iner- tial deep orientation-estimation and localization

    Scott Sun, Dennis Melamed, and Kris Kitani. Idol: Iner- tial deep orientation-estimation and localization. InProceed- ings of the AAAI Conference on Artificial Intelligence, pages 6128–6137, 2021. 1

    2021

  47. [55]

    Open- mask3d: Open-vocabulary 3d instance segmentation

    Ayc ¸a Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Open- mask3d: Open-vocabulary 3d instance segmentation. arXiv preprint arXiv:2306.13631, 2023. 3

    2023 arXiv

  48. [56]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105–6114. PMLR,

  49. [57]

    Mnas- net: Platform-aware neural architecture search for mobile

    Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnas- net: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2820–2828,

  50. [58]

    Strapdown inertial nav- igation technology

    David Titterton and John L Weston. Strapdown inertial nav- igation technology. IET, 2004. 1

    2004

  51. [59]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. 5

    2017

  52. [60]

    Retrieval-augmented egocentric video captioning

    Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Retrieval-augmented egocentric video captioning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 13525–13536, 2024. 3

    2024

  53. [61]

    Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding

    Le Xue, Mingfei Gao, Chen Xing, Roberto Mart ´ın-Mart´ın, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF Conference o...

  54. [62]

    Ridi: Robust imu double integration

    Hang Yan, Qi Shan, and Yasutaka Furukawa. Ridi: Robust imu double integration. InProceedings of the European Con- ference on Computer Vision (ECCV), pages 621–636, 2018. 2

    2018

  55. [63]

    Egolocate: Real-time motion capture, localization, and map- ping with sparse body-mounted sensors

    Xinyu Yi, Yuxiao Zhou, Marc Habermann, Vladislav Golyanik, Shaohua Pan, Christian Theobalt, and Feng Xu. Egolocate: Real-time motion capture, localization, and map- ping with sparse body-mounted sensors. ACM Transactions on Graphics (TOG), 42(4):1–17, 2023. 2

    2023

  56. [64]

    Automatic step detection in the accelerometer signal

    Hong Ying, Carmen Silex, Andreas Schnitzer, Steffen Leon- hardt, and Michael Schiek. Automatic step detection in the accelerometer signal. In 4th International Workshop on Wearable and Implantable Body Sensor Networks (BSN

  57. [65]

    Re- tailopt: Opt-in, easy-to-deploy trajectory estimation from smartphone motion data and retail facility information

    Ryo Yonetani, Jun Baba, and Yasutaka Furukawa. Re- tailopt: Opt-in, easy-to-deploy trajectory estimation from smartphone motion data and retail facility information. In Proceedings of the 2024 ACM International Symposium on Wearable Computers, pages 125–132, 2024. 2

    2024

  58. [66]

    Imunet: Efficient regression architecture for inertial imu nav- igation and positioning

    Behnam Zeinali, Hadi Zanddizari, and Morris J Chang. Imunet: Efficient regression architecture for inertial imu nav- igation and positioning. IEEE Transactions on Instrumenta- tion and Measurement, 2024. 1, 2, 5, 6

    2024

  59. [67]

    Helping hands: An object-aware ego-centric video recog- nition model

    Chuhan Zhang, Ankush Gupta, and Andrew Zisserman. Helping hands: An object-aware ego-centric video recog- nition model. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV) , pages 13901–13912, 2023. 3

    2023

  60. [68]

    Structural multiplane image: Bridging neural view synthesis and 3d reconstruction

    Mingfang Zhang, Jinglu Wang, Xiao Li, Yifei Huang, Yoichi Sato, and Yan Lu. Structural multiplane image: Bridging neural view synthesis and 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16707–16716, 2023. 3

    2023

  61. [69]

    Masked video and body-worn imu autoencoder for egocentric action recognition

    Mingfang Zhang, Yifei Huang, Ruicong Liu, and Yoichi Sato. Masked video and body-worn imu autoencoder for egocentric action recognition. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV) , pages 312–

  62. [70]

    Learning video representations from large language models

    Yue Zhao, Ishan Misra, Philipp Kr ¨ahenb¨uhl, and Rohit Gird- har. Learning video representations from large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6586–6597, 2023. 3 11 Egocentric Action-aware Inert...

    2023

  63. [73]

    While InternVL scores the highest on some metrics, SLIP generally attains stronger localization accuracy

    Supplementary Ablation Studies Different Vision-Language Encoder in Stage 1 We compare CLIP-Large[50], InternVL[7], and SLIP-Base[43] as possible backbones for the vision-language encoders. While InternVL scores the highest on some metrics, SLIP generally attains stronger loca...

  64. [74]

    The heatmaps featured in the video are interpreted similarly to those shown in Fig

    Supplementary Video We create a supplementary video that further illustrates the performance of our framework. The heatmaps featured in the video are interpreted similarly to those shown in Fig. 5, providing visual insight into the localization process. The video demonstrates ...

  65. [330]

    Springer, 2024. 2, 6

    2024

  66. [2007]

    Springer, 2007

    March 26–28, 2007 RWTH Aachen University, Ger- many, pages 80–85. Springer, 2007. 1, 2

    2007

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.