pith. sign in

arxiv: 2606.17741 · v1 · pith:523DCT55new · submitted 2026-06-16 · 📡 eess.SY · cs.HC· cs.SY

A Wearable Multimodal Ultrasound+Inertial System for Real-Time Virtual Reality Interaction

Giusy Spacone , Sebastian Frey , Enzo Baraldi , Mattia Orlandi , Luca Benini , Andrea Cossettini This is my paper

Pith reviewed 2026-06-26 23:02 UTC · model grok-4.3

classification 📡 eess.SY cs.HCcs.SY
keywords wearable sensingultrasoundinertial measurement unitvirtual realityhand pose estimationmultimodal interfacereal-time interactionforearm sensing
0
0 comments X

The pith

A wearable ultrasound and inertial system tracks hand poses for VR interaction with 80 percent accuracy and low power use.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a fully wearable system that combines A-mode ultrasound and inertial sensing on the forearm and upper arm to estimate hand poses and forearm positions for controlling a VR environment in real time. It demonstrates inter-session offline accuracies of 80±6% for hand pose and 77±7% for position, with online task success rates exceeding 88% after five minutes of fine-tuning across three tasks. This matters because it enables camera-free, long-duration wearable VR interfaces that consume only 19.9 mW, supporting continuous operation for over two and a half days on a small battery. The system uses an end-to-end framework for acquisition and Unity integration, evaluated on five subjects over multiple sessions.

Core claim

The multimodal interface achieves offline inter-session accuracies of 80±6% for hand pose estimation and 77±7% for forearm position estimation, and online success rates of 92.0±16.0%, 88.0±9.8%, and 96.0±8.0% for cylinder grasping, marble pinching, and liquid pouring tasks after 5 min fine-tuning, all at 19.9 mW power consumption.

What carries the argument

The multimodal learning pipeline that fuses concurrent ultrasound and inertial data for simultaneous hand pose and forearm position estimation in 2D space.

If this is right

  • The system supports three functional VR tasks: gross motor grasping, fine motor pinching, and liquid pouring.
  • Power draw of 19.9 mW allows more than 2.5 days of continuous operation on a 350 mAh battery.
  • Minimal 5-minute fine-tuning enables high online success rates across subjects.
  • Offline accuracies hold across multiple acquisition sessions on different days.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Such a system could be adapted for assistive devices in rehabilitation by mapping muscle activity to prosthetic control.
  • Combining with additional modalities like EMG might further enhance robustness without increasing power significantly.
  • Real-time performance suggests potential for integration into mobile VR headsets for untethered use.

Load-bearing premise

The multimodal learning pipeline, trained on five subjects across five sessions, generalizes to new sessions and users with only 5 minutes of fine-tuning.

What would settle it

An experiment where the system is tested on a new user with no fine-tuning or with fine-tuning on unrelated data, resulting in task success rates below 50%.

Figures

Figures reproduced from arXiv: 2606.17741 by Andrea Cossettini, Enzo Baraldi, Giusy Spacone, Luca Benini, Mattia Orlandi, Sebastian Frey.

Figure 1
Figure 1. Figure 1: Overview of Sensing System. The acquisition electronics (with the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the software architecture: WULPUS communicates [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of Hand Poses (top) and Forearm Positions (bottom) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Top: visualization of the Unity-based interaction environment [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: TSNE projections of the latent representations before the clas [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Average SR per task in the online validation with different amounts [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Average CT per task in the online validation with different amounts [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
read the original abstract

A-mode ultrasound (US) is a promising sensing modality for Virtual Reality (VR) interaction, as it enables the mapping of muscular activity into control commands while retaining the benefits of wearable sensing. However, existing approaches still face limitations in terms of wearability and interaction complexity, often relying on external hardware such as cameras. In this work, we propose a fully wearable multimodal interface for real-time VR-interaction, based on concurrent US and inertial (accelerometry) sensing from the forearm and upper arm. The system is built on the WULPUS platform and integrates an end-to-end software framework for real-time acquisition, visualization, and communication with a Unity-based VR environment. A multimodal learning pipeline is introduced for concurrent hand pose and forearm position estimation in 2D space. The interface is evaluated through offline and online experiments with five subjects, during the execution of three functional tasks: cylinder grasping (gross motor) and relocation, marble pinching (fine motor) and relocation, and liquid pouring. For offline experiments, we collect 5 acquisition sessions across multiple days, achieving an average inter-session accuracy across subjects of 80$\pm$6\% for hand pose estimation and 77$\pm$7\% for forearm position estimation. Online validation with minimal fine-tuning (5 min) demonstrates success rates of 92.0$\pm$16.0\%, 88.0$\pm$9.8\%, and 96.0$\pm$8.0\% for the three tasks, respectively. With a power consumption of only 19.9~mW, our system enables more than 2.5 days of continuous use on a small 350 mAh LiPo battery without the need for recharge, enabling truly wearable, multimodal, and functionally meaningful VR interaction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims to introduce a fully wearable multimodal A-mode ultrasound + inertial (accelerometry) sensing system from forearm and upper arm for real-time VR interaction. It reports offline inter-session accuracies of 80±6% (hand pose) and 77±7% (forearm position) across five subjects and five sessions, plus online task success rates of 92.0±16.0%, 88.0±9.8%, and 96.0±8.0% for cylinder grasping, marble pinching, and liquid pouring after only 5 min fine-tuning, all at 19.9 mW enabling >2.5 days continuous operation on a 350 mAh battery.

Significance. If the reported generalization via 5-min fine-tuning holds under subject-independent protocols, the work would advance wearable VR interfaces by showing that multimodal US+IMU fusion can deliver functionally meaningful, camera-free tracking at very low power. The end-to-end real-time framework with Unity integration is a practical strength.

major comments (2)
  1. [Offline experiments] Offline experiments section: the inter-session accuracies (80±6% hand pose, 77±7% forearm position) are reported without explicit subject-wise splits, leave-one-subject-out results, or confirmation that training and test sessions are fully disjoint per user; this directly affects whether the multimodal pipeline supports the claimed generalization to new sessions/users.
  2. [Online validation] Online validation section: the online success rates inherit large standard deviations (e.g., ±16.0% on first task) with n=5; the manuscript must clarify whether the 5-min fine-tuning data came from held-out users/sessions rather than the training cohort and provide per-subject breakdowns, as this is the load-bearing step for the central online performance claim.
minor comments (1)
  1. [Abstract] Abstract and methods: the exact sensor placement protocol on forearm/upper arm and the multimodal model architecture details (e.g., fusion strategy) should be expanded for reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the experimental protocols. We address each major point below and will revise the manuscript to improve transparency on data splits and per-subject results.

read point-by-point responses

  1. Referee: [Offline experiments] Offline experiments section: the inter-session accuracies (80±6% hand pose, 77±7% forearm position) are reported without explicit subject-wise splits, leave-one-subject-out results, or confirmation that training and test sessions are fully disjoint per user; this directly affects whether the multimodal pipeline supports the claimed generalization to new sessions/users.

    Authors: The inter-session evaluation is performed within each subject: for every user, models are trained on a subset of their five sessions and tested on the remaining held-out sessions from the same subject, ensuring complete disjointness between training and test data per user. Leave-one-subject-out was not performed because the study focuses on within-user session-to-session generalization rather than cross-subject generalization. We will revise the Offline experiments section to explicitly describe these subject-wise splits, confirm the disjoint session protocol, and report subject-specific accuracies alongside the aggregate figures. revision: yes

  2. Referee: [Online validation] Online validation section: the online success rates inherit large standard deviations (e.g., ±16.0% on first task) with n=5; the manuscript must clarify whether the 5-min fine-tuning data came from held-out users/sessions rather than the training cohort and provide per-subject breakdowns, as this is the load-bearing step for the central online performance claim.

    Authors: The 5-min fine-tuning data consists of new recordings collected from the same five subjects but drawn from sessions held out from the initial offline training sets (i.e., per-user held-out data). The reported standard deviations reflect genuine inter-subject variability in a small cohort, which is typical for wearable sensing studies. We will revise the Online validation section to explicitly state that fine-tuning uses held-out per-subject data, add a table or figure with per-subject success rates, and retain the aggregate statistics with their standard deviations. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical validation with no derivation chain

full rationale

The paper reports hardware design, data collection from five subjects over five sessions, training of a multimodal learning pipeline, and offline/online accuracy metrics on held-out or fine-tuned data. No equations, first-principles derivations, or predictions are present that could reduce to fitted inputs by construction. All reported figures (80±6%, 77±7%, online success rates) are direct experimental outcomes, not self-referential. Self-citation load-bearing, ansatz smuggling, or renaming of known results are absent. The generalization assumption is an empirical question, not a circularity issue.

Axiom & Free-Parameter Ledger

1 free parameters · 0 axioms · 0 invented entities

Abstract-only review limits visibility; the system depends on standard supervised ML training assumptions and consistent sensor placement, with model parameters fitted to subject data.

free parameters (1)
  • multimodal ML model parameters
    Weights and hyperparameters of the hand-pose and forearm-position estimator are fitted to the collected sessions.

pith-pipeline@v0.9.1-grok · 5884 in / 1269 out tokens · 51670 ms · 2026-06-26T23:02:39.677127+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references

  1. [1]

    Hand interaction designs in mixed and augmented reality head mounted display: a scoping review and classification,

    R. Nguyenet al., “Hand interaction designs in mixed and augmented reality head mounted display: a scoping review and classification,” Frontiers in Virtual Reality, vol. 4, p. 1171230, 2023

    2023

  2. [2]

    Virtual/augmented reality for rehabilitation applications using electromyography as control/biofeedback: System- atic literature review,

    C. L. Toledo-Peralet al., “Virtual/augmented reality for rehabilitation applications using electromyography as control/biofeedback: System- atic literature review,”Electronics, vol. 11, no. 14, 2022

    2022

  3. [3]

    A generic non-invasive neuromotor interface for human-computer interaction,

    P. Kaifoshet al., “A generic non-invasive neuromotor interface for human-computer interaction,”Nature, vol. 645, no. 8081, pp. 702– 711, 2025

    2025

  4. [4]

    A systematic review of commercial smart gloves: Current status and applications,

    M. Caeiro-Rodr ´ıguezet al., “A systematic review of commercial smart gloves: Current status and applications,”Sensors, vol. 21, no. 8, 2021

    2021

  5. [5]

    A methodological and structural review of hand gesture recognition across diverse data modalities,

    J. Shinet al., “A methodological and structural review of hand gesture recognition across diverse data modalities,”IEEE Access, vol. 12, pp. 142 606–142 639, 2024

    2024

  6. [6]

    Ultrasound as a neurorobotic interface: A review,

    X. Yanget al., “Ultrasound as a neurorobotic interface: A review,” IEEE Trans. Syst. Man Cybern. Syst., vol. 54, no. 6, pp. 3534–3546, 2024

    2024

  7. [7]

    A wearable multisensor fusion system for neuropros- thetic hand,

    Z. Yinet al., “A wearable multisensor fusion system for neuropros- thetic hand,”IEEE Sensors Journal, vol. 25, no. 8, pp. 12 547–12 558, 2025

    2025

  8. [8]

    Wearable and ultra-low-power fusion of emg and a-mode us for hand-wrist kinematic tracking,

    G. Spaconeet al., “Wearable and ultra-low-power fusion of emg and a-mode us for hand-wrist kinematic tracking,” in2025 IEEE BioCAS, 2025, pp. 314–318

    2025

  9. [9]

    Virtual reality interactions via a user-generic ultrasound human-machine interface for wrist and hand tracking,

    G. Sgambatoet al., “Virtual reality interactions via a user-generic ultrasound human-machine interface for wrist and hand tracking,” Nature Communications, vol. 16, no. 1, p. 11062, 2025

    2025

  10. [10]

    Wulpus: a wearable ultra low-power ultrasound probe for multi-day monitoring of carotid artery and muscle activity,

    S. Freyet al., “Wulpus: a wearable ultra low-power ultrasound probe for multi-day monitoring of carotid artery and muscle activity,” in2022 IEEE IUS, 2022, pp. 1–4

    2022

  11. [11]

    Towards wearable a-mode ultrasound sensing for real- time finger motion recognition,

    X. Yanget al., “Towards wearable a-mode ultrasound sensing for real- time finger motion recognition,”IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 26, no. 6, pp. 1199–1208, 2018

    2018

  12. [12]

    Wearable real-time gesture recognition scheme based on a-mode ultrasound,

    Z. Luet al., “Wearable real-time gesture recognition scheme based on a-mode ultrasound,”IEEE Trans. Neural Syst. Rehabil. Eng., vol. 30, pp. 2623–2629, 2022

    2022

  13. [13]

    Sonomyographic prosthetic interaction: Online simultaneous and proportional control of wrist and hand motions using semisupervised learning,

    X. Yang and othersi, “Sonomyographic prosthetic interaction: Online simultaneous and proportional control of wrist and hand motions using semisupervised learning,”IEEE TMECH, vol. 28, no. 2, pp. 804–813, 2022

    2022

  14. [14]

    High performance wearable ultrasound as a human-machine interface for wrist and hand kinematic tracking,

    B. G. Sgambatoet al., “High performance wearable ultrasound as a human-machine interface for wrist and hand kinematic tracking,”IEEE Trans. Biomed. Eng., vol. 71, no. 2, pp. 484–493, 2024

    2024

  15. [15]

    Synchronous gesture recognition and arm joint angle monitoring for human-machine interaction using multiple flexible ultrasonic patches,

    Z. Tanget al., “Synchronous gesture recognition and arm joint angle monitoring for human-machine interaction using multiple flexible ultrasonic patches,”Advanced Functional Materials, p. e05131, 2025

    2025

  16. [16]

    Tracking of wrist and hand kinematics with ultra low power wearable a-mode ultrasound,

    G. Spaconeet al., “Tracking of wrist and hand kinematics with ultra low power wearable a-mode ultrasound,”IEEE T-BioCAS, vol. 19, no. 3, pp. 536–548, 2025

    2025

  17. [17]

    https://amzn.eu/d/9dQ3nRG

    Hydrogel pad. https://amzn.eu/d/9dQ3nRG

  18. [18]

    Real-time motor unit tracking from semg signals with adaptive ica on a parallel ultra-low power processor,

    M. Orlandiet al., “Real-time motor unit tracking from semg signals with adaptive ica on a parallel ultra-low power processor,”IEEE Transactions on Biomedical Circuits and Systems, no. 4, pp. 771–782, 2024

    2024

  19. [19]

    Nordic power profiler kit,

    “Nordic power profiler kit,” https://www.nordicsemi.com/Products/ Development-hardware/Power-Profiler-Kit-2/Download

  20. [20]

    A wearable ultrasound system for sensing muscular morphological deformations,

    X. Yanget al., “A wearable ultrasound system for sensing muscular morphological deformations,”IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, no. 6, pp. 3370–3379, 2021

    2021

  21. [21]

    Portable ultrasound research system for use in automated bladder monitoring with machine-learning-based segmen- tation,

    M. Fournelleet al., “Portable ultrasound research system for use in automated bladder monitoring with machine-learning-based segmen- tation,”Sensors, vol. 21, no. 19, 2021

    2021