arxiv: 2604.16341 · v1 · submitted 2026-03-14 · 💻 cs.HC · cs.CV

Recognition: no theorem link

Deep Learning for Virtual Reality User Identification: A Benchmark

Davide Frizzo , Fabrizio Genilotti , David Petrovic , Arianna Stropeni , Francesco Borsatti , Davide Dalle Pezze , Riccardo De Monte , Manuel Barusco

show 1 more author

Gian Antonio Susto

Authors on Pith no claims yet

Pith reviewed 2026-05-15 11:24 UTC · model grok-4.3

classification 💻 cs.HC cs.CV

keywords virtual realityuser identificationdeep learningmotion trackingbehavioral biometricsstate space modelsbenchmarkauthentication

0 comments

The pith

A benchmark evaluates multiple deep learning architectures for identifying users from VR headset and controller motion data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper benchmarks established and emerging deep learning models to identify individual users based solely on their movements while using virtual reality equipment. It draws on time-series data collected from 71 users playing Half-Life: Alyx, including head and hand tracking from headsets and controllers. The work compares LSTM, GRU, CNN, TCN, Transformer, and state space models to set performance baselines. The central aim is to support secure, privacy-preserving authentication in VR applications, particularly in manufacturing settings where equipment access must be controlled without traditional credentials. A sympathetic reader would care because reliable motion-based identification could replace less convenient or less private methods in shared immersive environments.

Core claim

We evaluate both established architectures (Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Neural Network (CNN), Temporal Convolutional Network (TCN), Transformer) and the emerging SSMs on time series motion data from the Who is Alyx VR dataset with 71 users, providing the first comprehensive benchmark for VR user identification and baseline metrics for privacy preserving authentication systems in manufacturing environments.

What carries the argument

Comparison of time-series deep learning architectures on VR motion tracking sequences for user identification.

If this is right

Establishes baseline performance metrics that future work on VR identification can compare against.
Enables development of authentication systems that avoid storing traditional personal identifiers.
Applies directly to secure access control for VR equipment in manufacturing.
Includes state space models as a viable option alongside recurrent and convolutional networks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The benchmark could support real-time identification in multi-user VR training simulations.
Motion-based identification raises new questions about long-term privacy of movement data collected in consumer VR.
Similar techniques might transfer to augmented reality headsets that capture comparable tracking signals.

Load-bearing premise

VR motion tracking data from headsets and controllers contains sufficiently unique and stable user-specific patterns across sessions to support reliable identification without additional features or context.

What would settle it

An experiment showing that models trained on one VR session achieve no better than chance accuracy when tested on data from a separate session would falsify the stability of user-specific motion patterns.

Figures

Figures reproduced from arXiv: 2604.16341 by Arianna Stropeni, Davide Dalle Pezze, Davide Frizzo, David Petrovic, Fabrizio Genilotti, Francesco Borsatti, Gian Antonio Susto, Manuel Barusco, Riccardo De Monte.

**Figure 2.** Figure 2: Test sample size vs. Mean Reciprocal Rank (MRR) for the BRA, BRV and BR encodings [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Test sample size vs. MRR V. RESULTS In this section, we highlight the experimental results obtained on the Who is Alyx dataset. A. Performance Comparison Across Architectures First, we evaluate the capabilities of deep learning models in VR User Identification. In [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

read the original abstract

Virtual Reality (VR) applications require robust user identification systems to ensure secure access to equipment and protect worker identities. Motion tracking data from VR headsets and controllers has emerged as a powerful behavioral biometric, with recent studies demonstrating identification accuracies exceeding 94% across a large user base. However, the application of modern deep learning architectures, particularly State Space Models (SSM), to VR scenarios remains largely unexplored. In this work, we benchmark user identification performance across the large-scale Who is Alyx VR dataset, gathering data from 71 users playing the popular Half-Life:Alyx game. We evaluate both established architectures (Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Neural Network (CNN), Temporal Convolutional Network (TCN), Transformer) and the emerging SSMs on time series motion data. Our results provide the first comprehensive benchmark of state-of-the-art and novel architectures for VR user identification, establishing baseline performance metrics for future privacy preserving authentication systems in manufacturing environments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Benchmark applies standard time-series models including SSMs to VR motion data from 71 users but shows no results and likely skips cross-session testing.

read the letter

This paper benchmarks several deep learning models on VR motion tracking data to identify users from the Who is Alyx dataset collected during Half-Life: Alyx gameplay. It tests LSTM, GRU, CNN, TCN, Transformer, and state space models on headset and controller signals from 71 users, positioning the work as the first comprehensive comparison that brings in SSMs for this task. The practical framing around secure access in manufacturing environments is clear enough. What it does reasonably well is pull together a sizable real-world motion dataset and run a broad set of established plus newer architectures on the same time-series inputs, which can give other researchers a starting point for behavioral biometrics in VR. The inclusion of SSMs is a timely choice given their recent use in sequence tasks. The soft spots are straightforward. The abstract asserts high accuracies and a full benchmark yet supplies zero numerical results, training details, split information, or error analysis, so the central claims rest on nothing visible. The cross-session stability issue raised in the stress-test note looks real: if the evaluation stays within single recordings rather than holding out separate sessions, reported performance can be inflated by session artifacts and fail to support reliable authentication when users return later. That assumption sits at the core of the privacy-preserving angle. This work is aimed at HCI and biometrics researchers who need quick baselines on VR motion signals. A reader already familiar with time-series classification would get some value from the model lineup once the full experiments are shown. It deserves peer review so referees can examine the actual numbers, splits, and any generalization tests rather than desk-rejecting on the abstract alone.

Referee Report

2 major / 0 minor

Summary. The manuscript benchmarks several deep learning architectures (LSTM, GRU, CNN, TCN, Transformer, and State Space Models) for user identification from VR headset and controller motion tracking data on the Who is Alyx dataset collected from 71 users playing Half-Life: Alyx. It claims to deliver the first comprehensive benchmark of these models and to establish baseline performance metrics for privacy-preserving authentication systems in manufacturing environments.

Significance. If the performance claims are supported by detailed numerical results and proper cross-session validation, the work would supply useful empirical baselines for behavioral biometrics in VR, particularly by evaluating emerging State Space Models alongside established recurrent and convolutional architectures on time-series motion data. The scale of the 71-user dataset is a strength for establishing reference metrics in this application domain.

major comments (2)

[Dataset Description] Dataset section: The Who is Alyx dataset description gives no indication of session-level splits or temporal hold-outs. This is load-bearing for the central claim of reliable identification for authentication systems, because within-recording accuracy can exceed 94% due to session-specific artifacts while failing to generalize when a user returns in a new session.
[Abstract] Abstract and Results section: The abstract asserts high accuracies and a comprehensive benchmark yet supplies no numerical results, training details, validation splits, or error analysis. This leaves the central performance claims unsupported by visible evidence.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback, which has helped us improve the clarity and rigor of our manuscript. We address each major comment below and have revised the paper accordingly.

read point-by-point responses

Referee: [Dataset Description] Dataset section: The Who is Alyx dataset description gives no indication of session-level splits or temporal hold-outs. This is load-bearing for the central claim of reliable identification for authentication systems, because within-recording accuracy can exceed 94% due to session-specific artifacts while failing to generalize when a user returns in a new session.

Authors: We agree that session-level splits and temporal hold-outs are essential to demonstrate generalization beyond session-specific artifacts. The original manuscript described the overall dataset collection but did not explicitly detail the partitioning strategy. In the revised version, we have expanded the Dataset section to include a full description of our cross-session validation protocol, using leave-one-session-out splits to ensure models are evaluated on temporally distinct sessions. This directly supports the authentication use case by reporting performance that accounts for session variability. revision: yes
Referee: [Abstract] Abstract and Results section: The abstract asserts high accuracies and a comprehensive benchmark yet supplies no numerical results, training details, validation splits, or error analysis. This leaves the central performance claims unsupported by visible evidence.

Authors: We acknowledge that the original abstract was high-level and omitted specific metrics and methodological details. We have revised the abstract to include key numerical results (e.g., peak identification accuracy across models) along with a concise statement of the validation approach. The Results section has been expanded with training hyperparameters, explicit validation split descriptions (now including the session-level protocol), and an error analysis to provide the requested evidence and transparency. revision: yes

Circularity Check

0 steps flagged

No circularity: pure empirical benchmark study

full rationale

The paper is a standard empirical comparison of neural architectures (LSTM, GRU, CNN, TCN, Transformer, SSM) on the Who is Alyx motion dataset for user identification. No derivations, first-principles predictions, fitted parameters renamed as outputs, or self-citation load-bearing steps are present. All results are direct performance metrics on held-out data splits; the central claim is simply that the benchmark was performed and metrics were recorded. This matches the default expectation of a non-circular empirical study.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on standard machine-learning assumptions for time-series classification rather than new theoretical constructs. No invented physical entities or ad-hoc axioms are introduced.

free parameters (1)

model hyperparameters
Learning rates, batch sizes, layer dimensions, and training epochs are tuned per architecture but not enumerated in the abstract.

axioms (1)

domain assumption Motion sequences from VR controllers contain user-discriminative temporal patterns that neural networks can learn.
Invoked implicitly when claiming identification accuracies from raw tracking data.

pith-pipeline@v0.9.0 · 5499 in / 1148 out tokens · 43590 ms · 2026-05-15T11:24:11.798470+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 3 internal anchors

[1]

Unique identification of 50,000+ virtual reality users from head & hand motion data,

V . Nair, W. Guo, J. Mattern, R. Wang, J. F. O’Brien, L. Rosenberg, and D. Song, “Unique identification of 50,000+ virtual reality users from head & hand motion data,” in32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 895–910

work page 2023
[2]

Sok: Data privacy in virtual reality,

G. M. Garrido, V . Nair, and D. Song, “Sok: Data privacy in virtual reality,”arXiv preprint arXiv:2301.05940, 2023

work page arXiv 2023
[4]

Diagonal state spaces are as effective as structured state spaces,

A. Gupta, A. Gu, and J. Berant, “Diagonal state spaces are as effective as structured state spaces,” 2022. [Online]. Available: https://arxiv.org/abs/2203.14343

work page arXiv 2022
[5]

Sim- plified state space layers for sequence modeling.arXiv preprint arXiv:2208.04933, 2022

J. T. H. Smith, A. Warrington, and S. W. Linderman, “Simplified state space layers for sequence modeling,” 2023. [Online]. Available: https://arxiv.org/abs/2208.04933

work page arXiv 2023
[6]

Behavioural biometrics in vr: Identifying people from body motion and relations in virtual reality,

K. Pfeuffer, M. J. Geiger, S. Prange, L. Mecke, D. Buschek, and F. Alt, “Behavioural biometrics in vr: Identifying people from body motion and relations in virtual reality,” inProceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–12

work page 2019
[7]

Understanding user identification in virtual reality through behavioral biometrics and the effect of body normaliza- tion,

J. Liebers, M. Abdelaziz, L. Mecke, A. Saad, J. Auda, U. Gruenefeld, F. Alt, and S. Schneegass, “Understanding user identification in virtual reality through behavioral biometrics and the effect of body normaliza- tion,” inProceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1–11

work page 2021
[8]

Exploring the stability of behavioral biometrics in virtual reality in a remote field study: Towards implicit and continuous user identification through body movements,

J. Liebers, C. Burschik, U. Gruenefeld, and S. Schneegass, “Exploring the stability of behavioral biometrics in virtual reality in a remote field study: Towards implicit and continuous user identification through body movements,” inProceedings of the 29th ACM Symposium on Virtual Reality Software and Technology, 2023, pp. 1–12

work page 2023
[9]

Using siamese neural networks to perform cross-system behavioral authentication in virtual reality,

R. Miller, N. K. Banerjee, and S. Banerjee, “Using siamese neural networks to perform cross-system behavioral authentication in virtual reality,” in2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, 2021, pp. 140–149

work page 2021
[10]

Combining real-world constraints on user behavior with deep neural networks for virtual reality (vr) biometrics,

——, “Combining real-world constraints on user behavior with deep neural networks for virtual reality (vr) biometrics,” in2022 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, 2022, pp. 409–418

work page 2022
[11]

Motion-Based User Identification across XR and Metaverse Applications by Deep Classification and Similarity Learning

L. Schach, C. Rack, R. P. McMahan, and M. E. Latoschik, “Motion- based user identification across xr and metaverse applications by deep classification and similarity learning,”arXiv preprint arXiv:2509.08539, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[12]

Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition,

F. J. Ord ´o˜nez and D. Roggen, “Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition,”Sensors, vol. 16, no. 1, p. 115, 2016

work page 2016
[13]

Inceptiontime: Finding alexnet for time series classification,

H. Ismail Fawaz, B. Lucas, G. Forestier, C. Pelletier, D. F. Schmidt, J. Weber, G. I. Webb, L. Idoumghar, P.-A. Muller, and F. Petitjean, “Inceptiontime: Finding alexnet for time series classification,”Data Mining and Knowledge Discovery, vol. 34, no. 6, pp. 1936–1962, 2020

work page 1936
[14]

Efficiently Modeling Long Sequences with Structured State Spaces

A. Gu, K. Goel, and C. R ´e, “Efficiently modeling long sequences with structured state spaces,”arXiv preprint arXiv:2111.00396, 2021

work page internal anchor Pith review Pith/arXiv arXiv 2021
[15]

Who is alyx? a new behavioral biometric dataset for user identification in xr,

C. Rack, T. Fernando, M. Yalcin, A. Hotho, and M. E. Latoschik, “Who is alyx? a new behavioral biometric dataset for user identification in xr,” Frontiers in Virtual Reality, vol. 4, p. 1272234, 2023

work page 2023
[16]

Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State- Space Architectures from S4 to Mamba

S. Somvanshi, M. M. Islam, M. S. Mimi, S. B. B. Polock, G. Chhetri, and S. Das, “From s4 to mamba: A comprehensive survey on structured state space models,” 2025. [Online]. Available: https://arxiv.org/abs/2503.18970

work page internal anchor Pith review Pith/arXiv arXiv 2025