Multisensory cues facilitate coordination of stepping movements with a virtual reality avatar
Pith reviewed 2026-05-25 17:15 UTC · model grok-4.3
The pith
An avatar in VR guides human stepping when footstep sounds match the visuals.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Participants instructed to step in time with a virtual avatar showed slow drift in step asynchronies under visual-only conditions, but asynchronies stabilized when footstep sounds were included. A phase perturbation of 15% in the avatar's step cycle produced a clear corrective response in the auditory-visual conditions. The authors conclude that an avatar's movements can influence a person's gait provided relevant auditory cues are present to achieve suitable accuracy.
What carries the argument
Asynchrony measurement between participant step onsets and avatar step onsets, tested with and without added congruent footstep sounds and with a 15% phase perturbation applied to one avatar step cycle.
If this is right
- An avatar's movements can influence a person's own gait when congruent auditory cues are included.
- Visual cues alone produce unstable coordination that drifts over repeated steps.
- Congruent footstep sounds enable both stable timing and corrective adjustments when the avatar's movement timing changes.
- Humanoid avatars provide a feasible method for visually cued gait guidance once auditory cues are added.
Where Pith is reading between the lines
- The same multisensory approach might support coordination training in other rhythmic movements beyond walking.
- Designers of VR movement systems could test whether different sound types or timings further improve correction speed.
- The method could be extended to cases where the avatar leads or follows the user at varying ratios rather than strict synchrony.
Load-bearing premise
Any difference in stepping stability between the visual-only and auditory-visual conditions is caused only by the added footstep sounds and not by other differences in the experimental setup or instructions.
What would settle it
A replication in which the only change is the presence or absence of footstep sounds still shows equivalent drift in asynchronies and no corrective response to the phase perturbation.
Figures
read the original abstract
The effectiveness of simple sensory cues for retraining gait have been demonstrated, yet the feasibility of humanoid avatars for entrainment have yet to be investigated. Here, we describe the development of a novel method of visually cued training, in the form of a virtual partner, and investigate its ability to provide movement guidance in the form of stepping. Real stepping movements were mapped onto an avatar using motion capture data. The trajectory of one of the avatar step cycles was then accelerated or decelerated by 15% to create a perturbation. Healthy participants were motion captured while instructed to step in time to the avatar's movements, as viewed through a virtual reality headset. Step onset times were used to measure the timing errors (asynchronies) between them. Participants completed either a visual-only condition, or auditory-visual with footstep sounds included. Participants' asynchronies exhibited slow drift in the Visual-Only condition, but became stable in the Auditory-Visual condition. Moreover, we observed a clear corrective response to the phase perturbation in both auditory-visual conditions. We conclude that an avatar's movements can be used to influence a person's own gait, but should include relevant auditory cues congruent with the movement to ensure a suitable accuracy is achieved.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript describes an experiment in which participants stepped in time with a VR avatar whose movements were derived from motion capture. One step cycle was perturbed by accelerating or decelerating it 15%. Participants completed either a visual-only condition or an auditory-visual condition that added congruent footstep sounds. The reported results are that asynchronies between participant and avatar steps drifted slowly in the visual-only condition but remained stable in the auditory-visual condition, and a corrective response to the perturbation appeared in the auditory-visual conditions. The authors conclude that an avatar can influence gait but requires congruent auditory cues to achieve suitable accuracy.
Significance. If the empirical claims are robust, the work provides evidence that multisensory cues improve entrainment to a virtual partner during stepping, which could inform the design of VR-based gait training or rehabilitation systems. The perturbation method and direct measurement of asynchronies constitute a clear test of both stability and error correction.
major comments (3)
- [Abstract / Methods] Abstract and Methods: The description states that participants completed 'either a visual-only condition, or auditory-visual with footstep sounds included' but supplies no information confirming that visual rendering, avatar motion parameters, trial structure, instructions, or participant expectations were identical across conditions. Because the central claim attributes the difference in asynchrony drift and the presence of corrective responses specifically to the addition of auditory cues, any unmentioned covariation between conditions undermines the causal attribution.
- [Results] Results: The abstract and provided summary contain no participant numbers, statistical tests, measures of variability, or raw data summaries for the reported slow drift versus stability or for the corrective responses. Without these, it is not possible to assess whether the stabilization and corrective-response claims are supported by the measurements.
- [Conclusion] Conclusion: The recommendation that 'relevant auditory cues congruent with the movement' should be included to ensure accuracy rests on the assumption that the observed difference is caused by audition rather than other factors; the manuscript does not report controls that would support this attribution.
minor comments (1)
- [Abstract] Abstract: The phrase 'in both auditory-visual conditions' appears, yet only a single auditory-visual condition is described; clarify whether multiple AV conditions existed.
Simulated Author's Rebuttal
We thank the referee for these detailed comments, which help clarify the presentation of our methods and results. We address each major comment below.
read point-by-point responses
-
Referee: [Abstract / Methods] Abstract and Methods: The description states that participants completed 'either a visual-only condition, or auditory-visual with footstep sounds included' but supplies no information confirming that visual rendering, avatar motion parameters, trial structure, instructions, or participant expectations were identical across conditions. Because the central claim attributes the difference in asynchrony drift and the presence of corrective responses specifically to the addition of auditory cues, any unmentioned covariation between conditions undermines the causal attribution.
Authors: The Methods section specifies that the visual avatar rendering, motion-capture parameters, trial structure, and participant instructions were identical in both conditions; the sole difference was the addition of congruent footstep sounds. Participant expectations were controlled via the same instructions in both conditions. We will revise the Abstract and Methods to state this equivalence explicitly. revision: yes
-
Referee: [Results] Results: The abstract and provided summary contain no participant numbers, statistical tests, measures of variability, or raw data summaries for the reported slow drift versus stability or for the corrective responses. Without these, it is not possible to assess whether the stabilization and corrective-response claims are supported by the measurements.
Authors: The full Results section contains the participant count, the statistical tests performed on asynchronies and phase correction, and associated variability measures. We agree the Abstract should summarize these quantitative elements and will revise it accordingly. revision: yes
-
Referee: [Conclusion] Conclusion: The recommendation that 'relevant auditory cues congruent with the movement' should be included to ensure accuracy rests on the assumption that the observed difference is caused by audition rather than other factors; the manuscript does not report controls that would support this attribution.
Authors: The design held all factors constant except the auditory cues, as described in Methods. We will revise the Conclusion to explicitly note these controls and thereby support the attribution. revision: yes
Circularity Check
No circularity: purely empirical timing measurements with no derivations or self-referential predictions
full rationale
The paper reports a behavioral experiment measuring step asynchronies via motion capture in visual-only vs. auditory-visual conditions. No equations, models, fitted parameters, predictions, or derivation chains appear in the abstract or described methods. Results rest on direct timing data rather than any construction that reduces to its own inputs. No self-citations of uniqueness theorems or ansatzes are invoked. This matches the default expectation of a non-circular empirical study.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Step onset times provide a valid and sensitive measure of coordination with the avatar
- domain assumption Participants will attempt to follow the instruction to step in time with the avatar
Reference graph
Works this paper leans on
-
[1]
Rizzolatti G, Craighero L. the Mirror-Neuron System. Annu Rev Neurosci [Internet]. 2004;27(1):169–92. Available from: http://www.annualreviews.org/doi/abs/10.1146/annurev.neuro.27.070203.144230
-
[2]
Sensorimotor Foundations of Higher Cognition
Haggard P, Rossetti Y, Kawato M. Sensorimotor Foundations of Higher Cognition. Oxford University Press; 2008. 683 p
work page 2008
-
[3]
Neurophysiological mechanisms underlying the understanding and imitation of action
Rizzolatti G, Fogassi L, Gallese V. Neurophysiological mechanisms underlying the understanding and imitation of action. Nat Rev Neurosci. 2001 Sep;2(9):661
work page 2001
-
[4]
Sensorimotor synchronization with different metrical levels of point-light dance movements
Su Y. Sensorimotor synchronization with different metrical levels of point-light dance movements. Front Hum Neurosci. 2016;10(April):1–15
work page 2016
-
[5]
Khan O, Ahmed I, Rahhal M, Arvanitis TN, Elliott MT. Step in time: exploration of synchrony and timing correction in response to virtual reality avatars for gait re-training. In: Sharkey PM, Rizzo AA, editors. Proc 11th Intl Conf on Disability, Virtual Reality and Assoc Technologies [Internet]. 2016 [cited 2018 Jul 16]. p. 323–6. Available from: http://ce...
work page 2016
-
[6]
Writing SMART rehabilitation goals and achieving goal attainment scaling: a practical guide
Bovend’Eerdt TJH, Botell RE, Wade DT. Writing SMART rehabilitation goals and achieving goal attainment scaling: a practical guide. Clin Rehabil. 2009;23(4):352–61
work page 2009
-
[7]
Eng JJ, Fang Tang P. Gait training strategies to optimize walking ability in people with stroke: A synthesis of the evidence. Expert Rev Neurother. 2011;7(10):1417–36
work page 2011
-
[8]
Johnson L, Burridge JH, Demain SH. Internal and external focus of attention during gait re- education: an observational study of physical therapist practice in stroke rehabilitation. Phys Ther [Internet]. 2013;93(7):957–66. Available from: http://www.ncbi.nlm.nih.gov/pubmed/23559523
-
[9]
Campbell R. Why don’t patients do their exercises? Understanding non-compliance with physiotherapy in patients with osteoarthritis of the knee. J Epidemiol Community Heal [Internet]. 2001 Feb 1 [cited 2017 Oct 30];55(2):132–8. Available from: http://www.ncbi.nlm.nih.gov/pubmed/11154253
-
[10]
Effects of sensory cueing in virtual motor rehabilitation
Palacios-Navarro G, Albiol-Pérez S, García-Magariño García I. Effects of sensory cueing in virtual motor rehabilitation. A review. J Biomed Inform [Internet]. 2016;60:49–57. Available from: http://dx.doi.org/10.1016/j.jbi.2016.01.006
-
[11]
Hemiparetic stepping to the beat: asymmetric response to metronome phase shift during treadmill gait
Pelton TA, Johannsen L, Chen H, Wing AM. Hemiparetic stepping to the beat: asymmetric response to metronome phase shift during treadmill gait. Neurorehabil Neural Repair [Internet]. 2010;24(5):428–34. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19952366
-
[12]
Rhythmic auditory-motor facilitation of gait patterns in patients with Parkinson’s disease
McIntosh GC, Brown SH, Rice RR, Thaut MH. Rhythmic auditory-motor facilitation of gait patterns in patients with Parkinson’s disease. J Neurol Neurosurg Psychiatry. 1997;62(1):22– 6
work page 1997
-
[13]
Bank PJM, Roerdink M, Peper CE. Comparing the efficacy of metronome beeps and stepping stones to adjust gait: Steps to follow! Exp Brain Res. 2011 Mar;209(2):159–69
work page 2011
-
[14]
Zivotofsky AZ, Gruendlinger L, Hausdorff JM. Modality-specific communication enabling gait synchronization during over-ground side-by-side walking. Hum Mov Sci [Internet]. 2012 Oct [cited 2013 Nov 18];31(5):1268–85. Available from: http://www.sciencedirect.com/science/article/pii/S0167945712000127
work page 2012
-
[15]
How visual information influences coordination dynamics when following the leader
Meerhoff LA, De Poel HJ, Button C. How visual information influences coordination dynamics when following the leader. Neurosci Lett [Internet]. 2014 Oct [cited 2015 Jul 12];582:12–5. Available from: http://dx.doi.org/10.1016/j.neulet.2014.08.022
-
[16]
Sensorimotor synchronization with audio-visual stimuli: limited multisensory integration
Armstrong A, Issartel J. Sensorimotor synchronization with audio-visual stimuli: limited multisensory integration. Exp Brain Res [Internet]. 2014 Nov 16 [cited 2017 Oct 30];232(11):3453–63. Available from: http://link.springer.com/10.1007/s00221-014-4031-9
-
[17]
The supplementation of spatial information improves coordination
Armstrong A, Issartel J, Varlet M, Marin L. The supplementation of spatial information improves coordination. Neurosci Lett [Internet]. 2013 Aug [cited 2014 Jan 10];548:212–6. Available from: http://dx.doi.org/10.1016/j.neulet.2013.05.013
-
[18]
Compatibility of motion facilitates visuomotor synchronization
Hove MJ, Spivey MJ, Krumhansl CL. Compatibility of motion facilitates visuomotor synchronization. J Exp Psychol Hum Percept Perform. 2010 Dec;36(6):1525–34
work page 2010
-
[19]
Su Y-H. Audiovisual beat induction in complex auditory rhythms: Point-light figure movement as an effective visual beat. Acta Psychol (Amst) [Internet]. 2014 Sep [cited 2015 Jan 22];151:40–50. Available from: http://dx.doi.org/10.1016/j.actpsy.2014.05.016
-
[20]
Multisensory cues improve sensorimotor synchronisation
Elliott MT, Wing AM, Welchman AE. Multisensory cues improve sensorimotor synchronisation. Eur J Neurosci [Internet]. 2010 May 17 [cited 2017 Oct 30];31(10):1828–35. Available from: http://doi.wiley.com/10.1111/j.1460-9568.2010.07205.x
-
[22]
A gait analysis data collection and reduction technique
Davis RB, Ounpuu S, Tyburski D, Gage JR. A gait analysis data collection and reduction technique. Hum Mov Sci. 1991;10(5):575–87
work page 1991
-
[23]
Jacoby N, Tishby N, Repp BH, Ahissar M, Keller PE, Jacoby N, et al. Parameter Estimation of Linear Sensorimotor Synchronization Models : Phase Correction , Period Correction , and Ensemble Synchronization. Timing Time Percept [Internet]. 2015 May 25 [cited 2017 Sep 25];3(1–2):52–87. Available from: http://booksandjournals.brillonline.com/content/journals/...
-
[24]
Visual enhancement of auditory beat perception across auditory interference levels
Su Y-H. Visual enhancement of auditory beat perception across auditory interference levels. Brain Cogn [Internet]. 2014 Oct [cited 2015 May 6];90:19–31. Available from: http://www.sciencedirect.com/science/article/pii/S0278262614000864
work page 2014
-
[25]
Wright RL, Elliott MT. Stepping to phase-perturbed metronome cues: multisensory advantage in movement synchrony but not correction. Front Hum Neurosci [Internet]. 2014 [cited 2014 Dec 8];8(September):724. Available from: http://journal.frontiersin.org/journal/10.3389/fnhum.2014.00724/full
-
[26]
Perceiving and reenacting spatiotemporal characteristics of walking sounds
Young W, Rodger M, Craig CM. Perceiving and reenacting spatiotemporal characteristics of walking sounds. J Exp Psychol Hum Percept Perform. 2013
work page 2013
-
[27]
Multisensory processing in review: From physiology to behaviour
Alais D, Newell FN, Mamassian P. Multisensory processing in review: From physiology to behaviour. Seeing and Perceiving. 2010
work page 2010
-
[28]
Causal inference in multisensory perception
Körding KP, Beierholm U, Ma WJ, Quartz S, Tenenbaum JB, Shams L. Causal inference in multisensory perception. PLoS One. 2007
work page 2007
-
[29]
Moving in time: Bayesian causal inference explains movement coordination to auditory beats
Elliott MT, Wing AM, Welchman AE. Moving in time: Bayesian causal inference explains movement coordination to auditory beats. Proc R Soc B Biol Sci. 2014
work page 2014
-
[30]
The effects of rhythmic sensory cues on the temporal dynamics of human gait
Sejdić E, Fu Y, Pak A, Fairley J a., Chau T. The effects of rhythmic sensory cues on the temporal dynamics of human gait. PLoS One. 2012;7(8)
work page 2012
-
[31]
Chen HY, Wing AM, Pratt D. The synchronisation of lower limb responses with a variable metronome: The effect of biomechanical constraints on timing. Gait Posture. 2006;23(3):307–14
work page 2006
-
[32]
Variability of timing in expressive piano performance increases with interval duration
Repp BH. Variability of timing in expressive piano performance increases with interval duration. Psychon Bull Rev. 1997
work page 1997
-
[33]
Sensorimotor synchronization: A review of the tapping literature
Repp BH. Sensorimotor synchronization: A review of the tapping literature. Psychon Bull Rev [Internet]. 2005;12(6):969–92. Available from: http://www.springerlink.com/index/10.3758/BF03206433
-
[34]
Timing movements to interval durations specified by discrete or continuous sounds
Rodger MWM, Craig CM. Timing movements to interval durations specified by discrete or continuous sounds. Exp Brain Res [Internet]. 2011 Aug [cited 2015 Mar 5];214(3):393–402. Available from: http://link.springer.com/article/10.1007/s00221-011-2837-2
-
[35]
Peak velocity as a cue in audiovisual synchrony perception of rhythmic stimuli
Su Y-H. Peak velocity as a cue in audiovisual synchrony perception of rhythmic stimuli. Cognition [Internet]. 2014 Jun [cited 2014 Dec 17];131(3):330–44. Available from: http://www.sciencedirect.com/science/article/pii/S0010027714000298
work page 2014
-
[36]
Influence of stimulus velocity profile on rhythmic visuomotor coordination
Varlet M, Coey CA, Schmidt RC, Marin L, Bardy BG, Richardson MJ. Influence of stimulus velocity profile on rhythmic visuomotor coordination. J Exp Psychol Hum Percept Perform [Internet]. 2014 Oct;40(5):1849–60. Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/a0037417
-
[37]
Location but not amount of stimulus occlusion influences the stability of visuomotor coordination
Hajnal A, Richardson MJ, Harrison SJ, Schmidt RC. Location but not amount of stimulus occlusion influences the stability of visuomotor coordination. Exp Brain Res. 2012;221(3):351–5
work page 2012
-
[38]
Three-dimensional human gait pattern – reference data for normal men
Pietraszewski B, Winiarski S, Jaroszczuk S. Three-dimensional human gait pattern – reference data for normal men. Acta Bioeng Biomech. 2012;14(3):9–16
work page 2012
-
[39]
Properties of pedestrians walking in line
Jelić A, Appert-Rolland C, Lemercier S, Pettré J. Properties of pedestrians walking in line. II. Stepping behavior. Phys Rev E [Internet]. 2012 Oct [cited 2014 May 20];86(4):46111. Available from: http://link.aps.org/doi/10.1103/PhysRevE.86.046111
-
[40]
Repp BH, Keller PE, Jacoby N. Quantifying phase correction in sensorimotor synchronization: Empirical comparison of three paradigms. Acta Psychol (Amst) [Internet]. 2012;139(2):281–
work page 2012
-
[41]
Available from: http://dx.doi.org/10.1016/j.actpsy.2011.11.002
-
[42]
Follow the leader: Visual control of speed in pedestrian following
Rio KW, Rhea CK, Warren WH. Follow the leader: Visual control of speed in pedestrian following. J Vis [Internet]. 2014;14(2):4–4. Available from: http://jov.arvojournals.org/Article.aspx?doi=10.1167/14.2.4
-
[43]
Intersensory binding across space and time: A tutorial review - Springer
Chen L, Vroomen J. Intersensory binding across space and time: A tutorial review - Springer. Atten Percept Psychophys [Internet]. 2013 [cited 2013 May 28];75:790–811. Available from: http://link.springer.com/article/10.3758/s13414-013-0475-4/fulltext.html
-
[44]
Visuomotor synchronization: Analysis of the initiation and stable synchronization phases
Kurgansky A V. Visuomotor synchronization: Analysis of the initiation and stable synchronization phases. Hum Physiol [Internet]. 2008 Jun [cited 2015 Mar 17];34(3):289–98. Available from: http://0- link.springer.com.pugwash.lib.warwick.ac.uk/article/10.1134/S0362119708030043
-
[45]
Linear Phase-Correction in Synchronization: Predictions, Parameter Estimation, and Simulations
Vorberg D, Schulze H-H. Linear Phase-Correction in Synchronization: Predictions, Parameter Estimation, and Simulations. J Math Psychol [Internet]. 2002 Feb [cited 2017 Sep 25];46(1):56–87. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0022249601913756
work page 2002
-
[46]
Handbook of perception and action
Vorberg D, Wing A. Handbook of perception and action. Modeling variability and dependence in timing. 1996. Figure 1 Click here to access/download;Figure;Fig1.tiff Figure 2 Click here to access/download;Figure;Fig2.tiff Figure 3 Click here to access/download;Figure;Fig3.tiff Figure 4 Click here to access/download;Figure;Fig4.tiff Figure 5 Click here to acc...
work page 1996
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.