ReconViguRation: Reconfiguring Physical Keyboards in Virtual Reality
Pith reviewed 2026-05-24 19:41 UTC · model grok-4.3
The pith
Physical keyboards can be reconfigured in VR to support specialized input and visual output mappings.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By defining input and output mappings that reconfigure the virtual presentation of physical keyboards, individual keys can be reassigned to the same or different actions with visual output distributed in various ways, enabling nine VR-relevant applications that users can interact with effectively.
What carries the argument
Input and output mappings for reconfiguring the virtual presentation of physical keyboards, where physical key presses trigger remapped actions and visuals are altered accordingly.
If this is right
- Users can access emojis, special characters, and multiple languages via remapped keys on one physical keyboard.
- Secure password entry is possible by hiding actual key visuals.
- Tasks like window management and photo browsing can be controlled through keyboard remappings in VR.
- Games such as whack-a-mole can integrate physical keyboard input with VR visuals.
- Virtual text processing macros become accessible without switching devices.
Where Pith is reading between the lines
- Remapping could allow keyboards to serve as context-aware controllers in various VR scenarios beyond the tested apps.
- The technique might extend to other physical input devices beyond keyboards.
- Dynamic remapping based on VR context could reduce reliance on separate input hardware.
Load-bearing premise
VR tracking systems can identify and monitor individual physical keys with sufficient precision and low latency to enable reliable remapped interactions without errors.
What would settle it
If tracking errors cause frequent misidentification of pressed keys, leading to incorrect actions in the remapped applications during user testing.
Figures
read the original abstract
Physical keyboards are common peripherals for personal computers and are efficient standard text entry devices. Recent research has investigated how physical keyboards can be used in immersive head-mounted display-based Virtual Reality (VR). So far, the physical layout of keyboards has typically been transplanted into VR for replicating typing experiences in a standard desktop environment. In this paper, we explore how to fully leverage the immersiveness of VR to change the input and output characteristics of physical keyboard interaction within a VR environment. This allows individual physical keys to be reconfigured to the same or different actions and visual output to be distributed in various ways across the VR representation of the keyboard. We explore a set of input and output mappings for reconfiguring the virtual presentation of physical keyboards and probe the resulting design space by specifically designing, implementing and evaluating nine VR-relevant applications: emojis, languages and special characters, application shortcuts, virtual text processing macros, a window manager, a photo browser, a whack-a-mole game, secure password entry and a virtual touch bar. We investigate the feasibility of the applications in a user study with 20 participants and find that, among other things, they are usable in VR. We discuss the limitations and possibilities of remapping the input and output characteristics of physical keyboards in VR based on empirical findings and analysis and suggest future research directions in this area.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ReconViguRation, a system that remaps both input actions and visual output on physical keyboards inside VR to enable novel interactions. It defines a design space of input/output mappings, implements nine concrete VR applications (emoji entry, language/special-character switching, shortcuts, macros, window management, photo browsing, whack-a-mole, secure password entry, virtual touch bar), and evaluates feasibility via a 20-participant user study that concludes the applications are usable in VR.
Significance. If the usability findings hold, the work meaningfully enlarges the interaction design space for physical keyboards in immersive VR by exploiting visual reconfiguration rather than merely replicating desktop layouts. The nine applications demonstrate concrete, task-relevant uses; the empirical study supplies initial feasibility evidence that can guide subsequent VR input research.
major comments (2)
- [User Study / Evaluation] User Study / Evaluation section: the central usability claim rests on the 20-participant study, yet no quantitative metrics (task times, error rates, NASA-TLX scores, statistical tests, or exclusion criteria) are reported in the abstract or results summary, weakening the evidential support for generalizability beyond the specific hardware and setup.
- [Implementation / System] Implementation / System section: the feasibility of all nine remapped applications presupposes that existing VR tracking can identify and monitor individual physical keys with sufficient spatial precision and low latency; the manuscript supplies no measured key-detection accuracy, false-positive rates, or end-to-end latency figures under the remapped conditions, leaving the load-bearing tracking assumption unquantified.
minor comments (1)
- [Abstract] Abstract: the phrase “among other things, they are usable in VR” is vague; replace with a concise statement of the main quantitative or qualitative outcomes.
Simulated Author's Rebuttal
We thank the referee for the positive assessment of the work's significance and the recommendation for minor revision. We address the two major comments point by point below.
read point-by-point responses
-
Referee: [User Study / Evaluation] User Study / Evaluation section: the central usability claim rests on the 20-participant study, yet no quantitative metrics (task times, error rates, NASA-TLX scores, statistical tests, or exclusion criteria) are reported in the abstract or results summary, weakening the evidential support for generalizability beyond the specific hardware and setup.
Authors: The study was designed as an initial feasibility evaluation emphasizing qualitative participant feedback on the nine applications rather than a controlled comparative experiment. The results section details observations and comments supporting usability. We agree that the abstract and high-level summary would benefit from additional context. In revision we will expand the abstract with a concise statement on study outcomes and include a brief summary of key observations in the results overview. revision: yes
-
Referee: [Implementation / System] Implementation / System section: the feasibility of all nine remapped applications presupposes that existing VR tracking can identify and monitor individual physical keys with sufficient spatial precision and low latency; the manuscript supplies no measured key-detection accuracy, false-positive rates, or end-to-end latency figures under the remapped conditions, leaving the load-bearing tracking assumption unquantified.
Authors: The applications rely on standard VR headset and controller tracking of the physical keyboard, which proved sufficient for all nine implemented applications during the user study. Specific quantitative tracking benchmarks were outside the scope of this work, which centers on input/output reconfiguration. We will add an explicit discussion of the tracking assumptions and their potential limitations to the revised manuscript. revision: partial
Circularity Check
No circularity; empirical user study supplies independent evidence of usability
full rationale
The manuscript is an exploratory design paper that defines input/output mappings, implements nine concrete VR applications, and reports feasibility via a 20-participant user study. No equations, fitted parameters, or 'predictions' appear. The usability claim is grounded in the study results rather than any self-referential definition or self-citation chain. Prior keyboard-in-VR literature is cited only for context and does not substitute for the present evaluation. This matches the default case of a self-contained empirical contribution.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Existing VR tracking systems can identify and monitor individual physical keys with sufficient precision and low latency for remapped interactions.
- domain assumption Users can adapt to remapped keyboard layouts and visuals in VR without prohibitive cognitive load.
Reference graph
Works this paper leans on
-
[1]
F. A. Alsulaiman and A. El Saddik. A novel 3d graphical password schema. In 2006 IEEE Symposium on Virtual Environments, Human- Computer Interfaces and Measurement Systems , pp. 125–128. IEEE, 2006
work page 2006
-
[2]
Apple. Apple touch bar. https://developer.apple.com/macos/ touch-bar/. Last accessed 19.03.2019
work page 2019
-
[3]
M. Azmandian, M. Hancock, H. Benko, E. Ofek, and A. D. Wilson. Haptic retargeting: Dynamic repurposing of passive haptics for en- hanced virtual reality experiences. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems , CHI ’16, pp. 1968–1979. ACM, New York, NY , USA, 2016. doi: 10.1145/2858036. 2858226
- [4]
- [5]
- [6]
-
[7]
J. Brooke et al. Sus-a quick and dirty usability scale. Usability evalua- tion in industry, 189(194):4–7, 1996
work page 1996
-
[8]
E. Burns, S. Razzaque, M. Whitton, and F. Brooks. Macbeth: The avatar which i see before me and its movement toward my hand. in proceedings of the 2016 chi conference on human factors in computing systems. In Proceedings of IEEE Virtual Reality Conference, p. 295296. IEEE, 2007
work page 2016
-
[9]
D. Buschek, B. Roppelt, and F. Alt. Extending keyboard shortcuts with arm and wrist rotation gestures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems , p. 21. ACM, 2018
work page 2018
-
[10]
S. K. Card, T. P. Moran, and A. Newell. The psychology of human- computer interaction. Lawrence Erlbaum Associates, 1983
work page 1983
-
[11]
X. Chen, T. Grossman, D. J. Wigdor, and G. Fitzmaurice. Duet: ex- ploring joint interactions on a smart phone and a smart watch. In Proceedings of the SIGCHI Conference on Human Factors in Comput- ing Systems, pp. 159–168. ACM, 2014
work page 2014
-
[12]
O. E. H. C. B. H. Cheng, Lung-Pan and A. D Wilson. Sparse haptic proxy: Touch feedback in virtual environments using a general passive prop. in proceedings of the 2017 chi conference on human factors in computing systems. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, p. 37183728. ACM, 2017
work page 2017
-
[13]
F. Conti and O. Khatib. Spanning large workspaces using small haptic devices. in proceedings of first joint eurohaptics conference and sym- posium on haptic interfaces for virtual environment and teleoperator. In roceedings of First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator, p. 183188. ACM, 2005
work page 2005
-
[14]
C. Corsten, I. Avellino, M. Mllers, and J. Borchers. Instant user inter- faces: repurposing everyday objects as input devices. in proceedings of the acm international conference on interactive tabletops and sur- faces (its ’13). In Proceedings of the ACM international conference on Interactive tabletops and surfaces , pp. 71–80. ACM, 2013
work page 2013
- [15]
-
[16]
V . Dhakal, A. M. Feit, P. O. Kristensson, and A. Oulasvirta. Obser- vations on typing from 136 million keystrokes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems , CHI ’18, pp. 646:1–646:12. ACM, New York, NY , USA, 2018. doi: 10. 1145/3173574.3174220
-
[17]
P. H. Dietz, B. Eidelson, J. Westhues, and S. Bathiche. A practical pressure sensitive computer keyboard. In Proceedings of the 22nd annual ACM symposium on User interface software and technology , pp. 55–58. ACM, 2009
work page 2009
-
[18]
T. Dube and A. Arif. Text entry in virtual reality: A comprehensive review of the literature. In Proceedings of HCI International 2019 , 2019
work page 2019
- [19]
-
[20]
Flying with a vr headset isn’t as dorky as it sounds
Engadget. Flying with a vr headset isn’t as dorky as it sounds. https://www.engadget.com/2018/02/22/ htc-vive-focus-in-flight-vr/ . Last accessed 19.03.2019
work page 2018
-
[21]
W. Fallot-Burghardt, M. Fjeld, C. Speirs, S. Ziegenspeck, H. Krueger, and T. L ¨aubli. Touch&type: a novel pointing device for notebook computers. In Proceedings of the 4th Nordic conference on Human- computer interaction: changing roles , pp. 465–468. ACM, 2006
work page 2006
-
[22]
G. W. Fitzmaurice, H. Ishii, and A. S. W. Buxton. Bricks: laying the foundations for graspable user interfaces. in proceedings of the sigchi conference on human factors in computing systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , pp. 442–449. ACM, 1995
work page 1995
-
[23]
M.-C. Fluet, O. Lambercy, and R. Gassert. Effects of 2d/3d visual feedback and visuomotor collocation on motor performance in a virtual peg insertion test. In 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society , pp. 4776–4779. IEEE, 2012
work page 2012
-
[24]
S. Follmer, D. Leithinger, A. Olwal, A. Hogge, and H. Ishii. inform: dynamic physical affordances and constraints through shape and ob- ject actuation. in proceedings of the acm user interface software and technology symposium. In Proceedings of the ACM User Interface Software and Technology Symposium, p. 417426. ACM, 2013
work page 2013
-
[25]
M. J. Fu, A. D. Hershberger, K. Sano, and M. C. C ¸avus ¸o˘glu. Effect of visuomotor colocation on 3d fitts’ task performance in physical and virtual environments. Presence, 21(3):305–320, 2012
work page 2012
-
[26]
H. Gellersen and F. Block. Novel interactions on the keyboard. Com- puter, 45(4):36–40, 2012
work page 2012
- [27]
-
[28]
K. R. Gray. Facilitating keyboard use while wearing a head-mounted display. 2018
work page 2018
-
[29]
J. Grubert, E. Ofek, M. Pahud, and P. O. Kristensson. The office of the future: Virtual, portable, and global. IEEE computer graphics and applications, 38(6):125–133, 2018
work page 2018
-
[30]
J. Grubert, L. Witzani, E. Ofek, M. Pahud, M. Kranz, and P. O. Kris- tensson. Effects of hand representations for typing in virtual reality. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) , pp. 151–158, March 2018. doi: 10.1109/VR.2018.8446250
-
[31]
J. Grubert, L. Witzani, E. Ofek, M. Pahud, M. Kranz, and P. O. Kris- tensson. Text entry in immersive head-mounted display-based virtual reality using standard keyboards. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) , pp. 159–166, March 2018. doi: 10.1109/VR.2018.8446059
- [32]
-
[33]
A. Hettiarachchi and D. Wigdor. Annexing reality: Enabling oppor- tunistic use of everyday objects as tangible proxies in augmented reality. In Proceedings of the CHI Conference on Human Factors in Computing Systems, p. 19571967. ACM, 2016
work page 2016
-
[34]
K. Hinckley, R. Pausch, J. C. Goble, and N. F. Kassell. Passive real- world interface props for neurosurgical visualization. in proceedings of the sigchi conference on human factors in computing systems. In Pro- ceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 452–458. ACM, 1994
work page 1994
-
[35]
A. H. Hoppe, L. Otto, F. van de Camp, R. Stiefelhagen, and G. Unm¨ußig. qvrty: Virtual keyboard with a haptic, real-world represen- tation. In International Conference on Human-Computer Interaction , pp. 266–272. Springer, 2018
work page 2018
- [36]
-
[37]
I. U. T. Inc. A brief history of the lcd key technology. http://www. lcd-keys.com/english/history.htm. Last accessed 19.03.2019
work page 2019
-
[38]
J. Kato, D. Sakamoto, and T. Igarashi. Surfboard: keyboard with microphone as a low-cost interactive surface. In Adjunct proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 387–388. ACM, 2010
work page 2010
-
[39]
D. Kim, S. Izadi, J. Dostal, C. Rhemann, C. Keskin, C. Zach, J. Shotton, T. Large, S. Bathiche, M. Nießner, et al. Retrodepth: 3d silhouette sensing for high-precision input on and above physical surfaces. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems, pp. 1377–1386. ACM, 2014
work page 2014
-
[40]
P. Knierim, V . Schwind, A. M. Feit, F. Nieuwenhuizen, and N. Henze. Physical keyboards in virtual reality: Analysis of typing performance and effects of avatar hands. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems , p. 345. ACM, 2018
work page 2018
-
[41]
D. A. Kontarinis, J. S. Son, W. Peine, and R. D. Howe. A tactile shape sensing and display system for teleoperated manipulation. in proceedings of the ieee conference on robotics and automation. In Proceedings of the IEEE Conference on Robotics and Automation , p. 641646. IEEE, 1995
work page 1995
-
[42]
K. J. Kruszynski and R. van Liere. Tangible props for scientic vi- sualization: concept, requirements. Virtual reality, 13(4):235–244, 2009
work page 2009
-
[43]
T. Kurosawa, B. Shizuki, and J. Tanaka. Keyboard clawing: input method by clawing key tops. In International Conference on Human- Computer Interaction, pp. 272–280. Springer, 2013
work page 2013
-
[44]
D. M. Lane, H. A. Napier, S. C. Peres, and A. Sandor. Hidden costs of graphical user interfaces: Failure to make the transition from menus and icon toolbars to keyboard shortcuts. International Journal of Human-Computer Interaction, 18(2):133–144, 2005
work page 2005
- [45]
-
[46]
B. Lee, H. Park, and H. Bang. Multidirectional pointing input using a hardware keyboard. ETRI Journal, 35(6):1160–1163, 2013
work page 2013
-
[47]
J. R. Lewis. Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the asq. ACM Sigchi Bulletin, 23(1):78– 81, 1991
work page 1991
- [48]
-
[49]
R. W. Lindeman, J. L. Sibert, and J. K. Hahn. Hand-held windows: towards effective 2d interaction in immersive virtual environments. In Proceedings IEEE Virtual Reality (Cat. No. 99CB36316) , pp. 205–212. IEEE, 1999
work page 1999
-
[50]
Logitech g19 keyboard for gaming
Logitech. Logitech g19 keyboard for gaming. https://support. logitech.com/en_us/product/g19-keyboard-for-gaming . Last accessed 19.03.2019
work page 2019
-
[51]
K.-L. Low, G. Welch, A. Lastra, and H. Fuchs. Life-sized projector- based dioramas. in proceedings of the acm symposium on virtual reality software and technology (vrst ’01). In Proceedings of the IEEE Con- ference on Robotics and Automation , pp. 93–101. ACM, 2001
work page 2001
-
[52]
C. C. Loy, W. Lai, and C. Lim. Development of a pressure-based typing biometrics user authentication system. ASEAN Virtual Instrumentation Applications Contest Submission, 2005
work page 2005
- [53]
-
[54]
N. Marquardt, R. Jota, S. Greenberg, and J. A. Jorge. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In IFIP Conference on Human-Computer Interaction, pp. 461–476. Springer, 2011
work page 2011
-
[55]
M. McGill, D. Boland, R. Murray-Smith, and S. Brewster. A dose of reality: overcoming usability challenges in vr head-mounted displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , pp. 2143–2152. ACM, 2015. doi: 10.1145/ 2702123.2702382
-
[56]
P. Milgram and F. Kishino. A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems , 77(12):1321–1329, 1994
work page 1994
-
[57]
D. Norman. The design of everyday things: Revised and expanded edition. Basic books, 2013
work page 2013
-
[58]
P. K. Novak, J. Smailovi´c, B. Sluban, and I. Mozeti ˇc. Sentiment of emojis. PloS one, 10(12):e0144296, 2015
work page 2015
-
[59]
R. C. Omanson, C. S. Miller, E. Young, and D. Schwantes. Comparison of mouse and keyboard efficiency. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting , vol. 54, pp. 600–604. Sage Publications Sage CA: Los Angeles, CA, 2010
work page 2010
-
[60]
optimus. Optimus oled keyboards. https://www.artlebedev.com/ optimus/. Last accessed 19.03.2019
work page 2019
-
[61]
A. Otte, D. Schneider, T. Menzner, T. Gesslein, P. Gagel, and J. Grubert. Evaluating text entry in virtual reality using a touch-sensitive physical keyboard. In 2019 IEEE International Symposium on Mixed and Augmented Reality. IEEE, 2019
work page 2019
-
[62]
Keyboard Surface Interaction: Making the keyboard into a pointing device
J. Ramos, Z. Li, J. Rosas, N. Banovic, J. Mankoff, and A. Dey. Key- board surface interaction: Making the keyboard into a pointing device. arXiv preprint arXiv:1601.04029, 2016
work page internal anchor Pith review Pith/arXiv arXiv 2016
-
[63]
Razor deathstalker ultimate keyboard
Razor. Razor deathstalker ultimate keyboard. https://support.razer.com/gaming-keyboards/ razer-deathstalker-ultimate . Last accessed 19.03.2019
work page 2019
-
[64]
J. Rekimoto, T. Ishizawa, C. Schwesig, and H. Oba. Presense: in- teraction techniques for finger sensing input devices. InProceedings of the 16th annual ACM symposium on User interface software and technology, pp. 203–212. ACM, 2003
work page 2003
-
[65]
F. Roesner, T. Kohno, and D. Molnar. Security and privacy for aug- mented reality systems. Commun. ACM, 57(4):88–96, 2014
work page 2014
-
[66]
A. Roudaut, D. Krusteva, M. McCoy, A. Karnik, K. Ramani, and S. Subramanian. Cubimorph: designing modular interactive devices. in robotics and automation (icra). In Proceedings of the IEEE Conference on Robotics and Automation , p. 33393334. IEEE, 2016
work page 2016
-
[67]
A. Roudaut, R. Reed, T. Hao, and S. Subramanian. changibles: analyz- ing and designing shape changing constructive assembly. in proceed- ings of the 32nd annual acm conference on human factors in computing systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p. 25932596. ACM, 2014
work page 2014
-
[68]
M. A. Sasse and I. Flechais. Usable security: Why do we need it? how do we get it? O’Reilly, 2005
work page 2005
-
[69]
K. Sekimori, Y . Yamasaki, Y . Takagi, K. Murata, B. Shizuki, and S. Takahashi. Ex-space: Expanded space key by sliding thumb on home position. In International Conference on Human-Computer Interaction, pp. 68–78. Springer, 2018
work page 2018
-
[70]
Y . Shi, T. Vega G´alvez, H. Zhang, and S. Nanayakkara. Gestakey: Get more done with just-a-key on a keyboard. In Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 73–75. ACM, 2017
work page 2017
-
[71]
Y . Shi, H. Zhang, H. Rajapakse, N. T. Perera, T. Vega G ´alvez, and S. Nanayakkara. Gestakey: Touch interaction on individual keycaps. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p. 596. ACM, 2018
work page 2018
-
[72]
A. Strauss and J. Corbin. Basics of qualitative research. Sage publica- tions, 1990
work page 1990
- [73]
-
[74]
doi: 10.1145/2556288.2557030
- [75]
-
[76]
R. J. Teather, R. S. Allison, and W. Stuerzlinger. Evaluating vi- sual/motor co-location in fish-tank virtual reality. In2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC- STH), pp. 624–629. IEEE, 2009
work page 2009
-
[77]
Y .-C. Tung, T. Y . Cheng, N.-H. Yu, C. Wang, and M. Y . Chen. Flick- board: Enabling trackpad interaction with automatic mode switching on a capacitive-sensing keyboard. In Proceedings of the 33rd An- nual ACM Conference on Human Factors in Computing Systems , pp. 1847–1850. ACM, 2015
work page 2015
-
[78]
Introducing the logitech bridge sdk
VIVE. Introducing the logitech bridge sdk. https://blog.vive.com/us/2017/11/02/ introducing-the-logitech-bridge-sdk/ . Last accessed 19.03.2019
work page 2017
- [79]
- [80]
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.