Predictive Gaze Is Preserved but Reorganized toward Monitoring during Robot-Mediated Manipulation
Pith reviewed 2026-06-26 12:21 UTC · model grok-4.3
The pith
Predictive gaze persists during robot teleoperation but shifts toward monitoring the end-effector.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Gaze remains strongly aligned with task goals, preserving its predictive role even during robot-mediated manipulation. At the same time, teleoperation systematically redistributes visual attention toward the robotic end-effector and manipulated objects, increasing online monitoring. These findings show that predictive gaze is not lost under altered embodiment, but reorganized in response to changes in sensory feedback and control demands.
What carries the argument
Eye-tracking comparison of predictive gaze (task-goal aligned) versus monitoring gaze (end-effector focused) across direct versus teleoperated manipulation conditions.
If this is right
- Gaze can serve as an informative signal for inferring action intentions in human-robot interaction.
- The visuomotor system retains core predictive mechanisms despite disrupted embodiment.
- Redistributed attention toward the robot compensates for changes in sensory feedback and control.
- Teleoperation induces a measurable reorganization rather than a loss of anticipatory gaze behavior.
Where Pith is reading between the lines
- Teleoperation interfaces could be designed to reduce monitoring load by providing clearer end-effector feedback.
- Operator training might target strategies to balance predictive and monitoring gaze efficiently.
- Similar gaze reorganization may appear in other altered-embodiment contexts such as prosthetic control.
- Real-time gaze tracking could allow robots to anticipate user intent and offer assistance during manipulation.
Load-bearing premise
That distinctions between predictive and monitoring gaze can be reliably made from eye-tracking data alone and that observed shifts are specifically attributable to altered embodiment rather than differences in task demands, visual feedback quality, or individual strategy.
What would settle it
A replication in which the proportion of gaze directed at the end-effector shows no increase in the teleoperation condition compared with direct manipulation, or in which gaze alignment with task goals drops under teleoperation.
read the original abstract
Goal-directed eye movements are a fundamental component of visuomotor control, enabling humans to anticipate and guide their actions. Whether this anticipatory and task-driven behavior is preserved when actions are executed through a robot rather than through one's own body remains unclear. Here we address this question by investigating gaze behavior during goal-directed telemanipulation to determine how visuomotor control adapts to altered embodiment. Our findings show that gaze remains strongly aligned with task goals, preserving its predictive role even during robot-mediated manipulation. At the same time, teleoperation systematically redistributes visual attention toward the robotic end-effector and manipulated objects, increasing online monitoring. These findings show that predictive gaze is not lost under altered embodiment, but reorganized in response to changes in sensory feedback and control demands. More broadly, they reveal the flexibility of the human visuomotor system when the natural sensorimotor coupling is disrupted and identify gaze as an informative signal for inferring action intentions in human-robot interaction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports an eye-tracking study of goal-directed telemanipulation, claiming that gaze remains strongly aligned with task goals (preserving its predictive role) while attention is redistributed toward the robotic end-effector and manipulated objects (increasing online monitoring) relative to direct manipulation; the reorganization is attributed to changes in sensory feedback and control demands under altered embodiment.
Significance. If the empirical distinction between predictive and monitoring gaze holds and the shifts are attributable to embodiment rather than task or feedback confounds, the work would demonstrate flexibility of the human visuomotor system when natural sensorimotor coupling is disrupted and would position gaze as a useful signal for inferring action intentions in human-robot interaction.
major comments (2)
- [Abstract] Abstract: the central findings are stated but the abstract supplies no information on experimental protocol, participant numbers, eye-tracking methods, statistical tests, or data exclusion criteria. This prevents evaluation of whether the data support the claims that predictive gaze is preserved while attention redistributes toward monitoring.
- [Results / Discussion] The distinction between predictive goal-directed fixations and online effector monitoring is asserted on the basis of spatial-temporal alignment to task events, yet the manuscript does not address how this operationalization rules out confounds such as differences in task demands, visual feedback quality, or individual strategy (the weakest assumption identified in the stress-test note).
Simulated Author's Rebuttal
We thank the referee for their constructive comments, which highlight opportunities to improve clarity and rigor. We address each major comment below and outline planned revisions.
read point-by-point responses
-
Referee: [Abstract] Abstract: the central findings are stated but the abstract supplies no information on experimental protocol, participant numbers, eye-tracking methods, statistical tests, or data exclusion criteria. This prevents evaluation of whether the data support the claims that predictive gaze is preserved while attention redistributes toward monitoring.
Authors: We agree that the abstract lacks key methodological details necessary for independent evaluation. In the revised version we will expand the abstract to report participant numbers (N=XX), the eye-tracking hardware and sampling rate, the primary statistical tests (including effect sizes), and the data exclusion criteria applied to fixations and trials. revision: yes
-
Referee: [Results / Discussion] The distinction between predictive goal-directed fixations and online effector monitoring is asserted on the basis of spatial-temporal alignment to task events, yet the manuscript does not address how this operationalization rules out confounds such as differences in task demands, visual feedback quality, or individual strategy (the weakest assumption identified in the stress-test note).
Authors: The distinction is operationalized via fixation timing and location relative to discrete, objectively defined task events (e.g., grasp onset, transport start) that are identical across direct and teleoperated conditions. The within-subject design and matched visual scenes control for individual strategy and gross task demands. We acknowledge that the manuscript does not explicitly discuss residual confounds arising from altered visual feedback latency and resolution; we will add a dedicated paragraph in the Discussion section that (a) justifies the event-based classification against the literature on predictive gaze, (b) notes the inherent embodiment-related feedback differences, and (c) reports supplementary analyses that test robustness to alternative fixation definitions. revision: partial
Circularity Check
Empirical behavioral study with no derivations or self-referential structure
full rationale
The paper reports observational eye-tracking results from a telemanipulation experiment. No equations, fitted parameters, predictions derived from inputs, or load-bearing self-citations appear in the abstract or described methods/results. The central claims rest on direct measurement of gaze alignment to task events rather than any internal reduction or renaming of prior results. This is a standard empirical report whose logic does not reduce to its own inputs.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Michael Land, Neil Mennie, and Jennifer Rusted. 1999. The Roles of Vision and Eye Movements in the Control of Activities of D aily Living. Perception 28, 11 (November 1999), 1311–1328. https://doi.org/10.1068/p2935
-
[2]
Lavoie, Aïda M
Ewen B. Lavoie, Aïda M. Valevicius, Quinn A. Boser, Ognjen Kovic, Albert H. Vette, Patrick M. Pilarski, Jacqueline S. Hebert, and Craig S. Chapman
-
[3]
Using synchronized eye and motion tracking to determine high -precision eye-movement patterns during object -interaction tasks. J. Vis. 18, 6 (June 2018), 18. https://doi.org/10.1167/18.6.18
-
[4]
Anna Belardinelli. 2024. Gaze -Based Intention Estimation: Principles, Methodologies, and Applications in HRI. ACM Trans. Hum. Robot. Interact. 13, 3 (September 2024), 1–30. https://doi.org/10.1145/3656376
-
[5]
Johansson, Göran Westling, Anders Bäckström, and J
Roland S. Johansson, Göran Westling, Anders Bäckström, and J. Randall Flanagan. 2001. Eye–Hand Coordination in Object Manipulation. The Journal of Neuroscience 21, 17 (September 2001), 6917–6932. https://doi.org/10.1523/JNEUROSCI.21-17-06917.2001
-
[6]
Anne-Marie Brouwer, Volker H. Franz, and Karl R. Gegenfurtner. 2009. Differences in fixations between grasping and viewing objects. J. Vis. 9, 1 24 (January 2009), 18–18. https://doi.org/10.1167/9.1.18
-
[7]
Anna Belardinelli, Madeleine Y. Stepper, and Martin V. Butz. 2016. It’s in the eyes: Planning precise manual actions before execution. J. Vis. 16, 1 (January 2016), 18. https://doi.org/10.1167/16.1.18
-
[8]
Anna Belardinelli, Marissa Barabas, Marc Himmelbach, and Martin V. Butz. 2016. Anticipatory eye fixations reveal tool knowledge for tool interaction. Exp. Brain Res. 234, 8 (August 2016), 2415–2431. https://doi.org/10.1007/s00221-016-4646-0
-
[9]
Di Stasi, Carolina Diaz-Piedra, Héctor Rieiro, José M
Leandro L. Di Stasi, Carolina Diaz-Piedra, Héctor Rieiro, José M. Sánchez Carrión, Mercedes Martin Berrido, Gonzalo Olivares, and Andrés Catena
-
[10]
Gaze entropy reflects surgical task load. Surg. Endosc. 30, 11 (November 2016), 5034–5043. https://doi.org/10.1007/s00464-016-4851-8
-
[11]
Kourosh Darvish, Luigi Penco, Joao Ramos, Rafael Cisneros, Jerry Pratt, Eiichi Yoshida, Serena Ivaldi, and Daniele Pucci. 202 3. Teleoperation of Humanoid Robots: A Survey. IEEE Transactions on Robotics 39, 3 (June 2023), 1706–1727. https://doi.org/10.1109/TRO.2023.3236952
-
[12]
Funk, Florian Richter, Michael C
Patrick Barba, Joshua Stramiello, Emily K. Funk, Florian Richter, Michael C. Yip, and Ryan K. Orosco. 2022. Remote telesurger y in humans: a systematic review. Surg. Endosc. 36, 5 (May 2022), 2771–2777. https://doi.org/10.1007/s00464-022-09074-4
-
[13]
Jörn Vogel, Mattias Atzenhofer, Gabriel Quere, Maged Iskandar, Miriam Welser, Felix Schiel, Sebastian Jung, Samuel Bustamante , Werner Friedl, Wout Boerdijk, Freek Stulp, Alin Albu-Schäffer, and Annette Hagengruber. 2025. Shared control in assistive robotics: A Cybathlon-winning approach. Sci. Robot. 10, 107 (October 2025). https://doi.org/10.1126/scirobo...
-
[14]
Hochberg, Daniel Bacher, Beata Jarosiewicz, Nicolas Y
Leigh R. Hochberg, Daniel Bacher, Beata Jarosiewicz, Nicolas Y. Masse, John D. Simeral, Joern Vogel, Sami Haddadin, Jie Liu, Sydney S. Cash, Patrick van der Smagt, and John P. Donoghue. 2012. Reach and grasp by people with tetraplegia using a neurally cont rolled robotic arm. Nature 485, 7398 (May 2012), 372–375. https://doi.org/10.1038/nature11076
-
[15]
Peter Schmaus, Daniel Leidner, Thomas Kruger, Ralph Bayer, Benedikt Pleintinger, Andre Schiele, and Neal Y. Lii. 2020. Knowle dge Driven Orbit- to-Ground Teleoperation of a Robot Coworker. IEEE Robot. Autom. Lett. 5, 1 (January 2020), 143–150. https://doi.org/10.1109/LRA.2019.2948128
-
[16]
Voolstra, Khaled Nabil Salama, Michel L’Hour, and Vincent Creuze
Oussama Khatib, Xiyang Yeh, Gerald Brantner, Brian Soe, Boyeon Kim, Shameek Ganguly, Hannah Stuart, Shiquan Wang, Mark Cutkos ky, Aaron Edsinger, Phillip Mullins, Mitchell Barham, Christian R. Voolstra, Khaled Nabil Salama, Michel L’Hour, and Vincent Creuze . 2016. Ocean One: A Robotic Avatar for Oceanic Discovery. IEEE Robot. Autom. Mag. 23, 4 (December ...
-
[17]
Marasco, Keehoon Kim, James Edward Colgate, Michael A
Paul D. Marasco, Keehoon Kim, James Edward Colgate, Michael A. Peshkin, and Todd A. Kuiken. 2011. Robotic touch shifts percep tion of embodiment to a prosthesis in targeted reinnervation amputees. Brain 134, 3 (March 2011), 747–758. https://doi.org/10.1093/brain/awq361
-
[18]
Christian Antfolk, Marco D’Alonzo, Birgitta Rosén, Göran Lundborg, Fredrik Sebelius, and Christian Cipriani. 2013. Sensory fe edback in upper limb prosthetics. Expert Rev. Med. Devices 10, 1 (January 2013), 45–54. https://doi.org/10.1586/erd.12.68
-
[19]
Stanisa Raspopovic, Marco Capogrosso, Francesco Maria Petrini, Marco Bonizzato, Jacopo Rigosa, Giovanni Di Pino, Jacopo Carpa neto, Marco Controzzi, Tim Boretius, Eduardo Fernandez, Giuseppe Granata, Calogero Maria Oddo, Luca Citi, Anna Lisa Ciancio, Christian Cipriani, Maria Chiara Carrozza, Winnie Jensen, Eugenio Guglielmelli, Thomas Stieglitz, Paolo Ma...
-
[20]
Mario Selvaggio, Marco Cognetti, Stefanos Nikolaidis, Serena Ivaldi, and Bruno Siciliano. 2021. Autonomy in Physical Human-Robot Interaction: A Brief Survey. IEEE Robot. Autom. Lett. 6, 4 (October 2021), 7989–7996. https://doi.org/10.1109/LRA.2021.3100603
-
[21]
Canjun Yang, Yuanchao Zhu, and Yanhu Chen. 2022. A Review of Human –Machine Cooperation in the Robotics Domain. IEEE Trans. Hum. Mach. Syst. 52, 1 (February 2022), 12–25. https://doi.org/10.1109/THMS.2021.3131684
-
[22]
Giorgos Giannakakis, Dimitris Grigoriadis, Katerina Giannakaki, Olympia Simantiraki, Alexandros Roniotis, and Manolis Tsiknak is. 2022. Review on Psychological Stress Detection Using Biosignals. IEEE Trans. Affect. Comput. 13, 1 (January 2022), 440 –460. https://doi.org/10.1109/TAFFC.2019.2927337
-
[23]
Sai Krishna Allani, Brendan John, Javier Ruiz, Saurabh Dixit, Jackson Carter, Cindy Grimm, and Ravi Balasubramanian. 2016. Ev aluating human gaze patterns during grasping tasks: robot versus human hand. In Proceedings of the ACM Symposium on Applied Perception , July 22, 2016. ACM, New York, NY, USA, 45–52. https://doi.org/10.1145/2931002.2931007
-
[24]
Stefan Fuchs and Anna Belardinelli. 2021. Gaze -Based Intention Estimation for Shared Autonomy in Pick -and-Place Tasks. Front. Neurorobot. 15, (April 2021). https://doi.org/10.3389/fnbot.2021.647930
-
[25]
Anna Belardinelli, Anirudh Reddy Kondapally, Dirk Ruiken, Daniel Tanneberg, and Tomoki Watabe. 2022. Intention estimation from gaze and motion features for human-robot shared-control object manipulation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , October 23, 2022. IEEE, 9806–9813. https://doi.org/10.1109/IROS47612...
-
[26]
In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pp
Reuben M. Aronson, Nadia Almutlak, and Henny Admoni. 2021. Inferring Goals with Gaze during Teleoperated Manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , September 27, 2021. IEEE, 7307 –7314. https://doi.org/10.1109/IROS51168.2021.9636551
-
[27]
Ewen Lavoie, Jacqueline S. Hebert, and Craig S. Chapman. 2024. Comparing eye –hand coordination between controller-mediated virtual reality, and a real-world object interaction task. J. Vis. 24, 2 (February 2024), 9. https://doi.org/10.1167/jov.24.2.9
-
[28]
Randall Flanagan, and Roland S
Uta Sailer, J. Randall Flanagan, and Roland S. Johansson. 2005. Eye –Hand Coordination during Learning of a Novel Visuomotor Task. The Journal of Neuroscience 25, 39 (September 2005), 8833–8842. https://doi.org/10.1523/JNEUROSCI.2658-05.2005
-
[29]
Aronson, Thiago Santini, Thomas C
Reuben M. Aronson, Thiago Santini, Thomas C. Kübler, Enkelejda Kasneci, Siddhartha Srinivasa, and Henny Admoni. 2018. Eye -Hand Behavior in Human-Robot Shared Manipulation. In Proceedings of the 2018 ACM/IEEE International Conference on Human -Robot Interaction , February 26,
2018
-
[30]
https://doi.org/10.1145/3171221.3171287
ACM, New York, NY, USA, 4–13. https://doi.org/10.1145/3171221.3171287
-
[31]
Mikhail Startsev, Ioannis Agtzidis, and Michael Dorr. 2019. 1D CNN with BLSTM for automated classification of fixations, sacc ades, and smooth pursuits. Behav. Res. Methods 51, 2 (April 2019), 556–572. https://doi.org/10.3758/s13428-018-1144-2
-
[32]
Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie Weygandt Mathis, and Matthias Be thge. 2018. 25 DeepLabCut: markerless pose estimation of user -defined body parts with deep learning. Nat. Neurosci. 21, 9 (September 2018), 1281 –1289. https://doi.org/10.1038/s41593-018-0209-y
-
[33]
Sergio Garrido-Jurado, Rafael Muñoz-Salinas, Francisco J. Madrid-Cuevas, and Manuel J. Marín-Jiménez. 2014. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognit. 47, 6 (June 2014), 2280–2292. https://doi.org/10.1016/j.patcog.2014.01.005
-
[34]
David A. Rosenbaum, Kate M. Chapman, Matthias Weigelt, Daniel J. Weiss, and Robrecht van der Wel. 2012. Cognition, action, an d object manipulation. Psychol. Bull. 138, 5 (2012), 924–946. https://doi.org/10.1037/a0027839
-
[35]
J Randall Flanagan, Miles C Bowman, and Roland S Johansson. 2006. Control strategies in object manipulation tasks. Curr. Opin. Neurobiol. 16, 6 (December 2006), 650–659. https://doi.org/10.1016/j.conb.2006.10.005
-
[36]
Chris Miall, Orna Rosenthal, Kristin Ørstavik, Jonathan D
R. Chris Miall, Orna Rosenthal, Kristin Ørstavik, Jonathan D. Cole, and Fabrice R. Sarlegna. 2019. Loss of haptic feedback im pairs control of hand posture: a study in chronically deafferented individuals when grasping and lifting objects. Exp. Brain Res. 237, 9 (September 2019), 2167 –2184. https://doi.org/10.1007/s00221-019-05583-2
-
[37]
Jonathon W. Sensinger and Strahinja Dosen. 2020. A Review of Sensory Feedback in Upper -Limb Prostheses From the Perspective of Human Motor Control. Front. Neurosci. 14, (June 2020). https://doi.org/10.3389/fnins.2020.00345
-
[38]
Paul D. Marasco, Jacqueline S. Hebert, Jonathon W. Sensinger, Dylan T. Beckler, Zachary C. Thumser, Ahmed W. Shehata, Heather E. Williams, and Kathleen R. Wilson. 2021. Neurorobotic fusion of prosthetic touch, kinesthesia, and movement in bionic upper limbs promotes intrinsic brain behaviors. Sci. Robot. 6, 58 (September 2021). https://doi.org/10.1126/sci...
-
[39]
Kodi Y. Cheng, Heather E. Williams, Ahmed W. Shehata, Patrick M. Pilarski, Craig S. Chapman, and Jacqueline S. Hebert. 2025. Exploring the impact of myoelectric prosthesis controllers on visuomotor behavior. J. Neuroeng. Rehabil. 22, 1 (March 2025), 57. https://doi.org/10.1186/s12984 -025- 01604-0
-
[40]
Ewen Lavoie, Jacqueline S. Hebert, and Craig S. Chapman. 2025. How a lack of haptic feedback affects eye -hand coordination and embodiment in virtual reality. Sci. Rep. 15, 1 (July 2025), 25219. https://doi.org/10.1038/s41598-025-10319-0
-
[41]
Michael F. Land and Mary Hayhoe. 2001. In what ways do eye movements contribute to everyday activities? Vision Res. 41, 25–26 (November 2001), 3559–3565. https://doi.org/10.1016/S0042-6989(01)00102-X
-
[42]
Mohammad M D Sobuh, Laurence P J Kenney, Adam J Galpin, Sibylle B Thies, Jane McLaughlin, Jai Kulkarni, and Peter Kyberd. 2014. Visuomotor behaviours when using a myoelectric prosthesis. J. Neuroeng. Rehabil. 11, 1 (2014), 72. https://doi.org/10.1186/1743-0003-11-72
-
[43]
Eleftherios Triantafyllidis, Christopher Mcgreavy, Jiacheng Gu, and Zhibin Li. 2020. Study of Multimodal Interfaces and the I mprovements on Teleoperation. IEEE Access 8, (2020), 78213–78227. https://doi.org/10.1109/ACCESS.2020.2990080
-
[44]
Marc O. Ernst and Martin S. Banks. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 6870 (January 2002), 429–433. https://doi.org/10.1038/415429a
-
[45]
Jingjing Zhang, Mengjie Huang, Kai -Lun Liao, and Rui Yang. 2022. Exploring the Effect of Virtual Hand Realism on Embodiment by Subjective Ratings and Eye Tracking. In 2022 1st IEEE International Conference on Cognitive Aspects of Virtual Reality (CVR) , May 11, 2022. IEEE, 000017 – 000020. https://doi.org/10.1109/CVR55417.2022.9967571
-
[46]
Pradhan, Ivo V
Sara Falcone, Anne-Marie Brouwer, Dirk Heylen, Jan Van Erp, Liang Zhang, Saket S. Pradhan, Ivo V. Stuldreher, Ioana Cocu, Martijn Heuvel, Pieter S. de Vries, and Kaj Gijsbertse. 2022. Pupil diameter as an implicit measure to estimate sense of embodiment. I n Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 44 26 A APPENDICES A.1 Fi...
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.