REVIEW 4 major objections 6 minor 31 references
Where Do Humans Look When Demonstrating to Robots? Human Gaze Behavior in Pick-and-Place Tasks Across Demonstration Devices
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that demonstration devices that emulate a robot's embodiment or visual condition degrade the task-relevant information carried by a human demonstrator's gaze, and that gaze captured with a natural wearable camera—rather…
desk verdict The behavioral device comparison is the real contribution; the headline policy gain is confounded by PixMix augmentation and needs an ablation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a measurable quantity called 'task-relevant cue extraction capability': the number of trials, out of eight, in which a demonstrator fixates on the target during the pick phase or on the destination during the place phase. The framework compares seven demonstration setups along two axes—embodiment emulation (Wearable, UMI, Leader, Leader-Follower) and visual-condition emulation (Wearable, HMD-Ego, HMD-Top-down)—and tracks the competing tendency, gaze directed at the end-effector. In the policy comparison, the machinery is a gaze-conditioned augmentation pipeline: predicted gaze coordinates from a learned gaze estimation model are converted into a pixel weight map, pixels near the gaze point keep the original image, and pixels farther away are increasingly overwritten by PixMix augmentation before training a CNN-based Diffusion Policy. This makes gaze the mechanism that decides which visual details the policy must preserve.
What would settle it
Train the same Diffusion Policy with the same PixMix pipeline but replace predicted gaze with random or uniform weight maps; if the 18.8% to 68.8% out-of-distribution improvement persists, the claim that natural-gaze cues drive the gain is falsified. Separately, compare the gaze estimated on Leader-Follower demonstrations against the human fixations recorded in the behavioral study to check whether the estimates carry the same cue-quality differences.
Extended reading notes
Core claim
The paper's central discovery is that demonstration devices which emulate a robot's embodiment or visual condition impair a demonstrator's ability to extract task-relevant cues through gaze, and the impairment grows with the degree of emulation. Across the embodiment spectrum from Wearable to UMI to Leader to Leader-Follower, the number of trials in which the demonstrator fixates the target during pick or the destination during place decreases steadily, while gaze directed at the end-effector increases, and the same pattern appears when the viewpoint changes to an HMD-based top-down view. Workload, measured by NASA-TLX, also rises stepwise with emulation. In the policy comparison, gaze data from the Wearable condition produced task success rates close to an oracle using manually annotated gaze, lifting out-of-distribution pick success from 18.8% to 68.8% and out-of-distribution place success from 37.5% to 81.3%, while gaze from more robot-like devices sometimes performed worse than the non-gaze baseline.
Load-bearing premise
The load-bearing premise is that the large out-of-distribution gain comes from the gaze information itself, not from the image augmentation it is bundled with; the paper never runs a control with random gaze weighting or augmentation alone, so that attribution is an assumption.
Editorial extensions
If this is right
- A two-device data collection recipe follows from the trade-off: collect gaze with a wearable camera for cue extraction, and collect demonstration actions with the Leader-Follower for a small domain gap.
- Gaze-relevant instructions improve the capability metric across most devices without adding workload, so they are a low-cost addition to demonstration protocols.
- Gaze from the natural wearable condition raises out-of-distribution pick success from 18.8% to 68.8% and out-of-distribution place success from 37.5% to 81.3%, approaching an oracle that uses manually annotated gaze.
- Gaze from strongly robot-emulating devices can be worse than no gaze at all, as the UMI policy scored below the non-gaze baseline in place phases, so transferring gaze across data sources is not automatically beneficial.
Reading between the lines
- The paper leaves untested whether the same out-of-distribution gain would appear if the PixMix augmentation were applied with random or uniform weight maps; until that control is run, the gain should be read as the combined effect of gaze-guided augmentation rather than of gaze alone.
- A cheap screening rule follows from the behavioral result: record a few demonstrations on a candidate device, measure the ratio of object fixations to end-effector fixations, and expect gaze-conditioned policies to help only when object fixations dominate.
- The same experimental design could be pointed at subgoal extraction, as the paper's future-work section hints: wearable gaze may mark pick and place phases well enough to segment demonstrations for hierarchical policies without manual annotation.
- Because the zero-shot gaze estimator transferred poorly for UMI place phases, a domain-adaptive gaze estimator conditioned on the demonstration device is a natural next step that the paper does not test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how demonstration devices that emulate a robot's embodiment or visual condition affect human gaze during pick-and-place demonstrations. Two behavioral experiments compare Wearable, UMI, Leader, and Leader-Follower (embodiment experiment) and Wearable, HMD-Ego, and HMD-Top-down (visual condition experiment), measuring a 'capability' of extracting task-relevant cues defined as fixating on the target or destination. The results show a monotonic decline in capability with stronger embodiment emulation and a decline with the top-down view when gaze-relevant instructions are absent. A policy-based comparison then trains per-device GLC gaze estimators and uses gaze-weighted PixMix augmentation to train a CNN-based Diffusion Policy, reporting that gaze from the Wearable device improves out-of-distribution success from 18.8% to 68.8% in the pick task compared to a non-gaze baseline. The paper concludes that natural-device gaze improves policy robustness and provides design guidance for demonstration devices.
Significance. If the claims hold, the paper offers a useful, systematic framework for choosing demonstration devices when collecting gaze-informed imitation learning data, and it bridges cognitive science gaze literature with robot learning practice. The behavioral experiments are thoughtfully designed, with workload analysis using repeated-measures ANOVA, and the policy evaluation goes beyond simple behavioral metrics. Notable strengths include the explicit comparison across a spectrum of devices and the inclusion of an Oracle gaze condition. However, the central policy claim is weakened by a missing augmentation control and by the zero-shot application of gaze estimators across domains, while the behavioral claims lack inferential statistics and a precise fixation definition. The contribution is potentially valuable but requires substantial additional analysis and controls to be convincing.
major comments (4)
- [Sec. 5, Appendix D (Eqs. 1-2, Table 5)] The policy comparison does not isolate the effect of gaze information. The gaze-augmented policy trains with PixMix augmentation weighted by predicted gaze, whereas the Baseline uses standard training with no PixMix. Since PixMix is a known robustness regularizer, the 18.8→68.8 OOD-Pick improvement could be caused by the augmentation procedure alone rather than by the spatial distribution of gaze. A control with uniform PixMix weighting or with random gaze coordinate weighting is necessary to attribute the gain to gaze. Without this control, the headline claim that natural-device gaze improves policy robustness is not supported.
- [Sec. 4.1/4.2, Figs. 3 and 5] The capability metric (number of trials out of eight with a fixation on the task-relevant cue) is reported only as point summaries, without significance tests, confidence intervals, or effect sizes. With N=8 participants, the observed ordering across devices could be within sampling variability. The repeated-measures ANOVA reported for workload does not substitute for inference on the capability outcome. Additionally, the paper never defines the fixation criterion (e.g., a distance threshold between gaze and the annotated object, or a minimum fixation duration), making the metric unreproducible. Please provide appropriate nonparametric tests or confidence intervals and a precise fixation definition.
- [Sec. 5, Appendix D (Fig. 11, Table 5)] The policy comparison uses GLC gaze estimators trained on each device's image-gaze pairs and applied zero-shot to Leader-Follower demonstration images. This introduces a domain shift: Wearable images are first-person human camera views, while Leader-Follower images are third-person views of a robot. The paper does not validate that the predicted gaze on Leader-Follower data reproduces the measured behavioral gaze quality. Since the Oracle column uses manually annotated gaze, a straightforward validation would compare GLC predictions on the Leader-Follower data against those manual annotations. Without this, the policy comparison tests estimated gaze under a domain shift, not the human gaze measured in Sec. 4, and the interpretation of the Wearable result is ambiguous.
- [Appendix B, Procedure] The experiment always administers the 'without gaze-relevant instruction' condition before the 'with gaze-relevant instruction' condition for all participants. Since the same participants perform both conditions in this fixed order, the reported improvement with instructions is confounded with practice or familiarity. This affects the discussion's recommendation that 'providing gaze-related instructions is recommended whenever possible' (Sec. 4.1 Discussion). Please counterbalance instruction order across participants or explicitly treat the instruction effect as exploratory and discuss the confound.
minor comments (6)
- [Sec. 3.1] The section defines two Euclidean distance metrics (gaze-to-object and gaze-to-end-effector) but these distances are never reported in Sec. 4; only the fixation-count capability is used. Please either present the distance analyses or remove the definition.
- [Abstract and Table 5] The abstract states that gaze data 'improves the policy's task success rate from 18.8% to 68.8%,' but Table 5 shows this improvement for the OOD-Pick condition only; OOD-Place improves from 37.5% to 81.3%. Please specify the condition or report both improvements.
- [Sec. 4.1 and 4.2] The ANOVA results report only p-values (and one η² in Sec. 4.2). Please include F-statistics, degrees of freedom, and effect sizes for all reported tests, and state whether sphericity was assessed.
- [Table 9] The GLC sampling rate and length hyperparameters are device-dependent and appear without justification. Since these choices affect the estimated gaze, please provide a rationale or show robustness to reasonable variations.
- [Title] The title in the submission metadata ('Where Do Humans Look When Demonstrating to Robots?') differs from the title in the PDF ('Where Do We Look When We Teach?'); please align them.
- [Acknowledgments] The Acknowledgments section contains placeholder text and should be replaced with actual acknowledgments or removed before review.
Circularity Check
No significant circularity: the behavioral results are measured outcomes, and the policy comparison, while missing an augmentation control, does not fit or define its reported success rates in terms of its inputs.
full rationale
The paper's load-bearing claims are empirical rather than derivational. RQ1 and RQ2 capability is explicitly defined as fixating on the target during pick or the destination during place, measured with an eye tracker and manual annotation; the ordering Wearable > UMI > Leader > Leader-Follower is a measured outcome, not a fitted consequence. The workload analyses use NASA-TLX and repeated-measures ANOVA, which are external instruments. In the policy comparison of Sec. 5, Eqs. 1-2 construct PixMix augmentation weights from predicted gaze coordinates, but Table 5 success rates are measured from 16 trials and are not optimized by choosing r=30, sigma=100 or by the per-device GLC sampling rates in Table 9. Those hyperparameters could affect the results, but they are not fitted to the reported success outcomes, so the 18.8% to 68.8% OOD gain is not circular in the sense of a parameter renamed as a prediction. The only self-citation, [5], appears in the introduction as an illustrative example of shortcut learning and is not load-bearing for the paper's central claims. Appendix A explicitly limits the conclusions to individually emulated embodiment or visual conditions, and Sec. 5 acknowledges the limitation of zero-shot gaze prediction for UMI. Appendix D does lack a PixMix-only or random-gaze-weight control, which is a genuine confound and correctness risk for attributing the robustness gain to gaze, but it is not a definitional or self-citational circularity. No equation in the derivation chain equals an input by construction.
Assumptions & free parameters
free parameters (4)
- GLC sampling rate per device and phase =
Wearable pick 2, place 3; UMI 4, 5; Leader 9, 6; Leader-Follower 12, 15 (Table 9)
- Gaze augmentation radius r =
30.0 pixels
- Gaze augmentation sigma =
100.0
- Fixation threshold for capability metric =
not reported
assumptions (3)
- domain assumption Fixations directed at targets and destinations indicate task-relevant cue extraction (Land et al. 1999; Hayhoe and Ballard 2005).
- domain assumption The three device paradigms (wearable, embodiment-emulating, visual-condition-emulating) are representative of demonstration devices used in imitation learning.
- ad hoc to paper GLC gaze estimates on Leader-Follower demonstration images are a valid proxy for the gaze behavior measured in the behavioral experiments.
Cite this review
Pith. "Pith review of Where Do Humans Look When Demonstrating to Robots? Human Gaze Behavior in Pick-and-Place Tasks Across Demonstration Devices." pith.science (2026). https://pith.science/paper/L7MFUY5I
@misc{pith2026250605808,
author = {Pith},
title = {Pith review of: Where Do Humans Look When Demonstrating to Robots? Human Gaze Behavior in Pick-and-Place Tasks Across Demonstration Devices},
year = {2026},
howpublished = {\url{https://pith.science/paper/L7MFUY5I}},
note = {Machine review of arXiv:2506.05808}
}
read the original abstract
Imitation learning for generalizable performance often requires a large volume of demonstration data, making the process significantly costly. One promising strategy to address this challenge is to leverage the cognitive skills of human demonstrators with strong generalization capability, particularly by revealing the underlying task demands reflected in their gaze behavior. However, imitation learning typically involves humans collecting data using demonstration devices that emulate a robot's embodiment and visual condition. This raises the question of how such devices influence gaze behavior. We propose an experimental framework that systematically analyzes human demonstrators' gaze behavior across a spectrum of robot-emulating demonstration devices. Our experimental results show that certain device properties shift gaze from task-goal cues (e.g., objects) toward control-monitoring cues (e.g., the end-effector). Furthermore, these shifts directly affect the performance of typical gaze-based imitation learning models, sometimes degrading it below non-gaze baselines.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Haus- man, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li -Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang , and U. Zhilinsky. π 0: A vision-language-action flow model for general robot contro l. ArXiv, abs/2410.24164, 2024
-
[2]
C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, an d S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. ArXiv, abs/2303.04137, 2023
arXiv 2023
-
[3]
R. R. Selvaraju, A. Das, R. V edantam, M. Cogswell, D. Pari kh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128:336 – 359, 2016
work page 2016
-
[4]
R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. S. Zemel, W . Brendel, M. Bethge, and F. Wich- mann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2:665 – 673, 2020
work page 2020
- [5]
-
[6]
J. Pelz, M. Hayhoe, and R. Loeber. The coordination of eye , head, and hand movements in a natural task. Experimental brain research., 139:266–77, 2001
work page 2001
-
[7]
M. Hayhoe and D. Ballard. Eye movements in natural behavi or. Trends in Cognitive Sciences, 9(4):188–194, 2005
work page 2005
-
[8]
H. Kim, Y . Ohmura, A. Nagakubo, and Y . Kuniyoshi. Trainin g robots without robots: Deep imitation learning for master-to-robot policy transfer. IEEE Robotics and Automation Letters , 8:2906–2913, 2022
work page 2022
Show all 31 references
-
[9]
Land and D
M. Land and D. Lee. Where we look when we steer. Nature, 369:742–744, 1994
1994
-
[10]
M. F. Land and S. M. Furneaux. The knowledge base of the oc ulomotor system. Philosophical transactions of the Royal Society of London. Series B, Biolo gical sciences , 352 1358:1231– 1239, 1997
1997
-
[11]
M. Land, N. Mennie, and J. Rusted. The roles of vision and eye movements in the control of activities of daily living. Perception, 28:1311–1328, 02 1999
1999
-
[12]
M. M. Hayhoe, D. G. Bensinger, and D. H. Ballard. Task con straints in visual working memory. Vision Research, 38(1):125–137, 1998
1998
-
[13]
M. M. Hayhoe, A. Shrivastava, R. Mruczek, and J. B. Pelz. Visual memory and motor planning in a natural task. Journal of Vision, 3(1), 2003
2003
-
[14]
Ballard, M
D. Ballard, M. Hayhoe, and J. Pelz. Memory representati ons in natural tasks. Journal of Cognitive Neuroscience, 7:66–80, 12 1995
1995
-
[15]
C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching witho ut in-the-wild robots. In Proceedings of Robotics: Science and Systems (RSS) , 2024. 9
2024
-
[16]
T. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-g rained bimanual manipulation with low-cost hardware. ArXiv, abs/2304.13705, 2023
2023 arXiv
-
[17]
Z. Fu, T. Z. Zhao, and C. Finn. Mobile ALOHA: Learning bim anual mobile manipulation using low-cost whole-body teleoperation. In 8th Annual Conference on Robot Learning , 2024
2024
-
[18]
P . Wu, Y . Shentu, Z. Yi, X. Lin, and P . Abbeel. Gello: A gen eral, low-cost, and intuitive teleoperation framework for robot manipulators. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 12156–12163, 2024
2024
-
[19]
Cheng, J
X. Cheng, J. Li, S. Y ang, G. Y ang, and X. Wang. Open-telev ision: Teleoperation with immer- sive active visual feedback. In Conference on Robot Learning , 2024
2024
-
[20]
S. Chen, C. Wang, K. Nguyen, L. Fei-Fei, and C. K. Liu. Arc ap: Collecting high-quality human demonstrations for robot learning with augmented reality f eedback. ArXiv, abs/2410.08464, 2024
2024 arXiv
-
[21]
R. M. Aronson, T. Santini, T. C. K¨ ubler, E. Kasneci, S. S rinivasa, and H. Admoni. Eye- hand behavior in human-robot shared manipulation. In 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages 4–13, 2018
2018
-
[22]
A. L. Y arbus. Eye Movements and Vision. Plenum Press, 1967
1967
-
[23]
K. A. Turano, D. R. Geruschat, and F. H. Baker. Oculomoto r strategies for the direction of gaze tested with a real-world activity. Vision Research, 43(3):333–346, 2003
2003
-
[24]
Grauman, A
K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari , R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra , V . Cartillier, S. Crane, T....
2022
-
[25]
Kareer, D
S. Kareer, D. Patel, R. Punamiya, P . Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu. Egomimic: Scaling imitation learning via egocentric video . In CoRL W orkshop on Learning Robot Fine and Dexterous Manipulation: Perception and Cont rol, 2024
2024
-
[26]
Y . Li, M. Liu, and J. M. Rehg. In the eye of the beholder: Ga ze and actions in first person video. IEEE Transactions on Pattern Analysis and Machine Intellig ence, 45:6731–6747, 2020
2020
-
[27]
Brohan, N
A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jacks on, S. Jesmonth, N. J. Joshi, R. C. Julian, D. Kalashnikov, Y . Kuang, I. Leal, K.-H. Lee, S. Levine, Y . Lu, U. Malla...
2022 arXiv
-
[28]
A. Xie, L. Lee, T. Xiao, and C. Finn. Decomposing the gene ralization gap in imitation learning for visual robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 3153–3160, 2024. 10
2024
-
[29]
S. G. Hart and L. E. Staveland. Development of nasa-tlx ( task load index): Results of empirical and theoretical research. volume 52 of Advances in Psychology , pages 139–183. 1988
1988
-
[30]
B. Lai, M. Liu, F. Ryan, and J. M. Rehg. In the eye of transf ormer: Global–local correlation for egocentric gaze estimation and beyond. International Journal of Computer Vision, 132:854 – 871, 2022
2022
-
[31]
pick up the pink fork and place it on the green cup,
D. Hendrycks, A. Zou, M. Mazeika, L. Tang, B. Li, D. Song, and J. Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measur es. CVPR, 2022. 11 A Limitations Evaluation by individual emulation: We found that devices emulating (1) a robot’s embodiment or (2) v...
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.