Pith. sign in

REVIEW 4 major objections 5 minor 21 references

General Hand Guidance Framework using Microsoft HoloLens

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A HoloLens headset alone can hand-guide a standard industrial robot, with no torque sensors, external hardware, or dynamic model.

desk verdict The concept is appealing but a projection error in the joint-command law and thin experiments mean the central claims are not yet supported. read the letter →

arxiv 1908.04692 v1 pith:ZXMC7QNM submitted 2019-08-13 cs.RO cs.HC

David Puljiz , Erik Stöhr , Katharina S. Riesterer , Björn Hein , Torsten Kröger This is my paper · ORCID
classification cs.ROcs.HC
keywords handguidanceaugmentedrealityHoloLensindustrialmanipulatorICPregistrationkinematicchainhuman-robotinteractionnoexternalsensors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a Microsoft HoloLens headset alone can provide hand guidance for a standard industrial robot, removing the need for joint-torque sensors, external force-torque sensors, or a dynamic model. The system registers a holographic robot model to the real robot by matching it to the HoloLens spatial mesh, then reads the user's hand position with the headset's built-in tracking. Hand motion is converted into joint commands by decomposing it into rotations about successive joint axes and propagating the residual down the kinematic chain, so any robot with a kinematic description and 3D mesh can be guided or teleoperated. The authors demonstrate feasibility on a KUKA KR-5 industrial manipulator, showing that registration improves mean alignment error from 27.53 mm for manual guesses to 4.91 mm, and that the end-effector tracks hand motion.

What carries the argument

The load-bearing mechanism is the angle-update-and-propagate procedure expressed in equations (1) through (6). Given the hand positions in the current and previous frames, the joint position, and the joint axis, the hand displacement is projected onto the plane perpendicular to the axis; the angle between the projected vectors defines the change in joint angle. The previous hand position is then rotated around the joint axis by that same angle using a quaternion, and the residual displacement is passed to the next joint down the kinematic chain. This turns a single free-hand motion into a coordinated sequence of joint rotations without any model of robot dynamics.

What would settle it

Run the guidance loop with the operator's hand in front of a black surface or within a few centimeters of the robot body; the paper reports the built-in tracker loses the hand in those conditions, so a test showing the robot diverges or stops exactly when the hand is lost would falsify the central claim. A second check: compare commanded and actual joint angles while moving link 3; if the end-effector does not follow the hand with small lag, the mechanism fails.

Watch

Extended reading notes

Core claim

The central claim is that hand guidance can be generalized by replacing physical contact with spatial referencing and virtual interaction. The HoloLens builds a spatial mesh, the user places a seed hologram near the robot, and an ICP registration algorithm aligns the robot model to the scene point cloud. Once aligned, the user's tracked hand position relative to the robot generates joint commands: for the link being held, the hand displacement is projected onto the plane perpendicular to that joint's rotation axis, the resulting angle gives the joint update, and the remaining hand motion is rotated about the joint axis and propagated down the kinematic chain to the next joint. The robot follows the hand without being touched, requiring no force sensing and no dynamic or friction model, and the same mechanism supports both direct link pushing and end-effector dragging through a holographic handle.

Load-bearing premise

The entire control input is the HoloLens' built-in hand tracking, which the authors report has trouble seeing hands against black backgrounds and when hands are close to objects; if that tracking is unreliable in a real work cell, the robot cannot be guided.

Editorial extensions

If this is right

  • Non-collaborative industrial robots can be hand-guided without hardware retrofits, as long as a kinematic description and a 3D mesh exist.
  • Operators can choose between dragging a holographic end-effector sphere and pushing individual links, which lets redundant robots use their extra degrees of freedom.
  • No dynamic model, friction model, or identification experiments are needed before use, and the command output works with any downstream controller.
  • Because the robot is referenced virtually, the same setup can teleoperate an inaccessible robot by repositioning the hologram manually, including resizing it for very large machines.
  • The registration step alone improves overlay precision from a mean of 27.53 mm for manual placement to 4.91 mm, with tolerance to imprecise seed placement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: replacing the built-in depth hand tracker with the RGB-based tracker the authors point to would directly remove the stated failure mode of losing hands against black backgrounds and near objects, making the system far closer to a true plug-and-play teaching tool.
  • Editorial extension: the same registration-plus-hand-propagation mechanism could be re-run online to support mobile manipulators or moving robot bases, something the paper does not claim to handle.
  • Editorial extension: link-level pushing without physical contact could lower the barrier to collecting kinesthetic demonstrations for imitation learning, especially for redundant or humanoid robots where touching the robot is impractical.
  • Editorial extension: because the controller is generic and downstream, the hand-to-joint mapping could be compared head-to-head with joint-torque-sensor guidance in a user study; the authors list such a study as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes a hand-guidance system for a conventional industrial robot (KUKA KR-5) using a Microsoft HoloLens. The robot model is overlaid on the real robot through a semi-automatic ICP-based registration between the HoloLens spatial mesh and the robot's meshes. Hand motions are captured by the HoloLens' built-in hand tracking and are converted into joint commands by projecting consecutive hand positions onto planes perpendicular to joint axes, computing incremental joint angles, and propagating the residual motion down the kinematic chain. The authors claim that this removes the need for any sensors on the robot or in the cell and requires only a URDF model, thus enabling generalized hand guidance without a dynamic model. Two experiments are reported: registration robustness relative to user seed errors, and a single control trial in which link 3 is moved while joint commands and end-effector positions are recorded.

Significance. If the method works as claimed, it would be a practical contribution: it extends hand guidance to conventional industrial manipulators without torque sensors, and its reliance on standard URDF models and open-source ROS tools makes it portable across robot types. The registration study, showing robustness to seed placement errors, is a useful step. However, the kinematic derivation contains a geometric error that undermines the central decomposition claim, and the experimental evidence for the control claim is preliminary. The idea is timely and worth pursuing, but the manuscript in its current form does not establish the core algorithm or its reported precision.

major comments (4)
  1. [Section III, Eq. (1)] Equation (1) does not project the hand positions onto the plane through the joint center s_j,t as the text states. The expression p_j,t = h_t - (h_t·a_j,t / ||a_j,t||^2) a_j,t projects onto the plane through the world origin perpendicular to a_j,t, not onto the plane through s_j,t. The correct projection is p_j,t = h_t - (((h_t - s_j,t)·a_j,t) / ||a_j,t||^2) a_j,t. Because Eq. (2) then forms v_j,t = p_j,t - s_j,t, each v_j,t carries a spurious component (s_j,t·a_j,t / ||a_j,t||^2) a_j,t along the joint axis. The angle computed in Eq. (3) is therefore not the true angle of the hand about the joint. For example, with a=(0,0,1), s=(0,0,1), h_{t-1}=(1,0,1), and h_t=(0,1,1), the true rotation about the joint is 90 degrees, but Eqs. (1)-(3) give arccos(1/2)=60 degrees. Since Eq. (5) propagates this biased angle through the kinematic chain, the claim that hand motions are decomposed into orthogonal rotations is not supported by the presented mathematics. The correction is local, but it must be made and the derivation rechecked.
  2. [Section IV, Table I] The registration accuracy is evaluated by the RMS distance between closest points in the model point cloud and the scene point cloud. This is the same objective minimized by the ICP algorithm itself, so it cannot serve as an independent measure of absolute registration error. The additional visual inspection used to discard 'bad matches' is subjective and not reproducible. To support the claim that the system achieves the reported precision (e.g., mean 4.91 mm), the authors should compare against a ground-truth transform, for example from an external tracking system or a known fiducial-based calibration, and report the error distribution over multiple trials.
  3. [Section IV and Fig. 6] The control experiment is a single 15-second trial in which the user moves link 3 of the KR-5. The evaluation is qualitative: the authors state that the end-effector follows the hand and that the desired joint states are intuitively consistent, but no quantitative metric (e.g., RMS or maximum error between hand trajectory and end-effector trajectory) is reported, and there is no repeatability test or comparison with a JTS-based baseline. As the central claim of the paper is that this method is a usable alternative to sensor-based hand guidance, the experimental support is too thin.
  4. [Section V] The authors acknowledge that the built-in hand tracking 'had difficulties tracking the hand in front of black backgrounds... and when in close proximity to objects.' Since the control input is entirely derived from tracked hand positions, these failures can directly break the proposed interaction in industrial environments, where dark equipment and nearby objects are common. This is a limitation of the central method, not merely a future-work item; the paper should either demonstrate that the operating envelope covers the claimed use cases or clearly restrict the system's scope to conditions where the hand tracker is reliable.
minor comments (5)
  1. [Section V] The final paragraph of Section V ends mid-sentence with 'measuring the referencing error with respect to the'; this appears to be a truncation and should be completed.
  2. [Section III, Eq. (1)] The phrase 'the plane defined by s_j,t and a_j,t' is ambiguous; a plane is not uniquely defined by a point and an axis unless it is specified as the plane through s_j,t perpendicular to a_j,t. Please make this explicit in the text.
  3. [Figure 2 caption] The caption contains a typo in the numbered menu items: item 3 appears twice and item 4 is numbered incorrectly. The caption should be renumbered for clarity.
  4. [Contributions, Section I-B] The claim that the method requires 'no sensors on or around the robot' should be qualified: the system requires a HoloLens, a computer running ROS, and a network connection. These do not need to be mounted on the robot, but the statement as written could be misleading.
  5. [Figure 3] In the graphical representation, the quaternion q_j,t is used to denote both a rotation and the vector of the rotation axis; the legend should distinguish these notations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the hand-guidance command law is a hand-designed geometric heuristic, not a fitted or self-citation-derived result.

full rationale

The central derivation in Section III computes joint-angle updates directly from tracked hand positions, joint axes, and joint positions via Eqs. (1)-(6). No parameter is fitted to the evaluation trajectories, and no 'prediction' is statistically forced: the angle update Dq_j,t is defined by arccos of projected hand-motion vectors, with a tunable gain K set to 1 in the experiment. The referencing system uses standard external algorithms (ICP [17], Super4PCS [18], PCL [19]); no load-bearing uniqueness theorem or ansatz is imported from the authors' prior work. The one author self-citation (Reflexxes, [20], by Kröger) is used only as an off-the-shelf motion library, not as evidence for the derivation. The control evaluation compares executed joint/end-effector states against the same hand motions that generated the commands; that is a system consistency check, not a circular reduction of the claimed result. The registration precision metric, however, is self-referential: the paper reports 'the root mean square (RMS) distance between the closest points in the two point clouds' as the precision metric, which is exactly the objective ICP minimizes, and the authors list 'measuring the referencing error with respect to the ground truth' as future work. This weakens the evaluation of the referencing contribution but does not make the derivation circular. Separately, Eq. (1) omits the joint center in the projection, so the computed angles are geometrically biased; this is a correctness defect in an otherwise independent heuristic, not an equivalence between input and output. The Section V admission that HoloLens hand tracking fails on black backgrounds and near objects is a stated implementation limitation, not a circular step. Overall, the derivation does not reduce to its own inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of HoloLens hand tracking, staticity of the robot during registration, and ICP convergence. No new physical entities are introduced.

free parameters (2)
  • K = 1 (experiments); user-tunable
    Motion scaling factor in Eq. (4); chosen by hand, not fitted to data.
  • point cloud sampling density = small: 1,000 triangles/m3, 16,000 samples; big: 1,240,000 triangles/m3, 256,000 samples
    Two settings tested; the small one was chosen for comparable performance with less processing time (Section IV).
assumptions (4)
  • domain assumption The robot is static and joint values are known during referencing.
    Stated in Section II-A: 'We assume that the robot is static at the moment this is done and that the joint values are known to the controlling computer.' The registration and hand guidance depend on this.
  • domain assumption The HoloLens built-in hand tracking reliably detects hand position and the Hold gesture.
    The control input is derived entirely from tracked hand positions; the authors note failures with black backgrounds and close objects in Section V.
  • domain assumption ICP registration converges to the correct robot pose from the user-placed seed.
    The seed limits the search space, but ICP is local; the paper only tests on one scene and does not measure ground truth error.
  • standard math Standard linear algebra and quaternion operations in Eqs. (1)-(6).
    Projection, arccos, and quaternion rotation are standard; no novel math is claimed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of General Hand Guidance Framework using Microsoft HoloLens." pith.science (2026). https://pith.science/paper/ZXMC7QNM

@misc{pith2026190804692,
  author       = {Pith},
  title        = {Pith review of: General Hand Guidance Framework using Microsoft HoloLens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXMC7QNM}},
  note         = {Machine review of arXiv:1908.04692}
}
read the original abstract

Hand guidance emerged from the safety requirements for collaborative robots, namely possessing joint-torque sensors. Since then it has proven to be a powerful tool for easy trajectory programming, allowing lay-users to reprogram robots intuitively. Going beyond, a robot can learn tasks by user demonstrations through kinesthetic teaching, enabling robots to generalise tasks and further reducing the need for reprogramming. However, hand guidance is still mostly relegated to collaborative robots. Here we propose a method that doesn't require any sensors on the robot or in the robot cell, by using a Microsoft HoloLens augmented reality head mounted display. We reference the robot using a registration algorithm to match the robot model to the spatial mesh. The in-built hand tracking and localisation capabilities are then used to calculate the position of the hands relative to the robot. By decomposing the hand movements into orthogonal rotations and propagating it down through the kinematic chain, we achieve a generalised hand guidance without the need to build a dynamic model of the robot itself. We tested our approach on a commonly used industrial manipulator, the KUKA KR-5.

Figures

Figures reproduced from arXiv: 1908.04692 by the authors.

Figure 1
Figure 1. The user moving their hand next to the robot link they want to [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The UI of the application. On the top left one can see the main menu following the user. It allows to 1) open connections with a PC running [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. The influence of rotation on our registration algorithm. One can [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: Graphical representation of (1) - (6). The hand motion between the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 5
Figure 5. Figure 5: The results of the referencing algorithm; (left) The HoloLens spatial mesh and the seed hologram positioned by the user; (right) The hologram [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The Cartesian movements of the end-effector and joints 1-3 while moving the third link for 15 seconds. (left) The movements of the hand strongly [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 16 canonical work pages

  1. [1]

    The kuka-dlr lightweight robot arm-a new reference platform for robotics research and manufacturing,

    R. Bischoff, J. Kurth, G. Schreiber, R. Koeppe, A. Albu-Sch ¨affer, A. Beyer, O. Eiberger, S. Haddadin, A. Stemmer, G. Grunwald et al., “The kuka-dlr lightweight robot arm-a new reference platform for robotics research and manufacturing,” in Robotics (ISR), 2010 41st international symposium on and 2010 6th German conference on robotics (ROBOTIK). VDE, 201...

    work page 2010

  2. [2]

    The need for an intuitive teaching method for small and medium enterprises,

    R. D. Schraft and C. Meyer, “The need for an intuitive teaching method for small and medium enterprises,” VDI BERICHTE, vol. 1956, p. 95, 2006

    work page 1956

  3. [3]

    Hand-guiding robots along predefined geometric paths under hard joint constraints,

    M. Hanses, R. Behrens, and N. Elkmann, “Hand-guiding robots along predefined geometric paths under hard joint constraints,” in 2016 IEEE 21st International Conference on Emerging Technologies and Factory Automation (ETF A), Sept 2016, pp. 1–5

    work page 2016

  4. [4]

    A survey of robot learning from demonstration,

    B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and autonomous systems, vol. 57, no. 5, pp. 469–483, 2009

    2009

  5. [5]

    Perspectives on augmented reality based human-robot interaction with industrial robots,

    R. Bischoff and A. Kazi, “Perspectives on augmented reality based human-robot interaction with industrial robots,” in Intelligent Robots and Systems, 2004.(IROS 2004). Proceedings. 2004 IEEE/RSJ Inter- national Conference on , vol. 4. IEEE, 2004, pp. 3226–3231

    work page 2004

  6. [6]

    Intuitive robot tasks with augmented reality and virtual obstacles,

    A. Gaschler, M. Springer, M. Rickert, and A. Knoll, “Intuitive robot tasks with augmented reality and virtual obstacles,” in Robotics and Automation (ICRA), 2014 IEEE International Conference on . IEEE, 2014, pp. 6026–6031

    work page 2014

  7. [7]

    Robot programming through augmented trajectories in augmented reality,

    C. P. Quintero, S. Li, M. K. Pan, W. P. Chan, H. F. M. V . der Loos, and E. Croft, “Robot programming through augmented trajectories in augmented reality,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , Oct 2018, pp. 1838–1844

    work page 2018

  8. [8]

    Interactive robot knowledge patching using augmented reality,

    H. Liu, Y . Zhang, W. Si, X. Xie, Y . Zhu, and S.-C. Zhu, “Interactive robot knowledge patching using augmented reality,” in2018 IEEE 21st International Conference on Robotics and Automation (ICRA) , 2018, pp. 1947 –1954

    work page 2018

Show all 21 references
  1. [9]

    Alternative modes of interaction in proximal human-in-the-loop operation of robots,

    T. Chakraborti, S. Sreedharan, A. Kulkarni, and S. Kambhampati, “Alternative modes of interaction in proximal human-in-the-loop operation of robots,” CoRR, vol. abs/1703.08930, 2017. [Online]. Available: http://arxiv.org/abs/1703.08930

    2017 arXiv

  2. [10]

    Communicating robot motion intent with augmented reality,

    M. Walker, H. Hedayati, J. Lee, and D. Szafir, “Communicating robot motion intent with augmented reality,” in Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction , ser. HRI ’18. New York, NY , USA: ACM, 2018, pp. 316–324. [Online]. Available: ht...

    2018

  3. [11]

    Real-time hand guiding of industrial manipulator in 5 dof using microsoft kinect and accelerometer,

    S. Moe and I. Schjlberg, “Real-time hand guiding of industrial manipulator in 5 dof using microsoft kinect and accelerometer,” in 2013 IEEE RO-MAN , Aug 2013, pp. 644–649

    2013

  4. [12]

    Torque control based sensorless hand guiding for direct robot teaching,

    S. D. Lee, K. H. Ahn, and J. B. Song, “Torque control based sensorless hand guiding for direct robot teaching,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , Oct 2016, pp. 745–750

    2016

  5. [13]

    Accurate sensorless lead-through programming for lightweight robots in structured environments,

    M. Ragaglia, A. M. Zanchettin, L. Bascetta, and P. Rocco, “Accurate sensorless lead-through programming for lightweight robots in structured environments,” Robotics and Computer-Integrated Manufacturing, vol. 39, pp. 9 – 21, 2016. [Online]. Available: http://www.sciencedirect....

    2016

  6. [14]

    Sensorless friction-compensated passive lead- through programming for industrial robots,

    A. Stolt, F. B. Carlson, M. M. G. Ardakani, I. Lundberg, A. Roberts- son, and R. Johansson, “Sensorless friction-compensated passive lead- through programming for industrial robots,” in 2015 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS) , Sep. 20...

    2015

  7. [15]

    Ros: an open-source robot operating system,

    M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs, E. Berger, R. Wheeler, and A. Ng, “Ros: an open-source robot operating system,” in Proc. of the IEEE Intl. Conf. on Robotics and Automation (ICRA) Workshop on Open Source Robotics , Kobe, Japan, May 2009

    2009

  8. [16]

    ros control: A generic and simple control framework for ros,

    S. Chitta, E. Marder-Eppstein, W. Meeussen, V . Pradeep, A. Rodr ´ıguez Tsouroukdissian, J. Bohren, D. Coleman, B. Magyar, G. Raiola, M. L ¨udtke, and E. Fern ´andez Perdomo, “ros control: A generic and simple control framework for ros,” The Journal of Open Source Software , 2...

    2017

  9. [17]

    A method for registration of 3-d shapes,

    P. J. Besl and N. D. McKay, “A method for registration of 3-d shapes,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 14, no. 2, pp. 239–256, Feb 1992

    1992

  10. [18]

    Super 4pcs fast global pointcloud registration via smart indexing,

    N. Mellado, D. Aiger, and N. J. Mitra, “Super 4pcs fast global pointcloud registration via smart indexing,” Computer Graphics F orum, vol. 33, no. 5, pp. 205–215, 2014. [Online]. Available: http://dx.doi.org/10.1111/cgf.12446

    2014 doi

  11. [19]

    3D is here: Point Cloud Library (PCL),

    R. B. Rusu and S. Cousins, “3D is here: Point Cloud Library (PCL),” in IEEE International Conference on Robotics and Automation (ICRA) , Shanghai, China, May 9-13 2011

    2011

  12. [20]

    Opening the door to new sensor-based robot applicationsthe reflexxes motion libraries,

    T. Krger, “Opening the door to new sensor-based robot applicationsthe reflexxes motion libraries,” in 2011 IEEE International Conference on Robotics and Automation , May 2011, pp. 1–4

    2011

  13. [21]

    Ganerated hands for real-time 3d hand tracking from monocular rgb,

    F. Mueller, F. Bernard, O. Sotnychenko, D. Mehta, S. Sridhar, D. Casas, and C. Theobalt, “Ganerated hands for real-time 3d hand tracking from monocular rgb,” in Proceedings of Computer Vision and Pattern Recognition (CVPR) , June 2018. [Online]. Available: https://handtracker....

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.