Pith. sign in

REVIEW 4 major objections 6 minor 45 references

Natural Multimodal Fusion-Based Human-Robot Interaction: Application With Voice and Deictic Posture via Large Language Model

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Combining pointing with speech, parsed by a constrained LLM, gives faster and more robust robot interaction than unimodal baselines.

desk verdict A working voice+pointing HRI system with real robot trials, but the novelty overlaps heavily with the authors' own prior work and the 'significantly better' claim lacks statistical support and a fusion ablation. read the letter →

arxiv 2501.00785 v3 pith:F2YXHUZB submitted 2025-01-01 cs.RO

Yuzhi Lai , Shenghai Yuan , Youssef Nassar , Mingyu Fan , Atmaraaj Gopal , Arihiro Yorita , Naoyuki Kubota , Matthias Rätsch This is my paper · ORCID
classification cs.RO
keywords human-robotinteractiondeicticposturelargelanguagemodelmultimodalfusionvoicecommandobjectdetectionelderlycareroboticsroboticmanipulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that fusing two simple inputs—spoken fragments and a pointing arm—gives a faster, more accurate way for people to command a robot than any single channel. The intended audience is elderly users who cannot memorize gesture vocabularies or complex language syntax. The paper reports that its system took 50.6% less time than hand-gesture interaction, 53.3% less time than language interaction, and 54% less time than a vision-language model, while keeping high accuracy on pick, place, pour, and multi-step tasks. The reason this matters is that the result is achieved with off-the-shelf components—open-vocabulary detection, skeleton tracking, speech-to-text, and an LLM—so the interaction pattern could be reproduced widely.

What carries the argument

The load-bearing mechanism is a nearest-line object selector: for each detected object with 3D position $\xi_i$, compute $$d_i=\sqrt{\frac{|(r_2-r_1)\times(r_1-\xi_i)|^2}{|r_2-r_1|^2}}$$ where $r$ is the right-forearm ray, and choose the object with minimal $d_i$ as the intended target. Voice supplies action intent and object class, a pronoun (this/that) triggers the selection, and optional metric words add parameters such as pour angle. The two input streams are temporally aligned by the pronoun command, and the LLM prompt is divided into API constraints, action definitions, and example tasks, which restricts the model to generating executable, collision-checked action sequences.

What would settle it

Place six visually identical cups 25 cm apart (as in the paper's clutter test), have users point to a non-nearest cup while their forearm is partially occluded, and measure whether the selected bounding box is the intended cup; if the intended object is chosen less reliably than the paper's reported accuracy, the nearest-line assumption is the weak link.

Watch

Extended reading notes

Core claim

The discovery is a parallel multimodal command sequence: the user speaks a short action phrase (with optional class, pronoun, and metric words) while pointing with the right forearm, and a large language model compiles both streams, together with detected object positions, into a robot action sequence. Object intention is decided geometrically by taking the object with minimum distance to the forearm ray (Eq. 1); action intention comes from the spoken words. To keep the LLM safe, the model's output tokens are constrained to a fixed API set, action definitions, and example tasks, so hallucinations cannot escape into arbitrary control. On a real UR3e manipulator, the system completed the same commands in roughly half the interaction time of three unimodal baselines and maintained comparable or better accuracy across simple, causal, and sequential tasks.

Load-bearing premise

The intended object is always the one closest to the line of the user's right forearm at the moment the pronoun is spoken; when pointing is ambiguous, the arm is occluded, or skeleton/depth data are noisy, that selection picks the wrong target and the whole pipeline follows it.

Editorial extensions

If this is right

  • Users can express compound tasks such as "pick cup and pour into bowl at 90 degrees" through a short spoken phrase plus one pointing motion, with no gesture vocabulary to memorize.
  • The reported timing gains (50.6% vs gesture, 53.3% vs language, 54% vs VLM) suggest the parallel interface removes repeated dialogue and gesture rounds.
  • Structuring LLM output through API constraints and example tasks keeps generated action sequences inside a safe, executable set, reducing hallucination risk.
  • The system retains useful accuracy down to about 1 lux, making it practical in typical indoor lighting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the intention decoder only needs a 3D direction ray, the same pipeline could substitute gaze or head orientation for the forearm, which would help users with limited arm mobility; this is an extension, not something the paper tests.
  • The interaction-time advantage likely grows with task complexity, since compound commands avoid the repeated dialogue of VLM baselines; an experiment with three or more sub-actions would test that.
  • The constrained-prompt design is language-agnostic, but the participant pool was mostly English speakers, so the speed and accuracy numbers should be re-tested in other languages before generalizing to the elderly population at large.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces NMM-HRI, a multimodal human-robot interaction framework that fuses voice commands with deictic posture (pointing) input, processed by an LLM to generate robot action sequences. The system uses YOLO-World for object detection, an RGBD camera with OpenPose for skeleton and pointing-direction estimation, VOSK for speech-to-text, and GPT-4 with constrained prompt templates to produce executable actions. It is evaluated on a UR3e manipulator with 27 participants, including 6 elderly users, against gesture-based, NLP-based, and VLM-based baselines, with reported interaction-time reductions of about 50%, end-to-end accuracy comparisons, and robustness tests in cluttered and low-light conditions. The authors state that the system will be open-sourced.

Significance. If the reported results are established, the proposed interaction paradigm is a plausible step toward more natural HRI for elderly users, and the real-robot deployment with diverse participants and tasks is a positive feature. The paper also makes a reasonable contribution in constraining LLM outputs for robot control through structured prompts. However, the central claim—that parallel voice plus deictic posture is significantly more accurate, robust, and efficient than existing unimodal or VLM-based HRI—is not yet adequately supported: the key object-selection mapping is direction-blind, the efficiency comparison lacks statistical grounding, and the robustness analysis is mostly qualitative. The paper would be strengthened substantially by an isolated evaluation of the pointing/object-selection component and by same-pipeline ablations.

major comments (4)
  1. [§III-B, Eq. (1)-(2)] The object-intention mapping P is direction-blind and restricted to the right forearm. Eq. (1) computes the perpendicular distance to the infinite line through two points on the right forearm, so any object lying on the extension of that line behind the hand has a small distance and can be selected instead of the intended target. Section IV-C only mentions an app for users who cannot move their arms and does not address left-arm pointing or partial occlusion of the forearm. Because a single wrong object selection makes the generated action sequence wrong, the claimed accuracy and robustness advantages are not established without an isolated object-selection test that includes objects in front of and behind the hand along the pointing ray, with trials using both arms and with partial occlusion.
  2. [§VI-B, Fig. 7] The headline efficiency gains—50.6% less time than gesture-based HRI, 53.3% less than language-based HRI, and 54% less than VLM-based HRI—are reported as single percentages with no error bars, no per-condition variance, and no statistical significance tests. With 27 participants and no paired comparisons, the claim of 'significantly better' performance is unsupported. In addition, the three baselines are entirely different pipelines (Leap Motion gestures, language-conditioned imitation, and VLM dialogue), so the incremental benefit of the proposed fusion cannot be isolated. The authors should report mean and standard deviation for each condition, perform appropriate paired tests, and add voice-only and pointing-only ablations within the same perception-and-LLM pipeline.
  3. [§VI-D] The robustness claims are not quantitatively supported. The cluttered-environment experiment reports no success-rate numbers, error bars, or per-group statistics for the two participant groups; the low-light experiment says performance is 'comparable' across 1–600 lux without plotting accuracy versus illuminance or giving trial counts. The text itself concedes that robustness depends on skeleton-detection precision and point-cloud quality, yet no experiment isolates object-selection accuracy under those failure modes. Moreover, Section IV-C promises a separate evaluation of deictic-posture accuracy, but no such experiment appears in Section VI. An isolated pointing/object-selection experiment with confusion matrices under varying occlusion, distance, and lighting is needed to substantiate the robustness claim.
  4. [§VI-A, §VI-C] The baselines are dated and not matched to the proposed interaction modality: [6] is a 2023 Leap Motion gesture system, [17] is a 2020 language-conditioned imitation learning method, and [26] is a 2022 VLM system that requires gestures plus dialogue. Accuracy in Section VI-C is defined end-to-end as N_executed/N_trials and is reported without confidence intervals, trial counts per task category, or statistical comparisons. The statement in Section VI-D that 'lighting affects all baseline methods equally' is asserted without presenting baseline data under low light. The comparison should either use contemporaneous baselines with matched interaction conditions or explicitly justify why these baselines constitute a meaningful bound; at minimum, trial counts, confidence intervals, and per-category results should be provided.
minor comments (6)
  1. [§III-B, Eq. (1)] The phrase 'two random points from r' should be 'two distinct points on the pointing line,' and the notation should clarify that r denotes the line rather than the direction vector, since Eq. (1) uses the infinite line through r1 and r2.
  2. [§III-A, Eq. (2)] The notation P(r, ξ) is overloaded: P was originally defined as P(B, S), but Eq. (2) writes P(r, ξ) without specifying that ξ denotes the set of object positions in the scene. This should be made explicit to avoid ambiguity.
  3. [§IV-A] Reference [34] is cited for VOSK speech-to-text, but the reference list entry is a review on hand and arm gesture-based HRI; the VOSK citation appears mismatched and should be corrected.
  4. [§VI-D] The low-light robustness discussion says the system performs comparably above 1 lux, but no quantitative data or curve is shown; at minimum, a table of success rates at representative lux levels should be added.
  5. [§I] The phrase 'an Natural Multimodal fusion-based HRI framework' contains a grammatical error; it should read 'a Natural Multimodal fusion-based HRI framework.'
  6. [§VIII] The conclusion states that the system demonstrates 'excellent effectiveness, accuracy, and robustness,' which overstates the evidence presented in Section VI; the conclusion should be aligned with the actual quantitative support, especially given the limitations acknowledged in Section VII.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the core mappings are operational definitions, the cited prior work is peripheral, and the main claims are evaluated end-to-end against external baselines.

full rationale

The paper is an engineered HRI system rather than a derivation, and no load-bearing step reduces to its own inputs. Eq. (1)-(2) define the object intention IO as the detected object whose 3D position is closest to the right-forearm line; this is an explicit operational definition, not a claimed prediction derived from P, and the experiments assess successful execution/intent against user intent and external baselines (gesture [6], NLP [17], VLM [26]). The self-citations [8], [44], [45] are used for motivation, a future database idea, and future work, respectively, and are not load-bearing for the central speed/accuracy/robustness claims. The LLM prompt includes example tasks, so overlap with some test scenarios is a benchmarking confound rather than a definitional circularity: the system must still map live voice, posture, and depth data to executable API calls. One stated promise is not fulfilled in the reported text: Section IV-C says 'The accuracy of the deictic posture is evaluated in a separate experiment later,' but no isolated posture-accuracy experiment is presented; this is missing evidence, not circularity. Similarly, the acknowledged dependence on skeleton-detection precision and point-cloud quality (Section VI-D) is an external robustness limitation. No self-citation chain, uniqueness argument, or fitted-parameter-as-prediction pattern appears, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The system introduces no new mathematical objects or fitted constants. It depends on a chain of engineering assumptions about perception and LLM reliability, the most fragile being the pointing-ray target selection, which the paper itself identifies as a limiting factor.

assumptions (5)
  • domain assumption The object closest to the user's right forearm direction is the intended object (Eq. 1 and Eq. 2).
    This is the core mapping from posture to object intention; if the user points with another limb or the closest-object rule fails, the entire system selects the wrong target.
  • domain assumption YOLO-World can reliably detect the everyday objects used in the experiments (cup, bowl, plate, scissors, bottle).
    The system relies on open-vocabulary detection to form the object prior set S; a missed or false detection directly changes the target selection.
  • domain assumption GPT4 with the three-part prompt constraint (API constraints, action definition, example tasks) generates safe and executable action sequences.
    Section IV-D states this constraint structure is intended to mitigate hallucination; the paper provides no formal guarantee and only qualitative evidence.
  • domain assumption VOSK speech-to-text can parse fragmented, short verbal commands like 'pick' and 'this'.
    Section IV-A selects VOSK for partial input handling; the reference list does not actually point to VOSK, introducing an unverified tool assumption.
  • domain assumption The RGBD camera and OpenPose produce accurate 3D skeletons under the tested lighting conditions.
    Section VI-D says robustness depends on skeleton detection precision and point cloud quality; the paper does not provide accuracy numbers for skeleton estimation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Natural Multimodal Fusion-Based Human-Robot Interaction: Application With Voice and Deictic Posture via Large Language Model." pith.science (2026). https://pith.science/paper/F2YXHUZB

@misc{pith2026250100785,
  author       = {Pith},
  title        = {Pith review of: Natural Multimodal Fusion-Based Human-Robot Interaction: Application With Voice and Deictic Posture via Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F2YXHUZB}},
  note         = {Machine review of arXiv:2501.00785}
}
read the original abstract

Translating human intent into robot commands is crucial for the future of service robots in an aging society. Existing Human-Robot Interaction (HRI) systems relying on gestures or verbal commands are impractical for the elderly due to difficulties with complex syntax or sign language. To address the challenge, this paper introduces a multi-modal interaction framework that combines voice and deictic posture information to create a more natural HRI system. The visual cues are first processed by the object detection model to gain a global understanding of the environment, and then bounding boxes are estimated based on depth information. By using a large language model (LLM) with voice-to-text commands and temporally aligned selected bounding boxes, robot action sequences can be generated, while key control syntax constraints are applied to avoid potential LLM hallucination issues. The system is evaluated on real-world tasks with varying levels of complexity using a Universal Robots UR3e manipulator. Our method demonstrates significantly better performance in HRI in terms of accuracy and robustness. To benefit the research community and the general public, we will make our code and design open-source.

Figures

Figures reproduced from arXiv: 2501.00785 by the authors.

Figure 1
Figure 1. Proposed voice-posture fusion HRI method has superior efficiency and requires no memorization of key syntax, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. System Overview. V represents voice command, B represents human posture, M is mapping verbal features to action intention IA, P is mapping human posture and environment observation S into object intention IO. GPT4 decodes the multimodal commands and generates the action sequences A. Finally, the state of end-effector q is changed by the control APIs. own action intention IA and object intention IO. Our system contai… view at source ↗
Figure 4
Figure 4. Collision-free trajectory generation. D. Action Sequences Generation and Execution The human intention I is encoded into a robot action sequences A through the mapping A. Task planning and action sequence generation often require extensive domain knowledge about the state of robots and constraints. To streamline this process, we employ GPT4 to encode this high-level policy. We establish constraints on the output res… view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: Alternative ways of finding object reference. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 5
Figure 5. Figure 5: The prompt is segmented into three sections: basic API constraints, action definition, and example tasks. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: List of Gestures used in gesture-based HRI system [ [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Accuracy evaluation for various tasks across different [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: System robustness evaluation under varying real-world constraints. Lighting affects all baseline methods equally. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Experiments on adverse tasks and evaluations with diverse elderly participants and environments. to update the system via verbal identification and online adaptation. Future work will explore this approach [45]. VIII. CONCLUSIONS In this work, we introduced a system t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 34 canonical work pages

  1. [44]

    Nvp-hri: Zero shot natural voice and posture-based human–robot interaction via large language model,

    Y . Lai, S. Yuan, Y . Nassar, M. Fan, T. Weber, and M. R¨atsch, “Nvp-hri: Zero shot natural voice and posture-based human–robot interaction via large language model,” Expert Systems with Applications , vol. 268, p. 126360, 2025

    2025

  2. [8]

    Intuitive multi-modal human-robot interaction via posture and voice,

    Y . Lai, M. Radke, Y . Nassar, A. Gopal, T. Weber, Z. Liu, Y . Zhang, and M. R¨atsch, “Intuitive multi-modal human-robot interaction via posture and voice,” in International Conference on Robotics, Computer Vision and Intelligent Systems . Springer, 2024, pp. 441–456

    work page 2024

  3. [34]

    Hand and arm gesture- based human-robot interaction: A review,

    X. Wang, H. Shen, H. Yu, J. Guo, and X. Wei, “Hand and arm gesture- based human-robot interaction: A review,” in Proceedings of the 6th International Conference on Algorithms, Computing and Systems , 2022, pp. 1–7

    work page 2022

  4. [6]

    Communicating human intent to a robotic companion by multi-type gesture sentences,

    P. Vanc, J. K. Behrens, K. Stepanova, and V . Hlavac, “Communicating human intent to a robotic companion by multi-type gesture sentences,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023, pp. 9839–9845

    work page 2023

  5. [17]

    Language-conditioned imitation learning for robot ma- nipulation tasks,

    S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor, “Language-conditioned imitation learning for robot ma- nipulation tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 13 139–13 150, 2020

    2020

  6. [26]

    Interactive multimodal robot dialog using pointing gesture recogni- tion,

    S. Constantin, F. I. Eyiokur, D. Yaman, L. B ¨armann, and A. Waibel, “Interactive multimodal robot dialog using pointing gesture recogni- tion,” in Computer Vision - ECCV 2022 Workshops - Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI , ser. Lecture Notes in Computer Science, vol. 13806. Springer, 2022, pp. 640–657

    work page 2022

  7. [1]

    Air- embodied: An efficient active 3dgs-based interaction and reconstruc- tion framework with embodied large language model,

    Z. Qi, S. Yuan, F. Liu, H. Cao, T. Deng, J. Yang, and L. Xie, “Air- embodied: An efficient active 3dgs-based interaction and reconstruc- tion framework with embodied large language model,” arXiv preprint arXiv:2409.16019, 2024

    arXiv 2024

  8. [2]

    Handle object navi- gation as weighted traveling repairman problem,

    R. Liu, X. Xu, S. Yuan, and L. Xie, “Handle object navi- gation as weighted traveling repairman problem,” arXiv preprint arXiv:2503.06937, 2025

    arXiv 2025

Show all 45 references
  1. [3]

    Medical robots for infectious diseases: Lessons and challenges from the covid-19 pandemic,

    A. Di Lallo, R. Murphy, A. Krieger, J. Zhu, R. H. Taylor, and H. Su, “Medical robots for infectious diseases: Lessons and challenges from the covid-19 pandemic,” IEEE Robotics & Automation Magazine , vol. 28, no. 1, pp. 18–27, 2021

    2021

  2. [4]

    Jacquard v2: Refining datasets using the human in the loop data correction method,

    Q. Li and S. Yuan, “Jacquard v2: Refining datasets using the human in the loop data correction method,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) , 2024

    2024

  3. [5]

    Distilling location proposals of unknown objects through gaze information for human- robot interaction,

    D. Weber, T. Santini, A. Zell, and E. Kasneci, “Distilling location proposals of unknown objects through gaze information for human- robot interaction,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 11 086– 11 093

    2020

  4. [7]

    Analysis of the accuracy and robustness of the leap motion controller,

    F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler, “Analysis of the accuracy and robustness of the leap motion controller,” Sensors, vol. 13, no. 5, pp. 6380–6393, 2013

    2013

  5. [9]

    Large language models for human–robot interaction: A review,

    C. Zhang, J. Chen, J. Li, Y . Peng, and Z. Mao, “Large language models for human–robot interaction: A review,” Biomimetic Intelligence and Robotics, vol. 3, no. 4, p. 100131, 2023

    2023

  6. [10]

    Chat with the environment: Interactive multimodal perception using large language models,

    X. Zhao, M. Li, C. Weber, M. B. Hafez, and S. Wermter, “Chat with the environment: Interactive multimodal perception using large language models,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2023, pp. 3590–3596

    2023

  7. [11]

    Generating executable action plans with environmentally-aware language models,

    M. Gramopadhye and D. Szafir, “Generating executable action plans with environmentally-aware language models,” in 2023 IEEE/RSJ 10 International Conference on Intelligent Robots and Systems (IROS) , 2023, pp. 3568–3575

    2023

  8. [12]

    A fast and light-weight noniterative visual odometry with rgb-d cameras,

    Z. Yang, K. Xu, S. Yuan, and L. Xie, “A fast and light-weight noniterative visual odometry with rgb-d cameras,” Unmanned Systems, pp. 1–13, 2024

    2024

  9. [13]

    Sgba: Semantic gaussian mixture model-based lidar bundle adjustment,

    X. Ji, S. Yuan, J. Li, P. Yin, H. Cao, and L. Xie, “Sgba: Semantic gaussian mixture model-based lidar bundle adjustment,”IEEE Robotics and Automation Letters , vol. 9, no. 12, pp. 10 922–10 929, 2024

    2024

  10. [14]

    Survey on localization systems and algorithms for unmanned systems,

    S. Yuan, H. Wang, and L. Xie, “Survey on localization systems and algorithms for unmanned systems,” Unmanned Systems, vol. 9, no. 02, pp. 129–163, 2021

    2021

  11. [15]

    From macro to micro: Autonomous multiscale image fusion for robotic surgery,

    L. Zhang, M. Ye, P. Giataganas, M. Hughes, A. Bradu, A. Podoleanu, and G.-Z. Yang, “From macro to micro: Autonomous multiscale image fusion for robotic surgery,” IEEE Robotics & Automation Magazine , vol. 24, no. 2, pp. 63–72, 2017

    2017

  12. [16]

    A socially assistive robotic platform for upper-limb rehabilitation: A longitudinal study with pediatric patients,

    J. C. Pulido, C. Suarez-Mejias, J. C. Gonzalez, A. Duenas Ruiz, P. Ferrand Ferri, M. E. Martinez Sahuquillo, C. E. Ruiz De Vargas, P. Infante-Cossio, C. L. Parra Calderon, and F. Fernandez, “A socially assistive robotic platform for upper-limb rehabilitation: A longitudinal st...

    2019

  13. [18]

    Towards real-time physical human-robot interaction using skeleton information and hand gestures,

    O. Mazhar, S. Ramdani, B. Navarro, R. Passama, and A. Cherubini, “Towards real-time physical human-robot interaction using skeleton information and hand gestures,” in 2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS) , 2018, pp. 1–6

    2018

  14. [19]

    Control system shell of mobile robot with voice recognition module,

    A. Skrzypek, W. Panfil, M. Kosior, P. Przysta, et al., “Control system shell of mobile robot with voice recognition module,” in 2019 12th International Workshop on Robot Motion and Control (RoMoCo) . IEEE, 2019, pp. 191–196

    2019

  15. [20]

    Armar-6: A high- performance humanoid for human-robot collaboration in real-world scenarios,

    T. Asfour, M. Waechter, L. Kaul, S. Rader, P. Weiner, S. Ottenhaus, R. Grimm, Y . Zhou, M. Grotz, and F. Paus, “Armar-6: A high- performance humanoid for human-robot collaboration in real-world scenarios,” IEEE Robotics & Automation Magazine , vol. 26, no. 4, pp. 108–121, 2019

    2019

  16. [21]

    Unsupervised scene categorization, path segmentation and landmark extraction while traveling path,

    M. A. Esfahani, H. Wang, K. Wu, and S. Yuan, “Unsupervised scene categorization, path segmentation and landmark extraction while traveling path,” in 2020 16th International Conference on Control, Automation, Robotics and Vision (ICARCV) . IEEE, 2020, pp. 190– 195

    2020

  17. [22]

    Working with walt: How a cobot was developed and inserted on an auto assembly line,

    I. El Makrini, S. A. Elprama, J. Van den Bergh, B. Vanderborght, A.-J. Knevels, C. I. Jewell, F. Stals, G. De Coppel, I. Ravyse, J. Potargent, J. Berte, B. Diericx, T. Waegeman, and A. Jacobs, “Working with walt: How a cobot was developed and inserted on an auto assembly line,...

    2018

  18. [23]

    A tool for organizing key characteristics of virtual, augmented, and mixed reality for human–robot interaction systems: Synthesizing vam- hri trends and takeaways,

    T. R. Groechel, M. E. Walker, C. T. Chang, E. Rosen, and J. Z. Forde, “A tool for organizing key characteristics of virtual, augmented, and mixed reality for human–robot interaction systems: Synthesizing vam- hri trends and takeaways,” IEEE Robotics & Automation Magazine , vol...

    2022

  19. [24]

    An extensible architecture for robust multimodal human-robot communication,

    S. Rossi, E. Leone, M. Fiore, A. Finzi, and F. Cutugno, “An extensible architecture for robust multimodal human-robot communication,” in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2013, pp. 2208–2213

    2013

  20. [25]

    Intelligent robotic wheelchair with emg-, gesture-, and voice-based interfaces,

    I. Moon, M. Lee, J. Ryu, and M. Mun, “Intelligent robotic wheelchair with emg-, gesture-, and voice-based interfaces,” in 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , vol. 4, 2003, pp. 3453–3458

    2003

  21. [27]

    Adaptive-lio: Enhancing robustness and precision through environmental adaptation in lidar inertial odometry,

    C. Zhao, K. Hu, J. Xu, L. Zhao, B. Han, K. Wu, M. Tian, and S. Yuan, “Adaptive-lio: Enhancing robustness and precision through environmental adaptation in lidar inertial odometry,” IEEE Internet of Things Journal, 2024

    2024

  22. [28]

    Robust loop closure by textual cues in challenging environments,

    T. Jin, T.-M. Nguyen, X. Xu, Y . Yang, S. Yuan, J. Li, and L. Xie, “Robust loop closure by textual cues in challenging environments,” IEEE Robotics and Automation Letters , vol. 10, no. 1, pp. 812–819, 2025

    2025

  23. [29]

    Chatgpt for robotics: Design principles and model abilities,

    S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “Chatgpt for robotics: Design principles and model abilities,” Microsoft Auton. Syst. Robot. Res, vol. 2, p. 20, 2023

    2023

  24. [30]

    Ua-mpc: Uncertainty-aware model predictive control for motorized lidar odom- etry,

    J. Li, X. Xu, J. Liu, K. Cao, S. Yuan, and L. Xie, “Ua-mpc: Uncertainty-aware model predictive control for motorized lidar odom- etry,” IEEE Robotics and Automation Letters , 2025

    2025

  25. [31]

    Unsupervised uav 3d trajectories estimation with sparse point clouds,

    H. Liang, Y . Yang, J. Hu, J. Yang, F. Liu, and S. Yuan, “Unsupervised uav 3d trajectories estimation with sparse point clouds,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2025, pp. 1–5

    2025

  26. [32]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell,et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  27. [33]

    Evaluation of the efficiency of state-of-the-art speech recognition engines,

    A. Trabelsi, S. Warichet, Y . Aajaoun, and S. Soussilane, “Evaluation of the efficiency of state-of-the-art speech recognition engines,” Procedia Computer Science, vol. 207, pp. 2242–2252, 2022

    2022

  28. [35]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779– 788

    2016

  29. [36]

    Yolo-world: Real-time open-vocabulary object detection,

    T. Cheng, L. Song, Y . Ge, W. Liu, X. Wang, and Y . Shan, “Yolo-world: Real-time open-vocabulary object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 16 901–16 911

    2024

  30. [37]

    Realtime multi-person 2d pose estimation using part affinity fields,

    Z. Cao, T. Simon, S.-E. Wei, and Y . Sheikh, “Realtime multi-person 2d pose estimation using part affinity fields,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 7291–7299

    2017

  31. [38]

    Autonomous object level segmentation,

    S. Yuan and H. Wang, “Autonomous object level segmentation,” in 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV) . IEEE, 2014, pp. 33–37

    2014

  32. [39]

    Airslam: An efficient and illumination-robust point-line visual slam system,

    K. Xu, Y . Hao, S. Yuan, C. Wang, and L. Xie, “Airslam: An efficient and illumination-robust point-line visual slam system,” IEEE Transac- tions on Robotics , 2025

    2025

  33. [40]

    Relative localiz- ability and localization for multi-robot systems,

    L. Chen, C. Liang, S. Yuan, M. Cao, and L. Xie, “Relative localiz- ability and localization for multi-robot systems,” IEEE Transactions on Robotics, 2025

    2025

  34. [41]

    Large-scale uwb anchor calibration and one- shot localization using gaussian process,

    S. Yuan, B. Lou, T.-M. Nguyen, P. Yin, M. Cao, X. Xu, J. Li, J. Xu, S. Chen, and L. Xie, “Large-scale uwb anchor calibration and one- shot localization using gaussian process,” in Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA) , Atlanta...

    2025

  35. [42]

    Helmetposer: A helmet-mounted imu dataset for data-driven estimation of human head motion in diverse conditions,

    J. Li, Q. Leng, J. Liu, X. Xu, T. Jin, M. Cao, T.-M. Nguyen, S. Yuan, K. Cao, and L. Xie, “Helmetposer: A helmet-mounted imu dataset for data-driven estimation of human head motion in diverse conditions,” in Proceedings of the IEEE International Conference on Robotics and Auto...

    2025

  36. [43]

    Heterogeneous stereo: A human vision inspired method for general robotics sensing,

    H. Wang, S. Yuan, and K. Wu, “Heterogeneous stereo: A human vision inspired method for general robotics sensing,” in TENCON 2017-2017 IEEE region 10 conference . IEEE, 2017, pp. 793–798

    2017

  37. [45]

    Fam- hri: Foundation-model assisted multi-modal human-robot interaction combining gaze and speech,

    Y . Lai, S. Yuan, B. Zhang, B. Kiefer, P. Li, and A. Zell, “Fam- hri: Foundation-model assisted multi-modal human-robot interaction combining gaze and speech,” arXiv preprint arXiv:2503.16492 , 2025

    2025 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.