REVIEW 4 major objections 5 minor 1 cited by
Evaluating Efficiency and Engagement in Scripted and LLM-Enhanced Human-Robot Interactions
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read For a simple pick-and-place interaction, a fully scripted robot performs comparably to an LLM-enhanced version on objective efficiency and focus, with lower latency and energy.
desk verdict A modest, useful HRI comparison whose efficiency and energy conclusions survive contact with the data, but the 'focus' claim is weaker than the abstract suggests and needs either better evidence or softer wording. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central comparison is between two interaction controllers: a pre-programmed schedule (PPS), which uses fixed timing and thresholds (for example, starting the next instruction when the participant's walking speed drops below 0.3 m/s), and an LLM-enhanced controller that feeds gaze fixations and detected objects into a large language model via an API with chain-of-thought prompting to generate robot speech, pointing, and gaze commands in real time. The metrics that carry the argument are mobile eye-tracking (fixation durations, saccade velocity and amplitude, pupil diameter), two standard questionnaires for trust and robot perception, and power measurements across local computation, network nodes, and API calls, with training and inference energy estimated from published data.
What would settle it
A pre-registered replication with a larger sample (for example, N at least 40) and a pre-defined equivalence margin on task completion time and error rate would settle the comparability claim: if the scripted condition turns out to be significantly slower or more error-prone than the LLM condition beyond a small margin, the central conclusion fails. Separately, direct per-query power metering of the actual deployed model, rather than using GPT-3 as a proxy, could invalidate the energy edge if it shows the proxy overstates the LLM's cost.
Extended reading notes
Core claim
The paper's central claim is that augmenting a scripted human-robot interaction with an LLM backbone does not automatically improve interaction metrics. Comparing a pre-programmed schedule (PPS) with an LLM-enhanced controller in the same task, the authors found that while the LLM condition produced slightly higher subjective ratings on trust and perceived intelligence, objective measures from gaze tracking and time allocation showed comparable or better efficiency in the scripted condition during simple task execution. The LLM condition also carried higher latency (about 2.5 s per API response) and higher estimated energy use per interaction (538–580 Wh versus 506 Wh). The authors therefore argue for aligning the interaction modality with task demands: scripted control for predictable, efficiency-driven tasks and LLM adaptation for complex, dynamic scenarios.
Load-bearing premise
The load-bearing premise is that non-significant differences between conditions at N=15 can be read as evidence that the scripted condition performs comparably to the LLM condition; the study does not run an equivalence test or a power analysis to support that reading.
Editorial extensions
If this is right
- For simple, repetitive industrial interactions, scripted control offers comparable objective efficiency with lower latency and energy consumption, making it the cheaper default.
- LLM-enhanced interaction can raise subjective engagement and perceived human-likeness without producing measurable objective gains on simple tasks.
- The roughly 2.5 s API response latency of the LLM condition places a practical limit on real-time interaction, which matters for time-critical robot operations.
- Energy accounting that includes model training and inference shifts the cost-benefit balance against LLM deployment in resource-constrained settings.
- Interaction design should be task-dependent: dynamic adaptation is valuable for complex scenarios, while predictability and focus are better for simple ones.
Reading between the lines
- If the comparability claim holds, a hybrid controller that runs scripted by default and escalates to LLM reasoning only when an anomaly or unexpected state is detected could capture most of the efficiency of scripts with adaptability where it is actually needed.
- The gaze differences could reflect not only engagement but also comprehension cost: longer fixations in the LLM condition might indicate that users worked harder to parse variable, generated instructions; a follow-up measuring task errors would help separate these interpretations.
- Because the energy comparison substitutes GPT-3 training and inference figures for the newer model actually used, direct per-query power metering of GPT-4o-mini could shift the LLM condition's estimated energy cost in either direction.
- The study's task was deliberately simple; extending the same measurement protocol to a more complex, multi-step task with unpredictable states could reveal objective benefits of LLM adaptation that the current design cannot detect.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a within-subjects user study (N=15) comparing a fully scripted ('pre-programmed schedule', PPS) interaction with an LLM-enhanced interaction for a simulated industrial pick-and-place task involving a NAO robot mounted on a forklift. The authors collect gaze-tracking metrics, Trust and Godspeed questionnaires, task-time ratios, and energy-consumption estimates. The central claim is that while subjective ratings trend toward the LLM condition, objective metrics show the scripted condition performs comparably, 'particularly in efficiency and focus during simple tasks,' and that the scripted condition may be preferable on latency and energy grounds for simple repetitive interactions.
Significance. If the central claim is properly supported, the paper would provide useful practical guidance for HRI designers choosing between rule-based and LLM-driven control: use the cheaper, more predictable scripted approach for low-complexity tasks and reserve LLM enhancement for tasks that genuinely require adaptation. The study has strengths worth recognizing: a realistic within-subjects setup with counterbalancing, objective eye-tracking measures rather than only self-reports, and an explicit attempt to account for energy costs including training and inference. The manuscript also openly discloses several limitations, such as the GPT-3 proxy for GPT-4o-mini energy data. However, as detailed in my major comments, the focus conclusion is not currently supported by the quantitative gaze metrics, and the equivalence interpretation of non-significant subjective results needs additional statistical backing.
major comments (4)
- [Section IV and Table I] The conclusion that the PPS condition is 'more focused during task execution' is not supported by the paper's own quantitative gaze data. In Table I, Fixation Duration (Task Execution) is significantly longer in the LLM condition (79.08 s vs. 70.08 s, p<0.05), and Saccade Velocity and Saccade Amplitude during task execution are significantly lower in the LLM condition (p<0.05). Longer fixations and lower saccade velocity/amplitude are conventional indicators of sustained, non-exploratory attention, i.e., the opposite of the stated conclusion. The heatmap inspection described in Section III is qualitative and appears to conflict with these metrics. Please either provide a quantitative spatial-dispersion measure (e.g., fixation-map entropy, convex hull area, or k-density spread) that supports the PPS focus claim, or revise the abstract and discussion to align with the gaze metrics actually reported.
- [Section III, Trust and Godspeed results] The manuscript treats non-significant differences in the Trust scale (F=1.05, p=0.32) and Godspeed subscales (p>0.05) as evidence that the PPS condition is 'a viable, simpler option' and that subjective ratings are 'marginally higher' for LLM. With N=15, a non-significant p-value does not establish equivalence or comparability; it may simply reflect low power. To support the comparability claim, the authors should either (a) report an equivalence test such as TOST with a pre-specified margin, (b) provide a post-hoc power analysis showing that the study could detect a meaningful difference in these scales, or (c) soften the language to 'no statistically significant differences were found' and avoid drawing a positive comparability conclusion from absence of significance. As written, the central 'performs comparably' conclusion partly rests on this absence-of-evidence interpretation.
- [Table I and Section III, multiple gaze comparisons] Table I reports p-values for ten pairwise gaze comparisons without any correction for multiple testing. With eight metrics and two phases, the probability of at least one false positive is substantial. The authors should apply a multiple-comparison correction (e.g., Benjamini-Hochberg within each family of gaze metrics) and re-verify which asterisked entries remain significant. This matters because the significant saccade and fixation differences are currently used to infer engagement differences, and their status could change under correction.
- [Section III, Energy consumption] The energy comparison relies on GPT-3 training and inference figures from reference [11] as a proxy for GPT-4o-mini, which the authors acknowledge. However, the per-query calculation is fragile: it divides the total training energy (1,287,000 kWh) by an estimated annual query count (71.2B) and adds a per-query inference cost derived from a daily energy figure. This treats training cost as uniformly amortized across all queries and ignores differences in model size, inference hardware, and query complexity between GPT-3 and GPT-4o-mini. Given that the title and abstract mention energy as a reason to prefer the scripted condition, the estimate should at least be presented with a sensitivity analysis (e.g., varying query counts by an order of magnitude) or explicitly labeled as an illustrative back-of-the-envelope calculation rather than a measured comparison. As written, the claim that PPS 'may have an edge' in energy is appropriately hedged, but the numerical contrast (506 Wh vs. 538–580 Wh) is not robust enough to support the emphasis it receives.
minor comments (5)
- [Section II.A] The notation 'GPT 4o-mini' appears without a consistent hyphen or version formatting; please unify to 'GPT-4o-mini' throughout.
- [Section II.C] The Trust scale is analyzed with a one-way ANOVA while the Godspeed subscales are analyzed with Mann-Whitney U tests because of non-normality. Please justify the parametric test for the Trust data, or report a normality check for that variable as well.
- [Section III] The sentence 'The lower task time ratio in the PPS condition indicates less engagement with task elements' appears to contradict the later claim that the PPS condition is 'more focused during task execution.' Please clarify what is meant by 'engagement' versus 'focus' and ensure the two statements are logically consistent.
- [Section IV] The phrase 'Llama 40B' is not a standard model name; the authors likely mean 'Llama 3 70B' or 'Llama 2 70B'. Please correct the model reference.
- [Section IV] The discussion mentions 'output variability of the LLM requires prompt engineering' but does not report how many prompt iterations or what failure cases occurred. Adding a brief description of the prompt engineering process and any rejected LLM responses would strengthen the reproducibility of the LLM condition.
Circularity Check
No significant circularity; the comparison is empirical and the self-citations are to system components, not to load-bearing results.
full rationale
The paper performs an empirical within-subjects comparison between a pre-programmed schedule (PPS) and an LLM-enhanced interaction condition. No mathematical derivation is present, and no fitted parameter is renamed as a prediction. The self-citations ([5] for the PPS design, [13] for the custom LLM framework, [16] for an adapted trust scale) identify the system components being compared or minor measurement adaptations; they are not invoked as evidence for the paper's conclusions. The energy estimate explicitly uses GPT-3 data as a proxy for GPT-4o-mini and states this limitation, which is a weakness in external validity rather than circularity. The abstract's claim that the scripted condition 'performs comparably' rests on interpreting non-significant questionnaire results and gaze metrics; this is a statistical-inference concern (absence of evidence vs. evidence of equivalence), not a reduction of the conclusion to its inputs by construction. No step in the paper's argument is self-definitional, fitted-then-predicted, or dependent on a uniqueness theorem imported from the authors' prior work. The verdict is therefore no significant circularity, score 0.
Assumptions & free parameters
free parameters (5)
- Stationary speed threshold =
0.3 m/s
- Fixation duration threshold =
30 ms
- LLM trigger fixation time =
3 s
- Inactivity prompt threshold =
7 s
- PPS reaction delays =
1.5 s and 2.5 s
assumptions (3)
- domain assumption Tobii I-VT gaze filter with a 100 degrees per second classification threshold correctly identifies fixations, saccades and pupil diameter as measures of attention and cognitive load.
- domain assumption The LLM condition and PPS condition differ only in the control mechanism, so observed differences can be attributed to scripted versus LLM-enhanced response generation.
- domain assumption OpenAI GPT-4o-mini responses via Chat Completions with chain-of-thought prompting are stable enough across participants to represent the LLM condition.
Cite this review
Pith. "Pith review of Evaluating Efficiency and Engagement in Scripted and LLM-Enhanced Human-Robot Interactions." pith.science (2026). https://pith.science/paper/JZT4MSKZ
@misc{pith2026250112128,
author = {Pith},
title = {Pith review of: Evaluating Efficiency and Engagement in Scripted and LLM-Enhanced Human-Robot Interactions},
year = {2026},
howpublished = {\url{https://pith.science/paper/JZT4MSKZ}},
note = {Machine review of arXiv:2501.12128}
}
read the original abstract
To achieve natural and intuitive interaction with people, HRI frameworks combine a wide array of methods for human perception, intention communication, human-aware navigation and collaborative action. In practice, when encountering unpredictable behavior of people or unexpected states of the environment, these frameworks may lack the ability to dynamically recognize such states, adapt and recover to resume the interaction. Large Language Models (LLMs), owing to their advanced reasoning capabilities and context retention, present a promising solution for enhancing robot adaptability. This potential, however, may not directly translate to improved interaction metrics. This paper considers a representative interaction with an industrial robot involving approach, instruction, and object manipulation, implemented in two conditions: (1) fully scripted and (2) including LLM-enhanced responses. We use gaze tracking and questionnaires to measure the participants' task efficiency, engagement, and robot perception. The results indicate higher subjective ratings for the LLM condition, but objective metrics show that the scripted condition performs comparably, particularly in efficiency and focus during simple tasks. We also note that the scripted condition may have an edge over LLM-enhanced responses in terms of response latency and energy consumption, especially for trivial and repetitive interactions.
Figures
Forward citations
Cited by 1 Pith paper
-
Gaze-supported Large Language Model Framework for Bi-directional Human-Robot Interaction
A gaze- and speech-driven LLM framework for assistive robots matches a scripted interaction pipeline on task performance while slightly increasing user-perceived confidence, at higher energy cost.
Reference graph
Works this paper leans on
-
[11]
The growing energy footprint of artificial intelligence,
A. de Vries, “The growing energy footprint of artificial intelligence,” Joule, vol. 7, no. 10, pp. 2191–2194, 2023
2023
-
[1]
Human–robot interaction: status and challenges,
T. B. Sheridan, “Human–robot interaction: status and challenges,” Hu- man factors, vol. 58, no. 4, pp. 525–532, 2016
work page 2016
-
[2]
Robotic vision for human-robot interaction and collaboration: A survey and systematic review,
N. Robinson, B. Tidd, D. Campbell, D. Kuli ´c, and P. Corke, “Robotic vision for human-robot interaction and collaboration: A survey and systematic review,” ACM Transactions on Human-Robot Interaction , vol. 12, no. 1, pp. 1–66, 2023
2023
-
[3]
Human–robot interaction review and challenges on task planning and programming,
P. Tsarouchi, S. Makris, and G. Chryssolouris, “Human–robot interaction review and challenges on task planning and programming,”International Journal of Computer Integrated Manufacturing, vol. 29, no. 8, pp. 916– 931, 2016
work page 2016
-
[4]
NAO robot, transmitter of social cues: what impacts? The example with “endowment effect
O. Masson, J. Baratgin, and F. Jamet, “NAO robot, transmitter of social cues: what impacts? The example with “endowment effect”,” in Advances in Artificial Intelligence: From Theory to Practice: 30th Inter- national Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2017, Arras, France, June 27-30, 2017, Pr...
work page 2017
-
[5]
T. Schreiter, L. Morillo-Mendez, R. T. Chadalavada, A. Rudenko, E. Billing, M. Magnusson, K. O. Arras, and A. J. Lilienthal, “Advantages of Multimodal versus Verbal-Only Robot-to-Human Communication with an Anthropomorphic Robotic Mock Driver,” in 2023 32nd IEEE International Conference on Robot and Human Interactive Communica- tion (RO-MAN). IEEE, 2023, ...
work page 2023
-
[6]
10 Years of Human-NAO Interaction Research: A Scoping Review,
A. Amirova, N. Rakhymbayeva, E. Yadollahi, A. Sandygulova, and W. Johal, “10 Years of Human-NAO Interaction Research: A Scoping Review,” Frontiers in Robotics and AI , vol. 8, 2021
work page 2021
-
[7]
Wizard of oz studies in hri: a systematic review and new reporting guidelines,
L. D. Riek, “Wizard of oz studies in hri: a systematic review and new reporting guidelines,” Journal of Human-Robot Interaction, vol. 1, no. 1, pp. 119–136, 2012
work page 2012
Show all 18 references
-
[8]
The Child Factor in Child–Robot Interaction: Discovering the Impact of Developmental Stage and Individual Characteristics,
I. Rudenko, A. Rudenko, A. J. Lilienthal, K. O. Arras, and B. Bruno, “The Child Factor in Child–Robot Interaction: Discovering the Impact of Developmental Stage and Individual Characteristics,” International Journal of Social Robotics , vol. 16, no. 8, pp. 1879–1900, 2024
1900
-
[9]
Applying Large Language Models for Intelligent Industrial Automation: From Theory to Application: Towards Autonomous Systems with Large Language Models,
Y . Xia, N. Jazdi, and M. Weyrich, “Applying Large Language Models for Intelligent Industrial Automation: From Theory to Application: Towards Autonomous Systems with Large Language Models,” atp magazin , vol. 66, no. 6-7, pp. 62–71, 2024
2024
-
[10]
Understanding Large-Language Model (LLM)-powered Human-Robot Interaction,
C. Y . Kim, C. P. Lee, and B. Mutlu, “Understanding Large-Language Model (LLM)-powered Human-Robot Interaction,” ser. HRI ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 371–380. [Online]. Available: https://doi.org/10.1145/3610977.3634966
2024
-
[12]
Seamless human–robot collaboration in industrial applications,
S. Makris, G. Michalos, N. Dimitropoulos, J. Krueger, and K. Haninger, “Seamless human–robot collaboration in industrial applications,” in CIRP Novel Topics in Production Engineering: Volume 1 . Springer, 2024, pp. 39–73
2024
-
[13]
Bidirectional intent communication: A role for large foundation models,
T. Schreiter, R. Hazra, J. R ¨uppel, and A. Rudenko, “Bidirectional intent communication: A role for large foundation models,” 2024 33rd IEEE International Conference on Robot and Human Interactive Communica- tion (RO-MAN), arXiv preprint arXiv:2408.10589 , 2024
2024 arXiv
-
[14]
Chain-of-thought prompting elicits reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022
2022
-
[15]
The development of a scale to evaluate trust in industrial human-robot collaboration,
G. Charalambous, S. Fletcher, and P. Webb, “The development of a scale to evaluate trust in industrial human-robot collaboration,” International Journal of Social Robotics , vol. 8, no. 2, pp. 193–209, 2016
2016
-
[16]
The Effect of Anthropomorphism on Trust in an Industrial Human-Robot Interaction,
T. Schreiter, L. Morillo-Mendez, R. T. Chadalavada, A. Rudenko, E. A. Billing, and A. J. Lilienthal, “The Effect of Anthropomorphism on Trust in an Industrial Human-Robot Interaction,”2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO- MAN...
2022 arXiv
-
[17]
Measurement in- struments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots,
C. Bartneck, D. Kuli ´c, E. Croft, and S. Zoghbi, “Measurement in- struments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots,” International journal of social robotics, vol. 1, no. 1, pp. 71–81, 2009
2009
-
[18]
Pupil diameter and load on memory,
D. Kahneman and J. Beatty, “Pupil diameter and load on memory,” Science, vol. 154, pp. 1583 – 1585, 1966. [Online]. Available: https://api.semanticscholar.org/CorpusID:22762466
1966
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.