{"id":"6f041f09-59f2-4030-97be-8287f6060655","arxiv_id":"2412.06808","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A human-robot teaming framework with adjustable LLM feedback improves collaboration in easy and medium tasks, but overly frequent feedback from a less capable LLM hurts performance in hard tasks.","lead":"This paper builds a framework that lets a robot vary how often it talks to a human teammate, and tests four communication styles in a cooperative cooking game with 16 people. It finds that more talk helps in harder tasks up to a point, but if the AI's advice is unreliable, too much talk hurts the team.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feedback-mode conditions differ in robot planning as well as message frequency; the central causal claim may conflate communication with robot behavior.","rationale":"The central claim is causal: it states that the frequency and content of language feedback drive the observed preference and performance patterns. For that claim to hold, the four agent conditions must differ only in the messages delivered to the human, with robot planning and execution held constant. The paper asserts this in Section IV-E, but the condition descriptions show otherwise. In IFA the Coordinator is essentially absent; in AFA the Coordinator periodically generates coordination suggestions that, when accepted, modify the subtask graph and therefore the Manager's subtask allocation and the Greedy Planner's targets. This means the robot's physical behavior and the division of labor are part of the manipulation. The observed performance differences (Fig. 5) and preference shifts (Fig. 6) could therefore reflect the robot being a more helpful or more intrusive actor, rather than the human receiving more or fewer messages. The abstract's phrase 'noisy and inaccurate feedback from superactive robotic agents' presumes the feedback is the active ingredient, but the paper does not isolate it. This is the weakest load-bearing assumption and it is not tested. The lack of inferential statistics compounds the issue, but even with statistics the confound would remain. I therefore agree with the reader's conditional verdict: the framework is a useful contribution, but the stated design principle is not yet supported without addressing this attribution problem.","tokens_in":15513,"tokens_out":8612,"duration_ms":86221,"concrete_test":"Replay each logged trial's state sequence through the HRT-ML pipeline in IFA mode with all Coordinator-initiated suggestions and Manager instructions masked, using the same random seeds and initial graph, and compare the robot's resulting action sequence at each timestep to the originally recorded sequence. If the sequences differ, the feedback-mode manipulation changed the robot's planning or action policy, confirming the confound and invalidating the causal attribution to language feedback; if they are identical, the manipulation affected only the information available to the human, supporting the authors' interpretation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-E defines the four agents by whether the Coordinator and Manager produce language feedback, but it also changes whether the Coordinator can proactively modify the subtask graph. In IFA, 'Only the Manager generates target subtasks for the greedy planner' and no coordination conversation occurs; in AFA, the GPT-4o Coordinator is periodically asked to 'suggest coordination strategies,' which can add intermediate nodes, revise edges, and alter target positions. Because the Manager allocates subtasks based on this graph (Section IV-C), the robot's physical action sequence and the division of labor with the human differ across conditions, not just the text shown on the interface. Thus differences in game score (Fig. 5) and preference (Fig. 6) could be caused by differences in robot planning and coordination behavior rather than by the frequency or content of language feedback. The abstract's claim that 'noisy and inaccurate feedback' hinders performance presupposes that only feedback varies; the paper does not demonstrate that. The stated framework sharing (Section IV-E) is therefore not sufficient to isolate the communication channel.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HRT-ML, a human-robot teaming framework with two LLM-based modules (Coordinator and Manager) that generate multi-modal language feedback at different frequencies and to different degrees of proactivity. The authors report a user study (n=16) in an Overcooked environment with three layout difficulties (easy, medium, hard) and four feedback modes (IFA, PFA, AFA, SFA), measuring game score, satisfaction, engagement, trust, perceived intelligence, and stated preference. The central empirical claim is that humans prefer more frequent, proactive support as task complexity increases relative to human capability, but that overly frequent feedback from an LLM whose capacity is exceeded can hurt team performance. The paper concludes with a set of qualitative conditions (Section VII) for when passive, active, or superactive feedback should be used, based on the relative magnitudes of task complexity, human capability, and LLM capability. The contribution is positioned as the first systematic study of LLM-based multimodal feedback frequency on human-robot teaming performance across task complexities.","tokens_in":15692,"tokens_out":2741,"duration_ms":28605,"significance":"If the central claim holds, the paper would provide a useful, actionable design principle for setting the frequency and proactivity of language feedback in LLM-powered human-robot teams, with clear relevance to HRI and human-AI collaboration. The study design is nontrivial: it introduces a flexible DAG-based coordination framework, a two-level (Coordinator/Manager) LLM architecture, and a user study with three difficulty levels and four feedback modes. The paper also ships unusually detailed prompt and survey material in the appendix, which aids reproducibility. The reported raw measurements (game scores, Likert ratings) are concrete and lend themselves to further analysis. However, the validity of the main causal claim is currently threatened by a confound between feedback mode and robot planning behavior, and by the absence of inferential statistics for the central performance comparisons. The paper's proposed adaptation principle in Section VII is plausible but is a post-hoc summary of the same experimental data rather than a validated rule.","major_comments":[{"comment":"The four feedback conditions do not differ only in language feedback; they also differ in the robot's planning and coordination behavior. In IV-E, IFA is described as having only the Manager generate target subtasks for the greedy planner, with no coordination conversation, while AFA lets the GPT-4o Coordinator periodically suggest coordination strategies that can revise the subtask graph, add intermediate nodes, and modify target positions. Since the Manager allocates subtasks based on this graph (Section IV-C), the robot's actual action sequence and the human-robot division of labor change across conditions. Therefore, the differences in game scores in Fig. 5 and preferences in Fig. 6 cannot be attributed solely to the frequency or content of language feedback. The abstract's claim that 'noisy and inaccurate feedback' hinders performance presupposes that only feedback varies, which the current design does not establish. This confound is load-bearing and needs to be addressed, either by equating planning behavior across conditions or by adding a condition that controls for coordination behavior with feedback removed.","section":"IV-E and VI-B"},{"comment":"The paper makes strong comparative claims without reporting inferential statistics. For example, in VI-B it states that on the easy layout, PFA scored 83.6 on average and was 'significantly higher than IFA, AFA, and SFA,' but no t-test, ANOVA, nonparametric test, or multiple-comparison correction is reported anywhere in the results. Similar comparative statements about SFA outperforming IFA and PFA on the hard layout are made without supporting statistics. With n=16 participants and 12 trials each, the variability visible in the box plots makes these 'significant' claims unverifiable. The authors should either add the appropriate statistical analyses or explicitly soften the causal and comparative language to descriptive terms.","section":"VI-B, Fig. 5"},{"comment":"The statement that 'This strong correlation between satisfaction and engagement indicates that engagement is the cause of different satisfaction levels reported by participants' overstates what a correlation can show. A correlation coefficient of 0.93 with p=0.00275 establishes association, not causation. Moreover, satisfaction and engagement were both self-reported on Likert scales in the same post-trial survey, so shared method variance could inflate the correlation. The authors should rephrase this as a suggestive association and, if they want to support the engagement-as-cause story, use a mediation or path analysis or collect engagement as a behavioral measure.","section":"VI-C, Fig. 7"},{"comment":"The adaptation principle in Section VII is derived entirely from the same experimental data used to reach the earlier conclusions; it is not a prediction validated on holdout data or a new experiment. For instance, the claim that when Ch < T and Cl < T a 'more passive (PFA) or relative infrequent (AFA) feedback style would result in better teaming performance' is a restatement of the hard-layout results in Fig. 5. The paper should explicitly label this section as a post-hoc interpretation rather than a tested design principle, and should state what independent evidence would be needed to validate it. Relatedly, the AFA feedback interval (~20 s) is a free parameter of the study; the sensitivity of the results to this interval is not examined.","section":"VII"},{"comment":"The limitations section states that robots were set to a constant active level and could not dynamically adjust their support during a task, yet the abstract and introduction frame the contribution as 'adaptive communication support' and 'dynamically adjust their levels and frequencies of communications.' The paper evaluates fixed feedback modes, not an adaptive policy. This gap between the framing and the actual experimental manipulation should be reconciled, either by renaming the contribution to 'selecting a feedback level' or by adding a real-time adaptation mechanism that is tested across changing task conditions.","section":"IX and abstract"}],"minor_comments":[{"comment":"There are several typos and grammatical slips, e.g., 'robot to adopt their roles' in the abstract, 'preceived intelligence' in Section VI-A, 'avaiable' in Appendix C, and 'onlyh be' in the Coordinator prompt. A thorough proofreading pass would improve readability.","section":"Abstract and Introduction"},{"comment":"The survey questions in Appendix B use scales from 1 to 7, but Fig. 4 shows y-axis values from 0 to 6 and labels starting at 'Very Untrustworthy' at the bottom. Please make the scale anchors and the plotted axis range consistent across the text, appendix, and figures.","section":"Appendix B and Fig. 4"},{"comment":"The sentence 'Team with PFA performed best, scoring 83.6 in average' is ambiguous because Fig. 5 shows a range of scores up to 100 and the average score for the easy layout is reported earlier as 53.6 for all agent types. Clarify whether 83.6 is the mean, the median, or the mean for a subset of trials, and report the corresponding standard deviation or interquartile range.","section":"VI-B"},{"comment":"The notation T, Ch, and Cl is introduced informally in Section VII without formal definitions or thresholds. Even as a qualitative framework, the text should specify what quantities these symbols refer to and how a practitioner would estimate them, otherwise the proposed conditions are not actionable.","section":"VII"},{"comment":"The related-work section would benefit from a brief discussion of prior studies on feedback frequency or communication cost in human-robot interaction (beyond directionality), since the paper's contribution is specifically about frequency and proactivity. The current narrative focuses mostly on unidirectional vs. bidirectional communication.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting problem, and the framework and data have value. My main reservation is that the central causal claim is not currently supported because the feedback-mode manipulation also changes robot planning behavior, and because the key performance comparisons lack inferential statistics. Both issues are fixable within the scope of a revision: the authors could add a control analysis or a condition that equalizes planning, and they could add the missing statistical tests and temper the language accordingly. I would not reject the paper on these grounds, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a real system-building contribution: HRT-ML with a Coordinator and Manager is a sensible decomposition for adjustable LLM feedback, and running four feedback modes across three task difficulties is a legitimate first pass at the problem. The headline finding—that people want more proactive support as tasks get harder, but that overly active feedback can hurt when the LLM is over its depth—is plausible and practically useful.\n\nThe soft spots, though, are substantial.\n\nFirst, the four feedback modes are not purely feedback modes. In IFA the Coordinator never revises the subtask graph; the Manager simply generates targets for the greedy planner. In AFA and SFA the Coordinator can add intermediate nodes, revise edges, and change target positions. That means the robot's physical behavior—planning, division of labor, even its movement paths—differs across conditions, not just the messages shown on a panel. So the performance differences in Figure 5 cannot be cleanly attributed to communication frequency or content. The abstract's claim about 'noisy and inaccurate feedback' presupposes that only feedback varies. That is not demonstrated, and the framework-sharing statement in Section IV-E doesn't fix it.\n\nSecond, the statistics are missing. The paper says PFA performed 'significantly higher' than others on the easy layout, but no test is reported. With n=16 and 12 trials per participant, you need at least a repeated-measures ANOVA or mixed model and post-hoc tests with multiple-comparison correction. Without them, the central empirical claim is anecdote plus boxplots.\n\nThird, there's a correlation-causation slip: Section VI-C reports a correlation of 0.93 between engagement and satisfaction and then concludes that engagement 'is the cause' of satisfaction. That's backwards (or at least unsupported) and should be rewritten.\n\nFourth, the adaptation principle in Section VII is post-hoc, a summary of the same data rather than an independent validation. That's fine if stated as a hypothesis, but it is presented as a result.\n\nThe paper acknowledges that the robots had fixed activity levels, which is honest. The framework itself could be useful to other groups, and the question matters. But the empirical support for the main claim is currently weaker than the abstract suggests. A serious referee should engage with it, because the system is real and the question is timely; the path to a publishable version is clear: isolate the communication channel (or at least measure and report the planning behavior), add proper statistics, and temper the causal claims.","headline":"Useful framework, but the feedback conditions are confounded with robot planning behavior, so the central claim is not yet supported.","tokens_in":16215,"tokens_out":3216,"would_cite":false,"duration_ms":30095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-powered robot teammates should adjust how often they speak to how hard the task is: users want frequent proactive help as tasks get harder, but when the task exceeds the LLM's ability, superactive feedback becomes noise and hurts team…","keywords":["human-robot collaboration","large language models","language feedback","adaptive communication","task complexity","user study","Overcooked","proactive assistance"],"falsifier":"Run the four agents on the same hard layout with a fifth condition in which the superactive agent's suggestions are filtered to be always correct and delivered at the same high rate; if team scores recover, the hard-layout penalty comes from inaccurate content, while if scores stay low, the penalty comes from communication volume itself. A complementary check is to keep content identical and vary only delivery rate.","tokens_in":15330,"feed_emoji":"🤖","tokens_out":6370,"duration_ms":58375,"temperature":0.7,"pith_summary":"This paper tries to establish a design principle for how often an LLM-powered robot should talk to a human teammate. Through a user study in an Overcooked-style cooking game, the authors compare four feedback modes — inactive, passive, active, and superactive — across easy, medium, and hard layouts. They report that as tasks become harder relative to the human's skill, people prefer and benefit from more frequent, proactive robot communication. But when the task is hard enough that the LLM's own suggestions become unreliable, high-frequency feedback backfires: humans spend extra effort interpreting and responding to noisy messages, and team scores drop. If the principle holds, robot helpers should tune their communication frequency dynamically based on task complexity, human capability, and the language model's capability.","feed_headline":"Proactive robot help wins on hard tasks, backfires on easy ones","feed_subtitle":"A 16-person study shows the best feedback frequency tracks task complexity and the LLM's limits.","key_machinery":"The load-bearing mechanism is HRT-ML, a two-module framework that separates strategic from tactical communication. A Coordinator uses a directed acyclic graph of subtasks to hold low-frequency, high-level coordination conversations and can revise the graph structure, node attributes, or add temporary subtasks based on human input. A Manager performs high-frequency, subtask-level allocation, generating short instructions and updating graph status when an Interact action completes a subtask. Four feedback modes — Inactive, Passive, Active, and Superactive — vary only in when and how often the Manager's and Coordinator's language output reaches the human, so observed differences in score, trust, satisfaction, and preference are attributed to communication frequency and content.","core_discovery":"The central claim is that the value of language feedback in human-robot teams follows task complexity and model capability, not \"more is better.\" The paper's data show trust and perceived intelligence rise monotonically as the robot becomes more active, yet team score does not: passive agents score highest on the easy layout, superactive agents score highest on the medium layout, and on the hard layout superactive feedback lowers performance below that of merely active feedback. The authors interpret the pattern through flow theory and cognitive load: support helps when it closes the gap between task complexity and human skill, but becomes harmful when the LLM's suggestions are inaccurate or so frequent that responding to them consumes the human's attention. They generalize this into a four-case adaptation rule comparing task complexity $T$, human capability $C_h$, and LLM capability $C_l$.","pith_inferences":["The paper's own limitation notes that agents held a constant feedback level; a natural extension the authors leave open is real-time estimation of human cognitive load and LLM confidence to switch modes mid-task.","The four discrete modes imply an inverted-U relationship between feedback frequency and team performance, with the peak shifting right as difficulty increases; a continuous frequency sweep would test this directly.","The flow-theory explanation suggests some humans deliberately want to retain mastery in hard tasks, so an HRI system might need to let users set or signal an engagement preference rather than always maximizing score.","Since preference and performance diverge, designers may need separate optimization targets — objective efficiency and subjective flow — for LLM-powered teammates."],"forward_implications":["Robot teammates should not use a fixed communication policy; the preferred feedback level shifts from passive in easy tasks to active or superactive as task complexity grows relative to human skill.","When the task exceeds the LLM's capability, high-frequency feedback can reduce team efficiency and engagement, so less frequent or more passive communication becomes the safer default.","Human preference and objective team score can diverge: participants often preferred passive agents even when active agents scored higher, so satisfaction and performance need to be measured separately.","Trust and perceived intelligence improve monotonically with feedback activity, meaning humans can find a robot more credible and capable even when its chatter does not help them perform better.","The paper's adaptation rule — compare task complexity with human and LLM capability — provides a starting point for robots to select among Inactive, Passive, Active, and Superactive modes."],"supporting_citations":[{"why":"Supplies the coordination-focused cooking game environment used for the user study and team score measurement.","marker":"[5]"},{"why":"Grounds the hypothesis that LLM-powered robots can proactively initiate communication because LLMs can match human performance in many domains.","marker":"[30]"},{"why":"Provides the teaming-fluency metric used to design and select the easy, medium, and hard layouts.","marker":"[17]"},{"why":"Motivates trust as a key outcome in human-robot collaboration and informs the trust ratings.","marker":"[8]"},{"why":"Supplies flow theory, the framework the paper uses to explain why humans prefer different feedback levels at different difficulties.","marker":"[11]"},{"why":"Supports the claim that frequent messages impose cognitive burden that slows responses and reduces efficiency.","marker":"[21]"},{"why":"Provides the baseline finding that bidirectional communication makes robots seem more intelligent and collaborative, which this study extends to frequency and content.","marker":"[2]"}],"fun_headline_variants":["Robot feedback should adapt to task complexity and LLM limits","More robot talk isn't always better: context matters","Adaptive LLM feedback: key to human-robot team success","When robot advice hurts: task complexity and LLM accuracy","Proactive robot support: beneficial unless feedback gets noisy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that differences in score and ratings across the four agents are caused by the frequency and content of language feedback, because the same planning and action framework is shared across conditions; if the agents also differ in how subtasks are allocated or in the robot's behavior beyond what is spoken, the attribution to \"feedback\" would collapse.","fun_headline_variants_meta":{"raw":{"variants":["Robot feedback should adapt to task complexity and LLM limits","More robot talk isn't always better: context matters","Adaptive LLM feedback: key to human-robot team success","When robot advice hurts: task complexity and LLM accuracy","Proactive robot support: beneficial unless feedback gets noisy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3854,"prompt_tokens":985,"completion_tokens":2869,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":2788}},"tokens_in":601,"tokens_out":2869,"duration_ms":19327,"temperature":1.0,"reasoning_tokens":2788,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:37:59.354841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the four agents on the same hard layout with a fifth condition in which the superactive agent's suggestions are filtered to be always correct and delivered at the same high rate; if team scores recover, the hard-layout penalty comes from inaccurate content, while if scores stay low, the penalty comes from communication volume itself. A complementary check is to keep content identical and vary only delivery rate.","supporting_citations":[{"cited_title":"Trust-aware decision making for human-robot collaboration: Model learning and plan- ning","cited_arxiv_id":null,"evidence_quote":"Motivates trust as a key outcome in human-robot collaboration and informs the trust ratings."},{"cited_title":"On the utility of learning about humans for human-ai coordina- tion","cited_arxiv_id":null,"evidence_quote":"Supplies the coordination-focused cooking game environment used for the user study and team score measurement."},{"cited_title":"Evaluating fluency in human–robot collaboration","cited_arxiv_id":null,"evidence_quote":"Provides the teaming-fluency metric used to design and select the easy, medium, and hard layouts."},{"cited_title":"Beyond boredom and anxiety","cited_arxiv_id":null,"evidence_quote":"Supplies flow theory, the framework the paper uses to explain why humans prefer different feedback levels at different difficulties."},{"cited_title":"Cognitive burden of survey questions and response times: A psycholinguistic experiment","cited_arxiv_id":null,"evidence_quote":"Supports the claim that frequent messages impose cognitive burden that slows responses and reduces efficiency."},{"cited_title":"Effects of communication directionality and AI agent differences in human-AI interaction","cited_arxiv_id":null,"evidence_quote":"Provides the baseline finding that bidirectional communication makes robots seem more intelligent and collaborative, which this study extends to frequency and content."}],"review_version":1}