{"id":"b0db5217-a22c-4dc2-b703-511de4526997","arxiv_id":"2411.12483","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"In a simulated emergency response game, human teammates mostly exchanged automated sensor reports, confirmations, and action updates, while explicit 'why' explanations were rare.","lead":"This paper labels 1,000 chat messages from teams playing a simulated emergency response game, mapping them onto categories from explainable AI research. It reports that the largest message categories are automated sensor reports and confirmations, while explicit 'why' questions are rare, and it claims communication volume tracks task accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central result that most explanation-related messages seek clarification is a consequence of the codebook: auto, doing, and features are all pre-assigned to select/clarify, and these three labels dominate the annotated messages.","rationale":"I read the paper as an exploratory study of human-human communication in a simulated emergency response task, aiming to infer which XAI capabilities robots need. That is a worthwhile question, and the TeamCollab environment and data collection are concrete. The iterative codebook and four annotators provide some process evidence, and raw agreement (61% with at least three of four agreeing) is better than chance, although no chance-corrected statistic is given. However, the central claim fails because the taxonomy mapping is doing the work. The codebook is not the result of the annotation; it is the input. By assigning auto, doing, and features to select/clarify, the authors guarantee that select/clarify will dominate unless those labels are rare, and Fig. 3 shows they are the most common. An independent reader looking at an auto message would not naturally call it an explanation-related clarification: it is a pushed sensor report. Similarly, 'I'll explore in the next room' is a status update, not a request for or provision of an explanation. The paper's own annotators noted that explicit XAI questions rarely appear; the analysis then stretches 'implicit explanation' to include almost all task communication, which makes the conclusion broad but not evidence-based. The performance correlations are also not statistically grounded, but even if they were, they would not rescue the qualitative finding. So the most load-bearing concern is the construct validity of the codebook, and a re-coding experiment can settle it. I agree with the reader's weakest-assumption diagnosis and recommend keeping the REJECT verdict; if the re-coding supports the original mapping, the paper could be resubmitted with significance tests and become a conditional accept.","tokens_in":5493,"tokens_out":4445,"duration_ms":43652,"concrete_test":"Independently recluster the 1,000 annotated messages: give four new annotators only the four XAI category definitions from [2] (select/clarify, mutate/simulate, dialogue/progress, dialogue/answer) and ask them to classify each message, without providing the Table I mapping. Then compute the majority-label distribution and Fleiss kappa. If auto/doing/features no longer fall predominantly into select/clarify, the central result is an artifact of the mapping. As a complementary check, recompute the majority distribution with auto excluded and with doing/features reclassified as non-explanation coordination; if 'most messages seek clarification' still holds, the result survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline empirical claim is not an independent discovery: Section III.C defines a CodeBook (Table I) that assigns the labels 'doing', 'features', and 'auto' to the category 'select/clarify'. Section IV.B then reports that these three labels are the most frequent in the majority-label distribution (Fig. 3). Therefore the statement 'most explanation-related messages seek clarification in the decisions or actions taken' follows directly from the coding decision, not from a free categorization of the observed dialogue. The construct validity of this mapping is also weak: 'auto' messages are one-click sensor broadcasts (e.g., 'Object 18 ... Status Danger') with no element of user selection or clarification; 'doing' messages are action updates and coordination; 'features' messages report object attributes. Labeling these as explanation-related 'select/clarify' conflates information transmission with asking-for or giving explanations. The secondary performance claim is also under-supported: the three Pearson correlations in Section IV.A are reported without p-values or confidence intervals for 20 sessions, so 'confirm that messages have an impact' overstates the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript analyses 2,607 chat messages from 20 TeamCollab sessions to identify which explanation-related interaction types humans use when collaborating in a simulated emergency-response task. A subset of 1,000 messages was coded by four annotators using a codebook derived from a two-level XAI taxonomy (select/clarify, mutate/simulate, dialogue/progress, dialogue/answer). The main claims are that most explanation-related messages seek clarification in decisions or actions (Section IV.B) and that the volume of communication correlates with team performance, specifically a negative correlation with total objects collected and a positive correlation with the percentage of collected objects that are dangerous (Section IV.A).","tokens_in":5786,"tokens_out":2538,"duration_ms":24546,"significance":"If the findings were valid, they would provide actionable guidance for robot explainability in collaborative settings, suggesting that clarification and confirmation capabilities are more important than deep causal explanation. The paper addresses a relevant question and its dataset of human-human communication in a simulated task is a useful resource. The iterative codebook development with multiple annotators is a strength, and the authors are transparent about the annotators' observations that explicit 'why' questions were rare. However, the central qualitative claim is largely a consequence of the coding scheme, and the quantitative performance claim is not backed by inferential statistics. The significance for XAI design is therefore currently not established.","major_comments":[{"comment":"The main qualitative result that most explanation-related messages seek clarification is a direct consequence of the codebook definitions. Table I assigns the labels 'doing', 'features', and 'auto' to the category 'select/clarify', and Fig. 3 shows that these three labels are the most frequent in the majority-label distribution. Therefore the statement that 'most explanation-related messages seek clarification in the decisions or actions taken' follows by construction rather than from an independent categorization of the dialogue. This circularity is acknowledged implicitly in the discussion ('This is likely why messages labelled with auto and features are quite frequent'), but the manuscript still presents the finding as an empirical discovery. A re-analysis that distinguishes information transmission from explanation-seeking behavior, or a substantially qualified claim, would be needed.","section":"§III.C, Table I; §IV.B, Fig. 3"},{"comment":"The three Pearson correlations (PCC = -0.28, 0.05, 0.30) are reported without p-values, confidence intervals, or any inferential test. With n = 20 sessions, none of these correlations is likely to reach conventional significance levels, and the text overstates the evidence by saying the data 'confirm that messages have an impact on the performance of our simulated task'. At minimum, exact p-values (or bootstrap intervals) and a clear statement about statistical power are required before drawing conclusions about the relationship between communication volume and performance.","section":"§IV.A, Fig. 2"},{"comment":"Inter-annotator agreement is reported as '61% of samples were labelled with at least three annotators agreeing on the label', with 5% unclassified. This is not a chance-corrected measure such as Cohen's kappa or Fleiss' kappa, so it does not substantiate the claim of a 'good level of inter-annotator agreement'. Without such a measure, the reliability of the majority labels that drive the central distribution in Fig. 3 is not established. The authors should report kappa statistics and, if agreement is low, discuss how disagreements were resolved.","section":"§III.C.1, §IV.B"},{"comment":"The construct validity of mapping 'auto' messages to the 'select/clarify' category is weak. Autocompleted messages are one-click sensor broadcasts (e.g., 'Object 18 ... Status Danger'), not user-driven selections or clarifications. Similarly, 'doing' messages are action updates and coordination statements, and 'features' messages report object attributes. Labeling these as explanation-related 'select/clarify' conflates information transmission with asking for or giving explanations. This directly affects the headline claim about what types of explanations robots need, because the dominant categories may not correspond to explanation behavior at all.","section":"§III.C, Table I"}],"minor_comments":[{"comment":"The paper says 'qualitative linguist analysis'; this should be 'linguistic analysis'.","section":"§IV.B"},{"comment":"The procedure mentions 'inter-code agreement' but the paper later uses 'inter-annotator agreement'; please use consistent terminology throughout.","section":"§III.C.1"},{"comment":"The figures show scatter plots without any indication of uncertainty or regression lines; adding regression lines and confidence bands would improve readability, though the statistical issues in the major comments would still need to be addressed.","section":"Fig. 2"},{"comment":"The caption 'Majority Label Distribution ( ≥ 3 agreement level)' could clarify how the 'unclassified or no similarities' 5% is handled; currently it is unclear whether 'unclassified' is a separate bar or omitted from the denominator.","section":"Fig. 3"},{"comment":"Reference [2] is a CHI 2023 paper and should include the page range or article number for completeness, per the reference style used elsewhere.","section":"References"}],"recommendation":"reject","confidential_remarks":"The main reason for rejection is the circularity of the central qualitative claim: the codebook pre-assigns the three most frequent labels to the 'select/clarify' category, so the finding is not an independent discovery. The performance correlations are also underpowered and lack inferential statistics. The paper could potentially be revised into a more descriptive study of communication patterns in TeamCollab, but as written the explanation-related framing overreaches the evidence. I would not encourage a major revision at this venue unless the authors can substantially reframe the analysis and provide a chance-corrected agreement measure plus appropriate significance tests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central finding that most explanation-related messages seek clarification is a consequence of the codebook, not a discovery. The paper is a reasonable pilot but overstates its evidence.\n\nWhat's actually new: they took Bertrand et al.'s taxonomy and applied it to a new chat corpus from a collaborative simulation. That's a modest extension, but the label distribution (Fig. 3) and the session-level correlations are new empirical output. The paper is also clearly written, the codebook is explicit, and they openly note that explicit “why” questions were rare—that's consistent with prior work and worth knowing.\n\nWhere it goes soft: the main qualitative result is circular. Table I assigns auto, doing, and features to select/clarify, and those three labels dominate the distribution. So saying “most explanation-related messages seek clarification” is just restating the codebook. The construct validity is also weak: autocompleted sensor broadcasts aren't clarifications, and action updates aren't explanations. The performance correlations (PCC -0.28, 0.05, 0.30, n=20) are reported without p-values or confidence intervals, and none reach conventional significance, so the abstract's “confirm that messages have an impact” is overstatement. Inter-annotator agreement is only raw 61% majority agreement, with no kappa or other chance-corrected measure. And the selection of 1,000 messages from 15 sessions is not explained—how representative is that sample?\n\nWhat's still decent: the experimental setup is interesting, and for a pilot the data collection is careful enough to be worth reanalyzing. The discussion about implicit versus explicit explanations is thoughtful, and the direction—focusing robot explanation capabilities on clarification and coordination rather than deeper causal reasoning—is plausible, just not supported as stated.\n\nWho this is for: researchers working on human-robot teaming and interactive XAI, especially those wanting a cautionary example of coding-scheme circularity. It doesn't deserve publication as is, but I'd send it to review anyway—a serious referee can push the authors to either reframe the claims, add proper statistics, or reposition this as a preliminary study. I wouldn't cite it in its current form, but if they fix the analysis, it could become useful.\n\nRecommendation: send to peer review with a clear expectation of major revision.","headline":"The headline result is largely coded into the codebook, so the main claim doesn't stand; still, it's a clear, honest pilot that would benefit from reanalysis and significant revision.","tokens_in":6228,"tokens_out":1673,"would_cite":false,"duration_ms":18200,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In the TeamCollab emergency-response simulation, most explanation-related messages are clarification requests, not 'why' questions, and communication volume correlates with task accuracy.","keywords":["explanation-related interactions","explainable AI","human-robot teaming","collaborative perception-cognition-communication-action","qualitative coding","emergency response simulation","team communication","clarification dialogue"],"falsifier":"Re-annotate the same 1,000 messages with an independent codebook that distinguishes clarification-seeking messages (e.g., explicit questions like 'what are you doing?') from statements that merely share information; if select/clarify is no longer the majority category, the paper's ranking of explanation needs would collapse. Alternatively, run a controlled experiment with a mandatory one-message-per-action condition to test whether increased communication volume actually raises the percentage of dangerous objects collected.","tokens_in":5275,"feed_emoji":"💬","tokens_out":5960,"duration_ms":56426,"temperature":0.7,"pith_summary":"This paper analyses 1,000 text messages exchanged by human participants in a simulated emergency-response task and classifies them according to a taxonomy of interactive explainable-AI techniques. It claims that most explanation-related messages are seeking clarification about what teammates are doing or about object features, rather than asking for deep causal 'why' explanations. It also reports that teams that exchange more messages collect a higher percentage of dangerous objects, even though they collect fewer objects overall. If these findings hold, AI-equipped robots in such collaborative settings would most need capabilities for confirming, clarifying, and sharing sensor information, not complex causal reasoning. The study is positioned as a step toward deciding which explanation capabilities to build into robots that work alongside humans.","feed_headline":"Teammates mostly seek clarification, not deep 'why' answers","feed_subtitle":"In a simulated emergency-response task, communication volume tracks accuracy—pointing robot designers toward confirm-and-clarify dialogue.","key_machinery":"The coding scheme, a three-stage qualitative labelling procedure with four annotators and a codebook, maps each message to one of nine labels grouped into four categories from an interactive-XAI taxonomy (select/clarify, mutate/simulate, dialogue/progress, dialogue/answer). The select/clarify category carries the main argument: because auto, doing, and features—the three most frequent labels—all fall under it, the paper concludes that clarification is the dominant explanation need. Inter-annotator agreement (61% of messages having at least three of four annotators agree) provides the reliability check for this machinery.","core_discovery":"The paper's central discovery is that in human-human collaboration on a time-pressured physical task, the explanation-related communication that actually occurs is dominated by the select/clarify mode: autocompleted sharing of sensed object attributes, 'what are you doing' updates, and discussion of object features. Explicit justification questions ('why is X dangerous?') essentially did not occur, and annotators found no majority-agreement cases for the 'why' label. The paper also establishes a positive Pearson correlation (0.30) between the number of messages a team sends and the fraction of collected objects that are actually dangerous, which it interprets as communication enabling agents to cross-check sensor readings and avoid wasting effort on benign objects.","pith_inferences":["This suggests that a robot that simply broadcasts sensor data and periodically states its current action would satisfy most of the observed 'explanation' demand, before any causal reasoning is added—an inference the paper does not draw.","A testable extension is to give one robot a clarification-only dialogue policy and another a causal-explanation policy in the same simulation, and compare team accuracy, to directly test the paper's design recommendation.","The correlation between message volume and accuracy is modest (PCC 0.30), so a sensitivity analysis separating coordination messages from clarification-seeking ones would clarify whether clarification specifically drives accuracy—an analysis the paper does not report.","Because the task is physical and time-pressured, the low frequency of 'why' questions may not transfer to planning-heavy or diagnosis-heavy collaborative tasks, where causal explanation needs could be stronger."],"forward_implications":["Robot design implication: explanation capabilities for collaborative physical tasks should prioritise clarification and confirmation dialogues, such as answering 'what are you doing?' and sharing sensor data, over long-form causal explanations.","Performance implication: communication volume is not a net cost; teams that communicate more are more accurate in identifying dangerous objects, so encouraging message exchange can improve outcomes measured by accuracy.","The near-absence of explicit 'why' questions suggests that implicit explanation dialogues—messages that function as clarifications in context—are what matter in emergency-response-style teamwork.","The positive team-level correlation justifies using the TeamCollab simulation to test explanation-equipped robot teammates in the paper's stated next step of human-robot interaction experiments."],"supporting_citations":[{"why":"Provides the TeamCollab simulation environment and the message logs that constitute the study's dataset.","marker":"[1]"},{"why":"Supplies the interactive-XAI taxonomy that the codebook's select/clarify, mutate/simulate, and dialogue categories are adapted from.","marker":"[2]"},{"why":"Justifies interpreting messages as implicit explanations by showing that teammates communicate actionable information implicitly in human-AI teams.","marker":"[9]"},{"why":"Contributes the three-stage codebook development and inter-annotator reliability-check procedure used to label the 1,000 messages.","marker":"[13]"}],"fun_headline_variants":["Why questions? Almost never — teammates prefer clarifying updates","In emergency tasks, teammates ask 'what' not 'why' — design robots to clarify","More messages, better picks: 0.30 correlation in collaborative task","Human teams almost never ask 'why'—they ask 'what'—robots should too"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The finding that most explanation-related messages seek clarification largely follows from the coding decision to treat routine coordination messages (autocompleted sensor-sharing, 'doing' updates, and feature talk) as select/clarify 'explanation-related'; under a narrower definition of explanation, the majority might vanish.","fun_headline_variants_meta":{"raw":{"variants":["Why questions? Almost never — teammates prefer clarifying updates","In emergency tasks, teammates ask 'what' not 'why' — design robots to clarify","More messages, better picks: 0.30 correlation in collaborative task","Human teams almost never ask 'why'—they ask 'what'—robots should too"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2350,"prompt_tokens":789,"completion_tokens":1561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":1477}},"tokens_in":405,"tokens_out":1561,"duration_ms":12291,"temperature":1.0,"reasoning_tokens":1477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:27:12.126816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the same 1,000 messages with an independent codebook that distinguishes clarification-seeking messages (e.g., explicit questions like 'what are you doing?') from statements that merely share information; if select/clarify is no longer the majority category, the paper's ranking of explanation needs would collapse. Alternatively, run a controlled experiment with a mandatory one-message-per-action condition to test whether increased communication volume actually raises the percentage of dangerous objects collected.","supporting_citations":[{"cited_title":"TeamCollab: A framework for col- laborative Perception-Cognition-Communication-Action,","cited_arxiv_id":null,"evidence_quote":"Provides the TeamCollab simulation environment and the message logs that constitute the study's dataset."},{"cited_title":"On selective, mutable and dialogic XAI: a review of what users say about different types of interactive explanations,","cited_arxiv_id":null,"evidence_quote":"Supplies the interactive-XAI taxonomy that the codebook's select/clarify, mutate/simulate, and dialogue categories are adapted from."},{"cited_title":"Implicit commu- nication of actionable information in human-AI teams,","cited_arxiv_id":null,"evidence_quote":"Justifies interpreting messages as implicit explanations by showing that teammates communicate actionable information implicitly in human-AI teams."},{"cited_title":"De- veloping and using a codebook for the analysis of interview data: An example from a professional development research project,","cited_arxiv_id":null,"evidence_quote":"Contributes the three-stage codebook development and inter-annotator reliability-check procedure used to label the 1,000 messages."}],"review_version":1}