{"id":"56a6a79b-abdd-4aba-88eb-96329209f415","arxiv_id":"2411.12829","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors present and release a multi-layer symbolic annotation of the SCOUT human-robot dialogue corpus, covering utterance semantics, multi-floor dialogue structure, and visual context for common-ground research.","lead":"This paper describes layered symbolic annotations on hundreds of human-robot search-and-rescue dialogues, capturing utterance meaning, dialogue structure across two conversational floors, and visual context. It is a resource paper: the annotations and data are released on GitHub and are meant to help robots maintain common ground with remote human partners.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Wizard-of-Oz transfer to autonomous dialogue is asserted, not tested; the paper's own Experiment 4 data could settle it.","rationale":"The paper is primarily a corpus and annotation resource contribution, and its central value claim is that the multi-floor WoZ data and annotations make the dimensions of common ground accessible to autonomous systems. The strongest supporting evidence is ScoutBot, MultiBot, and JUDI, but these systems are retrieval-based dialogue managers that lack grounding and world models; only JUDI's integration with AMR-based grounding is described as ongoing. Consequently, the abstract's claim that the annotations enable autonomous bi-directional dialogue depends on transfer from the WoZ setup to automated components. The reader's weakest assumption identifies this WoZ fidelity concern, and I agree. I sharpen it by noting that the paper itself reports an Experiment 4 with an automated DM replacing the wizard, which is a natural quasi-experimental control for exactly this transfer. Not reporting any comparison from that experiment leaves the load-bearing assertion untested. This is not a soundness error in the annotation schemas — for Dialogue-AMR and Dialogue Structure, IAA is reported and reasonable — but it is a gap in evidence for the central claim. The visual-context annotations lacking IAA and covering only 30 of 287 dialogues are a secondary limitation, not the primary load-bearing issue. A conditional verdict remains appropriate: the resource contribution is credible, but the enablement claim should be either demonstrated against Experiment 4 or softened to 'intended to support' pending such evidence. No ad hominem is intended; the critique is about the strength of the evidence, not the integrity of the authors.","tokens_in":33319,"tokens_out":3746,"duration_ms":42655,"concrete_test":"Identify the fourth-experiment dialogues in the SCOUT release (or ARL records) in which the DM-Wizard was replaced by the automated dialogue manager. Compare these against the Wizard-of-Oz dialogues on Commander utterance length, vocabulary, clarification-request rates, and the distribution of Dialogue Structure TU/relation types (especially req-clar, offer, and translation-right variants). Then run the WoZ-trained retrieval-based dialogue manager on the Experiment 4 Commander utterances and measure instruction-level execution success. If the distributions and success rates are statistically comparable, the transfer concern is resolved; if they shift significantly, the claim that the annotations enable autonomous dialogue must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 justifies the multi-WoZ setup by asserting that it 'mimics how the physical motion component of an autonomous system would not speak directly to a human user but rather would receive its parsed instructions from a separate module.' The same section discloses that 'In the fourth and final experiment of our project developing SCOUT, the DM-Wizard was replaced with an automated dialogue manager.' No results from that experiment are reported. The central claim — that the annotations enable autonomous bi-directional dialogue — therefore rests on unverified transfer from human wizard behavior to automated components. This matters because the DM-wizard had live video, full language understanding, and world knowledge, so the annotated dialogue structures (clarification offers, translation-right instructions, ack types) reflect human-level competence. Commanders may speak differently to a system that makes different errors; if dialogue-structure distributions shift, the corpus annotations lose predictive value for the deployed system. ScoutBot, MultiBot, and JUDI do not close this gap: they are retrieval-based dialogue managers, with JUDI's AMR-based grounding still under development and no end-to-end task evaluation reported. The issue is not that WoZ collection is illegitimate, but that the paper's strongest claim exceeds its evidence. The missing comparison to Experiment 4 is the most direct way to test the transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a multi-layer symbolic annotation framework applied to SCOUT, a Wizard-of-Oz (WoZ) human-robot dialogue corpus in a remote search-and-navigation domain. Four layers are presented: Standard-AMR and Dialogue-AMR for utterance-level propositional semantics and illocutionary force; Dialogue Structure annotations (transaction units and relations) for multi-floor dialogue; and two visual-context annotations (photo-requesting strategies and LIDAR Exploration Maps). The paper reports inter-annotator agreement for Dialogue Structure (antecedents α=0.79–0.94, relation types 0.83–0.93, TUs 0.65–0.85 with the modified schema) and Smatch 86.6% for Dialogue-AMR on 290 utterances, and it discusses the ScoutBot, MultiBot, and JUDI dialogue systems as implementations that use the annotations. The central claim is that these annotations make the dimensions of meaning needed for common ground accessible to autonomous systems, enabling flexible two-way dialogue for human-robot teams.","tokens_in":33583,"tokens_out":4993,"duration_ms":50704,"significance":"If the annotation schemas prove usable for developing autonomous dialogue systems, this is a substantial resource: a large (89,056 utterances, 278 dialogues) multi-floor, multi-modal corpus with layered annotations, released on GitHub, plus concrete worked examples and explicit discussion of architectural challenges. Strong points include the reported IAA on dialogue structure, the public corpus release, the reproducibility of the annotation guidelines, and the implemented prototypes (ScoutBot, MultiBot, JUDI) that at least partially exploit the dialogue-structure annotations. However, the visual-context annotation is still small-scale (30 dialogues) with no agreement metrics, and the transfer from WoZ data collection to fully autonomous components is asserted rather than demonstrated, with the systems described as lacking grounding and world models and the AMR-based grounding still ongoing. The paper is best read as a corpus and schema description with an aspirational end-to-end goal, not as a demonstrated autonomous dialogue system.","major_comments":[{"comment":"The abstract claims the annotations 'enable physical robots to autonomously engage with humans in bi-directional dialogue and navigation,' and Section 2 asserts that the multi-WoZ setup 'mimics how the physical motion component of an autonomous system would not speak directly to a human user.' However, the fourth experiment that replaced the DM-Wizard with an automated dialogue manager is mentioned but no results are reported. The systems described in Section 6.1 (ScoutBot, MultiBot) are retrieval-based and explicitly lack grounding and world models; Section 6.2 states that AMR-based grounding is 'ongoing' with evaluation strategy still to be determined. The evidence therefore does not yet support the strong claim. Please either report results from Experiment 4 (even descriptive comparisons of dialogue-structure distributions or commander behavior) or temper the claims throughout to 'support the development of' and 'progress toward' such systems.","section":"Section 2, Section 6"},{"comment":"The Visual Context annotations are presented as a component of the multi-modal common-ground contribution, but no inter-annotator agreement is reported for either annotation task. Photo-requesting strategies were analyzed by one annotator, and Exploration Maps were annotated by one annotator and verified by a second, with no quantitative reliability measure such as Cohen's kappa or Krippendorff's alpha. Given the emphasis on IAA for the other annotation layers, the lack of agreement data makes the reliability of the visual annotations difficult to assess. Please report agreement metrics if available, or explicitly label these as preliminary exploratory analyses whose reliability remains to be established.","section":"Section 5.1, Section 5.2"},{"comment":"The IAA figures for Dialogue-AMR and Dialogue Structure are computed on the same SCOUT corpus on which the annotation guidelines were iteratively refined, as described in Sections 3.1.1 and 4.1. This measures internal consistency of the final guidelines on the development corpus, not external validity or generalizability to other domains or to fully autonomous behavior. The paper does not currently acknowledge this limitation. Please add an explicit statement of this point, and, if available, report any held-out or cross-domain annotation results (for example, the Minecraft Dialogue Corpus annotation mentioned in Section 3.2) that would speak to the schema's transferability.","section":"Section 3.2, Section 3.1.1, Section 4.1"},{"comment":"The annotation notation is inconsistent in the core examples. Section 3.1 states that new speech-act relations are denoted with a '-SA' suffix, and Figure 11 uses 'command-SA' with a numbered roleset 'go-02'; Table A2 likewise lists Command-SA and go-02. However, Table 3 and Table A1 show 'command-00' and 'go-01' in the Dialogue-AMR annotations for the same type of utterances. The reader cannot tell whether this reflects an intentional version difference or a typographical error. Please harmonize the notation across the paper, or explicitly state the mapping between the '-SA' relations and the numbered relation forms.","section":"Section 3.3, Table 3, Table A1, Figure 11"}],"minor_comments":[{"comment":"The corpus size is given as 278 dialogues in Section 2 but as 287 dialogues in Section 5.2 ('until all 287 dialogues are completed'); these numbers should be reconciled.","section":"Section 2 vs Section 5.2"},{"comment":"Typo: 'comparisong' should be 'comparing'; Section 3.1.3 and Table 2 contain 'repetoire' instead of 'repertoire'; Section 1 has 'interlocuter' instead of 'interlocutor'; Section 3.1.1 has 'Damsl guidelines' instead of 'DAMSL guidelines'.","section":"Section 3.2"},{"comment":"The first bullet point in the photo-requesting strategies begins with 'F ront' due to a formatting issue; this should read 'Front'.","section":"Section 5.1.1"},{"comment":"The phrase 'a translation-right' appears without the noun 'relation' in one sentence; this is grammatically awkward and should be rephrased.","section":"Section 4.2.2"},{"comment":"The description of how Dialogue Structure annotations are used as training data is at the level of 'instruction-response pairs' but does not specify which annotation layers (relations, TUs, antecedents) are leveraged in the retrieval model and the dialogue policy; a sentence clarifying the mapping would strengthen the reproducibility of the system description.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a valuable descriptive corpus/annotation resource, and the central schema contributions are defensible. The main reason for major revision is the mismatch between the strong end-to-end autonomy claims and the reported evidence: the WoZ-to-automated transfer is not tested, and the systems described are incomplete with respect to the stated goal. If the authors can add even a brief report on Experiment 4 or clearly restructure the claims as 'toward' autonomous dialogue, the paper would be suitable. I would not reject on the circularity issue, as iterative schema development is standard in annotation research, but it should be acknowledged explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid corpus-and-schema paper, worth a serious referee, but the abstract's claim that the annotations \"enable\" autonomous bi-directional dialogue is ahead of the evidence. The genuinely new pieces are the unified presentation of the full SCOUT annotation stack and the visual-context annotations (photo request strategy taxonomy, LIDAR Exploration Maps), which the earlier papers only described piecemeal. Having the whole stack together with worked examples and a released GitHub corpus is a real resource.\n\nCredit where due: all 89,056 utterances carry dialogue structure annotations with credible IAA (antecedents alpha 0.79–0.94, relations 0.83–0.93, TUs 0.65–0.85). Dialogue-AMR hits 86.6% Smatch on 290 utterances. The paper is transparent that the visual-context annotations are ongoing — 30 of 287 dialogues, one annotator verified by a second — and it openly says ScoutBot, MultiBot, and JUDI lack grounding and world models.\n\nThe soft spot is load-bearing: the paper says the multi-WoZ setup \"mimics\" a fully automated system, and the abstract promises autonomous two-way dialogue. But the implemented systems are retrieval-based, with JUDI's AMR grounding still under development and no end-to-end task evaluation reported. The paper itself mentions a fourth experiment where the DM-Wizard was replaced by an automated dialogue manager, and reports no results. That is the most direct test of the WoZ-to-automated transfer, and its absence leaves the central claim asserted rather than shown. Commanders will likely speak differently to a system that makes different errors — the wizards had live video, full language understanding, and world knowledge. This doesn't undermine the corpus, but the claims need to be scoped or the experiment reported.\n\nMinor stuff: the corpus says 278 dialogues one place and 287 in another; Table 3 uses command-00/assert-00 while the text and appendix say -SA. And since the schemas were iterated on the same SCOUT data used for IAA, those numbers measure internal consistency, not external validity — the paper acknowledges this implicitly but could be plainer.\n\nFor anyone working on human-robot dialogue or situated semantics, this is a useful reference and a genuine data release. The main revision is to align the claims with the evidence, or add the missing experiment. I'd accept it for review and ask for that.","headline":"A solid, transparent corpus-and-schema paper whose abstract oversells the autonomous-dialogue payoff; the missing Experiment 4 results are the obvious fix.","tokens_in":34173,"tokens_out":2976,"would_cite":true,"duration_ms":29512,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that symbolic annotations over multi-floor human-robot dialogue can make the dimensions of meaning needed for common ground accessible to autonomous systems, enabling flexible two-way dialogue for robot teams.","keywords":["human-robot dialogue","common ground","Abstract Meaning Representation","Dialogue-AMR","multi-floor dialogue","dialogue structure annotation","visual grounding","Wizard-of-Oz corpus"],"falsifier":"Re-run the original SCOUT Commander utterances through an automated pipeline that replaces the two human wizards with an automated dialogue manager using the dialogue-structure annotations and an autonomous navigation module using the Dialogue-AMR-derived behavior specifications, then measure how often the robot executes the commander's intent successfully compared with the wizard-mediated original. If the automated execution rate falls far below the wizard-mediated rate on the same utterances, the corpus would not transfer to real autonomous components; conversely, matching or exceeding wizard performance would support the paper's central claim.","tokens_in":33099,"feed_emoji":"🤖","tokens_out":9987,"duration_ms":85868,"temperature":0.7,"pith_summary":"The paper's aim is to give autonomous robots the symbolic representations they need to converse with a human partner in remote search-and-rescue settings, where video is unavailable but dialogue and occasional still images can establish common ground. It argues that three dimensions of meaning must be made explicit: the content and intent of each utterance, the way utterances across separate conversational floors relate to one another, and the connection between language and the robot's visual and LIDAR inputs. To capture these, the authors annotate the SCOUT corpus of 278 human-robot dialogues with Dialogue-AMR, a Dialogue Structure schema of transaction units and relations, and visual-context annotations. If the schemas hold up, the released annotations become training and specification data for building dialogue systems in which a commander can flexibly re-task one or more robots by talking to them, rather than tele-operating them.","feed_headline":"Robot dialogue needs three meaning layers, and they are now annotated","feed_subtitle":"The SCOUT corpus adds speech-act and structure layers to human-robot talk, supporting two-way rescue dialogue.","key_machinery":"The load-bearing machinery is the three-layer annotation schema over multi-floor dialogue. The first layer, Dialogue-AMR, augments Abstract Meaning Representation with a speech-act root (e.g., command-SA, offer-SA), tense and aspect features, and a normalized lexicon in which varied surface forms like 'turn,' 'rotate,' and 'pivot' map to a single robot concept such as Rotation, so that unconstrained language can be distilled into executable behavior parameters. The second layer, Dialogue Structure, segments the dialogue into Transaction Units, which are clusters of utterances from multiple speakers and floors that together realize one intent, and labels each utterance with a relation type and an antecedent, including translation relations that capture how the dialogue manager passes instructions to the navigation module and relays acknowledgements back. The third layer, Visual Context, annotates photo-request strategies and LIDAR exploration maps, linking language to the shared visual environment. Together these layers are intended to give a dialogue system the information it needs to establish common ground and initiate repairs when the commander's and robot's views of the environment diverge.","core_discovery":"The paper's central claim is that a robot can only act as a cooperative partner in a physically situated task if it can access several distinct dimensions of meaning at once: propositional content and illocutionary force within a single utterance, the discourse relations between utterances across multiple conversational floors, and the grounding of language in images and LIDAR. The authors demonstrate this through a multi-layer annotation of the SCOUT corpus, in which Commander utterances are parsed into Dialogue-AMR graphs with a speech-act root, grouped into Transaction Units with typed antecedent relations (including translations between the Commander-DM floor and the DM-RN floor), and aligned with photo-request strategies and annotated LIDAR exploration maps. They further report that these annotations have been used to train three successive dialogue systems, ScoutBot, MultiBot, and JUDI, which accept natural-language navigation instructions and respond with feedback or clarification, with grounding added through an AMR-based pipeline. The claim is therefore not just that the annotations describe human-robot dialogue, but that they make the dimensions of meaning accessible to autonomous systems in a form that supports implementation.","pith_inferences":["Editorial inference: the multi-floor 'translation' relations are a general mechanism for any relayed instruction chain, so the schema could find use in human-only teams where a coordinator passes orders to field operators, a setting the paper does not discuss.","Editorial inference: if the Dialogue-AMR normalization is as consistent as reported, it could be used not only to parse commands but to generate clarification questions, for instance composing 'Do you mean the door on the left?' from the speech-act and action parameters.","Editorial inference: the paper's finding that commanders who used the cardinal or degrees-of-rotation photo strategies were more successful suggests a robot could learn to propose such strategies proactively, but the paper only reports the correlation, not a causal test."],"forward_implications":["A system built from these annotations can take a commander's natural-language instruction, distill it to a robot behavior and its parameters through Dialogue-AMR, and pass it to a separate navigation module, mirroring the multi-floor architecture of a deployed system.","The fully annotated dialogue-structure data, covering all 89,056 utterances in SCOUT, lets a dialogue manager learn instruction-response pairs, so it can provide feedback or ask for clarification without hand-written rules.","The visual-context annotations give a robot the ability to notice when a commander's last photo no longer matches the robot's current position, making repair dialogue possible.","The progression from ScoutBot to MultiBot to JUDI is evidence that the annotation approach generalizes from one robot to heterogeneous robot teams and from lab conditions to situations without a cloud connection."],"supporting_citations":[{"why":"Supplies the SCOUT corpus, the human-robot dialogue dataset that all annotation layers are applied to.","marker":"Lukin et al., 2024"},{"why":"Defines Standard AMR, the representation that Dialogue-AMR augments with speech acts and tense/aspect.","marker":"Banarescu et al., 2013"},{"why":"Introduces the Dialogue-AMR speech-act augmentation for human-robot dialogue.","marker":"Bonial et al., 2019"},{"why":"Sets out the original multi-floor dialogue-structure annotation guidelines that the TU and relation schema refines.","marker":"Traum et al., 2018"},{"why":"Adds situated relation subtypes that capture environment-dependent translations across floors.","marker":"Bonial et al., 2021"},{"why":"Reports ScoutBot, the first dialogue system trained from SCOUT dialogue data.","marker":"Lukin et al., 2018"},{"why":"Describes MultiBot, showing the dialogue manager generalizes to heterogeneous robot teams.","marker":"Marge et al., 2019"},{"why":"Describes JUDI, the deployed dialogue interface used with the robot front end.","marker":"Marge et al., 2023"},{"why":"Reports the AMR-based grounding experiments that execute natural-language instructions on a robot platform.","marker":"Bonial et al., 2023"}],"fun_headline_variants":["Three meaning layers let robots build common ground","SCOUT corpus adds speech-act and structure to robot talk","Dialogue-AMR and multi-floor schema enable two-way rescue talk","From annotations to JUDI: Robot dialogue gets three layers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human wizards who stood in for the dialogue manager and the robot navigator during data collection interpret and execute commander instructions about as well as the automated components they stand in for, so that the annotated patterns will transfer to real deployed systems.","fun_headline_variants_meta":{"raw":{"variants":["Three meaning layers let robots build common ground","SCOUT corpus adds speech-act and structure to robot talk","Dialogue-AMR and multi-floor schema enable two-way rescue talk","From annotations to JUDI: Robot dialogue gets three layers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1327,"prompt_tokens":1002,"completion_tokens":325,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":618,"tokens_out":325,"duration_ms":4070,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:09:31.744095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the original SCOUT Commander utterances through an automated pipeline that replaces the two human wizards with an automated dialogue manager using the dialogue-structure annotations and an autonomous navigation module using the Dialogue-AMR-derived behavior specifications, then measure how often the robot executes the commander's intent successfully compared with the wizard-mediated original. If the automated execution rate falls far below the wizard-mediated rate on the same utterances, the corpus would not transfer to real autonomous components; conversely, matching or exceeding wizard performance would support the paper's central claim.","supporting_citations":[{"cited_title":"Bonial, M","cited_arxiv_id":null,"evidence_quote":"Supplies the SCOUT corpus, the human-robot dialogue dataset that all annotation layers are applied to."},{"cited_title":"Henry, S","cited_arxiv_id":null,"evidence_quote":"Sets out the original multi-floor dialogue-structure annotation guidelines that the TU and relation schema refines."},{"cited_title":"Gervits, C.J","cited_arxiv_id":null,"evidence_quote":"Reports ScoutBot, the first dialogue system trained from SCOUT dialogue data."},{"cited_title":"Nogar, C.J","cited_arxiv_id":null,"evidence_quote":"Describes MultiBot, showing the dialogue manager generalizes to heterogeneous robot teams."},{"cited_title":"Bonial, S","cited_arxiv_id":null,"evidence_quote":"Describes JUDI, the deployed dialogue interface used with the robot front end."}],"review_version":1}