{"id":"4ea0dbca-1a35-456d-b10c-f5839d71276c","arxiv_id":"1908.09921","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper introduces a question-answer annotation scheme with five question types, seven answer types, and semantic features, and shows it can be applied across three languages with moderate annotator agreement.","lead":"This paper proposes a layered annotation scheme for labeling the content and discourse function of question-answer pairs in conversation, and tests it on English, Spanish, and Dutch dialogues. It also reports preliminary machine learning experiments for automating question type annotation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-linguistic validity is untested: inter-annotator agreement is reported only for English, while the headline claim covers Spanish and Dutch.","rationale":"The reader's weakest assumption matches the most load-bearing gap: the multilingual claim is not backed by multilingual agreement data. My read of the paper confirms this: the IAA table is explicitly computed on the SCoSE subset, and the guidelines and feature mapping are English-only. The conclusion's mention of language-specific guidelines as future work is an honest acknowledgment but does not validate the current multilingual scope. I do not see a more fundamental internal inconsistency; the scheme, English IAA, and error analysis are coherent, and the public repository is a real resource. The ML results are preliminary and would benefit from error bars, but they are not the central contribution. Thus the verdict should remain CONDITIONAL, with the condition being per-language validation.","tokens_in":6035,"tokens_out":5194,"duration_ms":54724,"concrete_test":"Sample 150 QAPs from CallFriend (Spanish) and 150 from CGN (Dutch). Have two native-speaker annotators per language apply the existing English guidelines, translating only the wh-word cue list (e.g., dónde/waar for LOCATION, por qué/waarom for RE). Compute per-language Cohen's kappa for question types and features. If either language's question-type kappa falls below 0.6, or if feature kappa is substantially lower than the English 0.67, the cross-linguistic transfer assumption fails and the paper should be revised to claim English-only validation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that the scheme captures QAP content in English, Spanish, and Dutch rests on cross-linguistic validity, but the only agreement evidence in Section 4.1.3, Table 4, comes from 690 SCoSE (English) utterances jointly annotated by all three annotators; no per-language kappa is reported for CallFriend (Spanish) or CGN (Dutch). The annotation guidelines are written only in English (Section 4.1.1), and the feature mapping is explicitly from English wh-words (Section 3.2, Table 2). Applying the scheme to Spanish and Dutch therefore requires an untested transfer assumption: that the English-derived categories, precedence order, and wh-word-to-feature cues remain usable after translation. The conclusion lists language-specific guidelines as future work, which partially concedes this gap, but the paper's stated contribution still includes the multilingual application. The current data support the scheme as demonstrated on English, with exploratory annotations in Spanish and Dutch, not as a validated cross-linguistic scheme.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a multi-layer semantic annotation scheme for question-answer pairs in conversation, consisting of five question types (YN, WH, CS, PQ, DQ), seven answer types, and feature tags (semantic roles) for wh- and disjunctive questions. The authors describe corpus-driven development, write annotation guidelines, apply the scheme to English (SCoSE), Spanish (CallFriend), and Dutch (CGN), report inter-annotator agreement on a 690-utterance English subset, and run decision-tree, bag-of-words, and RNN classifiers for English question-type prediction. The reported agreement is moderate (kappa = 0.63 question, 0.67 features, 0.49 answers), and the decision tree outperforms the majority baseline (accuracy 0.73 vs 0.47; macro-F1 0.58 vs 0.31).","tokens_in":6233,"tokens_out":4954,"duration_ms":47828,"significance":"If the scheme is validated, it would be a useful, small tagset for annotating question-answer content in dialogue and for training automatic dialogue model components. The paper's strengths are its explicit and public annotation materials, its corpus-driven design grounded in three languages, the inclusion of a confusion matrix and feature set, and the honest reporting of disagreement categories. These make the resource easy to reuse and critique. The main caveat is that the cross-linguistic validity is currently asserted rather than demonstrated, and the ML results are partly powered by features that are definitional for the question types.","major_comments":[{"comment":"The inter-annotator agreement that supports the scheme is computed on 690 SCoSE (English) utterances, yet the paper's contribution is stated as applying the scheme to dialogues in English, Spanish, and Dutch (Abstract). No per-language kappa is reported for CallFriend or CGN, and the annotation guidelines (Section 4.1.1) and wh-word-to-feature mapping (Section 3.2, Table 2) are English-specific. The Spanish and Dutch annotations are therefore exploratory, not evidence of cross-linguistic validity. The authors should either present per-language agreement with confidence intervals and language-specific guideline adaptations, or restrict the validation claim to English and describe the other two as a pilot.","section":"§4.1.3, Table 4"},{"comment":"Several hand-designed features encode the formal definitions of the question types: has_wh, has_or, has_inversion, has_tag, and has_cliche are the surface cues used in the manual precedence order (Section 4.1.1) to identify WH, DQ, YN, and PQ. The decision tree's accuracy of 0.73 (F1 0.58) versus majority baseline 0.47 (F1 0.31) is therefore partly a check that these cues are learnable, not evidence that the semantic content of the QAP tagset is captured. I do not see this as invalidating the scheme, but the ML claim should be rephrased, and an ablation leaving out definitional cues would make the contribution of context features such as last_utt_similar measurable.","section":"§4.2, Table 5"},{"comment":"The answer layer is not operationalized to the same standard as the question layer. The text says overlapping tags are decided based on context and prosody but gives no decision rules or worked examples for distinguishing PA/NA from DA, or UA from UT, and Table 3 assigns most answer types to all question types with no eligibility constraints. This likely contributes to the low answer kappa (0.49, Table 4), but the paper does not report which confusions drove it. The answer layer should be strengthened with explicit decision procedures or acknowledged as a preliminary draft.","section":"§3.3, Table 3"}],"minor_comments":[{"comment":"The relationship between the 4,939-utterance subset, the 3,578-utterance gold standard, the 690 jointly annotated utterances, and the ML test set (184 questions) is not stated; please clarify the annotation protocol and data splits.","section":"§4.1.2"},{"comment":"The test set contains 184 questions but the per-class support for DQ is 3 and CS is 1, making macro-F1 noisy; report per-class precision, recall, and standard errors.","section":"§4.2, Table 6"},{"comment":"There are minor inconsistencies in notation: 'SCoSe/Amy' in Section 3.1 should be 'SCoSE/Amy', and the citation 'ELAN (2017)' should appear consistently with the reference list entry.","section":"§3.1 and References"},{"comment":"The statement that 'the agreement values obtained for question types were over 0.6 (for all annotators combined)' is ambiguous because Table 4 reports a single kappa of 0.63; specify whether this is averaged over pairwise annotator combinations and report the range.","section":"§4.1.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a suitable preliminary resource paper, but the gap between the multilingual framing and the English-only agreement evidence is a substantive issue that the authors can fix by reframing or by adding per-language agreement. The ML circularity concern is real but not disqualifying. No concerns about academic integrity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious but small annotation paper. It ships a tagset, annotation guidelines, a manually annotated multilingual corpus of 701 questions, and a decision-tree baseline. Read the headline as 'validated on English, exploratory for Spanish and Dutch.' The stress-test note is correct: the only inter-annotator agreement is on 690 English utterances; the guidelines are written in English; the wh-word-to-feature mapping is from English. No per-language kappa is reported. The conclusion does list language-specific guidelines as future work, so the gap is partly acknowledged, but the stated contribution still includes the multilingual application. That is the main weakness.\n\nWhat is genuinely good: the question/answer/feature layers are clearly defined, with a precedence order and discussion of overlap. The corpus and guidelines are in a public repo. The agreement scores (kappa 0.63 questions, 0.67 features, 0.49 answers) are moderate and honestly presented. The authors examine disagreements and refine the guidelines. The ML part is clearly labeled preliminary, uses simple hand-designed features, and reports macro-F1 with a confusion matrix. That is fine as a proof of concept.\n\nThe other soft spots are minor. The ML results are single-point estimates with no significance testing or confidence intervals, but the paper does not overclaim. The hand-designed features encode definitional properties of the tagset, so the above-baseline accuracy is partly circular; the paper acknowledges the features are superficial. The scheme overlaps with DAMSL and Freed's typology, and the authors say so. Novelty lies in the specific combination and the application to three languages, not in a new framework.\n\nOverall: a solid preliminary resource paper. The central annotation claim holds for English; the cross-linguistic part needs either per-language agreement data or a more careful scope. I would send this to peer review——it is exactly the kind of shared resource the dialogue-annotation subfield can use, and the repository makes it reproducible. Expect a revision that either adds per-language numbers or narrows the claims.\n\nFor my own work I would not cite it in the next year, but if I were doing dialogue annotation I would keep it in mind.","headline":"A modest, honest annotation resource for question-answer pairs, but the cross-linguistic claim runs ahead of the evidence.","tokens_in":6748,"tokens_out":2513,"would_cite":false,"duration_ms":25441,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An annotation scheme with five question types and seven answer types labels question-answer pairs in English, Spanish, and Dutch with moderate human agreement, and a decision tree already beats the majority baseline.","keywords":["question-answer pairs","annotation scheme","dialogue acts","semantic roles","inter-annotator agreement","decision tree classification","multilingual dialogue","question types"],"falsifier":"A controlled study in which bilingual annotators annotate the same Spanish and Dutch dialogues using only the English guidelines and wh-word-to-feature mapping, compared with a second pass using language-specific adapted guidelines, would settle the transfer question: if feature agreement on the English-guideline pass is at or near chance for either language, the cross-linguistic claim fails.","tokens_in":5876,"feed_emoji":"💬","tokens_out":6604,"duration_ms":61790,"temperature":0.7,"pith_summary":"This paper proposes a semantic annotation scheme for labeling question-answer pairs in dialogue. The scheme assigns each question a type from a five-way tagset (yes/no, wh-, disjunctive, completion suggestion, phatic) and, where relevant, a semantic-role feature for the questioned constituent, plus a seven-way tagset for answers. The authors test the scheme on English, Spanish, and Dutch spoken dialogues and report moderate inter-annotator agreement on question types and features, with lower agreement on answer types. They also show that a decision tree using surface features can classify English question types at accuracy 0.73 and F1 0.58, well above a majority-class baseline. If the scheme holds up in broader data, it would give dialogue systems a compact, multilingual way to represent what a question asks and what an answer contributes.","feed_headline":"Question-answer tagging hits moderate agreement in 3 languages","feed_subtitle":"Human annotators agree on question types and semantic roles; a decision tree classifies English questions above baseline.","key_machinery":"The load-bearing machinery is the type-feature pair: every question gets a question type, every wh- or disjunctive question additionally gets a feature tag naming the semantic role of the questioned constituent (location, time, agent, reason, and so on), and the question type restricts which answer tags are possible. A wh-word-to-feature mapping adapted from earlier speech processing work turns wh-words such as 'where' into feature cues, and a precedence order (wh-, disjunctive, yes/no, completion suggestion, phatic) lets annotators resolve ambiguous cases by checking the most syntactically specific type first. This combination lets a few tags cover a wide range of dialogue phenomena while keeping the annotation task tractable.","core_discovery":"The central claim is that the content and discourse contribution of a question-answer pair can be captured by a small layered tagset: a question type, an optional semantic-role feature for the questioned constituent, and an answer type that is constrained by the question type. On 701 questions from spoken corpora in English, Spanish, and Dutch, human annotators applied the scheme with observed agreement 0.73 for question types and 0.90 for features, corresponding to Cohen's kappa of 0.63 and 0.67; answer types reached observed agreement 0.59 and kappa 0.49. For automatic annotation, a decision tree using surface features such as wh-word presence, inversion, tag questions, and utterance length classifies English question types with accuracy 0.73 and macro F1 0.58. The hardest class is phatic questions, which are difficult to separate from other question types on short utterances.","pith_inferences":["The English wh-word-to-feature mapping may be more language-general than the paper claims, since many Spanish and Dutch wh-words carry the same semantic-role cues as English; a direct test on Spanish and Dutch wh-questions would show whether transfer holds or needs adaptation.","Because most decision-tree errors came from short utterances misclassified as phatic, adding discourse context or prosodic features could improve automatic annotation more than adding data alone.","If the scheme is combined with dialogue act taxonomies, the answer types could serve as a bridge between semantic content and dialogue state, for instance treating an uncertainty answer to a yes/no question as a cue for clarification or repair."],"forward_implications":["A dialogue system could use the type-feature representation as a compact preprocessing layer: knowing that a question is a wh-question about location or an uncertainty answer narrows the space of appropriate responses.","Automatic question type tagging is feasible with shallow features alone, since the decision tree's accuracy of 0.73 and F1 of 0.58 exceed the majority baseline by a wide margin, and more training data may push the neural models past this level.","The moderate agreement scores on English, Spanish, and Dutch suggest that the same tagset can be applied across languages, and the documented disagreement patterns, especially around phatic questions, give concrete targets for improving annotation guidelines.","Answer type restrictions derived from question types give a systematic taxonomy for evaluating dialogue systems: a good response should match the expected answer type for the question asked."],"supporting_citations":[{"why":"Supplies the question form/function continuum and the yes/no versus wh-question prototypes that the tagset starts from.","marker":"Freed (1994)"},{"why":"Provides the dialogue-act question types and automatic tagging setting that the scheme extends and compares against.","marker":"Stolcke et al. (2000)"},{"why":"Defines the DAMSL multi-layer dialogue annotation approach on which the layered question-answer structure builds.","marker":"Allen and Core (1997)"},{"why":"Supplies the wh-word-to-feature mapping adapted for the feature layer.","marker":"Boritchev (2017)"},{"why":"Original source of the wh-word-to-feature mapping as cited by the paper.","marker":"Jurafsky and Martin (2000)"},{"why":"Provides the standard moderate-agreement interpretation used to evaluate the kappa scores.","marker":"Landis and Koch (1977)"},{"why":"Supplies the decision-tree algorithm used for automatic question type classification.","marker":"Quinlan (1986)"},{"why":"Provides the Spanish CallFriend telephone conversation corpus used to test the scheme.","marker":"Canavan and Zipperlen (1996)"},{"why":"Provides the Spoken Dutch Corpus telephone conversations used for the Dutch test.","marker":"Oostdijk (2001)"},{"why":"Provides the SCoSE English face-to-face conversation corpus used for the English test.","marker":"Norrick (2017)"}],"fun_headline_variants":["QA pair tagging: new scheme, moderate agreement in 3 languages","Annotating question-answer content: multi-lingual scheme, kappa up to 0.67","Automatic QA tagging: decision tree hits 73% on English question types","Semantic tagset for dialogue QA: tested on English, Spanish, Dutch","Layered tagset maps question-answer pairs across three languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cross-linguistic validity of the scheme rests on the untested assumption that annotation guidelines written in English and an English wh-word-to-feature mapping carry over to Spanish and Dutch without language-specific adaptation.","fun_headline_variants_meta":{"raw":{"variants":["QA pair tagging: new scheme, moderate agreement in 3 languages","Annotating question-answer content: multi-lingual scheme, kappa up to 0.67","Automatic QA tagging: decision tree hits 73% on English question types","Semantic tagset for dialogue QA: tested on English, Spanish, Dutch","Layered tagset maps question-answer pairs across three languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001447,"raw_usage":{"total_tokens":5733,"prompt_tokens":755,"completion_tokens":4978,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":371,"completion_tokens_details":{"reasoning_tokens":4875}},"tokens_in":371,"tokens_out":4978,"duration_ms":37777,"temperature":1.0,"reasoning_tokens":4875,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:29:15.194882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study in which bilingual annotators annotate the same Spanish and Dutch dialogues using only the English guidelines and wh-word-to-feature mapping, compared with a second pass using language-specific adapted guidelines, would settle the transfer question: if feature agreement on the English-guideline pass is at or near chance for either language, the cross-linguistic claim fails.","supporting_citations":[{"cited_title":"Shriberg, Rebecca","cited_arxiv_id":null,"evidence_quote":"Provides the dialogue-act question types and automatic tagging setting that the scheme extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DAMSL multi-layer dialogue annotation approach on which the layered question-answer structure builds."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the wh-word-to-feature mapping adapted for the feature layer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Original source of the wh-word-to-feature mapping as cited by the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Spanish CallFriend telephone conversation corpus used to test the scheme."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Spoken Dutch Corpus telephone conversations used for the Dutch test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SCoSE English face-to-face conversation corpus used for the English test."}],"review_version":1}