REVIEW 3 major objections 4 minor 18 references
Toward Dialogue Modeling: A Semantic Annotation Scheme for Questions and Answers
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An annotation scheme with five question types and seven answer types labels question-answer pairs in English, Spanish, and Dutch with moderate human agreement, and a decision tree already beats the majority baseline.
desk verdict A modest, honest annotation resource for question-answer pairs, but the cross-linguistic claim runs ahead of the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the type-feature pair: every question gets a question type, every wh- or disjunctive question additionally gets a feature tag naming the semantic role of the questioned constituent (location, time, agent, reason, and so on), and the question type restricts which answer tags are possible. A wh-word-to-feature mapping adapted from earlier speech processing work turns wh-words such as 'where' into feature cues, and a precedence order (wh-, disjunctive, yes/no, completion suggestion, phatic) lets annotators resolve ambiguous cases by checking the most syntactically specific type first. This combination lets a few tags cover a wide range of dialogue phenomena while keeping the annotation task tractable.
What would settle it
A controlled study in which bilingual annotators annotate the same Spanish and Dutch dialogues using only the English guidelines and wh-word-to-feature mapping, compared with a second pass using language-specific adapted guidelines, would settle the transfer question: if feature agreement on the English-guideline pass is at or near chance for either language, the cross-linguistic claim fails.
Extended reading notes
Core claim
The central claim is that the content and discourse contribution of a question-answer pair can be captured by a small layered tagset: a question type, an optional semantic-role feature for the questioned constituent, and an answer type that is constrained by the question type. On 701 questions from spoken corpora in English, Spanish, and Dutch, human annotators applied the scheme with observed agreement 0.73 for question types and 0.90 for features, corresponding to Cohen's kappa of 0.63 and 0.67; answer types reached observed agreement 0.59 and kappa 0.49. For automatic annotation, a decision tree using surface features such as wh-word presence, inversion, tag questions, and utterance length classifies English question types with accuracy 0.73 and macro F1 0.58. The hardest class is phatic questions, which are difficult to separate from other question types on short utterances.
Load-bearing premise
The cross-linguistic validity of the scheme rests on the untested assumption that annotation guidelines written in English and an English wh-word-to-feature mapping carry over to Spanish and Dutch without language-specific adaptation.
Editorial extensions
If this is right
- A dialogue system could use the type-feature representation as a compact preprocessing layer: knowing that a question is a wh-question about location or an uncertainty answer narrows the space of appropriate responses.
- Automatic question type tagging is feasible with shallow features alone, since the decision tree's accuracy of 0.73 and F1 of 0.58 exceed the majority baseline by a wide margin, and more training data may push the neural models past this level.
- The moderate agreement scores on English, Spanish, and Dutch suggest that the same tagset can be applied across languages, and the documented disagreement patterns, especially around phatic questions, give concrete targets for improving annotation guidelines.
- Answer type restrictions derived from question types give a systematic taxonomy for evaluating dialogue systems: a good response should match the expected answer type for the question asked.
Reading between the lines
- The English wh-word-to-feature mapping may be more language-general than the paper claims, since many Spanish and Dutch wh-words carry the same semantic-role cues as English; a direct test on Spanish and Dutch wh-questions would show whether transfer holds or needs adaptation.
- Because most decision-tree errors came from short utterances misclassified as phatic, adding discourse context or prosodic features could improve automatic annotation more than adding data alone.
- If the scheme is combined with dialogue act taxonomies, the answer types could serve as a bridge between semantic content and dialogue state, for instance treating an uncertainty answer to a yes/no question as a cue for clarification or repair.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a multi-layer semantic annotation scheme for question-answer pairs in conversation, consisting of five question types (YN, WH, CS, PQ, DQ), seven answer types, and feature tags (semantic roles) for wh- and disjunctive questions. The authors describe corpus-driven development, write annotation guidelines, apply the scheme to English (SCoSE), Spanish (CallFriend), and Dutch (CGN), report inter-annotator agreement on a 690-utterance English subset, and run decision-tree, bag-of-words, and RNN classifiers for English question-type prediction. The reported agreement is moderate (kappa = 0.63 question, 0.67 features, 0.49 answers), and the decision tree outperforms the majority baseline (accuracy 0.73 vs 0.47; macro-F1 0.58 vs 0.31).
Significance. If the scheme is validated, it would be a useful, small tagset for annotating question-answer content in dialogue and for training automatic dialogue model components. The paper's strengths are its explicit and public annotation materials, its corpus-driven design grounded in three languages, the inclusion of a confusion matrix and feature set, and the honest reporting of disagreement categories. These make the resource easy to reuse and critique. The main caveat is that the cross-linguistic validity is currently asserted rather than demonstrated, and the ML results are partly powered by features that are definitional for the question types.
major comments (3)
- [§4.1.3, Table 4] The inter-annotator agreement that supports the scheme is computed on 690 SCoSE (English) utterances, yet the paper's contribution is stated as applying the scheme to dialogues in English, Spanish, and Dutch (Abstract). No per-language kappa is reported for CallFriend or CGN, and the annotation guidelines (Section 4.1.1) and wh-word-to-feature mapping (Section 3.2, Table 2) are English-specific. The Spanish and Dutch annotations are therefore exploratory, not evidence of cross-linguistic validity. The authors should either present per-language agreement with confidence intervals and language-specific guideline adaptations, or restrict the validation claim to English and describe the other two as a pilot.
- [§4.2, Table 5] Several hand-designed features encode the formal definitions of the question types: has_wh, has_or, has_inversion, has_tag, and has_cliche are the surface cues used in the manual precedence order (Section 4.1.1) to identify WH, DQ, YN, and PQ. The decision tree's accuracy of 0.73 (F1 0.58) versus majority baseline 0.47 (F1 0.31) is therefore partly a check that these cues are learnable, not evidence that the semantic content of the QAP tagset is captured. I do not see this as invalidating the scheme, but the ML claim should be rephrased, and an ablation leaving out definitional cues would make the contribution of context features such as last_utt_similar measurable.
- [§3.3, Table 3] The answer layer is not operationalized to the same standard as the question layer. The text says overlapping tags are decided based on context and prosody but gives no decision rules or worked examples for distinguishing PA/NA from DA, or UA from UT, and Table 3 assigns most answer types to all question types with no eligibility constraints. This likely contributes to the low answer kappa (0.49, Table 4), but the paper does not report which confusions drove it. The answer layer should be strengthened with explicit decision procedures or acknowledged as a preliminary draft.
minor comments (4)
- [§4.1.2] The relationship between the 4,939-utterance subset, the 3,578-utterance gold standard, the 690 jointly annotated utterances, and the ML test set (184 questions) is not stated; please clarify the annotation protocol and data splits.
- [§4.2, Table 6] The test set contains 184 questions but the per-class support for DQ is 3 and CS is 1, making macro-F1 noisy; report per-class precision, recall, and standard errors.
- [§3.1 and References] There are minor inconsistencies in notation: 'SCoSe/Amy' in Section 3.1 should be 'SCoSE/Amy', and the citation 'ELAN (2017)' should appear consistently with the reference list entry.
- [§4.1.3] The statement that 'the agreement values obtained for question types were over 0.6 (for all annotators combined)' is ambiguous because Table 4 reports a single kappa of 0.63; specify whether this is averaged over pairwise annotator combinations and report the range.
Circularity Check
Decision-tree accuracy partly re-encodes the scheme's own formal cues; the annotation scheme itself is independently validated.
-
self definitional
[Section 4.2, Table 5 (decision-tree features) vs. Section 3.1 (question-type definitions)]
"has wh Contains a wh-constituent; has or Contains the word “or”; has tag Contains a tag (‘isn’t it’, ‘right’); has cliche Contains a phatic marker (‘you know?’, ‘really?’)... The features are quite superficial and do not take into account the discourse context of a question. Still, the algorithm achieves an accuracy score of 0.73 and an F1-score of 0.58, outperforming our majority-class baseline algorithm."
The decision tree's informative features are the formal cues that Section 3.1 uses to define the question types: WH questions are prototypically identified by a wh-word, disjunctive questions by a disjunction, phatic questions by tag or cliché markers, and completion suggestions by similarity to an incomplete previous utterance. Feeding these definitional cues into a decision tree and reporting above-baseline accuracy therefore partially re-encodes the tagset's own criteria rather than testing an independent linguistic hypothesis. The reduction is not complete: labels also require semantic and pragmatic judgments, and the authors disclose that the features are superficial.
full rationale
The central contribution is not circular: the annotation scheme was developed from corpus data, applied to external corpora (SCoSE, CallFriend, CGN), and evaluated by human inter-annotator agreement (kappa 0.63 for questions, 0.67 for features, 0.49 for answers). That agreement is independent of the ML features and does not reduce to any fitted input. The only partial reduction I find is in the automatic-annotation proof-of-concept: Table 5's hand-designed features overlap with the formal properties used in Section 3.1 to define the question types, so the reported accuracy of 0.73 / F1 0.58 partly reflects the typology's own formal correlates. The authors transparently call the features superficial, and the experiment is framed as preliminary, so this is a minor self-definitional overlap rather than a load-bearing circularity. The wh-word-to-feature mapping cited from Boritchev (2017) is adapted from Jurafsky and Martin (2000), an external source, so the self-citation is not the load-bearing element. The untested transfer of English-only guidelines to Spanish and Dutch is a genuine scope and validity limitation, but it is not a circularity.
Assumptions & free parameters
free parameters (1)
- Last utterance similarity threshold =
50% word overlap
assumptions (3)
- domain assumption Question type categories (YN, WH, CS, PQ, DQ) are exhaustive for natural dialogue and distinguishable by formal/functional criteria.
- ad hoc to paper The wh-word-to-feature mapping from English (Boritchev 2017) transfers to Spanish and Dutch.
- domain assumption Answer types are determined by question types (Table 3).
Cite this review
Pith. "Pith review of Toward Dialogue Modeling: A Semantic Annotation Scheme for Questions and Answers." pith.science (2026). https://pith.science/paper/IBNV46BW
@misc{pith2026190809921,
author = {Pith},
title = {Pith review of: Toward Dialogue Modeling: A Semantic Annotation Scheme for Questions and Answers},
year = {2026},
howpublished = {\url{https://pith.science/paper/IBNV46BW}},
note = {Machine review of arXiv:1908.09921}
}
read the original abstract
The present study proposes an annotation scheme for classifying the content and discourse contribution of question-answer pairs. We propose detailed guidelines for using the scheme and apply them to dialogues in English, Spanish, and Dutch. Finally, we report on initial machine learning experiments for automatic annotation.
Reference graph
Works this paper leans on
-
[1]
James Allen and Mark Core. 1997. Draft of DAMSL : Dialog act markup in several layers. https://www.cs.rochester.edu/research/speech/damsl/RevisedManual/, accessed January 22, 2019
work page 1997
-
[2]
Maria Boritchev. 2017. Approaching dialogue modeling in a dynamic framework . Master's thesis, Universit\'e de Lorraine
work page 2017
-
[3]
Alexandra Canavan and George Zipperlen. 1996. https://catalog.ldc.upenn.edu/LDC96S58 CALLFRIEND , Spanish-Non-Caribbean Dialect ( LDC Catalog Number: LDC96S58 )
work page 1996
-
[4]
Jacob Cohen. 1960. https://doi.org/10.1177/001316446002000104 A coefficient of agreement for nominal scales . Educational and Psychological Measurement, 20(1):37--46
-
[5]
ELAN (version 5.2) . 2017. The Language Archive, Max Planck Institute for Psycholinguistics, Nijmegen , The Netherlands. https://tla.mpi.nl/tools/tla-tools/elan/
work page 2017
-
[6]
Alice F. Freed. 1994. The form and function of questions in informal dyadic conversation. Journal of Pragmatics, 21(6):621 -- 644
work page 1994
-
[7]
Art Graesser, Vasile Rus, and Zhiqiang Cai. 2008. Question classification schemes. In Proceedings of the Workshop on Question Generation
work page 2008
-
[8]
Daniel Jurafsky and James H. Martin. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 1st edition. Prentice Hall PTR, Upper Saddle River, NJ, USA
work page 2000
Show all 18 references
-
[9]
Richard Landis and Gary G
J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159--174
1977
-
[10]
Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING '02 Proceedings of the 19th international conference on computational linguistics, pages 1--7
2002
-
[11]
Neal Norrick. 2017. https://ca.talkbank.org/access/SCoSE.html SCoSE part 1: Complete conversations . English Linguistics, Department of English at Saarland University
2017
-
[12]
Nelleke Oostdijk. 2001. The design of the Spoken Dutch Corpus . Language and Computers, 36:105--112
2001
-
[13]
Ross Quinlan
J. Ross Quinlan. 1986. Induction of decision trees. Machine learning, 1(1):81--106
1986
-
[14]
Gunter Senft. 2009. Phatic communion. In Gunter Senft, Jan-Ola \"O stman, and Jef Verschueren, editors, Culture and language use, pages 226--233. John Benjamins Publishing, Amsterdam/Philadelphia
2009
-
[15]
Han Sloetjes and Peter Wittenburg. 2008. Annotation by category: ELAN and ISO DCR . In LREC
2008
-
[16]
Shriberg, Rebecca
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth. Shriberg, Rebecca. Bates, Daniel. Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational lingu...
2000
-
[17]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[18]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.