Pith. sign in

REVIEW 3 major objections 6 minor 27 references

MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversations

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A dialog act annotation scheme designed for open-domain human-machine spoken conversation reaches 94% annotator agreement and 0.79 F1 on automatic prediction.

desk verdict MIDAS is a legitimate resource paper for human-machine dialog act annotation, but the evaluation needs a documented conversation-level split before the headline F1 is fully trustable. read the letter →

arxiv 1908.10023 v1 pith:WCZF7ADR submitted 2019-08-27 cs.CL

Dian Yu , Zhou Yu This is my paper · ORCID
classification cs.CL
keywords dialogactannotationhuman-machineconversationmulti-labelclassificationopen-domaindialoguehierarchicalschemespokensystemBERTtransferlearningutterancesegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing dialog act schemes were built for human-human dialogue, in which both sides understand language perfectly. This paper argues that machines, which understand less, need an annotation vocabulary matched to their limitations, and proposes MIDAS, a hierarchical multi-label dialog act scheme for open-domain human-machine spoken conversation. The authors annotate 24K utterances from real social-bot conversations: two annotators reach $\kappa = 0.94$, and a BERT-based transfer-learning classifier reaches 0.79 F1. The claim is that machine-directed speech differs enough from human-human talk to deserve its own tags, and that those tags can be defined, reliably assigned, and automatically predicted.

What carries the argument

The load-bearing object is the MIDAS scheme itself: 23 dialog act tags arranged as leaf nodes under semantic request and functional request subtrees, with multi-label support capped at two tags. The hierarchy guides annotators to the right tag; the multi-label rule captures utterances that do more than one thing, such as a negative answer that is also a task command; context completion lets annotators resolve ellipsis before tagging; and the priority rule keeps only the two tags most useful for dialog planning. On the prediction side, the mechanism is a BERT encoder fine-tuned on in-domain unlabeled conversation before supervised multi-label training, with context formed by concatenating the previous system segment, previous user segment, and current user segment.

What would settle it

Annotate a held-out sample of user turns twice: once after automatic ASR segmentation and once after manual human segmentation, then measure MIDAS label agreement on the two versions. If classifier F1 or annotator kappa on manually segmented units falls clearly below the reported 0.79 and $\kappa = 0.94$, the usability claim is undermined.

Watch

Extended reading notes

Core claim

MIDAS organizes dialog acts into two trees, a semantic request type and a functional request type, with classes, categories, and 23 leaf-node tags; utterances receive up to two tags, selected by a priority order that favors answer, command, opinion, statement non-opinion, and question. Annotators reconstruct elliptical meaning from context before tagging, though the original text stays unchanged. Applied to 24K segments from Gunrock conversations, the scheme is reliable enough for two annotators to agree at $\kappa = 0.94$, and a multi-label classifier trained with in-domain fine-tuned BERT and textual context from previous system and user turns predicts the tags at 0.79 F1.

Load-bearing premise

The load-bearing premise is that the automatic ASR segmentation model, at 84.43% micro-F1, produces usable utterance units; if its boundaries are systematically wrong, the $\kappa = 0.94$ agreement and the 0.79 F1 describe units that do not match the user's intended utterance boundaries.

Editorial extensions

If this is right

  • A dialog system can use MIDAS tags directly for policy decisions: follow a proposed topic, answer a question, execute a command, or recognize a complaint.
  • The 24K annotated segments provide a training resource for open-domain dialog act prediction that does not rely on the poorly transferring human-human Switchboard labels.
  • The 0.79 F1 result indicates automatic dialog act prediction on raw ASR output is feasible for open-domain social bots, not just task-oriented systems.
  • Separating semantic request tags from functional request tags gives a system two orthogonal signals: what topic the user wants and what discourse move the user is making.
  • The reported 47.38% accuracy of a BERT model trained on SWBD-DAMSL and tested on human-machine conversation motivates replacing human-human schemes with machine-directed ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the segmentation model is the main identified error source, jointly training utterance segmentation and dialog act prediction, as the paper lists for future work, would likely raise the 0.79 F1; the paper does not demonstrate this.
  • The priority order answer > command > opinion > statement > question encodes a dialog-management reflex: detect compliance first, then topic shifts; that ordering could be evaluated downstream by measuring task success under alternative label-selection rules.
  • Nothing in the tag definitions ties MIDAS to Gunrock, so the scheme should transfer to other voice assistants; a cross-bot annotation study would test that portability.
  • Automating the annotators' ellipsis completion step, rather than leaving it to humans, could improve prediction on short answers such as 'the great gatsby'; the paper leaves this to future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces MIDAS, a hierarchical multi-label dialog act annotation scheme designed for open-domain human-machine spoken conversations. The authors describe the scheme's structure, collect and annotate 24K utterances from real Gunrock (Alexa Prize) conversations, report an inter-annotator agreement of kappa = 0.94 on a pilot subset, and train multi-label dialog act prediction models with transfer learning, achieving a micro-F1 score of 0.79. The paper also releases the annotated data and trained models.

Significance. If the results are reliable, MIDAS would be a useful resource for dialog system development and discourse analysis. The paper has notable strengths: it uses real user-system interactions rather than scripted data, reports results averaged over six random seeds, provides per-tag counts and annotated examples in the appendix, and publicly releases the data and code. However, two evaluation concerns—the potential for training/test leakage through the context representation and the reliance on imperfect automatic segmentation—mean that the headline numbers need further validation before the practical applicability of the scheme is fully established.

major comments (3)
  1. [Section 6, Setting] The paper reports 10.3K training and 2.6K test user segments but does not state that the split is at the conversation level. Because the context representation in Section 5.2 includes the previous system unit and previous user unit from the same dialogue, a random segment-level split can place a test utterance in the same conversation as training utterances. The model can then exploit conversation-specific lexical patterns or near-duplicate adjacent turns, inflating the central F1=0.79 result. The authors must either specify a conversation-level split and re-run the experiments, or provide a controlled experiment showing that overlapping contexts do not affect the reported results.
  2. [Section 4 and Section 7] All dialog act annotation and predictions are performed on automatic segmentation results from a model with 84.43% micro-F1, and the paper itself notes that some incorrectly segmented units led to inaccurate dialog act prediction. Consequently, the reported kappa and F1 characterize agreement and performance on machine-produced units rather than on true utterance boundaries. The authors should quantify how segmentation errors affect tag distribution and model performance, or at least delimit the claims to the automatic-segmentation setting.
  3. [Section 4] The inter-annotator agreement kappa = 0.94 is computed on only 1,185 pilot utterances, after which the two annotators annotated the remaining 24K utterances separately. Thus, the reliability of the final corpus is not directly measured. The paper should provide at least a small random sample of double-annotated utterances from the main annotation phase, or explicitly acknowledge this as a limitation.
minor comments (6)
  1. [Introduction] The phrase "inter-annotated agreement" should be "inter-annotator agreement" (also appears in Section 4).
  2. [Section 2] The text says "two annotators reached 0.94 in Kapa"; this should be "kappa".
  3. [Table 1] The example contains "the great gastby", which should be "the great gatsby".
  4. [Section 8] The word "syntatic" should be "syntactic", and "fine-turning" should be "fine-tuning".
  5. [Section 7] The phrase "complimentary information" should be "complementary information".
  6. [Section 3.3] The maximum of two tags per utterance is a somewhat arbitrary constraint; a more detailed discussion of its trade-off would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MIDAS is an empirical resource paper whose kappa and F1 results are measured, not derived from the scheme's definitions.

full rationale

The paper makes no derivation chain that reduces a prediction to its own inputs. The central claims are empirical: two annotators reached kappa = 0.94 on 1,185 utterances, and a classifier trained on 24K MIDAS-annotated utterances reached micro-F1 = 0.79 on a held-out test set. These are measurements on data annotated with the proposed scheme, which is the normal, non-circular mode of validation for a resource paper. The hand-built SWBD-DAMSL-to-MIDAS mapping is used only for auxiliary transfer learning and is not claimed to force the headline F1 result. The BERT and LSTM models are standard architectures with no parameter fitted to the test labels, and the context representation is defined from previous turns rather than from the target label. Potential threats such as automatic ASR segmentation errors or a non-conversation-level train/test split would be external validity or evaluation concerns, not circularity: neither is a step in which an output is equivalent to an input by definition or by self-citation. No load-bearing argument reduces to an author self-citation; the cited Gunrock data paper supplies the corpus, not the conclusion. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

Most of what the paper contributes is the taxonomy itself, so the ledger records the main design and data assumptions rather than hidden free parameters. The fitted threshold (0.5) and the two-tag cap directly affect reported scores; the automatic segmentation assumption is the biggest external dependency; the taxonomy's validity is asserted rather than benchmarked downstream.

free parameters (2)
  • Multi-label decision threshold = 0.5
    Set empirically in Section 5.1; controls whether a second predicted label is kept and therefore directly affects the reported precision, recall, and F1.
  • Maximum number of dialog act tags per utterance = 2
    Hand-chosen in Section 3.3 to keep annotation and prediction tractable; the paper cites Bunt's evidence that functional segments average four to five functions, so this cap is an ad hoc simplification that bounds label coverage.
assumptions (4)
  • domain assumption Automatic ASR segmentation approximates true utterance boundaries closely enough for annotation and evaluation.
    Section 4 applies a segmentation model with 84.43% micro-F1 to the full corpus and performs all annotation and prediction on its output; a systematic segmentation error would distort tags and F1.
  • ad hoc to paper Two dialog act labels per utterance are sufficient to capture user intent.
    Section 3.3 restricts utterances to at most two labels and uses a priority order (answer, command, opinion, statement non-opinion, question) to discard others, despite DIT++ evidence cited in the paper that segments average four to five functions.
  • domain assumption The semantic/functional two-tree taxonomy with 23 tags is the right ontology for human-machine dialog understanding.
    The taxonomy is the paper's central proposal, but the paper provides no downstream comparison (for example, end-to-end dialog quality with and without MIDAS) to show this ontology is better than SWBD-DAMSL or Khatri's 14-tag scheme.
  • domain assumption Ellipsis completion by annotators does not bias label assignment.
    Section 3.1 asks annotators to fill in ellipsis using context before tagging, but labels are assigned to completed interpretations while the original surface form is unchanged; the effect of this transformation on annotation reliability is not measured.
invented entities (1)
  • MIDAS tag taxonomy (23 dialog act tags, including new human-machine labels such as invalid command and nonsense) independent evidence
    purpose: Provides the label vocabulary for understanding user utterances in human-machine social dialog.
    The taxonomy is the paper's contribution rather than a hidden postulate; it has a reproducible handle because annotators applied it with kappa = 0.94 on a pilot set and the appendix lists per-tag counts and examples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversations." pith.science (2026). https://pith.science/paper/WCZF7ADR

@misc{pith2026190810023,
  author       = {Pith},
  title        = {Pith review of: MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WCZF7ADR}},
  note         = {Machine review of arXiv:1908.10023}
}
read the original abstract

Dialog act prediction is an essential language comprehension task for both dialog system building and discourse analysis. Previous dialog act schemes, such as SWBD-DAMSL, are designed for human-human conversations, in which conversation partners have perfect language understanding ability. In this paper, we design a dialog act annotation scheme, MIDAS (Machine Interaction Dialog Act Scheme), targeted on open-domain human-machine conversations. MIDAS is designed to assist machines which have limited ability to understand their human partners. MIDAS has a hierarchical structure and supports multi-label annotations. We collected and annotated a large open-domain human-machine spoken conversation dataset (consists of 24K utterances). To show the applicability of the scheme, we leverage transfer learning methods to train a multi-label dialog act prediction model and reach an F1 score of 0.79.

Figures

Figures reproduced from arXiv: 1908.10023 by the authors.

Figure 1
Figure 1. Semantic request tree. Scheme types, classes, categories, and sub-categories are in green, blue, purple, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Functional request tree. We remove class and category nodes if there is only one label under them. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 19 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jan Alexandersson, Bianka Buschbeck-Wolf, Tsutomu Fujinami, Michael Kipp, Stephan Koch, Elisabeth Maier, Norbert Reithinger, Birte Schmitz, and Melanie Siegel. 1998. Dialogue acts in verbmobil-2 - second edition. In DFKI Saarbrücken

    work page 1998

  4. [4]

    Harry Bunt. 1997. Dynamic interpretation and dialogue theory

    work page 1997

  5. [5]

    Harry Bunt. 2009. The dit++ taxonomy for functional dialogue markup. In AAMAS 2009 Workshop, Towards a Standard Markup Language for Embodied Dialogue Acts, pages 13--24

    work page 2009

  6. [6]

    Harry Bunt, Jan Alex, Jean Carletta, Jae woong Choe, Alex Chengyu Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-belis, Laurent Romary, Claudia Soria, and David Traum. 2010. Towards an iso standard for dialogue act annotation. In 1st Proceedings of Alexa Prize

    work page 2010

  7. [7]

    Harry Bunt, Volha Petukhova, and Alex Fang. 2017. Revisiting the iso standard for dialogue act annotation. In Proceedings 13th Joint ISO - ACL Workshop on Interoperable Semantic Annotation (isa-11), pages 37--50. Research Unit(s) information for this publication is provided by the author(s) concerned

    work page 2017

  8. [8]

    Chun-Yen Chen, Dian Yu, Weiming Wen, Yi Mang Yang, Jiaping Zhang, Mingyang Zhou, Kevin Jesse, Austin Chau, Antara Bhowmick, Shreenath Iyer, Giritheja Sreenivasulu, Runxiang Cheng, Ashwin Bhandare, and Zhou Yu. 2018. Gunrock: Building a human-like social bot by leveraging large scale real user data. In 2nd Proceedings of Alexa Prize

    work page 2018

Show all 27 references
  1. [9]

    Eunah Cho, Kevin Kilgour, Jan Niehues, and Alexander H. Waibel. 2015. Combination of nn and crf models for joint detection of punctuation and disfluencies. In INTERSPEECH

    2015

  2. [10]

    Core and James F

    Mark G. Core and James F. Allen. 1997. Coding dialogs with the damsl annotation scheme. In Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, pages 28--35

    1997

  3. [11]

    Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011

    2011

  4. [12]

    Jacob Devlin, Ming-Wei Chan, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  5. [13]

    Favre, D

    B. Favre, D. Hakkani-Tur, S. Petrov, and D. Klein. 2008. Efficient sentence segmentation using syntactic features. In 2008 IEEE Spoken Language Technology Workshop, pages 77--80

    2008

  6. [14]

    Godfrey, Edward C

    John J. Godfrey, Edward C. Holliman, and Jane McDaniel. 1992. Switchboard: telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517--520 vol.1

    1992

  7. [15]

    Dan Jurafsky, Liz Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual . Technical Report Draft 13, University of Colorado, Institute of Cognitive Science

    1997

  8. [16]

    Chandra Khatri, Rahul Goel, Behnam Hedayatni, Angeliki Metanillou, Anushree Venkatesh, Raefer Gabriel, and Arindam Mandal. 2018. Contextual topic modeling for dialog systems. In IEEE 2018 Spoken Language Technology

    2018

  9. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2014. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . CoRR, abs/1412.6980

    2014 arXiv

  10. [18]

    Klejch, P

    O. Klejch, P. Bell, and S. Renals. 2017. Sequence-to-sequence models for punctuated transcription combining lexical and acoustic features. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5700--5704

    2017

  11. [19]

    Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. Using context information for dialog act classification in dnn framework. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2170--2178. Association for Computational Linguistics

    2017

  12. [20]

    Minh - Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. http://arxiv.org/abs/1508.04025 Effective approaches to attention-based neural machine translation . CoRR, abs/1508.04025

    2015 arXiv

  13. [21]

    Stefano Mezza, Alessandra Cervone, Evgeny Stepanov, Giuliano Tortoreto, and Giuseppe Riccardi. 2018. Iso-standard domain-independent dialogue act tagging for conversational agents. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3539--35...

    2018

  14. [22]

    Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)

    2018

  15. [23]

    Stephan Peitz, Markus Freitag, Arne Mauser, and Hermann Ney. 2011. Modeling punctuation prediction as machine translation. In IWSLT

    2011

  16. [24]

    Lina Maria Rojas - Barahona, Milica Gasic, Nikola Mrksic, Pei - Hao Su, Stefan Ultes, Tsung - Hsien Wen, and Steve J. Young. 2016. http://arxiv.org/abs/1610.04120 Exploiting sentence and context representations in deep neural models for spoken language understanding . CoRR, ab...

    2016 arXiv

  17. [25]

    Aditya Siddhant, Anuj Kumar Goyal, and Angeliki Metallinou. 2018. http://arxiv.org/abs/1811.05370 Unsupervised transfer learning for spoken language understanding in intelligent agents . CoRR, abs/1811.05370

    2018 arXiv

  18. [26]

    Andreas Stolcke, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. 2000. https://doi.org/10.1162/089120100561737 Dialogue act modeling for automatic tagging and recognition of conve...

    doi

  19. [27]

    Thompson, Anne Anderson, Ellen Gurman Bard, Gwyneth Doherty-Sneddon, Alison Newlands, and Cathy Sotillo

    Henry S. Thompson, Anne Anderson, Ellen Gurman Bard, Gwyneth Doherty-Sneddon, Alison Newlands, and Cathy Sotillo. 1993. The hcrc map task corpus: Natural dialogue for speech recognition. In Proceedings of the Workshop on Human Language Technology, HLT '93, pages 25--30, Stroud...

    1993

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.