REVIEW 3 major objections 6 minor 35 references
CoMuMDR: Code-mixed Multi-modal Multi-domain corpus for Discourse paRsing in conversations
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that CoMuMDR is the first code-mixed, multi-modal, multi-domain discourse corpus for conversations, and that state-of-the-art parsers perform poorly on it, especially on relation classification.
desk verdict A genuinely new code-mixed discourse corpus with real value as a resource, but the benchmark-based claims about difficulty are not supported by the paper's own numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing artifact is the CoMuMDR corpus: 799 two-party call-center dialogues, transcribed and diarized into 8,811 utterances, manually segmented into elementary discourse units (clauses), and linked into a directed acyclic discourse graph whose edges carry one of nine discourse relation labels derived from Segmented Discourse Representation Theory (SDRT), a theory of how clauses connect through discourse relations, plus a newly added Question-Answer Complaint Pair. The paper also relies on the standard evaluation machinery of F1 scores for link prediction and link+relation classification, alongside inter-annotator metrics (Kappa, Jaccard, span exact/partial match) to characterize the difficulty of the annotation task.
What would settle it
Re-annotate a sample of CoMuMDR with careful adjudication, then measure agreement between the gold and a fresh expert annotation and compute the resulting human upper-bound F1; if that upper bound is close to the reported model scores, the claim of model underperformance is weakened because the task itself would be under-specified.
Extended reading notes
Core claim
The central claim is that CoMuMDR is the first corpus to combine code-mixing, multiple modalities, and multiple domains for discourse parsing in two-party dialogues, and that it exposes a real gap: state-of-the-art discourse parsers trained on this corpus achieve markedly lower F1 in relation classification than they do on STAC or Molweni, while link prediction is comparatively less affected. The authors attribute the gap to code-mixed language, imperfect ASR transcription and diarization, and domain variation, and they provide per-relation error analyses showing that rare relation types are rarely or never predicted.
Load-bearing premise
The benchmark conclusion assumes the gold annotations are reliable enough to make low model scores evidence of task difficulty, even though annotators disagree substantially on relation labels (Relationship Kappa is 0.319).
Editorial extensions
If this is right
- Model scores on CoMuMDR provide a lower-bound benchmark for discourse parsing in realistic code-mixed, speech-derived conversations.
- The nine-relation scheme, including Question-Answer Complaint Pair, offers a compact adaptation of SDRT that can be reused for other two-party customer-service corpora.
- Per-relation F1 tables show that rare relations receive zero F1 across models, so progress on this corpus depends on more data or a different learning signal for those classes.
- Because the released audio features accompany the transcripts, CoMuMDR supports testing whether acoustic information improves discourse parsing over text alone.
Reading between the lines
- Interpreted charitably, the low inter-annotator agreement suggests that the boundary between relations like Acknowledgment and Question-Answer Pair is intrinsically ambiguous; a controlled experiment that asks human annotators to choose among the nine labels for the same spans could reveal how much of the model gap is label noise rather than model limitation.
- The paper merges several SDRT labels into Continuation, so CoMuMDR cannot evaluate distinctions that matter in other genres; direct F1 comparisons with STAC and Molweni are therefore approximate rather than apples-to-apples.
- A natural follow-up is to translate CoMuMDR transcripts into English-only text and retrain the same parsers, which would separate the effect of code-mixing from the effect of domain shift.
- The audio-conditioned direction the paper flags is directly testable: if a parser that consumes the released audio features does not beat the text-only parser, the bottleneck likely lies in the annotation scheme rather than in the missing modality.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. CoMuMDR is presented as a new Hindi-English code-mixed, multi-modal (audio plus text), multi-domain discourse corpus of two-party customer call-center conversations, annotated with nine SDRT-inspired discourse relations at the EDU-span level. The paper describes corpus construction, reports inter-annotator agreement, and evaluates five discourse parsers plus GPT-4o under English-only and multilingual embeddings, comparing against STAC and Molweni. The headline claim is that state-of-the-art parsers underperform on CoMuMDR, demonstrating the difficulty of code-mixed multi-domain discourse parsing.
Significance. The resource fills a genuine gap: existing conversational discourse corpora are monolingual, written, single-domain, and not audio-based. The authors are transparent about the low inter-annotator agreement, provide per-relation breakdowns, and plan to release code, transcriptions, embeddings, and audio features, which is a useful commitment. Running the same five parsers on CoMuMDR, STAC, and Molweni is also a reasonable first benchmark. However, the paper's principal claim that SoTA models underperform on CoMuMDR is currently not supported with sufficient rigor, because the comparison is confounded by annotation reliability, label imbalance, link-distance distributions, and missing significance testing.
major comments (3)
- [§3, Table 2] The benchmark interpretation in §4 rests on CoMuMDR being a reliable gold standard, but the reported Relationship Kappa is 0.3190 and Structured Kappa is 0.4044, and the Limitations section itself concedes that label confusion and EDU-span differences produced low agreement. The paper does not report post-adjudication agreement or any human upper bound on link+relation F1, so it is impossible to tell whether the poor model scores reflect task difficulty or label noise. Please add post-adjudication agreement (gold vs. each annotator), per-relation agreement, and a human-annotation upper bound computed on the same evaluation metric.
- [§4, Tables 3, 4, 8] The claim that SoTA models underperform on CoMuMDR is confounded by label sparsity and link locality. Table 4 shows that every relation with support at or below 32 receives F1 0.00 from most models, whereas the high-support relations reach F1 0.40-0.70, which is comparable to the overall link+relation scores on STAC and Molweni. Table 8 shows that CoMuMDR links are predominantly local (mean distances 0.95-1.07 versus 0.67-1.86 for STAC/Molweni), which explains the higher link-only F1 on CoMuMDR (e.g., multilingual SSP-BERT 0.9531 vs. 0.7250 and 0.8205). The comparison should be repeated with macro-averaged scores conditioned on relation support, with an evaluation restricted to shared high-frequency relations, and/or with link-distance-matched subsets.
- [§4, Results paragraph] The sentence 'For both the settings, CoMuMDR scores are lowest across all the models' is contradicted by the paper's own tables. English-only Hierarchical link+relation F1 is 0.5786 on CoMuMDR versus 0.5704 on STAC, English-only SSP-BERT is 0.5675 versus 0.5623, and GPT-4o link+relation F1 on CoMuMDR (0.2808) exceeds both STAC (0.2729) and Molweni (0.1474) in Table 5. Please qualify the claim and provide confidence intervals or significance tests, since differences below 0.01 on an 81-dialogue test set are within noise.
minor comments (6)
- [Abstract and §1] The phrase 'The current discourse parsing datasets ... consists' should be 'consist'; also consider making the relation between the audio modality and the text-only baselines explicit in the abstract.
- [Table 1] The row 'Annotator Metrics Kappa Kappa Kappa, Jaccard' is hard to parse; spell out which metrics apply to which corpus and provide the STAC and Molweni Kappa values for direct comparison.
- [§3 and Table 6] The relation 'Diarization Continuation' is introduced as an annotation category and appears in Figure 7, but it is not listed in Table 6; make its role and exclusion from evaluation consistent across the main text, tables, and appendix.
- [Appendix F] The GPT-4o prompt instructs the model to use 'dia-continuation' for same-speaker split utterances, but this relation is excluded from scoring; clarify how such predictions were handled in the reported numbers.
- [Appendix C.5, Algorithm 2] The threshold parameter for span partial match is not given a value; state the threshold so that the reported agreement scores are reproducible.
- [Figure 1] The relation arcs and labels (QACPAck, dc) are difficult to follow; please redraw with clear arrows and a legend.
Circularity Check
No significant circularity: the corpus and its benchmark results are empirically constructed and externally testable, with no fitted parameter renamed as a prediction.
full rationale
This is a resource paper, not a derivation. The central claims are that CoMuMDR is a new code-mixed, multi-modal, multi-domain discourse corpus and that existing discourse parsers and GPT-4o perform poorly on it. Neither claim reduces to its inputs. The corpus was built through ASR transcription, diarization, anonymization, and manual annotation; no gold links or discourse labels were fitted to, or derived from, the downstream model predictions. The baselines are external published systems (Shi and Huang 2019; Liu and Chen 2021; Wang et al. 2021; Yu et al. 2022; Chi and Rudnicky 2022) implemented from their papers and trained from scratch on STAC, Molweni, and CoMuMDR, so the F1 comparisons are external benchmark results rather than constructional identities. The only overlapping-author citation is the ASR system (Verma et al., 2023) used to produce transcripts; this is a pipeline component and is not invoked to justify the underperformance claim, nor does it force any benchmark outcome by definition. The reported low inter-annotator agreement (Structured Kappa 0.4044, Relationship Kappa 0.3190) and the potential confounds in Tables 3 and 4 (rare labels with zero F1, mostly local links) concern the validity or interpretability of the benchmark comparison, i.e., correctness or experimental-design risk, not circularity. Similarly, the statement that 'For both the settings, CoMuMDR scores are lowest across all the models' is contradicted by some rows in Table 3 (English-only Hierarchical 0.5786 and SSP-BERT 0.5675 link+relation F1 on CoMuMDR exceed the corresponding STAC scores), but that is a presentational/statistical issue, not circular reasoning. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via citation, and no known result is merely renamed. The derivation chain, such as it is, is therefore self-contained and the results are externally testable.
Assumptions & free parameters
free parameters (1)
- Span partial match threshold =
not reported
assumptions (4)
- domain assumption SDRT is an appropriate annotation scheme for two-party call-center conversations.
- domain assumption Transcribed and diarized text is an adequate proxy for audio for discourse annotation.
- ad hoc to paper Linking an EDU only to the head of a CDU preserves the discourse structure.
- domain assumption The low Kappa values are an acceptable measure of annotation reliability.
invented entities (1)
-
Question-Answer Complaint Pair (QACP) relation
Cite this review
Pith. "Pith review of CoMuMDR: Code-mixed Multi-modal Multi-domain corpus for Discourse paRsing in conversations." pith.science (2026). https://pith.science/paper/OTAZDPSM
@misc{pith2026250608504,
author = {Pith},
title = {Pith review of: CoMuMDR: Code-mixed Multi-modal Multi-domain corpus for Discourse paRsing in conversations},
year = {2026},
howpublished = {\url{https://pith.science/paper/OTAZDPSM}},
note = {Machine review of arXiv:2506.08504}
}
read the original abstract
Discourse parsing is an important task useful for NLU applications such as summarization, machine comprehension, and emotion recognition. The current discourse parsing datasets based on conversations consists of written English dialogues restricted to a single domain. In this resource paper, we introduce CoMuMDR: Code-mixed Multi-modal Multi-domain corpus for Discourse paRsing in conversations. The corpus (code-mixed in Hindi and English) has both audio and transcribed text and is annotated with nine discourse relations. We experiment with various SoTA baseline models; the poor performance of SoTA models highlights the challenges of multi-domain code-mixed corpus, pointing towards the need for developing better models for such realistic settings.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Stergos Afantenos, Eric Kow, Nicholas Asher, and J \'e r \'e my Perret. 2015. https://doi.org/10.18653/v1/D15-1109 Discourse parsing for multi-party chat dialogues . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 928--937, Lisbon, Portugal. Association for Computational Linguistics
-
[4]
Nicholas Asher, Julie Hunter, Mathieu Morey, Benamara Farah, and Stergos Afantenos. 2016. https://aclanthology.org/L16-1432 Discourse structure and dialogue acts in multiparty dialogue: the STAC corpus . In Proceedings of the Tenth International Conference on Language Resources and Evaluation ( LREC '16) , pages 2721--2727, Portoro z , Slovenia. European ...
work page 2016
-
[5]
Nicholas Asher and Alex Lascarides. 2005. https://www.cambridge.org/de/universitypress/subjects/languages-linguistics/computational-linguistics/logics-conversation Logics of Conversation . Cambridge University Press, Cambridge, England, UK
work page 2005
-
[6]
Jiaao Chen and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.454 Controllable conversation generation with conversation structures via diffusion models . In Findings of the Association for Computational Linguistics: ACL 2023, pages 7238--7251, Toronto, Canada. Association for Computational Linguistics
-
[7]
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. https://doi.org/10.1109/JSTSP.2022.3188113 Wavlm: Large-scale self-supervised pre-training for full stack speech proces...
arXiv 2022
-
[8]
Alexander Chernyavskiy and Dmitry Ilvovsky. 2023. https://doi.org/10.18653/v1/2023.sigdial-1.48 Transformer-based multi-party conversation generation using dialogue discourse acts planning . In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 519--529, Prague, Czechia. Association for Computational Linguistics
Show all 35 references
-
[9]
Ta-Chung Chi and Alexander Rudnicky. 2022. https://doi.org/10.18653/v1/2022.sigdial-1.32 Structured dialogue discourse parsing . In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 325--335, Edinburgh, UK. Association for Co...
2022 doi
-
[10]
Sabit Hassan and Malihe Alikhani. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-main.28 D is CG en: A framework for discourse-informed counterspeech generation . In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of ...
2023 doi
-
[11]
Kenneth Heafield. 2011. https://aclanthology.org/W11-2123/ K en LM : Faster and smaller language model queries . In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187--197, Edinburgh, Scotland. Association for Computational Linguistics
2011
-
[12]
Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. https://aclanthology.org/C18-2002 The INCE p TION platform: Machine-assisted and knowledge-oriented interactive annotation . In Proceedings of the 27th International Confer...
2018
-
[13]
Nithin Rao Koluguri, Taejin Park, and Boris Ginsburg. 2021. https://arxiv.org/abs/2110.04410 Titanet: Neural model for speaker representation with 1d depth-wise separable convolutions and global context . Preprint, arXiv:2110.04410
2021 arXiv
-
[14]
Jiaqi Li, Ming Liu, Min-Yen Kan, Zihao Zheng, Zekun Wang, Wenqiang Lei, Ting Liu, and Bing Qin. 2020. https://doi.org/10.18653/v1/2020.coling-main.238 Molweni: A challenge multiparty dialogues-based machine reading comprehension dataset with discourse structure . In Proceeding...
2020 doi
-
[15]
Jiaqi Li, Ming Liu, Bing Qin, and Ting Liu. 2022. A survey of discourse parsing. Frontiers of Computer Science, 16(5):165329
2022
-
[16]
Jiaqi Li, Ming Liu, Bing Qin, Zihao Zheng, and Ting Liu. 2019. https://arxiv.org/abs/1911.03514 An annotation scheme of a large-scale multi-party dialogues dataset for discourse parsing and machine comprehension . Preprint, arXiv:1911.03514
2019 arXiv
-
[17]
Junyi Jessy Li, Kapil Thadani, and Amanda Stent. 2016. https://doi.org/10.18653/v1/W16-3617 The role of discourse units in near-extractive summarization . In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 137--147, Los Ang...
2016 doi
-
[18]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . CoRR, abs/1907.11692
2019 arXiv
-
[19]
Zhengyuan Liu and Nancy Chen. 2021. https://doi.org/10.18653/v1/2021.codi-main.11 Improving multi-party dialogue discourse parsing via domain integration . In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 122--127, Punta Cana, Dominican Republ...
2021 doi
-
[20]
Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021. https://doi.org/10.18653/v1/2021.codi-main.15 DMRST : A joint framework for document-level multilingual RST discourse segmentation and parsing . In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 154-...
2021 doi
-
[21]
Michal Lukasik, Boris Dadachev, Kishore Papineni, and Gon c alo Sim \ o es. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.380 Text segmentation by cross segment attention . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p...
2020 doi
-
[22]
Mann and Sandra A
William C. Mann and Sandra A. Thompson. 1988. https://doi.org/doi:10.1515/text.1.1988.8.3.243 Rhetorical structure theory: Toward a functional theory of text organization . Text - Interdisciplinary Journal for the Study of Discourse, 8(3):243--281
1988 doi
-
[23]
Mary L. McHugh. 2012. https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052 Interrater reliability: the kappa statistic . Biochemia Medica, 22(3):276
2012
-
[24]
Philippe Muller, Stergos Afantenos, Pascal Denis, and Nicholas Asher. 2012. https://aclanthology.org/C12-1115 Constrained decoding for text-level discourse parsing . In Proceedings of COLING 2012 , pages 1883--1900, Mumbai, India. The COLING 2012 Organizing Committee
2012
-
[25]
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. https://openreview.net/forum?id=HkAClQgA- A deep reinforced model for abstractive summarization . In International Conference on Learning Representations
2018
-
[26]
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 G lo V e: Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1532--1543, Doh...
2014 doi
-
[27]
Alan Lee Rashmi Prasad, Bonnie Webber and Aravind Joshi. 2019. https://catalog.ldc.upenn.edu/LDC2019T05 Penn Discourse Treebank Version 3.0 - Linguistic Data Consortium
2019
-
[28]
Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...
2019 doi
-
[29]
Zhouxing Shi and Minlie Huang. 2019. https://doi.org/10.1609/aaai.v33i01.33017007 A deep sequential model for discourse parsing on multi-party dialogues . In AAAI 2019, AAAI'19/IAAI'19/EAAI'19. AAAI Press
2019 doi
-
[30]
Tushar Verma, Atul Shree, and Ashutosh Modi. 2023. https://doi.org/10.21437/Interspeech.2023-757 Asr for low resource and multilingual noisy code-mixed speech . In INTERSPEECH 2023, pages 3242--3246
2023 doi
-
[31]
Ante Wang, Linfeng Song, Hui Jiang, Shaopeng Lai, Junfeng Yao, Min Zhang, and Jinsong Su. 2021. https://doi.org/10.24963/ijcai.2021/543 A structure self-aware model for discourse parsing on multi-party dialogues . In Proceedings of the Thirtieth International Joint Conference ...
2021 doi
-
[32]
Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018. https://doi.org/10.18653/v1/D18-1116 Toward fast and accurate neural discourse segmentation . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 962--967, Brussels, Belgium. Associa...
2018 doi
-
[33]
Chang-Sung Yu. 1986. https://doi.org/10.1002/net.3230160110 Graph theory, by w. t. tutte, encyclopedia of mathematics and its applications, volume 21, addison-wesley publishing company, menlo park, ca., 1984, 333 pp. price: 45.00 . Networks, 16(1):107--108
1986 doi
-
[34]
Nan Yu, Guohong Fu, and Min Zhang. 2022. https://aclanthology.org/2022.coling-1.477 Speaker-aware discourse parsing on multi-party dialogues . In Proceedings of the 29th International Conference on Computational Linguistics, pages 5372--5382, Gyeongju, Republic of Korea. Inter...
2022
-
[35]
Duzhen Zhang, Feilong Chen, and Xiuyi Chen. 2023. https://doi.org/10.18653/v1/2023.acl-long.408 D ual GAT s: Dual graph attention networks for emotion recognition in conversations . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol...
2023 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.