Pith. sign in

REVIEW 4 major objections 6 minor 59 references

CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Synthetic miscommunication turns can be added to benchmark dialogues automatically without breaking downstream performance.

desk verdict Useful augmentation pipeline and dataset, but the 'consistency preserving' promise is asserted without direct verification. read the letter →

arxiv 2412.07515 v1 pith:TCWO2AHP submitted 2024-12-10 cs.CL

Sebastian Steindl , Ulrich Schäfer , Bernd Ludwig This is my paper · ORCID
classification cs.CL
keywords task-orienteddialoguedataaugmentationmiscommunicationrepairLLM-basedevaluationMultiWOZCoPrUSconsistencypreserving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a fully automatic two-step LLM pipeline can take existing Wizard-of-Oz benchmark dialogues and insert realistic miscommunication and repair turns without changing the user's underlying goal. It focuses on three under-represented error types: misunderstandings, non-understandings, and vaguely related questions. Applied to MultiWOZ 2.1, the method produced CoPrUS-MultiWOZ, a dataset of nearly 1,900 modified dialogues, and the paper argues that downstream NLU, NLG and DST model performance stays roughly unchanged. If true, this gives the dialogue-research community a scalable way to build training data that includes the off-happy-path conversations real users have, rather than only sterile success paths.

What carries the argument

The method's load-bearing mechanism is a two-step prompting scheme organised by a three-way error taxonomy: misunderstanding (MU), where the user misinterprets an earlier system turn and only later realises; non-understanding (NU), where the user immediately asks for clarification; and vaguely related question (VQ), an in-domain question that does not advance the booking goal. Each error type has its own prompt, and a second prompt repairs the turn by replacing a masked system utterance, constrained to only use information from before the mask. An LLM judge filters candidates, and because the inserted turns are never supposed to change slot values, the original dialogue-state annotations are copied over unchanged.

What would settle it

Take the released CoPrUS-MultiWOZ dialogues and have annotators check each inserted miscommunication and repair against the propagated dialogue state; if a non-negligible share of inserted turns implies a different slot value or a changed user goal, the consistency-preserving claim fails and the downstream NLU and DST numbers become unreliable.

Watch

Extended reading notes

Core claim

The central claim is that current instruction-tuned LLMs can generate consistent miscommunication and repair turns post-hoc, so that the original slot annotations can be propagated unchanged onto the modified dialogues. The generation is split into two prompts: the first produces the user's misunderstanding, non-understanding or vaguely related question from a short context window; the second masks the upcoming system turn and asks the model to write a repair that lets the dialogue continue with the original next user utterance. A second LLM, Prometheus 2, scores each candidate on a 1-5 rubric and only the best candidates are kept, with a small human study showing acceptable alignment with those scores. On the released dataset, task models trained on CoPrUS-MultiWOZ perform essentially the same as on the original MultiWOZ, which the paper reads as evidence that the augmentation adds realism without degrading existing capabilities.

Load-bearing premise

The central assumption is that inserting a synthetic misunderstanding or repair never changes what the user wants or which slot values are active, so the original dialogue annotations can be copied onto the modified dialogue unchanged.

Editorial extensions

If this is right

  • CoPrUS-MultiWOZ gives researchers a ready-made benchmark for studying error recovery, since the modified dialogues contain natural repair trajectories that the original MultiWOZ lacks.
  • Training on the augmented data does not harm standard task metrics, so the extra realism comes at little or no cost to existing NLU, NLG, and DST pipelines.
  • The automatic quality-assurance loop means the pipeline can be re-run on other Wizard-of-Oz datasets without involving human annotators.
  • The taxonomy itself offers a compact vocabulary for describing and generating miscommunications in written task-oriented dialogue, distinguishing errors by the user's awareness and by whether the question is goal-relevant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the pipeline never verifies slot consistency directly, an obvious extension is to add automatic state-checking after insertion and to measure how often the LLM judge lets through a turn that silently shifts a slot value.
  • The chosen error distribution (20% MU, 20% VQ, 60% NU) is posited rather than measured; a natural follow-up would be to collect real chat-based booking conversations and count actual error-type frequencies.
  • The same two-step repair scaffold could be applied to other deviation types beyond the three studied, such as user turn-backs or negative feedback, which earlier work has identified as missing from MultiWOZ.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces CoPrUS, a fully automatic two-step LLM pipeline that inserts synthetic miscommunications (misunderstandings, non-understandings, and vaguely related questions) and corresponding repair turns into Wizard-of-Oz benchmark dialogues, using Llama-3.1-70B for generation and Prometheus 2 for automatic quality control. The method is applied to MultiWOZ 2.1, producing a released dataset (CoPrUS-MultiWOZ) with nearly 1900 modified dialogues. The evaluation consists of qualitative examples, a 100-sample human rating study, and downstream NLU/NLG/DST training experiments. The central claim is that current LLMs can aid in post-hoc miscommunication augmentation without harming downstream task performance.

Significance. If the consistency-preserving premise holds, this is a useful and timely contribution: it addresses a real gap in TOD benchmarks, provides a concrete and reproducible pipeline, and releases a dataset that can support future research on error recovery. The paper is transparent about its limitations, and the inclusion of human evaluation and downstream experiments is a strength. However, the central claim of consistency preservation is asserted rather than directly verified, and the evaluation does not measure the intended downstream benefit (robustness to miscommunications). These gaps currently limit the strength of the contribution.

major comments (4)
  1. [Sec. 5, Eq. (2)] The premise that original MultiWOZ annotations remain valid after inserting synthetic turns is asserted, not verified. The paper states in Section 5: 'we do not change the existing annotation but propagate the annotation of the real utterances with the utterances themselves.' This is load-bearing because the inserted turns (ũ_i and s̃_{i+1} in Eq. (2)) could in principle alter the user's goal or introduce contradictory slot values, which would invalidate the released dataset's annotations and make Table 2's no-drop results uninterpretable. The paper's own qualitative analysis (Section 6) acknowledges that 'some dialogues remain where the generated utterances seem illogical,' and Section 10 repeats that 'the dialogue flow can be disrupted from a logical point of view.' Neither the Prometheus 2 judge (Section 4.5) nor the human evaluation (Section 7) checks whether the full dialogue trajectory preserves the dialogue state: the LLM judge scores single utterances against rubrics, and the human judges rated 100 individual candidates in isolation with equal numbers of accepted and rejected candidates. A direct consistency check is needed, e.g., running a state tracker on the modified dialogues and verifying that the belief state after the inserted turns matches the original state at the corresponding point, or at least a manual analysis of a sample of complete modified dialogues for state consistency.
  2. [Sec. 8, Table 2] The empirical evaluation demonstrates that standard NLU/NLG/DST metrics do not degrade on CoPrUS-MultiWOZ, but it does not show that the augmentation serves the paper's stated purpose of helping dialogue systems handle miscommunications. The authors explicitly state in Section 10: 'Our evaluation does not measure or try to improve this ability but rather lays the foundation for such research in future work.' Since the motivation (Section 1) is that miscommunications are underrepresented and systems need to learn to repair them, the central claim of usefulness as data augmentation would be substantially strengthened by an experiment showing that a model trained on CoPrUS-MultiWOZ is better able to detect or recover from the inserted error types (e.g., by testing on held-out dialogues containing miscommunications, or by measuring repair-success rates). As it stands, the 'no performance drop' result is evidence against harm, but not evidence for the claimed benefit.
  3. [Sec. 4.4, Fig. 3] The repair generation prompt includes the next real user utterance as PART 2, and the model is instructed not to use information from PART 2. However, there is no verification that the generated repair does not leak information from the subsequent turn. If such leakage occurs, the synthetic dialogue contains an unrealistic dependency, because the system's repair utterance would appear to anticipate the user's next utterance, which is not possible in a real conversation. This also affects the consistency claim, since the propagated annotation of the following user utterance may not reflect the information actually available. The paper should either report a leakage analysis (e.g., counting cases where the repair explicitly references entities or details first mentioned in PART 2) or restrict the generation to prevent conditioning on the future turn.
  4. [Sec. 7] The human evaluation rates individual candidate utterances on an absolute 1-5 scale, but it does not compare the synthetic miscommunications and repairs against naturally occurring or human-authored counterparts. The paper's central claim is that CoPrUS produces 'more realistic' dialogues (title, abstract), yet there is no reference point: judges are never asked whether a synthetic utterance is distinguishable from a real miscommunication or from an original MultiWOZ utterance. A paired preference test or a discrimination experiment would provide direct evidence for the realism claim. Without such a comparison, the absolute ratings are difficult to interpret, especially since the judges are volunteers and their rating scale may be subject to leniency.
minor comments (6)
  1. [Eq. (2)] The expression for the final CoPrUS dialogue appears to contain a duplicated s_i ('... ui, si, . . .') instead of the intended insertion after s_i and before u_i; please clarify the indexing.
  2. [Table 3 and surrounding text] The text defines 'FP (model accepts, human rejects) and False Negatives (FP, model rejects, human accepts)'; the second 'FP' should be 'FN', and the claim that the FP rate (0.22) is 'low' is not clearly supported by the absolute number.
  3. [Sec. 7] The human evaluation does not report the number of judges, their demographic information, or inter-annotator agreement, which would be useful for assessing the reliability of the rating comparison.
  4. [Fig. 5] The prompt in the appendix contains a typo: 'User A is not surprised by by something he himself said or initiated earlier' has a duplicated 'by'.
  5. [Sec. 4.3] The error-type distribution (pMU = 0.2, pVQ = 0.2, pNU = 0.6) is a free parameter and is admittedly 'posited'; the paper would benefit from a sensitivity analysis or at least a discussion of how this distribution affects the resulting dataset.
  6. [Sec. 4.5] The acceptance rate of the Prometheus 2 judge and the distribution of scores (e.g., how many candidates were accepted on the first try vs. after ten tries) are not reported, making the quality-assurance step difficult to reproduce or evaluate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CoPrUS's generation, quality control, and downstream evaluation are self-contained and externally anchored.

full rationale

We walked the claimed derivation chain: the error taxonomy comes from external linguistic literature (Grice, Paek, Schegloff), the utterances are generated by Llama-3.1, filtered by a Prometheus-2 LLM judge, validated against a human evaluation, and finally assessed by retraining standard NLU/NLG/DST models on the released data. No equation or construction makes a predicted quantity equal to an input by definition. The key sentence in Sec. 5, "we do not change the existing annotation but propagate the annotation of the real utterances with the utterances themselves," is a design assumption about how the dataset is labeled, not a fitted parameter renamed as a prediction, and it does not force the downstream results in Table 2. The paper contains no load-bearing self-citations by the authors. The LLM-as-judge procedure does create a self-evaluation risk, but the authors explicitly measure its alignment with human judges in Sec. 7, and the central usefulness claim is checked against external benchmark tasks in Sec. 8. The admitted limitation in Sec. 10, "the dialogue flow can be disrupted from a logical point of view," undermines the strength of the 'consistency preserving' label, but that is a verification gap or correctness risk, not circularity. We therefore find no specific circular step and assign score 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No mathematical derivation is attempted; the claim is empirical. The ledger shows hand-set sampling parameters and domain assumptions, the most important being the unverified annotation propagation premise. These choices shape the dataset and the evaluation, so they are counted even though none is a fitted model parameter.

free parameters (4)
  • Miscommunication insertion rate = 18% of dialogues
    Chosen in Section 4.3 by hand, extrapolated from spoken dialogue error rates and then reduced for written chat.
  • Error type distribution = pMU=0.2, pVQ=0.2, pNU=0.6
    Posited in Section 4.3 because no prior work studies the same setting.
  • Context window = 5 previous turns
    Design choice in Section 4.6, selected by the authors to focus generation on recent context.
  • LLM acceptance threshold = score >= 4 of 5
    Used in Section 4.5 to accept generated candidates; chosen by hand without calibration.
assumptions (4)
  • domain assumption Grice's Cooperative Principle and maxims provide the normative baseline from which communication errors are defined.
    Sections 2.3 and 4.1 define the three error types as deviations from Grice's maxims; if this framework does not capture relevant miscommunications, the taxonomy is incomplete.
  • ad hoc to paper MultiWOZ annotations remain valid when synthetic miscommunication and repair turns are inserted; user goal and slot values do not change.
    Section 5 asserts annotations are propagated unchanged. This is load-bearing for the NLU and DST evaluations and is not verified.
  • domain assumption The miscommunication rate of 18% and the type distribution pMU=0.2, pVQ=0.2, pNU=0.6 approximate real chat-based task-oriented dialogues.
    Section 4.3 extrapolates from spoken dialogue studies and adjusts by hand; the authors acknowledge this distribution is posited.
  • domain assumption Prometheus 2 ratings are a valid proxy for human judgment of utterance realism at the chosen threshold.
    Sections 4.5 and 7 justify this with a 100-example human evaluation, but no large-scale validation or inter-annotator agreement is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues." pith.science (2026). https://pith.science/paper/TCWO2AHP

@misc{pith2026241207515,
  author       = {Pith},
  title        = {Pith review of: CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TCWO2AHP}},
  note         = {Machine review of arXiv:2412.07515}
}
read the original abstract

Large-scale Wizard-Of-Oz dialogue datasets have enabled the training of deep learning-based dialogue systems. While they are successful as benchmark datasets, they lack certain types of utterances, which would make them more realistic. In this work, we investigate the creation of synthetic communication errors in an automatic pipeline. Based on linguistic theory, we propose and follow a simple error taxonomy. We focus on three types of miscommunications that could happen in real-world dialogues but are underrepresented in the benchmark dataset: misunderstandings, non-understandings and vaguely related questions. Our two-step approach uses a state-of-the-art Large Language Model (LLM) to first create the error and secondly the repairing utterance. We perform Language Model-based evaluation to ensure the quality of the generated utterances. We apply the method to the MultiWOZ dataset and evaluate it both qualitatively and empirically as well as with human judges. Our results indicate that current LLMs can aid in adding post-hoc miscommunications to benchmark datasets as a form of data augmentation. We publish the resulting dataset, in which nearly 1900 dialogues have been modified, as CoPrUS-MultiWOZ to facilitate future work on dialogue systems.

Figures

Figures reproduced from arXiv: 2412.07515 by the authors.

Figure 1
Figure 1. Example MultiWOZ dialogue after applica [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Two examples of miscommunication and repairing utterances for each type. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Prompting procedure with shortened prompts. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Overview of the CoPrUS workflow. We apply the CoPrUS method to the MultiWOZ 2.1 (Eric et al., 2020) dataset to create CoPrUS￾MultiWOZ. MultiWOZ 2.1 is an updated version 7 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The full prompts used for the Llama model. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The prompt used for Prometheus 2. {instruction} is replaced by the prompt used for the Llama model, and [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 34 canonical work pages

  1. [1]

    Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, S \'e bastien Bubeck, Qin Cai, Martin Cai, Caio C \'e sar Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Yen-Chun Chen, Yi-L...

    arXiv doi:10.48550/arxiv.2404.14219 2024

  2. [2]

    Namo Bang, Jeehyun Lee, and Myoung-Wan Koo. 2023. https://doi.org/10.18653/v1/2023.findings-acl.464 Task-optimized adapters for an end-to-end task-oriented dialogue system . In Findings of the Association for Computational Linguistics: ACL 2023, pages 7355--7369, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-acl.464 2023

  3. [3]

    Serge Br \'e dart. 1991. Word interruption in self-repairing. Journal of psycholinguistic research, 20:123--138

    work page 1991

  4. [4]

    Pawe Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, I \ n igo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Ga s i \'c . 2018. https://doi.org/10.18653/v1/D18-1547 M ulti WOZ - a large-scale multi-domain W izard-of- O z dataset for task-oriented dialogue modelling . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proce...

    doi:10.18653/v1/d18-1547 2018

  5. [5]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. https://arxiv.org/abs/2308.07201 Chateval: Towards better llm-based evaluators through multi-agent debate . Preprint, arXiv:2308.07201

    arXiv 2023

  6. [6]

    Willy Chung, Samuel Cahyawijaya, Bryan Wilie, Holy Lovenia, and Pascale Fung. 2023. https://doi.org/10.18653/v1/2023.nlint-1.1 I nstruct TODS : Large language models for end-to-end task-oriented dialogue systems . In Proceedings of the Second Workshop on Natural Language Interfaces, pages 1--21, Bali, Indonesia. Association for Computational Linguistics

    doi:10.18653/v1/2023.nlint-1.1 2023

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Laila Dybkj r, Niels Ole Bernsen, and Hans Dybkj r. 1996. Grice incorporated: Cooperativity in spoken dialogue. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics

    work page 1996

Show all 59 references
  1. [9]

    Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. https://aclanthology.org/2020.lrec-1.53 M ulti WOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and ...

    2020

  2. [10]

    Petra Gieselmann. 2006. Comparing error-handling strategies in human-human and human-robot dialogues. In Proc. 8th Conf. Nat . Language Process .( KONVENS ). Konstanz , Germany , pages 24--31

    2006

  3. [11]

    Marta Gonzalez-Lloret. 2005. Reconstructing ns/nns communication. Book?? The Consequences of Mobility: Linguistic and Sociocultural Contact Zones?, pages 1--27

    2005

  4. [12]

    H. P. Grice. 1975. Logic and Conversation . Syntax and Semantics, 3:43--58

    1975

  5. [13]

    Pamela Grimm. 2010. Social desirability bias. Wiley international encyclopedia of marketing

    2010

  6. [14]

    Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang, Fei Huang, and Luo Si. 2022. Galaxy: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection. In Proceedings of the AAAI Con...

    2022

  7. [15]

    Ryuichiro Higashinaka, Masahiro Araki, Hiroshi Tsukahara, and Masahiro Mizukami. 2019. https://doi.org/10.1007/978-981-13-9443-0_29 Improving Taxonomy of Errors in Chat-Oriented Dialogue Systems . In Luis Fernando D'Haro, Rafael E. Banchs, and Haizhou Li, editors, 9th Internat...

    2019 doi

  8. [16]

    Ryuichiro Higashinaka, Masahiro Araki, Hiroshi Tsukahara, and Masahiro Mizukami. 2021. https://doi.org/10.18653/v1/2021.sigdial-1.10 Integrated taxonomy of errors in chat-oriented dialogue systems . In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Dis...

    2021 doi

  9. [17]

    Ryuichiro Higashinaka, Kotaro Funakoshi, Masahiro Araki, Hiroshi Tsukahara, Yuka Kobayashi, and Masahiro Mizukami. 2015. Towards taxonomy of errors in chat-oriented dialogue systems. In Proceedings of the 16th annual meeting of the special interest group on discourse and dialo...

    2015

  10. [18]

    Graeme Hirst, Susan McRoy, Peter Heeman, Philip Edmonds, and Diane Horton. 1994. https://doi.org/10.1016/0167-6393(94)90073-6 Repairing conversational misunderstandings and non-understandings . Speech Communication, 15(3):213--229

    1994 doi

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L \'e lio Renard Lavaud, Lucile Saulnier, Marie-...

    2024 doi arXiv

  12. [20]

    J. F. Kelley. 1984. https://doi.org/10.1145/357417.357420 An iterative design methodology for user-friendly natural language office information applications . ACM Transactions on Information Systems, 2(1):26--41

    1984

  13. [21]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. https://arxiv.org/abs/2405.01535 Prometheus 2: An open source language model specialized in evaluating other language models ....

    2024 arXiv

  14. [22]

    Takyoung Kim, Yukyung Lee, Hoonsang Yoon, Pilsung Kang, Junseong Bang, and Misuk Kim. 2022. https://doi.org/10.18653/v1/2022.seretod-1.1 Oh My Mistake !: Toward Realistic Dialogue State Tracking including Turnback Utterances . In Proceedings of the Towards Semi-Supervised and ...

    2022 doi

  15. [23]

    Takyoung Kim, Jamin Shin, Young-Ho Kim, Sanghwan Bae, and Sungdong Kim. 2023. https://doi.org/10.48550/arXiv.2305.13857 Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation . Preprint, arxiv:2305.13857

    2023 doi arXiv

  16. [24]

    Zekun Li, Zhiyu Chen, Mike Ross, Patrick Huber, Seungwhan Moon, Zhaojiang Lin, Xin Dong, Adithya Sagar, Xifeng Yan, and Paul Crook. 2024. https://doi.org/10.18653/v1/2024.acl-long.471 Large language models as zero-shot dialogue state tracker through function calling . In Proce...

    2024 doi

  17. [25]

    Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.273 M in TL : Minimalist transfer learning for task-oriented dialogue systems . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language...

    2020 doi

  18. [26]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  19. [27]

    Koh Mitsuda, Ryuichiro Higashinaka, Tingxuan Li, and Sen Yoshida. 2022. https://doi.org/10.18653/v1/2022.acl-short.50 Investigating person-specific errors in chat-oriented dialogue systems . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguis...

    2022 doi

  20. [28]

    Mitzner, Julie B

    Tracy L. Mitzner, Julie B. Boron, Cara Bailey Fausset, Anne E. Adams, Neil Charness, Sara J. Czaja, Katinka Dijkstra, Arthur D. Fisk, Wendy A. Rogers, and Joseph Sharit. 2010. Older adults talk technology: Technology usage and attitudes. Computers in human behavior, 26(6):1710--1721

    2010

  21. [29]

    Sebastian M \"o ller, Klaus-Peter Engelbrecht, and Antti Oulasvirta. 2007. Analysis of communication failures for spoken dialogue systems. In INTERSPEECH , pages 134--137

    2007

  22. [30]

    Clifford Nass and Li Gong. 1999. Maximized modality or constrained consistency? In AVSP'99-International Conference on Auditory-Visual Speech Processing

    1999

  23. [31]

    Clifford Nass and Youngme Moon. 2000. Machines and mindlessness: Social responses to computers. Journal of social issues, 56(1):81--103

    2000

  24. [32]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  25. [33]

    Tim Paek. 2003. Toward a taxonomy of communication errors. In ISCA Tutorial and Research Workshop on Error Handling in Spoken Dialogue Systems

    2003

  26. [34]

    Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2021. https://doi.org/10.1162/tacl_a_00399 Soloist: Building task bots at scale with transfer learning and machine teaching . Transactions of the Association for Computational Linguistics, 9:807--824

    2021 doi

  27. [35]

    Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.17 Few-shot natural language generation for task-oriented dialog . In Findings of the Association for Computational Linguistics...

    2020 doi

  28. [36]

    Kun Qian, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. 2021. https://doi.org/10.18653/v1/2021.sigdial-1.35 Annotation inconsistency and entity bias in MultiWOZ . In Proceedings of the 22nd Annual Meeting of the Special Interest Gr...

    2021 doi

  29. [37]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  30. [38]

    Caroline L Rieger. 2003. Repetitions as self-repair strategies in english and german conversations. Journal of pragmatics, 35(1):47--69

    2003

  31. [39]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyo...

    2022

  32. [40]

    Emanuel A Schegloff. 2000. When'others' initiate repair. Applied linguistics, 21(2):205--243

    2000

  33. [41]

    Emanuel A Schegloff, Gail Jefferson, and Harvey Sacks. 1977. The preference for self-correction in the organization of repair in conversation. Language, 53(2):361--382

    1977

  34. [42]

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. https://doi.org/10.1038/s41586-024-07566-y AI models collapse when trained on recursively generated data . Nature, 631(8022):755--759

    2024 doi

  35. [43]

    Gabriel Skantze. 2005. Exploring human error recovery strategies: Implications for spoken dialogue systems. Speech Communication, 45(3):325--341

    2005

  36. [44]

    Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2022. https://doi.org/10.18653/v1/2022.acl-long.319 Multi-task pre-training for plug-and-play task-oriented dialogue system . In Proceedings of the 60th Annual Meeting of the Association for C...

    2022 doi

  37. [45]

    Haipeng Sun, Junwei Bao, Youzheng Wu, and Xiaodong He. 2023. https://doi.org/10.18653/v1/2023.findings-acl.708 Mars: Modeling Context & State Representations with Contrastive Learning for End-to-End Task-Oriented Dialog . In Findings of the Association for Computational Lingui...

    2023 doi

  38. [46]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 doi arXiv

  39. [47]

    Dennis Ulmer, Elman Mansimov, Kaixiang Lin, Lijia Sun, Xibin Gao, and Yi Zhang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.566 Bootstrapping LLM -based task-oriented dialogue agents via self-talk . In Findings of the Association for Computational Linguistics: ACL 2024...

    2024 doi

  40. [48]

    Van Der Goot and Tyler Pilgrim

    Margot J. Van Der Goot and Tyler Pilgrim. 2020. https://doi.org/10.1007/978-3-030-39540-7_12 Exploring Age Differences in Motivations for and Acceptance of Chatbot Communication in a Customer Service Context . In Asbj rn F lstad, Theo Araujo, Symeon Papadopoulos, Effie Lai-Cho...

    2020 doi

  41. [49]

    Carel van Niekerk, Andrey Malinin, Christian Geishauser, Michael Heck, Hsien-chin Lin, Nurul Lubis, Shutong Feng, and Milica Gasic. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.623 Uncertainty measures in neural belief tracking and the effects on dialogue policy performan...

    2021 doi

  42. [50]

    Edda Weigand. 1999. https://doi.org/10.1016/S0378-2166(98)00068-X Misunderstanding: The standard case . Journal of Pragmatics, 31(6):763--785

    1999 doi

  43. [51]

    Puhai Yang, Heyan Huang, Wei Wei, and Xian-Ling Mao. 2022. https://doi.org/10.1145/3534678.3539385 Toward Real-life Dialogue State Tracking Involving Negative Feedback Utterances . In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages ...

    2022

  44. [52]

    Williams

    Steve Young, Milica Ga s i \'c , Blaise Thomson, and Jason D. Williams. 2013. https://doi.org/10.1109/JPROC.2012.2225812 POMDP-Based statistical spoken dialog systems: A review . Proceedings of the IEEE, 101(5):1160--1179

    2013

  45. [53]

    Changhong Yu, Chunhong Zhang, and Qi Sun. 2022. https://arxiv.org/abs/2205.05886 A Chit-Chats Enhanced Task-Oriented Dialogue Corpora for Fuse-Motive Conversation Systems . Preprint, arxiv:2205.05886

    2022 arXiv

  46. [54]

    Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu, Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022. https://doi.org/10.48550/arXiv.2201.08904 Description- Driven Task-Oriented Dialog Modeling . Preprint, arxiv:2201.08904

    2022 doi arXiv

  47. [55]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Proce...

    2023

  48. [56]

    Qi Zhu, Christian Geishauser, Hsien chin Lin, Carel van Niekerk, Baolin Peng, Zheng Zhang, Michael Heck, Nurul Lubis, Dazhen Wan, Xiaochen Zhu, Jianfeng Gao, Milica Gašić, and Minlie Huang. 2022. http://arxiv.org/abs/2211.17148 Convlab-3: A flexible dialogue system toolkit bas...

    2022 arXiv

  49. [57]

    Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao, Xiaoyan Zhu, and Minlie Huang. 2020. https://doi.org/10.18653/v1/2020.acl-demos.19 C onv L ab-2: An open-source toolkit for building, evaluating, and diagnosing dialogue systems ....

    2020 doi

  50. [58]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.