{"id":"da4d2a78-cdce-4669-9f08-313378dbffe3","arxiv_id":"2412.07515","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-based pipeline adds synthetic misunderstandings, non-understandings, and vaguely related questions plus repair responses to MultiWOZ dialogues, and the released CoPrUS-MultiWOZ dataset preserves task performance when used for training.","lead":"This paper describes an automatic pipeline that inserts realistic miscommunication turns, such as misunderstandings and clarification questions, into an existing task-oriented dialogue benchmark using large language models. The authors release a modified version of MultiWOZ with nearly 1900 affected dialogues, aimed at helping dialogue systems train on less perfect conversations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Consistency-preserving claim is asserted, not verified: CoPrUS inserts unlabeled miscommunication/repair turns while copying original annotations, yet the paper's own Sec. 6 and Sec. 10 concede illogical flows; no direct state-consistency check supports the title's promise.","rationale":"The reader's weakest assumption identifies the same gate: copied annotations require the inserted turns to leave the user goal and slot values intact. I examined whether any other element is more load-bearing. The realism claim lacks a human-authored baseline, and the code is not released, but those affect strength of contribution, not validity. The consistency assumption, by contrast, is built into the dataset construction and the empirical evaluation: Table 2's metrics are computed against annotations that are asserted, not checked, to still match the modified dialogues. The authors' own qualitative analysis and limitations section explicitly concede illogical flows remain, which is an in-text admission that the inserted turns can break dialogue logic. Because the paper provides no direct consistency metric or audit, the central promise in the title is currently unsupported. This does not make the work unserious or the method useless; it makes the claimed property contingent on an unverified assumption. A manual consistency audit on a stratified sample would settle the point. The reader's CONDITIONAL verdict therefore remains appropriate; no adjustment is needed.","tokens_in":17549,"tokens_out":6513,"duration_ms":62819,"concrete_test":"Run an annotator study on 100 stratified CoPrUS-MultiWOZ dialogues: for each inserted (miscommunication, repair) pair, annotators mark any introduced slot-value change, contradiction with earlier system turns, or break in the original user's follow-up turn, using the original MultiWOZ state annotations as reference. Report per-type violation rate and inter-annotator agreement; if the violation rate is non-negligible (e.g., >5%) or agreement is low, the 'Consistency Preserving' claim and the annotation-copying in Sec. 5 are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 says 'we do not change the existing annotation but propagate the annotation of the real utterances with the utterances themselves.' This is the load-bearing premise: if the inserted turns (Eq. 2: s_i, ũ_i, s̃_{i+1}, u_i) alter the user's goal or slot values, then the released dataset's annotations no longer describe the dialogue, and both the dataset's utility and Table 2's no-drop results are uninterpretable as evidence of consistency preservation. The paper offers no direct verification. Sec. 6 admits 'some dialogues remain where the generated utterances seem illogical'; Sec. 10 repeats that 'the dialogue flow can be disrupted from a logical point of view.' The LLM judge (Sec. 4.5) scores candidate utterances against the prompt and rubrics, not against the full dialogue state; the human evaluation (Sec. 7) rates 100 individual candidates in isolation, with equal accepted/rejected candidates, so it does not test whether the synthetic exchange preserves the state trajectory. Thus the central claim is unsupported at exactly the point where the annotation copy must hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoPrUS, a fully automatic two-step LLM pipeline that inserts synthetic miscommunications (misunderstandings, non-understandings, and vaguely related questions) and corresponding repair turns into Wizard-of-Oz benchmark dialogues, using Llama-3.1-70B for generation and Prometheus 2 for automatic quality control. The method is applied to MultiWOZ 2.1, producing a released dataset (CoPrUS-MultiWOZ) with nearly 1900 modified dialogues. The evaluation consists of qualitative examples, a 100-sample human rating study, and downstream NLU/NLG/DST training experiments. The central claim is that current LLMs can aid in post-hoc miscommunication augmentation without harming downstream task performance.","tokens_in":17769,"tokens_out":8207,"duration_ms":76305,"significance":"If the consistency-preserving premise holds, this is a useful and timely contribution: it addresses a real gap in TOD benchmarks, provides a concrete and reproducible pipeline, and releases a dataset that can support future research on error recovery. The paper is transparent about its limitations, and the inclusion of human evaluation and downstream experiments is a strength. However, the central claim of consistency preservation is asserted rather than directly verified, and the evaluation does not measure the intended downstream benefit (robustness to miscommunications). These gaps currently limit the strength of the contribution.","major_comments":[{"comment":"The premise that original MultiWOZ annotations remain valid after inserting synthetic turns is asserted, not verified. The paper states in Section 5: 'we do not change the existing annotation but propagate the annotation of the real utterances with the utterances themselves.' This is load-bearing because the inserted turns (ũ_i and s̃_{i+1} in Eq. (2)) could in principle alter the user's goal or introduce contradictory slot values, which would invalidate the released dataset's annotations and make Table 2's no-drop results uninterpretable. The paper's own qualitative analysis (Section 6) acknowledges that 'some dialogues remain where the generated utterances seem illogical,' and Section 10 repeats that 'the dialogue flow can be disrupted from a logical point of view.' Neither the Prometheus 2 judge (Section 4.5) nor the human evaluation (Section 7) checks whether the full dialogue trajectory preserves the dialogue state: the LLM judge scores single utterances against rubrics, and the human judges rated 100 individual candidates in isolation with equal numbers of accepted and rejected candidates. A direct consistency check is needed, e.g., running a state tracker on the modified dialogues and verifying that the belief state after the inserted turns matches the original state at the corresponding point, or at least a manual analysis of a sample of complete modified dialogues for state consistency.","section":"Sec. 5, Eq. (2)"},{"comment":"The empirical evaluation demonstrates that standard NLU/NLG/DST metrics do not degrade on CoPrUS-MultiWOZ, but it does not show that the augmentation serves the paper's stated purpose of helping dialogue systems handle miscommunications. The authors explicitly state in Section 10: 'Our evaluation does not measure or try to improve this ability but rather lays the foundation for such research in future work.' Since the motivation (Section 1) is that miscommunications are underrepresented and systems need to learn to repair them, the central claim of usefulness as data augmentation would be substantially strengthened by an experiment showing that a model trained on CoPrUS-MultiWOZ is better able to detect or recover from the inserted error types (e.g., by testing on held-out dialogues containing miscommunications, or by measuring repair-success rates). As it stands, the 'no performance drop' result is evidence against harm, but not evidence for the claimed benefit.","section":"Sec. 8, Table 2"},{"comment":"The repair generation prompt includes the next real user utterance as PART 2, and the model is instructed not to use information from PART 2. However, there is no verification that the generated repair does not leak information from the subsequent turn. If such leakage occurs, the synthetic dialogue contains an unrealistic dependency, because the system's repair utterance would appear to anticipate the user's next utterance, which is not possible in a real conversation. This also affects the consistency claim, since the propagated annotation of the following user utterance may not reflect the information actually available. The paper should either report a leakage analysis (e.g., counting cases where the repair explicitly references entities or details first mentioned in PART 2) or restrict the generation to prevent conditioning on the future turn.","section":"Sec. 4.4, Fig. 3"},{"comment":"The human evaluation rates individual candidate utterances on an absolute 1-5 scale, but it does not compare the synthetic miscommunications and repairs against naturally occurring or human-authored counterparts. The paper's central claim is that CoPrUS produces 'more realistic' dialogues (title, abstract), yet there is no reference point: judges are never asked whether a synthetic utterance is distinguishable from a real miscommunication or from an original MultiWOZ utterance. A paired preference test or a discrimination experiment would provide direct evidence for the realism claim. Without such a comparison, the absolute ratings are difficult to interpret, especially since the judges are volunteers and their rating scale may be subject to leniency.","section":"Sec. 7"}],"minor_comments":[{"comment":"The expression for the final CoPrUS dialogue appears to contain a duplicated s_i ('... ui, si, . . .') instead of the intended insertion after s_i and before u_i; please clarify the indexing.","section":"Eq. (2)"},{"comment":"The text defines 'FP (model accepts, human rejects) and False Negatives (FP, model rejects, human accepts)'; the second 'FP' should be 'FN', and the claim that the FP rate (0.22) is 'low' is not clearly supported by the absolute number.","section":"Table 3 and surrounding text"},{"comment":"The human evaluation does not report the number of judges, their demographic information, or inter-annotator agreement, which would be useful for assessing the reliability of the rating comparison.","section":"Sec. 7"},{"comment":"The prompt in the appendix contains a typo: 'User A is not surprised by by something he himself said or initiated earlier' has a duplicated 'by'.","section":"Fig. 5"},{"comment":"The error-type distribution (pMU = 0.2, pVQ = 0.2, pNU = 0.6) is a free parameter and is admittedly 'posited'; the paper would benefit from a sensitivity analysis or at least a discussion of how this distribution affects the resulting dataset.","section":"Sec. 4.3"},{"comment":"The acceptance rate of the Prometheus 2 judge and the distribution of scores (e.g., how many candidates were accepted on the first try vs. after ten tries) are not reported, making the quality-assurance step difficult to reproduce or evaluate.","section":"Sec. 4.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and the dataset release is valuable. The main concern is the unverified consistency-preserving premise; if the authors add a direct state-consistency check and a positive downstream experiment, the contribution would be substantially strengthened. No concerns about citation patterns or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper does something genuinely useful: it builds a two-step LLM pipeline that inserts three types of miscommunication (misunderstanding, non-understanding, vaguely related question) plus repair turns into MultiWOZ dialogues, and it releases the resulting ~1900-dialogue dataset. The method is clearly described, the prompts are in the appendix, and the downstream experiments show that training on the augmented data does not degrade NLU, NLG, or DST. The taxonomy is grounded in prior work (Grice, Paek), and the limitation section is honest about remaining flaws. That is real work and a reasonable contribution.\n\nThe soft spot is exactly where the title makes its strongest promise. Section 5 says annotations are propagated unchanged because the synthetic turns should not alter the user's goal or slot values. But the paper never checks this. The LLM judge scores candidates against the prompt and rubric, not against the full dialogue state. The human evaluation rates 100 candidate utterances in isolation, not the state trajectory across the inserted exchange. And the paper itself admits in Sections 6 and 10 that some dialogues remain illogical or disrupted from a logical standpoint. So the load-bearing claim, that consistency is preserved, is unsupported at the point where it matters. If the inserted turns do change the state, the copied annotations are wrong and the downstream DST numbers become hard to interpret.\n\nThat said, this is not a fatal flaw. The consistency problem is addressable: a manual or automated check of dialogue state before and after insertion would settle it. The downstream no-degradation result suggests the annotations are mostly usable, but it is not the same as verification. Other weaknesses are minor. No pipeline code is shipped, only the data. The error-type distribution (0.2/0.2/0.6) is posited without empirical support. There is no comparison against human-authored miscommunications, so the realism claim is only partially supported. The paper also does not measure error-recovery ability, which it acknowledges.\n\nThe paper deserves a serious referee. It is a solid dataset contribution for the task-oriented dialogue community, and the consistency gap is fixable with a modest addition. I would not reject it, but I would make the state-consistency verification a requirement for acceptance. If you do dialogue robustness or data augmentation work, this dataset and pipeline are worth citing.","headline":"Useful augmentation pipeline and dataset, but the 'consistency preserving' promise is asserted without direct verification.","tokens_in":18289,"tokens_out":2680,"would_cite":true,"duration_ms":27994,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Synthetic miscommunication turns can be added to benchmark dialogues automatically without breaking downstream performance.","keywords":["task-oriented dialogue","data augmentation","miscommunication","dialogue repair","LLM-based evaluation","MultiWOZ","CoPrUS","consistency preserving"],"falsifier":"Take the released CoPrUS-MultiWOZ dialogues and have annotators check each inserted miscommunication and repair against the propagated dialogue state; if a non-negligible share of inserted turns implies a different slot value or a changed user goal, the consistency-preserving claim fails and the downstream NLU and DST numbers become unreliable.","tokens_in":17325,"feed_emoji":"💬","tokens_out":4131,"duration_ms":34977,"temperature":0.7,"pith_summary":"The paper claims that a fully automatic two-step LLM pipeline can take existing Wizard-of-Oz benchmark dialogues and insert realistic miscommunication and repair turns without changing the user's underlying goal. It focuses on three under-represented error types: misunderstandings, non-understandings, and vaguely related questions. Applied to MultiWOZ 2.1, the method produced CoPrUS-MultiWOZ, a dataset of nearly 1,900 modified dialogues, and the paper argues that downstream NLU, NLG and DST model performance stays roughly unchanged. If true, this gives the dialogue-research community a scalable way to build training data that includes the off-happy-path conversations real users have, rather than only sterile success paths.","feed_headline":"LLMs can inject realistic miscommunication into benchmark dialogues","feed_subtitle":"Synthetic repair turns added to 1,900 MultiWOZ dialogues leave NLU, NLG and DST scores intact.","key_machinery":"The method's load-bearing mechanism is a two-step prompting scheme organised by a three-way error taxonomy: misunderstanding (MU), where the user misinterprets an earlier system turn and only later realises; non-understanding (NU), where the user immediately asks for clarification; and vaguely related question (VQ), an in-domain question that does not advance the booking goal. Each error type has its own prompt, and a second prompt repairs the turn by replacing a masked system utterance, constrained to only use information from before the mask. An LLM judge filters candidates, and because the inserted turns are never supposed to change slot values, the original dialogue-state annotations are copied over unchanged.","core_discovery":"The central claim is that current instruction-tuned LLMs can generate consistent miscommunication and repair turns post-hoc, so that the original slot annotations can be propagated unchanged onto the modified dialogues. The generation is split into two prompts: the first produces the user's misunderstanding, non-understanding or vaguely related question from a short context window; the second masks the upcoming system turn and asks the model to write a repair that lets the dialogue continue with the original next user utterance. A second LLM, Prometheus 2, scores each candidate on a 1-5 rubric and only the best candidates are kept, with a small human study showing acceptable alignment with those scores. On the released dataset, task models trained on CoPrUS-MultiWOZ perform essentially the same as on the original MultiWOZ, which the paper reads as evidence that the augmentation adds realism without degrading existing capabilities.","pith_inferences":["Because the pipeline never verifies slot consistency directly, an obvious extension is to add automatic state-checking after insertion and to measure how often the LLM judge lets through a turn that silently shifts a slot value.","The chosen error distribution (20% MU, 20% VQ, 60% NU) is posited rather than measured; a natural follow-up would be to collect real chat-based booking conversations and count actual error-type frequencies.","The same two-step repair scaffold could be applied to other deviation types beyond the three studied, such as user turn-backs or negative feedback, which earlier work has identified as missing from MultiWOZ."],"forward_implications":["CoPrUS-MultiWOZ gives researchers a ready-made benchmark for studying error recovery, since the modified dialogues contain natural repair trajectories that the original MultiWOZ lacks.","Training on the augmented data does not harm standard task metrics, so the extra realism comes at little or no cost to existing NLU, NLG, and DST pipelines.","The automatic quality-assurance loop means the pipeline can be re-run on other Wizard-of-Oz datasets without involving human annotators.","The taxonomy itself offers a compact vocabulary for describing and generating miscommunications in written task-oriented dialogue, distinguishing errors by the user's awareness and by whether the question is goal-relevant."],"supporting_citations":[{"why":"Supplies the original MultiWOZ dialogues that CoPrUS modifies.","marker":"Budzianowski et al., 2018"},{"why":"Defines the MultiWOZ 2.1 version used as the base dataset and the fixed train/develop/test splits.","marker":"Eric et al., 2020"},{"why":"Provides Llama-3.1-70B-Instruct, the LLM that generates the miscommunication and repair utterances.","marker":"Dubey et al., 2024"},{"why":"Provides Prometheus 2, the LLM judge used for automatic quality assurance of generated utterances.","marker":"Kim et al., 2024"},{"why":"Supplies the empirical error-frequency estimates (28-32% overall, 19% without ASR errors) that justify sampling 18% of dialogues.","marker":"Möller et al., 2007"},{"why":"Grounds the error taxonomy in the cooperative principle and the maxims whose deviations count as miscommunication.","marker":"Grice, 1975"},{"why":"Supplies the four-level channel-signal-intention-conversation model that locates the three studied error types.","marker":"Paek, 2003"},{"why":"Establishes the repair concept, including self- and other-initiated repair, which motivates the repairing utterance step.","marker":"Schegloff et al., 1977"}],"fun_headline_variants":["LLMs add realistic miscommunication to benchmark dialogues","Synthetic dialogue errors: LLMs make benchmarks more realistic","Injecting miscommunication with LLMs, performance intact","LLM-generated repairs enrich dialogue benchmarks with realism","CoPrUS: Adding realistic miscommunication to MultiWOZ with LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that inserting a synthetic misunderstanding or repair never changes what the user wants or which slot values are active, so the original dialogue annotations can be copied onto the modified dialogue unchanged.","fun_headline_variants_meta":{"raw":{"variants":["LLMs add realistic miscommunication to benchmark dialogues","Synthetic dialogue errors: LLMs make benchmarks more realistic","Injecting miscommunication with LLMs, performance intact","LLM-generated repairs enrich dialogue benchmarks with realism","CoPrUS: Adding realistic miscommunication to MultiWOZ with LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1892,"prompt_tokens":925,"completion_tokens":967,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":887}},"tokens_in":541,"tokens_out":967,"duration_ms":8900,"temperature":1.0,"reasoning_tokens":887,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:45:51.266209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released CoPrUS-MultiWOZ dialogues and have annotators check each inserted miscommunication and repair against the propagated dialogue state; if a non-negligible share of inserted turns implies a different slot value or a changed user goal, the consistency-preserving claim fails and the downstream NLU and DST numbers become unreliable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MultiWOZ 2.1 version used as the base dataset and the fixed train/develop/test splits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the four-level channel-signal-intention-conversation model that locates the three studied error types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the repair concept, including self- and other-initiated repair, which motivates the repairing utterance step."}],"review_version":1}