{"id":"6df6e21c-c5d4-479d-8aec-4d0efcdd7b07","arxiv_id":"1908.08486","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Using dialogue act prediction as an auxiliary task during training lets a dialogue coherence model rank dialogues without explicit dialogue act labels at evaluation.","lead":"This paper trains a dialogue coherence model together with a dialogue act prediction task, so it can rank dialogues by coherence without needing dialogue act labels at test time. On DailyDialog it beats strong baselines by a wide margin, while on SwitchBoard it mostly matches them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic perturbation protocol in Section 4.2 is the load-bearing assumption: all reported gains are on artificially constructed incoherent dialogues, and the paper itself defers human-ranking correlation to future work; without it, the central practical claim is unsecured.","rationale":"The reader's weakest_assumption correctly identifies the synthetic perturbation protocol as the key unvalidated premise. My analysis agrees: the paper's experiments are entirely built on artificially constructed incoherence, and the authors themselves list human-correlation evaluation as future work. This is the load-bearing concern because the paper's central claim is about assessing dialogue coherence in practice, not merely about separating a fixed set of perturbations. The lack of human validation does not by itself disprove the method, but it makes the current evidence conditional: the method may be learning perturbation-specific cues rather than general coherence. The paper does have independent support: source code is released, the MTL formulation is clean, the comparison against strong baselines is fairly set up, and the cross-domain evaluation partly addresses overfitting to a single perturbation type. However, cross-domain evaluation still operates within the same synthetic regime, so it does not resolve the fundamental concern. The abstract's quantitative overclaims are secondary and would be corrected by a more careful calibration; they do not change the verdict because the core method remains plausible and the missing human study is an addressable limitation. Since the reader already flagged the same weakest assumption and issued a CONDITIONAL verdict, my stress-test does not change the recommended verdict. The concrete test I propose would settle whether the concern actually lands: a human-ranking correlation study on both perturbed and naturally incoherent dialogues, with per-perturbation human agreement checks.","tokens_in":91,"tokens_out":4170,"duration_ms":108303,"concrete_test":"Collect human coherence ratings on a stratified sample of original and perturbed dialogues from both DailyDialog and SwitchBoard, covering all four perturbation types, plus a separate set of naturally low-coherence dialogues (e.g., generated by a dialogue system or by random utterance selection from different conversations). Compute rank correlation (Spearman or Kendall) between M-DiCoh scores and human ratings, and compare against S-DiCoh, CoSim, and EAGrid. If M-DiCoh's correlation is not significantly better than baselines, or if its high synthetic accuracy does not transfer to naturally incoherent dialogues, the central claim is not supported. Also measure human agreement with the assumption that original dialogues are more coherent than their perturbations; low agreement would directly invalidate the synthetic task.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every experiment in Sections 4.4 and 4.5 constructs positive/negative examples by perturbing original dialogues with Utterance Ordering, Utterance Insertion, Utterance Replacement, or Even Utterance Ordering, then trains and tests a pairwise ranker on those synthetic pairs. This assumes (a) that the original dialogue is always more coherent than its perturbation, and (b) that ranking accuracy on such pairs is a valid proxy for dialogue coherence assessment. If M-DiCoh and the baselines exploit shallow artifacts of the perturbation procedure, such as word-order anomalies, disrupted entity transitions, or DA-transition irregularities, the reported gains on DailyDialog and the par performance on SwitchBoard may not transfer to real dialogue evaluation settings. The paper explicitly acknowledges this gap in Section 5: 'For future work, we would like to deeply study the impacts of our perturbations on the coherence of the examined dialogues. We will also investigate to what extent the rankings of dialogues obtained by our model correlate with human-provided rankings.' Because the central claim of the paper is about practical dialogue coherence assessment without DA labels, this untested assumption is the weakest link. A secondary calibration concern is that the abstract's 'more than 20 accuracy points' does not hold for UR on DailyDialog (83.02 vs 68.49, about 14.5 points) and 'on par' on SwitchBoard does not hold for UR (EAGrid 75.61 vs M-DiCoh 58.67). These overclaims do not invalidate the method but further emphasize the need for a human-grounded evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses dialogue coherence assessment by training a coherence ranker (DiCoh) jointly with a dialogue act prediction (DAP) auxiliary task in a multi-task learning framework. The utterance encoder is shared, and at evaluation only utterance text is needed, so explicit DA labels are not required. Experiments on DailyDialog and SwitchBoard under four synthetic perturbation domains (UO, UI, UR, EUO) compare M-DiCoh with baselines including EAGrid, ASeq, CoSim, and random; additionally, cross problem-domain evaluation on DailyDialog and DA prediction accuracy are reported. The paper claims substantial gains on DailyDialog and on-par performance on SwitchBoard, and releases source code.","tokens_in":13827,"tokens_out":4977,"duration_ms":49512,"significance":"The idea of using DA prediction as an auxiliary task to enrich utterance representations for coherence is sensible and novel relative to prior entity-grid and DA-transition approaches. The paper ships machine-checked experiments with five random seeds, multiple perturbation protocols, and cross-domain evaluation, and the code is released. If the synthetic evaluation reflects coherence, the method would be practically useful because it removes the need for gold DA labels at evaluation time. However, the absence of any human correlation study and the abstract's overstatement of results temper the significance.","major_comments":[{"comment":"All experiments train and test on pairs of original dialogues and their synthetic perturbations (UO, UI, UR, EUO), under the assumption that the original dialogue is always more coherent than the perturbed one. This assumption is load-bearing for the paper's central claim that the model assesses dialogue coherence without DA labels, yet no human correlation or direct validation that the perturbations systematically degrade coherence is reported; the paper explicitly defers this to future work in Section 5. Without such evidence, the reported gains could reflect shallow artifacts of the perturbation functions rather than coherence per se. Please add a human-ranking correlation study or otherwise validate the perturbation protocol, or temper the claims of practical dialogue coherence assessment.","section":"§4.2 and §5"},{"comment":"The abstract states the model 'substantially (more than 20 accuracy points) outperforms its strong competitors on the DailyDialogue corpus, and performs on par with them on the SwitchBoard corpus.' In Table 3, M-DiCoh's margin over EAGrid on DailyDialog UR is only 83.02 - 68.49 = 14.53 points, below 20. On SwitchBoard UR, M-DiCoh (58.67) is far below EAGrid (75.61) and ASeq (63.48), so 'on par' does not hold for UR. The empirical summary should be corrected to match the reported numbers.","section":"Abstract and Table 3"}],"minor_comments":[{"comment":"There are typographical errors: 'utt2 He looks strong.' is missing a colon after 'utt2', and 'utt1::' has a double colon.","section":"Table 1"},{"comment":"Hyperparameter tuning was performed only on the UO problem-domain of DailyDialog, and the same hyperparameters were used for SwitchBoard without tuning. This should be stated more prominently in the main text, as it may affect the SwitchBoard comparisons.","section":"Section C"},{"comment":"Standard deviations are reported only for S-DiCoh and M-DiCoh; baselines are reported without variance. Consider reporting variance for all models if multiple runs were performed, or justify the single-run reporting for baselines.","section":"Table 3"},{"comment":"The labels 'diali' and 'dialj' in the figure are not defined in the caption; please define them as the two dialogues in the input pair.","section":"Figure 2"},{"comment":"The result that M-DAP performs on par with S-DAP is interpreted as showing coherence information does not help dialogue act prediction. This point could be stated more clearly, since the same section also emphasizes the benefits of the auxiliary task in the reverse direction.","section":"Section 4.6"},{"comment":"The attention-weight analysis is qualitative and based on a single example. A quantitative analysis over many examples would strengthen the claim that the auxiliary task focuses attention on salient words.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid conference-style contribution with a clear idea and comprehensive synthetic experiments. For a journal, the lack of any validation against human judgments is the main gap; the overclaim in the abstract is a separate issue that must be corrected. I would be willing to reconsider after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one to know: this paper shows you can train a dialogue coherence ranker with dialogue act prediction as an auxiliary task, so at evaluation you only need the raw text. That's a genuinely useful move—prior work like Cervone et al. fed gold DA labels into the model at test time, which is expensive and brittle. The architecture itself is a standard BiLSTM with self-attention, but the MTL framing is new.\n\nWhat's good: the experiments are reasonably thorough. Four perturbation domains, five seeds, code released. On DailyDialog, M-DiCoh beats EAGrid by wide margins on every perturbation. Cross-domain transfer is also consistently better than EAGrid, which is a nice robustness point.\n\nNow the soft spots. First, the abstract overclaims. The 'more than 20 accuracy points' doesn't hold for UR on DailyDialog (83.02 vs 68.49, about 14.5 points), and 'on par' on SwitchBoard ignores UR, where EAGrid gets 75.61 and M-DiCoh gets 58.67. If you cite these numbers, cite the table, not the abstract.\n\nSecond, the evaluation is entirely on synthetic perturbations. All positive/negative pairs come from UO, UI, UR, EUO, and the model is just asked to tell which dialogue is the original. That could be solved by shallow artifacts rather than true coherence. The authors acknowledge this and list human correlation as future work, which is honest, but it means the practical claim—'suitable for evaluating dialogue agents in real scenarios'—is not yet backed. This is the load-bearing limitation if you care about real-world use.\n\nMinor: hyperparameters are tuned only on DailyDialog UO and applied to SwitchBoard unchanged, so SwitchBoard performance is probably not fully optimized. Not a fatal flaw, but worth noting.\n\nBottom line: it's a solid method paper that will be useful to anyone working on dialogue evaluation or coherence modeling. It deserves a serious referee and publication after claim calibration and clearer limitation framing. I'd cite it for the MTL trick.\n\nRecommendation: send to peer review; ask for a corrected abstract and a short human-correlation experiment if feasible.","headline":"MTL with dialogue act prediction as an auxiliary task is a real improvement for dialogue coherence assessment, but the claims need calibration and the synthetic-only evaluation limits the practical reach.","tokens_in":14385,"tokens_out":2676,"would_cite":true,"duration_ms":24593,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dialogue coherence can be ranked without dialogue act labels at evaluation time, using act prediction only as a training-side auxiliary task.","keywords":["dialogue coherence","dialogue act prediction","multi-task learning","utterance representation","coherence ranking","perturbation-based evaluation","DailyDialog","SwitchBoard"],"falsifier":"Collect human coherence rankings for unperturbed dialogues from DailyDialog and SwitchBoard and compare them with M-DiCoh's scores: if the model's ranking correlates with human judgments no better than a model trained without the dialogue-act auxiliary task, the central claim fails. A simpler version is to score real dialogues that differ only by a single swapped turn and check whether human-preferred dialogues receive higher scores.","tokens_in":13318,"feed_emoji":"💬","tokens_out":8079,"duration_ms":73683,"temperature":0.7,"pith_summary":"Dialogue coherence assessment usually needs to know each utterance's speech-act type at scoring time, either from annotators or a separate classifier. This paper argues that those labels are only needed during training: it couples the coherence ranker with a dialogue-act predictor in a multi-task setup, sharing an utterance encoder, so the act labels shape the representations without being inputs at evaluation. On DailyDialog the resulting M-DiCoh model beats the entity-grid-plus-act baseline EAGrid by more than 20 accuracy points on most of the tested perturbation tasks, and on SwitchBoard it is broadly on par, though weaker on utterance replacement. If this holds, coherence evaluation of dialogue systems can drop expensive act annotation and avoid propagating act-classifier errors into the score.","feed_headline":"Dialogue acts can stay out of coherence ranking at test time","feed_subtitle":"On DailyDialog the multi-task model beats entity-grid baselines by over 20 accuracy points; on SwitchBoard it ties them.","key_machinery":"The machinery is a multi-task learner with two heads on one utterance encoder. An utterance is encoded by word embeddings, a BiLSTM, and self-attention; the coherence head (DiCoh) runs another BiLSTM and attention over the utterance vectors and maps the dialogue to a scalar score, while the dialogue-act head (DAP) is a softmax over act types. The two are trained jointly with the weighted loss from Equation (11), pairing a margin ranking loss on dialogue pairs with average cross-entropy over act labels, where the trade-off weights $\\gamma_1, \\gamma_2$ are learned following the uncertainty-weighting idea. Dialogue act labels therefore act as a regularizer that pushes the shared encoder to attend to act-salient words during training, and are not needed at inference.","core_discovery":"Dialogue coherence ranking can be done without supplying dialogue act (DA) labels at evaluation time, provided DA prediction is used as an auxiliary training signal. The paper's M-DiCoh model trains a pairwise coherence scorer (DiCoh) together with a softmax DA predictor (DAP) over a shared BiLSTM-plus-attention utterance encoder; DA labels appear only in the auxiliary cross-entropy loss. In the perturbation-based ranking tests, M-DiCoh reaches 95.92% accuracy on utterance ordering for DailyDialog against 71.72% for EAGrid and 68.21% for ASeq, with analogous margins on insertion and even-utterance ordering. On SwitchBoard, where DA labels are far more fine-grained and imbalanced, M-DiCoh scores 99.41% on utterance ordering and 85.04% on insertion, close to or above EAGrid, but trails on utterance replacement (58.67% vs. 75.61%).","pith_inferences":["A testable extension of the paper's logic is that the auxiliary DA task helps mainly by focusing attention on question verbs, adjectives, and content nouns; ablating attention or replacing the DA head with a turn-boundary prediction head would isolate whether the gain comes from act semantics or from any discourse-level regularizer.","The paper's observation that coherence supervision does not improve DA prediction (M-DAP is on par with S-DAP) suggests the transfer is one-directional; one could test pretraining the shared encoder on a large DA corpus before coherence training as a cheaper substitute for joint multi-task learning.","If the reason for the SwitchBoard gap is fine-grained, imbalanced DA labels, then mapping SwitchBoard acts onto a coarse taxonomy before multi-task training should recover most of the margin; this is directly checkable.","The same multi-task recipe could be applied to other dialogue quality dimensions, such as using sentiment or topic labels as auxiliary tasks for coherence, since the method only requires a cheap auxiliary label that correlates with the target quality."],"forward_implications":["Coherence scoring no longer requires gold or predicted dialogue act labels at evaluation time, removing an annotation bottleneck.","The model can compare open-domain dialogue agents for coherence without an upstream dialogue-act tagger, so act-classifier errors do not propagate into the coherence score.","The two new perturbation types, utterance replacement and even-utterance ordering, add harder, dialogue-specific tests to the evaluation protocol.","On DailyDialog the multi-task formulation beats the entity-grid baseline by large margins; on SwitchBoard the gap closes, indicating robustness depends on DA granularity and utterance length.","Because act labels are only a training signal, the same architecture can be used in domains where acts are annotated for a small training set but not for evaluation."],"supporting_citations":[{"why":"Defines the EAGrid entity-plus-DA baseline and the perturbation-based evaluation protocol that the proposed model must beat.","marker":"Cervone et al. (2018)"},{"why":"Provides the DailyDialog corpus with its four coarse dialogue act labels and train/dev/test splits.","marker":"Li et al. (2017)"},{"why":"Provides the SwitchBoard dialogue act annotation scheme and corpus used for the second evaluation.","marker":"Jurafsky and Shriberg (1997)"},{"why":"Supplies the entity-grid coherence idea and the utterance ordering and insertion perturbation method that this work adapts.","marker":"Barzilay and Lapata (2005)"},{"why":"Supplies the uncertainty-based weighting of multi-task losses used in the total objective.","marker":"Kendall et al. (2018)"},{"why":"Supplies the pairwise preference ranking loss used for coherence scoring.","marker":"Gao et al. (2019)"}],"fun_headline_variants":["Coherence ranking without dialogue act labels at test time","Drop dialogue act labels: coherence scored via auxiliary task","Auxiliary dialogue acts train coherence scorer for label-free ranking","Dialogue coherence scores drop label dependency with multi-task training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that every synthetically perturbed dialogue is always less coherent than the original, so accuracy in telling original from perturbed is a valid measure of coherence; if the model is instead exploiting surface artifacts of the perturbations, the rankings would not transfer to real dialogue systems.","fun_headline_variants_meta":{"raw":{"variants":["Coherence ranking without dialogue act labels at test time","Drop dialogue act labels: coherence scored via auxiliary task","Auxiliary dialogue acts train coherence scorer for label-free ranking","Dialogue coherence scores drop label dependency with multi-task training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1372,"prompt_tokens":897,"completion_tokens":475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":409}},"tokens_in":513,"tokens_out":475,"duration_ms":5283,"temperature":1.0,"reasoning_tokens":409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:37:54.617181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect human coherence rankings for unperturbed dialogues from DailyDialog and SwitchBoard and compare them with M-DiCoh's scores: if the model's ranking correlates with human judgments no better than a model trained without the dialogue-act auxiliary task, the central claim fails. A simpler version is to score real dialogues that differ only by a single swapped turn and check whether human-preferred dialogues receive higher scores.","supporting_citations":[],"review_version":1}