REVIEW 3 major objections 4 minor 33 references
A Unified Neural Coherence Model
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A single neural framework combining sentence grammar, inter-sentence relations, and global patterns beats prior coherence models on both local and global discrimination.
desk verdict A useful new benchmark and a plausible architecture, but the below-chance L&H baseline and an inconsistent loss equation mean the headline accuracy claims need verification. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the adaptive window-level pairwise ranking loss, a training objective that compares positive and negative documents window by window and backs off the margin to zero whenever two corresponding local windows are identical, so only genuinely changed windows produce gradient. This loss trains a Siamese network built from a bi-LSTM sentence encoder with an auxiliary language-model objective and no parser, a bilinear layer that scores inter-sentence relations between consecutive sentences, and a lightweight depth-wise convolution module with global average pooling that supplies topic and attention context. The final coherence score for each window is a linear combination of the local bilinear features and the global pooled features, and the document score is the sum over windows; this makes local sensitivity a built-in property of both scoring and learning.
What would settle it
Retrain the distributed sentence model (L&H) and the non-star lexical neural grid on the same local discrimination training set with the same embedding and optimization budget used for the unified model, then compare accuracies on Dw=1,2,3; if the L&H model jumps from 6.76% to near chance or above and approaches the proposed model's 77.07%, the central claim that existing models fail on local contexts is refuted. A quick sanity check is whether any reimplementation of L&H on this task exceeds the 50% random baseline, since its reported 6.76% is below chance.
Extended reading notes
Core claim
The paper claims that a unified model, scoring local windows with explicit inter-sentence relations while also reading global document structure, outperforms prior coherence models on both local and global discrimination. The key component is the adaptive window-level pairwise ranking loss: positive and negative documents are compared window by window, and a rank margin is applied only when the corresponding windows differ, so locally coherent spans of a negative document are not unfairly penalized. With ELMo embeddings the full model reaches 93.19% on standard and 96.78% on inverse-order global discrimination, and 77.07% on the combined local discrimination set, against a best lexical neural grid baseline of 88.56% standard and 88.23% inverse. Ablations show the language-model loss helps consistently and that adding the global module improves local accuracy when that loss is present, though on inverse-order discrimination the global module hurts, pointing to order-sensitive local features as the carrier of the reversed-document signal.
Load-bearing premise
The load-bearing premise is that the baseline models, especially the distributed sentence model that scored 6.76% on the combined local task, were trained or tuned on the local discrimination task with comparable effort; if that baseline was merely evaluated as-is from global training, the paper's conclusion that existing models fail on local contexts would be an artifact of unequal evaluation.
Editorial extensions
If this is right
- A model of this kind can be applied directly to candidate ranking in dialogue and to reranking in machine translation, where a single misplaced or mismatched sentence should lower the score.
- Entity-grid neural models that pool over the entire document are shown to be insensitive to local window mismatches, so the unified model fills a gap those models leave open.
- Because the adaptive loss only penalizes windows that differ, the same trained scorer can be used on documents of any length without re-training.
- The language-model auxiliary loss is part of the recipe: the global module helps local accuracy only when the LM loss is present, so future coherence models should keep an explicit sentence-level objective.
Reading between the lines
- My inference: the adaptive window-level loss transfers to other pairwise ranking problems in which positive and negative examples share most of their structure, such as sentence-ordering in summarization or grammaticality reranking, because unchanged regions stop producing gradient noise.
- My inference: the inverse-order degradation when the global module is added suggests that average pooling discards order information; an order-sensitive global readout, such as position-aware attention, is a testable variant that might retain global topic cues without hurting reversed-document accuracy.
- My inference: the 6.76% accuracy of the distributed-sentence baseline on the combined local task, noted at Table 3, is below chance and the paper does not state whether that baseline was retrained on the local task; this particular number should not be read as a property of the model family until a fair re-tuning is run.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a unified neural coherence model that combines a bi-LSTM sentence encoder trained with an explicit language-model loss, a bilinear layer for inter-sentence discourse relations, and a lightweight convolution-pooling module for global coherence patterns. The model is trained with a window-level adaptive pairwise ranking loss and evaluated on the WSJ benchmark for global discrimination (standard and inverse order) as well as on newly created local discrimination tasks where positive and negative documents differ only in one or more permuted 3-sentence windows. The authors report large improvements over prior neural and entity-grid baselines and claim state-of-the-art results on both global and local coherence discrimination.
Significance. If the empirical claims hold, the paper makes a useful contribution by demonstrating that a single neural architecture can capture local window sensitivity and global document structure better than existing specialized models. The local discrimination task is a reasonable harder test that is relevant to ranking and generation applications, and the paper's ablations (Tables 5 and 6) give useful insight into the contributions of the global module and the language-model loss. The release of code, the five-seed averaging, and the comparison with multiple re-implemented baselines are also positive features. However, the reported below-chance performance of the L&H baseline and the mismatch between the described adaptive loss and its mathematical form are serious concerns that must be resolved before the central claims can be accepted.
major comments (3)
- [Section 5.3, Table 3; Section 5.4, Table 4] The reported L&H baseline scores are far below the 50% chance level: 6.76% on Dw=1,2,3 and 4.28% on Dw=1 in Table 3, and 17.39% and 18.11% on the standard and inverse global tasks in Table 4. For a pairwise discrimination task measured in accuracy, such numbers strongly indicate an evaluation-direction error, a sign inversion, or a training/evaluation mismatch rather than a genuine property of the model. Section 5.1 does not report how the L&H baseline was trained or adapted to the local task, only that the re-implementation differs from the original (bi-LSTM rather than RNN, Adam rather than AdaGrad). Since the paper's claim that existing models 'fail' on local contexts rests on this baseline, the authors must rerun the comparison with a verifiably correct protocol and document the training details; otherwise the headline improvement over L&H is unsupported.
- [Section 3.5, Eq. (8)] The adaptive margin with phi=0 for equal windows does not implement the stated behavior that the loss is 'active only for local windows that differ' between the positive and negative documents. With phi=0 the hinge term becomes max(0, Omega(D_neg^l) - Omega(D_pos^l)), which is nonzero whenever the negative window receives a higher score than the positive window. Because each window score in Eq. (6) contains the global document feature u, an identical local window from D_pos and D_neg can receive different scores due to different global contexts, so equal windows are still penalized in practice. The intended masking should be an explicit indicator function multiplying the hinge term, or an equivalent cancellation, rather than a zero margin. This issue is load-bearing because the adaptive loss is presented as a central methodological novelty.
- [Section 4, Local Discrimination Dataset] The description of the local discrimination dataset does not state whether cases are excluded in which a randomly permuted window happens to reproduce the original sentence order. The global discrimination task explicitly excludes permutations that match the original document, but the local task description only excludes overlapping windows and limits the number of negative samples per article. Such coincidental matches would create false negatives and could distort the reported accuracies. The authors should check and report whether this filtering was applied, and if not, quantify its effect.
minor comments (4)
- [Section 3.5] The notation is overloaded: the last paragraph writes 'Our total loss, L_Theta = L_Theta + L_lm', using L_Theta for both the ranking loss and the total loss. Please rename one of them.
- [Section 4] The sentence 'The number of local windows that we want to permute' defines w but the datasets are then named Dw=1, Dw=2, Dw=3, and Dw=1,2,3; clarify that Dw=1,2,3 is the concatenation and not a separate setting.
- [Section 5.2] The optimal hyperparameter values are said to be presented in the supplementary document; since the manuscript is intended to be read standalone, the key values (minibatch size, embedding size, kernel size, bilinear output dimension, margin tau) should be given in the main text or an appendix.
- [Figure 1] The figure is dense and the superscript 'negative outputs' notation is hard to distinguish visually; clearer alignment or a separate caption explanation would improve readability.
Circularity Check
No material circularity: the model is trained on paired permuted documents and evaluated on held-out sections, with no fitted constant reused as a prediction.
full rationale
The paper's derivation chain is an empirical neural modeling pipeline rather than a formal derivation from assumptions. The proposed model is trained end-to-end on pairs of coherent and permuted WSJ documents with a window-level pairwise ranking loss plus an auxiliary language-model loss, and all reported accuracies are on held-out test sections with no parameter fitted to the target metric or to the test labels. The architectural components (bi-LSTM sentence encoder, bilinear inter-sentence relation layer, lightweight convolution with average pooling, and linear scoring) are presented as modeling choices, and their contributions are evaluated through ablations in Tables 5 and 6. The adaptive margin in Eq. (8) is a standard masked pairwise ranking loss, not a hidden reuse of the evaluation answer. The only self-citations are to prior neural entity-grid models by the same group (Nguyen and Joty 2017; Mohiuddin et al. 2018); those are used as baselines and related work, their code is publicly available, and the paper reports both reproduced and original-paper results, so the citations do not smuggle in an assumption equivalent to the paper's conclusion. The below-chance L&H baseline in Table 3 raises a legitimate question about evaluation fairness, but that is a correctness concern, not circular reasoning: nothing in the paper's equations or definitions forces that baseline score. Overall, the central claim is supported by independent, held-out evaluation of a newly trained model, so no significant circularity is present.
Assumptions & free parameters
free parameters (5)
- margin constant tau =
not reported in main text (tuned on dev set)
- sentence embedding size =
selected from {128, 256}
- lightweight convolution kernel size =
selected from {3, 5, 7, 9}; global model uses kernel 5 in one described variant
- bilinear output dimension =
selected from {32, 64}
- minibatch size =
selected from {5, 10, 20, 25}
assumptions (4)
- domain assumption Sentence-order permutation is a valid proxy for incoherence.
- domain assumption The WSJ corpus and its train/test split support generalization claims.
- domain assumption The bi-LSTM with LM loss captures 'sentence grammar' and the bilinear layer captures 'discourse relations'.
- domain assumption Lightweight convolution with average pooling captures global entity and topic coherence.
Cite this review
Pith. "Pith review of A Unified Neural Coherence Model." pith.science (2026). https://pith.science/paper/RSHBMXOC
@misc{pith2026190900349,
author = {Pith},
title = {Pith review of: A Unified Neural Coherence Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/RSHBMXOC}},
note = {Machine review of arXiv:1909.00349}
}
read the original abstract
Recently, neural approaches to coherence modeling have achieved state-of-the-art results in several evaluation tasks. However, we show that most of these models often fail on harder tasks with more realistic application scenarios. In particular, the existing models underperform on tasks that require the model to be sensitive to local contexts such as candidate ranking in conversational dialogue and in machine translation. In this paper, we propose a unified coherence model that incorporates sentence grammar, inter-sentence coherence relations, and global coherence patterns into a common neural framework. With extensive experiments on local and global discrimination tasks, we demonstrate that our proposed model outperforms existing models by a good margin, and establish a new state-of-the-art.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL '05, pages 141--148, Ann Arbor, Michigan. Association for Computational Linguistics
work page 2005
-
[4]
Regina Barzilay and Mirella Lapata. 2008. http://www.aclweb.org/anthology/J08-1001 Modeling local coherence: An entity-based approach . Computational Linguistics, 34(1):1--34
work page 2008
-
[5]
Regina Barzilay and Lillian Lee. 2004. https://www.aclweb.org/anthology/N04-1015 Catching the drift: Probabilistic content models, with applications to generation and summarization . In HLT-NAACL 2004: Main Proceedings, pages 113--120, Boston, Massachusetts, USA. Association for Computational Linguistics
2004
-
[6]
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard S\" a ckinger, and Roopak Shah. 1993. http://dl.acm.org/citation.cfm?id=2987189.2987282 Signature verification using a "siamese" time delay neural network . In Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS'93, pages 737--744, San Francisco, CA, USA. Morgan Ka...
-
[7]
Fran c ois Chollet. 2016. http://arxiv.org/abs/1610.02357 Xception: Deep learning with depthwise separable convolutions . CoRR, abs/1610.02357
arXiv 2016
-
[8]
Ronan Collobert, Jason Weston, L \'e on Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493--2537
work page 2011
Show all 33 references
-
[9]
Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007. https://www.aclweb.org/anthology/N07-1055 A unified local and global model for discourse coherence . In Human Language Technologies 2007: The Conference of the North A merican Chapter of the Association for Computatio...
2007
-
[10]
Micha Elsner and Eugene Charniak. 2011 a . http://dl.acm.org/citation.cfm?id=2002472.2002622 Disentangling chat with local coherence models . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT...
2011
-
[11]
Micha Elsner and Eugene Charniak. 2011 b . Extending the entity grid with entity-specific features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT '11, pages 125--129, Portla...
2011
-
[12]
Vanessa Wei Feng and Graeme Hirst. 2012. Extending the entity-based coherence model with multiple ranks. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL '12, pages 315--324, Avignon, France. Association for C...
2012
-
[13]
Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. 2014. The impact of deep hierarchical discourse structures in the evaluation of text coherence. In COLING
2014
-
[14]
Grosz and Candace L
Barbara J. Grosz and Candace L. Sidner. 1986. https://www.aclweb.org/anthology/J86-3001 Attention, intentions, and the structure of discourse . Computational Linguistics, 12(3):175--204
1986
-
[15]
Grosz, Scott Weinstein, and Aravind K
Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995. Centering: A framework for modeling the local coherence of discourse. Comput. Linguist., 21(2):203--225
1995
-
[16]
Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. 2007. Hidden topic markov models. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, volume 2 of Proceedings of Machine Learning Research, pages 163--170, San Juan, Puerto Rico. PMLR
2007
-
[17]
Camille Guinaudeau and Michael Strube. 2013. Graph-based local coherence modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers , pages 93--103
2013
-
[18]
Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780
1997
-
[19]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. 2015. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings
2015 arXiv
-
[20]
Jiwei Li and Eduard Hovy. 2014. http://www.aclweb.org/anthology/D14-1218 A model of coherence based on distributed sentence representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2039--2048, Doha, Qatar. Associ...
2014
-
[21]
Jiwei Li and Dan Jurafsky. 2017. Neural net models of open-domain discourse coherence. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 198--209, Copenhagen, Denmark. Association for Computational Linguistics
2017
-
[22]
Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically evaluating text coherence using discourse relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT '11, pages 997--1006, Port...
2011
-
[23]
Annie Louis and Ani Nenkova. 2012. http://dl.acm.org/citation.cfm?id=2390948.2391078 A coherence model based on syntactic patterns . In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMN...
2012
-
[24]
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. https://doi.org/10.18653/v1/W15-4640 The U buntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems . In Proceedings of the 16th Annual Meeting of the Special Interest Group o...
2015 doi
-
[25]
Mann and S
W. Mann and S. Thompson. 1988. Rhetorical Structure Theory: Toward a Functional Theory of Text Organization . Text, 8(3):243--281
1988
-
[26]
Mohsen Mesgar and Michael Strube. 2018. https://www.aclweb.org/anthology/D18-1464 A neural local coherence model for text quality assessment . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4328--4339, Brussels, Belgium. Associ...
2018
-
[27]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111--3119
2013
-
[28]
Muhammad Tasnim Mohiuddin, Shafiq Joty, and Dat Tien Nguyen. 2018. https://www.aclweb.org/anthology/P18-1052 Coherence modeling of asynchronous conversations: A neural entity grid approach . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis...
2018
-
[30]
Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL
2018
-
[31]
Emily Pitler and Ani Nenkova. 2008. https://www.aclweb.org/anthology/D08-1020 Revisiting readability: A unified framework for predicting text quality . In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 186--195, Honolulu, Hawaii. ...
2008
-
[32]
Radu Soricut and Daniel Marcu. 2006. Discourse generation using utility-trained coherence models. In Proceedings of the COLING/ACL on Main Conference Poster Sessions, COLING-ACL '06, pages 803--810, Sydney, Australia. Association for Computational Linguistics
2006
-
[33]
Dat Tien Nguyen and Shafiq Joty. 2017. https://doi.org/10.18653/v1/P17-1121 A neural local coherence model . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1320--1330, Vancouver, Canada. Association for...
2017 doi
-
[34]
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. https://openreview.net/forum?id=SkVhlh09tX Pay less attention with lightweight and dynamic convolutions . In International Conference on Learning Representations
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.