Pith. sign in

REVIEW 3 major objections 4 minor 33 references

A Unified Neural Coherence Model

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A single neural framework combining sentence grammar, inter-sentence relations, and global patterns beats prior coherence models on both local and global discrimination.

desk verdict A useful new benchmark and a plausible architecture, but the below-chance L&H baseline and an inconsistent loss equation mean the headline accuracy claims need verification. read the letter →

arxiv 1909.00349 v1 pith:RSHBMXOC submitted 2019-09-01 cs.CL cs.LGstat.ML

classification cs.CLcs.LGstat.ML
keywords unifiedneuralcoherencemodeldiscourselocaldiscriminationglobalentitygridnetworkrankinglosstext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that existing neural coherence models, trained with document-level losses and global pooling, miss local disordering that matters for real applications such as dialogue candidate ranking and machine-translation reranking. The authors propose one neural architecture that combines sentence grammar (a bi-LSTM with language-model loss), inter-sentence discourse relations (a bilinear layer), and global topic and attention patterns (lightweight convolution with average pooling), trained with an adaptive window-level pairwise ranking loss. On the Wall Street Journal benchmark they report accuracies up to 77.07% on the combined local discrimination set and up to 96.78% on inverse-order global discrimination, exceeding all baselines. The central bet is that the local 3-sentence window is the right unit for both scoring and training; if true, this yields a coherence scorer that is sensitive to local context rather than only to global sentence order.

What carries the argument

The central mechanism is the adaptive window-level pairwise ranking loss, a training objective that compares positive and negative documents window by window and backs off the margin to zero whenever two corresponding local windows are identical, so only genuinely changed windows produce gradient. This loss trains a Siamese network built from a bi-LSTM sentence encoder with an auxiliary language-model objective and no parser, a bilinear layer that scores inter-sentence relations between consecutive sentences, and a lightweight depth-wise convolution module with global average pooling that supplies topic and attention context. The final coherence score for each window is a linear combination of the local bilinear features and the global pooled features, and the document score is the sum over windows; this makes local sensitivity a built-in property of both scoring and learning.

What would settle it

Retrain the distributed sentence model (L&H) and the non-star lexical neural grid on the same local discrimination training set with the same embedding and optimization budget used for the unified model, then compare accuracies on Dw=1,2,3; if the L&H model jumps from 6.76% to near chance or above and approaches the proposed model's 77.07%, the central claim that existing models fail on local contexts is refuted. A quick sanity check is whether any reimplementation of L&H on this task exceeds the 50% random baseline, since its reported 6.76% is below chance.

Watch

Extended reading notes

Core claim

The paper claims that a unified model, scoring local windows with explicit inter-sentence relations while also reading global document structure, outperforms prior coherence models on both local and global discrimination. The key component is the adaptive window-level pairwise ranking loss: positive and negative documents are compared window by window, and a rank margin is applied only when the corresponding windows differ, so locally coherent spans of a negative document are not unfairly penalized. With ELMo embeddings the full model reaches 93.19% on standard and 96.78% on inverse-order global discrimination, and 77.07% on the combined local discrimination set, against a best lexical neural grid baseline of 88.56% standard and 88.23% inverse. Ablations show the language-model loss helps consistently and that adding the global module improves local accuracy when that loss is present, though on inverse-order discrimination the global module hurts, pointing to order-sensitive local features as the carrier of the reversed-document signal.

Load-bearing premise

The load-bearing premise is that the baseline models, especially the distributed sentence model that scored 6.76% on the combined local task, were trained or tuned on the local discrimination task with comparable effort; if that baseline was merely evaluated as-is from global training, the paper's conclusion that existing models fail on local contexts would be an artifact of unequal evaluation.

Editorial extensions

If this is right

  • A model of this kind can be applied directly to candidate ranking in dialogue and to reranking in machine translation, where a single misplaced or mismatched sentence should lower the score.
  • Entity-grid neural models that pool over the entire document are shown to be insensitive to local window mismatches, so the unified model fills a gap those models leave open.
  • Because the adaptive loss only penalizes windows that differ, the same trained scorer can be used on documents of any length without re-training.
  • The language-model auxiliary loss is part of the recipe: the global module helps local accuracy only when the LM loss is present, so future coherence models should keep an explicit sentence-level objective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the adaptive window-level loss transfers to other pairwise ranking problems in which positive and negative examples share most of their structure, such as sentence-ordering in summarization or grammaticality reranking, because unchanged regions stop producing gradient noise.
  • My inference: the inverse-order degradation when the global module is added suggests that average pooling discards order information; an order-sensitive global readout, such as position-aware attention, is a testable variant that might retain global topic cues without hurting reversed-document accuracy.
  • My inference: the 6.76% accuracy of the distributed-sentence baseline on the combined local task, noted at Table 3, is below chance and the paper does not state whether that baseline was retrained on the local task; this particular number should not be read as a property of the model family until a fair re-tuning is run.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a unified neural coherence model that combines a bi-LSTM sentence encoder trained with an explicit language-model loss, a bilinear layer for inter-sentence discourse relations, and a lightweight convolution-pooling module for global coherence patterns. The model is trained with a window-level adaptive pairwise ranking loss and evaluated on the WSJ benchmark for global discrimination (standard and inverse order) as well as on newly created local discrimination tasks where positive and negative documents differ only in one or more permuted 3-sentence windows. The authors report large improvements over prior neural and entity-grid baselines and claim state-of-the-art results on both global and local coherence discrimination.

Significance. If the empirical claims hold, the paper makes a useful contribution by demonstrating that a single neural architecture can capture local window sensitivity and global document structure better than existing specialized models. The local discrimination task is a reasonable harder test that is relevant to ranking and generation applications, and the paper's ablations (Tables 5 and 6) give useful insight into the contributions of the global module and the language-model loss. The release of code, the five-seed averaging, and the comparison with multiple re-implemented baselines are also positive features. However, the reported below-chance performance of the L&H baseline and the mismatch between the described adaptive loss and its mathematical form are serious concerns that must be resolved before the central claims can be accepted.

major comments (3)
  1. [Section 5.3, Table 3; Section 5.4, Table 4] The reported L&H baseline scores are far below the 50% chance level: 6.76% on Dw=1,2,3 and 4.28% on Dw=1 in Table 3, and 17.39% and 18.11% on the standard and inverse global tasks in Table 4. For a pairwise discrimination task measured in accuracy, such numbers strongly indicate an evaluation-direction error, a sign inversion, or a training/evaluation mismatch rather than a genuine property of the model. Section 5.1 does not report how the L&H baseline was trained or adapted to the local task, only that the re-implementation differs from the original (bi-LSTM rather than RNN, Adam rather than AdaGrad). Since the paper's claim that existing models 'fail' on local contexts rests on this baseline, the authors must rerun the comparison with a verifiably correct protocol and document the training details; otherwise the headline improvement over L&H is unsupported.
  2. [Section 3.5, Eq. (8)] The adaptive margin with phi=0 for equal windows does not implement the stated behavior that the loss is 'active only for local windows that differ' between the positive and negative documents. With phi=0 the hinge term becomes max(0, Omega(D_neg^l) - Omega(D_pos^l)), which is nonzero whenever the negative window receives a higher score than the positive window. Because each window score in Eq. (6) contains the global document feature u, an identical local window from D_pos and D_neg can receive different scores due to different global contexts, so equal windows are still penalized in practice. The intended masking should be an explicit indicator function multiplying the hinge term, or an equivalent cancellation, rather than a zero margin. This issue is load-bearing because the adaptive loss is presented as a central methodological novelty.
  3. [Section 4, Local Discrimination Dataset] The description of the local discrimination dataset does not state whether cases are excluded in which a randomly permuted window happens to reproduce the original sentence order. The global discrimination task explicitly excludes permutations that match the original document, but the local task description only excludes overlapping windows and limits the number of negative samples per article. Such coincidental matches would create false negatives and could distort the reported accuracies. The authors should check and report whether this filtering was applied, and if not, quantify its effect.
minor comments (4)
  1. [Section 3.5] The notation is overloaded: the last paragraph writes 'Our total loss, L_Theta = L_Theta + L_lm', using L_Theta for both the ranking loss and the total loss. Please rename one of them.
  2. [Section 4] The sentence 'The number of local windows that we want to permute' defines w but the datasets are then named Dw=1, Dw=2, Dw=3, and Dw=1,2,3; clarify that Dw=1,2,3 is the concatenation and not a separate setting.
  3. [Section 5.2] The optimal hyperparameter values are said to be presented in the supplementary document; since the manuscript is intended to be read standalone, the key values (minibatch size, embedding size, kernel size, bilinear output dimension, margin tau) should be given in the main text or an appendix.
  4. [Figure 1] The figure is dense and the superscript 'negative outputs' notation is hard to distinguish visually; clearer alignment or a separate caption explanation would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the model is trained on paired permuted documents and evaluated on held-out sections, with no fitted constant reused as a prediction.

full rationale

The paper's derivation chain is an empirical neural modeling pipeline rather than a formal derivation from assumptions. The proposed model is trained end-to-end on pairs of coherent and permuted WSJ documents with a window-level pairwise ranking loss plus an auxiliary language-model loss, and all reported accuracies are on held-out test sections with no parameter fitted to the target metric or to the test labels. The architectural components (bi-LSTM sentence encoder, bilinear inter-sentence relation layer, lightweight convolution with average pooling, and linear scoring) are presented as modeling choices, and their contributions are evaluated through ablations in Tables 5 and 6. The adaptive margin in Eq. (8) is a standard masked pairwise ranking loss, not a hidden reuse of the evaluation answer. The only self-citations are to prior neural entity-grid models by the same group (Nguyen and Joty 2017; Mohiuddin et al. 2018); those are used as baselines and related work, their code is publicly available, and the paper reports both reproduced and original-paper results, so the citations do not smuggle in an assumption equivalent to the paper's conclusion. The below-chance L&H baseline in Table 3 raises a legitimate question about evaluation fairness, but that is a correctness concern, not circular reasoning: nothing in the paper's equations or definitions forces that baseline score. Overall, the central claim is supported by independent, held-out evaluation of a newly trained model, so no significant circularity is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on standard supervised-learning assumptions: the synthetic permutation tasks are valid coherence measures, the WSJ evaluation is representative, and the architectural components correspond to the linguistic constructs they are named after. There are no new physical entities or fitted constants. The main free parameters are ordinary hyperparameters tuned on a development set, but their values are not given in the arXiv version, which weakens reproducibility.

free parameters (5)
  • margin constant tau = not reported in main text (tuned on dev set)
    Appears in Eq. (8); controls the hinge penalty size. The paper says hyperparameters are tuned on a development set, but the value is deferred to the supplementary document.
  • sentence embedding size = selected from {128, 256}
    Listed as a hyperparameter in Section 5.2 and tuned on the development set.
  • lightweight convolution kernel size = selected from {3, 5, 7, 9}; global model uses kernel 5 in one described variant
    Tuned on the development set; the kernel size changes the context window over sentences.
  • bilinear output dimension = selected from {32, 64}
    Tuned on the development set per Section 5.2.
  • minibatch size = selected from {5, 10, 20, 25}
    Tuned on the development set per Section 5.2.
assumptions (4)
  • domain assumption Sentence-order permutation is a valid proxy for incoherence.
    The local and global discrimination tasks define negatives by permuting sentence order within a window or across a document, and accuracy on these synthetic negatives is treated as coherence-modeling ability (Sections 4 and 5).
  • domain assumption The WSJ corpus and its train/test split support generalization claims.
    All results are reported on a single news domain (WSJ sections 00-13 train, 14-24 test), yet the introduction claims relevance to dialogue and machine translation without cross-domain evaluation.
  • domain assumption The bi-LSTM with LM loss captures 'sentence grammar' and the bilinear layer captures 'discourse relations'.
    These architectural mappings are asserted in Sections 3.1 and 3.2 but are not validated with intrinsic probes or linguistic analyses.
  • domain assumption Lightweight convolution with average pooling captures global entity and topic coherence.
    The global module is described as modeling attention and topic structures, but average pooling discards positional information, and the paper itself notes degradation on the inverse-order task (Section 5.5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Unified Neural Coherence Model." pith.science (2026). https://pith.science/paper/RSHBMXOC

@misc{pith2026190900349,
  author       = {Pith},
  title        = {Pith review of: A Unified Neural Coherence Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RSHBMXOC}},
  note         = {Machine review of arXiv:1909.00349}
}
read the original abstract

Recently, neural approaches to coherence modeling have achieved state-of-the-art results in several evaluation tasks. However, we show that most of these models often fail on harder tasks with more realistic application scenarios. In particular, the existing models underperform on tasks that require the model to be sensitive to local contexts such as candidate ranking in conversational dialogue and in machine translation. In this paper, we propose a unified coherence model that incorporates sentence grammar, inter-sentence coherence relations, and global coherence patterns into a common neural framework. With extensive experiments on local and global discrimination tasks, we demonstrate that our proposed model outperforms existing models by a good margin, and establish a new state-of-the-art.

Figures

Figures reproduced from arXiv: 1909.00349 by the authors.

Figure 1
Figure 1. An overview of the proposed coherence model (best viewed in color). The superscript ‘ [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Depth-wise convolution for kernel size k = 3. The convolutions are done over the input dimensions. Compared to the regular convolutions,depth-wise convolutions reduces the number of parameters from d 2k to dk (note that d = 2p in our case). Light-weight Convolutions make the depth-wise convolution even simpler by sharing groups of out￾put channels and normalizing weights across the temporal dimension using a softmax… view at source ↗
Figure 3
Figure 3. Sample data in the local permutation data [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Regina Barzilay and Mirella Lapata. 2005. Modeling local coherence: An entity-based approach. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL '05, pages 141--148, Ann Arbor, Michigan. Association for Computational Linguistics

  4. [4]

    Regina Barzilay and Mirella Lapata. 2008. http://www.aclweb.org/anthology/J08-1001 Modeling local coherence: An entity-based approach . Computational Linguistics, 34(1):1--34

  5. [5]

    Regina Barzilay and Lillian Lee. 2004. https://www.aclweb.org/anthology/N04-1015 Catching the drift: Probabilistic content models, with applications to generation and summarization . In HLT-NAACL 2004: Main Proceedings, pages 113--120, Boston, Massachusetts, USA. Association for Computational Linguistics

  6. [6]

    a ckinger, and Roopak Shah. 1993. http://dl.acm.org/citation.cfm?id=2987189.2987282 Signature verification using a

    Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard S\" a ckinger, and Roopak Shah. 1993. http://dl.acm.org/citation.cfm?id=2987189.2987282 Signature verification using a "siamese" time delay neural network . In Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS'93, pages 737--744, San Francisco, CA, USA. Morgan Ka...

  7. [7]

    Fran c ois Chollet. 2016. http://arxiv.org/abs/1610.02357 Xception: Deep learning with depthwise separable convolutions . CoRR, abs/1610.02357

  8. [8]

    Ronan Collobert, Jason Weston, L \'e on Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493--2537

Show all 33 references
  1. [9]

    Micha Elsner, Joseph Austerweil, and Eugene Charniak. 2007. https://www.aclweb.org/anthology/N07-1055 A unified local and global model for discourse coherence . In Human Language Technologies 2007: The Conference of the North A merican Chapter of the Association for Computatio...

  2. [10]

    Micha Elsner and Eugene Charniak. 2011 a . http://dl.acm.org/citation.cfm?id=2002472.2002622 Disentangling chat with local coherence models . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT...

  3. [11]

    Micha Elsner and Eugene Charniak. 2011 b . Extending the entity grid with entity-specific features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT '11, pages 125--129, Portla...

  4. [12]

    Vanessa Wei Feng and Graeme Hirst. 2012. Extending the entity-based coherence model with multiple ranks. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL '12, pages 315--324, Avignon, France. Association for C...

  5. [13]

    Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. 2014. The impact of deep hierarchical discourse structures in the evaluation of text coherence. In COLING

  6. [14]

    Grosz and Candace L

    Barbara J. Grosz and Candace L. Sidner. 1986. https://www.aclweb.org/anthology/J86-3001 Attention, intentions, and the structure of discourse . Computational Linguistics, 12(3):175--204

  7. [15]

    Grosz, Scott Weinstein, and Aravind K

    Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995. Centering: A framework for modeling the local coherence of discourse. Comput. Linguist., 21(2):203--225

  8. [16]

    Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. 2007. Hidden topic markov models. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, volume 2 of Proceedings of Machine Learning Research, pages 163--170, San Juan, Puerto Rico. PMLR

  9. [17]

    Camille Guinaudeau and Michael Strube. 2013. Graph-based local coherence modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers , pages 93--103

  10. [18]

    Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

  11. [19]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  12. [20]

    Jiwei Li and Eduard Hovy. 2014. http://www.aclweb.org/anthology/D14-1218 A model of coherence based on distributed sentence representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2039--2048, Doha, Qatar. Associ...

  13. [21]

    Jiwei Li and Dan Jurafsky. 2017. Neural net models of open-domain discourse coherence. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 198--209, Copenhagen, Denmark. Association for Computational Linguistics

  14. [22]

    Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically evaluating text coherence using discourse relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT '11, pages 997--1006, Port...

  15. [23]

    Annie Louis and Ani Nenkova. 2012. http://dl.acm.org/citation.cfm?id=2390948.2391078 A coherence model based on syntactic patterns . In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMN...

  16. [24]

    Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. https://doi.org/10.18653/v1/W15-4640 The U buntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems . In Proceedings of the 16th Annual Meeting of the Special Interest Group o...

  17. [25]

    Mann and S

    W. Mann and S. Thompson. 1988. Rhetorical Structure Theory: Toward a Functional Theory of Text Organization . Text, 8(3):243--281

  18. [26]

    Mohsen Mesgar and Michael Strube. 2018. https://www.aclweb.org/anthology/D18-1464 A neural local coherence model for text quality assessment . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4328--4339, Brussels, Belgium. Associ...

  19. [27]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111--3119

  20. [28]

    Muhammad Tasnim Mohiuddin, Shafiq Joty, and Dat Tien Nguyen. 2018. https://www.aclweb.org/anthology/P18-1052 Coherence modeling of asynchronous conversations: A neural entity grid approach . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis...

  21. [30]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL

  22. [31]

    Emily Pitler and Ani Nenkova. 2008. https://www.aclweb.org/anthology/D08-1020 Revisiting readability: A unified framework for predicting text quality . In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 186--195, Honolulu, Hawaii. ...

  23. [32]

    Radu Soricut and Daniel Marcu. 2006. Discourse generation using utility-trained coherence models. In Proceedings of the COLING/ACL on Main Conference Poster Sessions, COLING-ACL '06, pages 803--810, Sydney, Australia. Association for Computational Linguistics

  24. [33]

    Dat Tien Nguyen and Shafiq Joty. 2017. https://doi.org/10.18653/v1/P17-1121 A neural local coherence model . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1320--1330, Vancouver, Canada. Association for...

  25. [34]

    Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. https://openreview.net/forum?id=SkVhlh09tX Pay less attention with lightweight and dynamic convolutions . In International Conference on Learning Representations

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.