{"id":"406e97af-a9cc-4031-b296-62f228a822c8","arxiv_id":"1909.00349","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A unified neural model with a bi-LSTM sentence encoder, bilinear relation layer, lightweight-convolution global module, and an adaptive window-level ranking loss achieves state-of-the-art coherence discrimination on WSJ and on a new local-order benchmark.","lead":"A neural coherence model combines sentence, discourse-relation, and global-context signals and reports large accuracy gains over earlier models on sentence-order discrimination. The paper also introduces a harder local-permutation benchmark that prior models struggle with, which may matter for dialogue and translation ranking.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The below-chance L&H baseline (6.76%) indicates a broken or asymmetric evaluation, so the 'existing models fail' claim is not yet supported.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I would flag: the L&H baseline score of 6.76% on the local discrimination task is below chance and unexplained, so the paper's claim that existing neural models 'fail' on local contexts is not yet established. My read does not change the reader's CONDITIONAL verdict: the concern is real, but it is a threat to the external validity of one comparison, not a demonstration that the proposed model is wrong. The other issues noted by the reader are less central. The apparent contradiction around Eq. (8) and the prose 'active only if windows differ' is weaker than it looks because the network shares parameters, so identical windows usually receive identical scores unless the global feature differs; the real question is whether the global feature can make identical windows differ, which is an implementation detail. The missing confidence intervals and deferred hyperparameters affect reproducibility but do not by themselves overturn the main result. The 'Our Best Model' issue in Table 4 matters for the global-discrimination claim, but the local-discrimination claim and the proposed framework's core comparison still depend most heavily on Table 3, where the broken L&H baseline is the most visible red flag. The concrete retraining check would settle whether the headline margin over existing models is real or an artifact of unequal evaluation.","tokens_in":12862,"tokens_out":11241,"duration_ms":109488,"concrete_test":"Retrain the re-implemented L&H model on the local discrimination training set (Dw=1,2,3) under the same paired examples, the same pairwise ranking objective, the same optimizer and epoch budget, and the same early-stopping used for the proposed local model, and verify that training accuracy exceeds chance before testing. Report test accuracy with standard deviation over the same five seeds used for the proposed model. If L&H then scores above 50% and near the proposed local model, the Table 3 comparison is not a fair test of existing neural coherence models; if it remains below 50%, inspect the score direction and rerun with a sanity check that the training loss decreases.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that existing coherence models 'fail' on local discrimination rests on Table 3, where the L&H baseline scores 6.76% on Dw=1,2,3 and 4.28% on Dw=1, far below the 50% chance level. Since L&H is a window-based neural model whose output is a coherence score, below-chance accuracy cannot be a genuine property of the model: a random model would score 50%, and any model with even weak signal would score above 50% if the evaluation direction is correct. The paper does not state whether L&H was retrained on the local discrimination task, how many epochs, which loss, or whether it was evaluated with the same score direction as the proposed model. Section 5.1 only notes that their implementation differs from the original (bi-LSTM instead of RNN, Adam instead of AdaGrad), so the re-implementation is not verified against the original model. One possibility is that L&H was trained only on global discrimination and evaluated on local pairs, so the comparison is not apples-to-apples; another is a sign inversion in computing document scores. Either way, the striking under-performance of this baseline, used as evidence that 'most neural models fail on harder tasks', may be an artifact of the evaluation protocol rather than a limitation of existing models. Because the headline improvement over L&H is roughly 68 points on Dw=1,2,3, the fairness of this baseline is load-bearing for the paper's central empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified neural coherence model that combines a bi-LSTM sentence encoder trained with an explicit language-model loss, a bilinear layer for inter-sentence discourse relations, and a lightweight convolution-pooling module for global coherence patterns. The model is trained with a window-level adaptive pairwise ranking loss and evaluated on the WSJ benchmark for global discrimination (standard and inverse order) as well as on newly created local discrimination tasks where positive and negative documents differ only in one or more permuted 3-sentence windows. The authors report large improvements over prior neural and entity-grid baselines and claim state-of-the-art results on both global and local coherence discrimination.","tokens_in":13001,"tokens_out":6330,"duration_ms":60502,"significance":"If the empirical claims hold, the paper makes a useful contribution by demonstrating that a single neural architecture can capture local window sensitivity and global document structure better than existing specialized models. The local discrimination task is a reasonable harder test that is relevant to ranking and generation applications, and the paper's ablations (Tables 5 and 6) give useful insight into the contributions of the global module and the language-model loss. The release of code, the five-seed averaging, and the comparison with multiple re-implemented baselines are also positive features. However, the reported below-chance performance of the L&H baseline and the mismatch between the described adaptive loss and its mathematical form are serious concerns that must be resolved before the central claims can be accepted.","major_comments":[{"comment":"The reported L&H baseline scores are far below the 50% chance level: 6.76% on Dw=1,2,3 and 4.28% on Dw=1 in Table 3, and 17.39% and 18.11% on the standard and inverse global tasks in Table 4. For a pairwise discrimination task measured in accuracy, such numbers strongly indicate an evaluation-direction error, a sign inversion, or a training/evaluation mismatch rather than a genuine property of the model. Section 5.1 does not report how the L&H baseline was trained or adapted to the local task, only that the re-implementation differs from the original (bi-LSTM rather than RNN, Adam rather than AdaGrad). Since the paper's claim that existing models 'fail' on local contexts rests on this baseline, the authors must rerun the comparison with a verifiably correct protocol and document the training details; otherwise the headline improvement over L&H is unsupported.","section":"Section 5.3, Table 3; Section 5.4, Table 4"},{"comment":"The adaptive margin with phi=0 for equal windows does not implement the stated behavior that the loss is 'active only for local windows that differ' between the positive and negative documents. With phi=0 the hinge term becomes max(0, Omega(D_neg^l) - Omega(D_pos^l)), which is nonzero whenever the negative window receives a higher score than the positive window. Because each window score in Eq. (6) contains the global document feature u, an identical local window from D_pos and D_neg can receive different scores due to different global contexts, so equal windows are still penalized in practice. The intended masking should be an explicit indicator function multiplying the hinge term, or an equivalent cancellation, rather than a zero margin. This issue is load-bearing because the adaptive loss is presented as a central methodological novelty.","section":"Section 3.5, Eq. (8)"},{"comment":"The description of the local discrimination dataset does not state whether cases are excluded in which a randomly permuted window happens to reproduce the original sentence order. The global discrimination task explicitly excludes permutations that match the original document, but the local task description only excludes overlapping windows and limits the number of negative samples per article. Such coincidental matches would create false negatives and could distort the reported accuracies. The authors should check and report whether this filtering was applied, and if not, quantify its effect.","section":"Section 4, Local Discrimination Dataset"}],"minor_comments":[{"comment":"The notation is overloaded: the last paragraph writes 'Our total loss, L_Theta = L_Theta + L_lm', using L_Theta for both the ranking loss and the total loss. Please rename one of them.","section":"Section 3.5"},{"comment":"The sentence 'The number of local windows that we want to permute' defines w but the datasets are then named Dw=1, Dw=2, Dw=3, and Dw=1,2,3; clarify that Dw=1,2,3 is the concatenation and not a separate setting.","section":"Section 4"},{"comment":"The optimal hyperparameter values are said to be presented in the supplementary document; since the manuscript is intended to be read standalone, the key values (minibatch size, embedding size, kernel size, bilinear output dimension, margin tau) should be given in the main text or an appendix.","section":"Section 5.2"},{"comment":"The figure is dense and the superscript 'negative outputs' notation is hard to distinguish visually; clearer alignment or a separate caption explanation would improve readability.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The below-chance L&H results in both Tables 3 and 4 are the main risk to the paper's empirical claims. If a corrected baseline closes the gap, the local-discrimination novelty may still stand, but the margin over prior work will need to be restated. The Eq. (8) masking issue is a clear correctness problem that should be fixed by using an explicit indicator mask, and the consequence for the reported local results should be assessed by rerunning the experiments. The paper is otherwise within scope, and the code release and ablation studies are strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know: this paper introduces a genuinely useful new evaluation setup for coherence—local discrimination with window-wise permutations—and the unified architecture (LSTM sentence encoder + bilinear relations + lightweight conv pooling) is a sensible integration of components that mostly exist in the literature. But the central empirical claim that existing models 'fail' on local tasks rests on a baseline number that is below chance and almost certainly broken. That makes the paper's headline story unreliable until the comparison is redone.\n\nWhat's good: the task design is a real contribution. Permuting 1–3 local windows gives a more sensitive probe for dialogue ranking and MT evaluation, and the adaptive window-level pairwise loss is a neat idea even if the implementation is muddled. The ablation study is honest about cases where the global component hurts. Code is released.\n\nSoft spots, in order of severity. First, Table 3 reports Li & Hovy's distributed sentence model at 6.76% on Dw=1,2,3 and 4.28% on Dw=1; Table 4 reports the same model at 17.39% on standard global discrimination. Those are far below chance. A model with any signal cannot be below chance unless there is a sign inversion or a severe mismatch between training and evaluation. The paper never explains these numbers, and since L&H is the baseline that supports the 'most neural models fail' narrative, this is load-bearing. Before accepting the local benchmark results, I'd want to see L&H re-trained and evaluated under the same protocol with the same score direction.\n\nSecond, Eq. (8) does not implement the prose. With φ=0 for equal windows, the hinge still pushes the positive score above the negative score even when the windows are identical. You'd need a mask or a different formulation to make the loss active only for differing windows. This ambiguity matters because the adaptive loss is claimed as a contribution.\n\nThird, Table 4's 'Our Best Model' is not a single configuration. From the ablations, the standard task benefits from the global model and LM loss, while the inverse task is best with local + LM loss. Reporting the best per task overstates the unified model. A single architecture with fixed hyperparameters should be used.\n\nMinor point: no confidence intervals despite averaging over five runs, and hyperparameters are in a supplementary that isn't described here.\n\nBottom line: the benchmark is worth a serious look, but the empirical claims need a corrected baseline and a clarified loss before they can be taken at face value. Send it to review, but require a faithful L&H rerun and a precise statement of the training objective.\n\nRegards.","headline":"A useful new benchmark and a plausible architecture, but the below-chance L&H baseline and an inconsistent loss equation mean the headline accuracy claims need verification.","tokens_in":13666,"tokens_out":5464,"would_cite":true,"duration_ms":51016,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single neural framework combining sentence grammar, inter-sentence relations, and global patterns beats prior coherence models on both local and global discrimination.","keywords":["unified neural coherence model","discourse coherence","local discrimination","global discrimination","entity grid","neural network","ranking loss","text coherence"],"falsifier":"Retrain the distributed sentence model (L&H) and the non-star lexical neural grid on the same local discrimination training set with the same embedding and optimization budget used for the unified model, then compare accuracies on Dw=1,2,3; if the L&H model jumps from 6.76% to near chance or above and approaches the proposed model's 77.07%, the central claim that existing models fail on local contexts is refuted. A quick sanity check is whether any reimplementation of L&H on this task exceeds the 50% random baseline, since its reported 6.76% is below chance.","tokens_in":12515,"feed_emoji":"📈","tokens_out":11800,"duration_ms":97924,"temperature":0.7,"pith_summary":"This paper argues that existing neural coherence models, trained with document-level losses and global pooling, miss local disordering that matters for real applications such as dialogue candidate ranking and machine-translation reranking. The authors propose one neural architecture that combines sentence grammar (a bi-LSTM with language-model loss), inter-sentence discourse relations (a bilinear layer), and global topic and attention patterns (lightweight convolution with average pooling), trained with an adaptive window-level pairwise ranking loss. On the Wall Street Journal benchmark they report accuracies up to 77.07% on the combined local discrimination set and up to 96.78% on inverse-order global discrimination, exceeding all baselines. The central bet is that the local 3-sentence window is the right unit for both scoring and training; if true, this yields a coherence scorer that is sensitive to local context rather than only to global sentence order.","feed_headline":"Unified neural model outranks prior coherence systems on every task","feed_subtitle":"One window-level loss lets one model handle local and global coherence, hitting 93.2% global and 77.1% local.","key_machinery":"The central mechanism is the adaptive window-level pairwise ranking loss, a training objective that compares positive and negative documents window by window and backs off the margin to zero whenever two corresponding local windows are identical, so only genuinely changed windows produce gradient. This loss trains a Siamese network built from a bi-LSTM sentence encoder with an auxiliary language-model objective and no parser, a bilinear layer that scores inter-sentence relations between consecutive sentences, and a lightweight depth-wise convolution module with global average pooling that supplies topic and attention context. The final coherence score for each window is a linear combination of the local bilinear features and the global pooled features, and the document score is the sum over windows; this makes local sensitivity a built-in property of both scoring and learning.","core_discovery":"The paper claims that a unified model, scoring local windows with explicit inter-sentence relations while also reading global document structure, outperforms prior coherence models on both local and global discrimination. The key component is the adaptive window-level pairwise ranking loss: positive and negative documents are compared window by window, and a rank margin is applied only when the corresponding windows differ, so locally coherent spans of a negative document are not unfairly penalized. With ELMo embeddings the full model reaches 93.19% on standard and 96.78% on inverse-order global discrimination, and 77.07% on the combined local discrimination set, against a best lexical neural grid baseline of 88.56% standard and 88.23% inverse. Ablations show the language-model loss helps consistently and that adding the global module improves local accuracy when that loss is present, though on inverse-order discrimination the global module hurts, pointing to order-sensitive local features as the carrier of the reversed-document signal.","pith_inferences":["My inference: the adaptive window-level loss transfers to other pairwise ranking problems in which positive and negative examples share most of their structure, such as sentence-ordering in summarization or grammaticality reranking, because unchanged regions stop producing gradient noise.","My inference: the inverse-order degradation when the global module is added suggests that average pooling discards order information; an order-sensitive global readout, such as position-aware attention, is a testable variant that might retain global topic cues without hurting reversed-document accuracy.","My inference: the 6.76% accuracy of the distributed-sentence baseline on the combined local task, noted at Table 3, is below chance and the paper does not state whether that baseline was retrained on the local task; this particular number should not be read as a property of the model family until a fair re-tuning is run."],"forward_implications":["A model of this kind can be applied directly to candidate ranking in dialogue and to reranking in machine translation, where a single misplaced or mismatched sentence should lower the score.","Entity-grid neural models that pool over the entire document are shown to be insensitive to local window mismatches, so the unified model fills a gap those models leave open.","Because the adaptive loss only penalizes windows that differ, the same trained scorer can be used on documents of any length without re-training.","The language-model auxiliary loss is part of the recipe: the global module helps local accuracy only when the LM loss is present, so future coherence models should keep an explicit sentence-level objective."],"supporting_citations":[{"why":"defines the entity-grid model and the standard global discrimination task that the paper benchmarks against.","marker":"(Barzilay and Lapata, 2008)"},{"why":"provides the distributed sentence baseline that the paper claims fails on local discrimination.","marker":"(Li and Hovy, 2014)"},{"why":"supplies the extended entity-grid baselines used for global discrimination.","marker":"(Elsner and Charniak, 2011b)"},{"why":"introduces the neural entity grid with convolution and pooling that the paper identifies as insensitive to local mismatches.","marker":"(Tien Nguyen and Joty, 2017)"},{"why":"supplies the lexicalized neural grid baseline and the inverse-order evaluation setup.","marker":"(Mohiuddin et al., 2018)"},{"why":"provides ELMo contextual embeddings used in the paper's best-performing configurations.","marker":"(Peters et al., 2018)"},{"why":"introduces lightweight convolution, the parameter-efficient global module in the unified model.","marker":"(Wu et al., 2019)"},{"why":"provides the pairwise ranking approach the paper adapts into its adaptive window-level loss.","marker":"(Collobert et al., 2011)"}],"fun_headline_variants":["One model beats all coherence tasks, local and global","Unified coherence net tops prior models on all tasks","Single neural model wins both local and global coherence","New coherence model: one loss, top scores everywhere","Adaptive ranking loss lifts neural coherence to SOTA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the baseline models, especially the distributed sentence model that scored 6.76% on the combined local task, were trained or tuned on the local discrimination task with comparable effort; if that baseline was merely evaluated as-is from global training, the paper's conclusion that existing models fail on local contexts would be an artifact of unequal evaluation.","fun_headline_variants_meta":{"raw":{"variants":["One model beats all coherence tasks, local and global","Unified coherence net tops prior models on all tasks","Single neural model wins both local and global coherence","New coherence model: one loss, top scores everywhere","Adaptive ranking loss lifts neural coherence to SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000562,"raw_usage":{"total_tokens":2618,"prompt_tokens":845,"completion_tokens":1773,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":1698}},"tokens_in":461,"tokens_out":1773,"duration_ms":77833,"temperature":1.0,"reasoning_tokens":1698,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:54:33.499030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the distributed sentence model (L&H) and the non-star lexical neural grid on the same local discrimination training set with the same embedding and optimization budget used for the unified model, then compare accuracies on Dw=1,2,3; if the L&H model jumps from 6.76% to near chance or above and approaches the proposed model's 77.07%, the central claim that existing models fail on local contexts is refuted. A quick sanity check is whether any reimplementation of L&H on this task exceeds the 50% random baseline, since its reported 6.76% is below chance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the entity-grid model and the standard global discrimination task that the paper benchmarks against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the distributed sentence baseline that the paper claims fails on local discrimination."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the lexicalized neural grid baseline and the inverse-order evaluation setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the pairwise ranking approach the paper adapts into its adaptive window-level loss."}],"review_version":1}