{"id":"aa1bd650-ec21-4545-94fe-a283518afe9d","arxiv_id":"1908.10721","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A self-attention reading comprehension model that injects discourse, semantic role, and coreference annotations into dedicated attention heads improves NarrativeQA question answering by up to 3.4 Rouge-L over the QANet baseline.","lead":"This paper adds discourse and semantic annotations (causal relations, event roles, coreferent mentions) as extra inputs to the self-attention heads of a question-answering model. On NarrativeQA, the modified model beats its plain QANet baseline by up to 3.4 Rouge-L, with the best gains from discourse relations and semantic role labels.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +3.4 Rouge-L gain over QANet is not statistically supported: single runs, no confidence intervals, and many hand-selected head configurations make the improvement potentially a multiple-comparisons artifact.","rationale":"The reader's assigned weakest assumption is annotation accuracy. That is a plausible concern, but the paper's own NoSense and Sent-span controls partially address it by showing that sense labels add over scope masks, even though those controls are not significance-tested. The more load-bearing threat is that the headline improvement itself may not be statistically reliable. A single run per configuration, with 355 independent documents and roughly fifteen hand-selected head allocations, means the reported +3.4 Rouge-L could arise from chance or from selecting the best of many comparisons. This directly undermines the central claim, independent of annotation quality. The reader mentions the absence of significance testing in the rationale, but the identified weakest assumption is different; I therefore mark partial agreement. The paper earns credit for the internal controls and the clear description of the oracle, which make the comparison to the own baseline reasonably fair, but the missing variance estimation is the decisive gap. The proposed concrete test would settle whether the gain is real; until then, the conditional verdict stands.","tokens_in":16036,"tokens_out":7040,"duration_ms":71945,"concrete_test":"Run the QANet baseline and the DR (Exp) configuration with at least 5 random seeds. For each run, compute per-document Rouge-L on the test set. Then perform a paired bootstrap over the 355 documents (resample documents, not questions, 10,000 times) to obtain a 95% confidence interval for the mean Rouge-L difference between DR (Exp) and QANet. Apply a Bonferroni correction for the number of configurations reported in Figures 4-5. If the corrected confidence interval excludes zero, the headline gain is reliable; otherwise the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the observed +3.4 Rouge-L improvement over QANet is real and attributable to the discourse-semantic annotations. The paper provides no statistical support: a single test-set run per configuration, no confidence intervals, no significance tests, and no adjustment for the fact that the 10.5k test questions are clustered within only 355 documents. Table 2 lists roughly fifteen hand-selected head-allocation configurations, and the paper highlights the best performer (DR (Exp), +3.43 in Figure 4). The maximum of many comparisons is expected to exceed the true effect even under the null, so the headline gain may be inflated by selection. The internal controls (DR (Exp NoSense), DR (NonE NoSense), Sent span 3) are also single runs and are not compared with significance tests; without variance estimates, we cannot tell whether the semantic sense labels contribute beyond the attention masks or whether the differences among configurations are noise. The conclusion's use of the word 'significant' is thus unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Discourse-Aware Semantic Self-Attention (DASSA), an extension of the QANet self-attention encoder for reading comprehension. The model augments selected self-attention heads with linguistic annotation types: semantic role labeling (SRL), coreference clusters, and shallow discourse relations (explicit and non-explicit). The annotations are provided as token-level label embeddings plus attention scope masks. The authors evaluate on the NarrativeQA summary setting, using a span-extraction formulation with their own Rouge-L-based oracle. They report up to +3.4 Rouge-L improvement over a QANet baseline and present fine-grained results by question type and context length, claiming that intra-sentential (explicit) discourse relations, SRL, and long-distance coreference are most beneficial.","tokens_in":16243,"tokens_out":3142,"duration_ms":35119,"significance":"If the central empirical claim is reliable, this is a useful contribution: it is one of the first attempts to inject discourse-level information into a self-attention reading-comprehension model, and the proposed mechanism is lightweight, adding only label embeddings and attention masks without increasing computational complexity. The paper is also transparent in describing its oracle construction, the annotation tools, and the head-allocation configurations, and it includes control conditions (NoSense, Sent span 3) that attempt to separate the contribution of sense labels from raw attention scope. However, the headline result rests on single runs with no variance estimates or significance testing, and the control comparisons are not statistically supported, so the empirical case is not yet convincing enough to support the paper's central claim as stated. The code-release commitment and the fine-grained evaluation are strengths, but they do not by themselves fix the statistical weakness of the main comparison.","major_comments":[{"comment":"All reported results are single test-set runs, with no confidence intervals, significance tests, or repeated-run variance. The conclusion in Section 7 states that DASSA 'yields significant (+3.43 Rouge-L) improvement', but 'significant' is used in the everyday sense and is not backed by any statistical procedure. Since the 10.5k test questions are clustered within 355 documents, a paired test or bootstrap by document would be feasible and appropriate. The absence of such support is load-bearing because the central claim is precisely that the observed gain is real and attributable to the annotations.","section":"Section 6 and Figure 4"},{"comment":"The control conditions 'DR (Exp NoSense)', 'DR (NonE NoSense)', and 'Sent span 3' are intended to isolate the effect of sense labels from attention scope. For example, DR (Exp) shows +3.43 while DR (Exp NoSense) shows +1.34. With single runs, it is impossible to know whether this gap is signal or noise, especially given that roughly fifteen configurations are compared and the best performer is highlighted. The paper should provide variance estimates or a proper multiple-comparison-aware analysis before concluding that semantic content, rather than the attention mask alone, drives the improvement.","section":"Section 6.2, Figure 4"},{"comment":"The oracle used for training supervision, 'Oracle (Ours)', reaches 70.82 Rouge-L, whereas the original 'Oracle (original)' from Kocisky et al. reaches 59.92. This means the QANet and DASSA results are not directly comparable to previously published NarrativeQA systems (e.g., RMR, DecaProp) that likely used the original oracle or a different span-mapping procedure. The absolute positions in Table 1 should be interpreted with this caveat, and the paper should either re-run previous systems under the same oracle or explicitly refrain from claiming that the proposed model outperforms prior work.","section":"Section 4 and Table 1"},{"comment":"The paper does not report any accuracy or error analysis for the three automatic annotation tools (SRL from He et al., neuralcoref, and the discourse parser of Mihaylov and Frank) on narrative text. Since the entire method depends on these annotations being accurate enough to provide useful training signal, the lack of any annotation-quality evaluation leaves open the possibility that the gains come from the attention scope masks or from annotation noise rather than from the intended semantic relations. At minimum, a sample-based accuracy estimate or a comparison with gold annotations for a subset would strengthen the interpretation.","section":"Section 2"}],"minor_comments":[{"comment":"The metric name appears both as 'Rouge-L' and 'Rouge-L'; the standard spelling is ROUGE-L and should be used consistently.","section":"Abstract and throughout"},{"comment":"The notation 'mask softmax(..., M_t)' is not defined precisely; please specify whether the mask is added to the logits before softmax or applied multiplicatively after softmax, and how the mask values are normalized.","section":"Section 3.2, Eq. (1)"},{"comment":"The 'Sent span 3' configuration is described only in prose; please specify in the text or a caption how Sent1/Sent2/Sent3 labels are assigned and why this serves as a control for attention scope.","section":"Section 6.2, Figure 4"},{"comment":"The 'DR (Exp) EMA' row is not explained beyond 'EMA is Exponential Moving Average'; please provide the EMA decay schedule or cite the reference for this training technique.","section":"Section 6.1, Table 1"},{"comment":"There is a typo: 'devided' should be 'divided'.","section":"Section 4"},{"comment":"Khashabi et al. 2018 appears as both '2018a' and '2018b' in the text and references, but the entries appear to refer to the same work; please consolidate.","section":"Section 5, related work"},{"comment":"The claim that 'all models show improvement over length 800-1000' is based on a single run per model; please temper the wording until variance is reported.","section":"Section 6.2, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is interesting and the architectural idea is sound, but the empirical evaluation does not yet support the strength of the claims. The single-run, many-configuration setup is a genuine concern because the paper explicitly uses the word 'significant' in the conclusion. In addition, the Oracle (Ours) differs substantially from the original NarrativeQA oracle, so the comparisons to previous work in Table 1 are likely unfair; the authors should either re-evaluate prior systems under the same protocol or de-emphasize those comparisons. If the authors add proper statistical support (e.g., document-level bootstrap confidence intervals), tone down the selection-based claims, and clarify the oracle differences, this could become a solid contribution. I recommend major revision rather than rejection because these issues are fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper to know about: it modifies QANet's self-attention to add dedicated heads for SRL, coreference, and discourse relations, applied to NarrativeQA. The reported +3.4 Rouge-L over their own QANet baseline is the headline, and the stress-test note is right: that number is not statistically supported. Single runs, no confidence intervals, no significance tests, and roughly fifteen hand-picked head allocations mean the best configuration could be a multiple-comparisons artifact. I would not judge the method on that number alone.\n\nWhat is genuinely new: this is the first work I know that puts discourse relations into a self-attention encoder for reading comprehension, and the design of annotation-specific heads with attention masks is clean and computationally practical. The per-question-type and context-length analysis is useful—it suggests which structures help where, e.g., SRL for who/when, discourse relations for why/where. The controls (NoSense, Sent span 3) are a good idea, even if they are only single runs without variance estimates.\n\nSoft spots beyond the statistics: the Oracle used for training supervision is noticeably different from prior work (70.7 vs. 54.1 Rouge-L for the original oracle), so absolute comparisons to previously published numbers are confounded. They do not report annotation accuracy or any error analysis, so it is unclear whether the gains come from the semantic labels themselves or mostly from the attention scope masks. The self-cited discourse parser is off-the-shelf and not fitted to the metric, so I do not see a circularity problem there.\n\nWho this is for: researchers working on integrating linguistic structure into neural reading comprehension. The architecture is a solid starting point and the analysis gives testable hypotheses. It deserves a serious referee, but the right outcome would be major revision: add multiple seeds or confidence intervals, account for the configuration search, and report annotation quality. If the +3.4 gain survives that, the contribution is real.\n\nRecommendation: send to peer review with a request for statistical grounding; do not desk reject.","headline":"A well-motivated architecture for injecting discourse annotations into self-attention, but the headline Rouge-L gain rests on unsupported single runs and hand-selected configurations.","tokens_in":16745,"tokens_out":2031,"would_cite":true,"duration_ms":21473,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding discourse and semantic relation labels to self-attention heads improves narrative reading comprehension by up to 3.4 Rouge-L points over the QANet baseline.","keywords":["reading comprehension","self-attention","discourse relations","semantic role labeling","coreference resolution","NarrativeQA","QANet","linguistic annotations"],"falsifier":"Train the DASSA model with the same attention-scope masks but with relation labels randomly permuted across tokens; if Rouge-L stays at the reported +3.4 improvement over QANet, the semantic content of the annotations is not what drives the gain. Alternatively, measure the accuracy of the SRL, coreference, and discourse-parsing tools on a sample of NarrativeQA summaries; if accuracy is near chance on the spans that carry the answer, the annotations cannot be supplying the claimed signal.","tokens_in":15839,"feed_emoji":"📖","tokens_out":5704,"duration_ms":47418,"temperature":0.7,"pith_summary":"This paper claims that a reading-comprehension model trained on narrative stories answers questions better when its self-attention heads are guided by linguistic annotations: semantic roles (who did what to whom), coreference (which mentions are the same character), and discourse relations (causation, temporal succession) between clauses and sentences. The authors extend the QANet model so that each attention head receives a token-level label embedding and a scope mask that restricts attention to the tokens involved in a given relation. On the NarrativeQA summary task, the best configuration improves Rouge-L by up to 3.4 points over the base QANet model, with the largest and most consistent gains from intra-sentential explicit discourse relations, followed by sentence-internal semantic roles; coreference helps most in mid-sized contexts. If true, the result shows that structured linguistic knowledge can be folded into transformer-style encoders cheaply, without multi-task training, and that the attention-scope restriction itself may be part of the benefit.","feed_headline":"Linguistic annotations boost self-attention QA by 3.4 Rouge-L","feed_subtitle":"Adding discourse, role, and coreference clues to attention heads beats plain QANet on NarrativeQA.","key_machinery":"The central object is the Discourse-Aware Semantic Self-Attention (DASSA) encoder block, a modification of the standard transformer encoder block. Each attention head is augmented with two inputs: an embedding vector for the linguistic annotation type (e.g., 'SRL Arg1', 'DiscRel Cause.Reason Arg2', or a coreference-cluster label) concatenated to the query and key inputs, and a symmetric attention-scope mask $M_t$ that restricts which tokens the head may attend to (one sentence for SRL; two neighboring sentences for non-explicit discourse relations; the full context for coreference). This combination encodes relational knowledge as flat token labels plus a scope mask, avoiding the $n^2$ memory cost of explicit graph edge representations and reducing memory to $2 n d_h H$ per head. The mechanism lets the model dedicate some heads to local, discourse-structured reasoning while keeping other heads free for full-context attention.","core_discovery":"The paper's central discovery is that dedicating self-attention heads to discourse-semantic relations improves answer-span prediction on long narratives. Concretely, replacing standard multi-head self-attention in QANet's modeling layer with Discourse-Aware Semantic Self-Attention — where each head focuses on a specific relation type, such as an SRL argument role, a PDTB-style discourse relation, or a coreference cluster — yields up to +3.4 Rouge-L on the NarrativeQA test set. The authors find that the strongest and most consistent gains come from explicit discourse relations that hold within a sentence, followed by semantic role relations within a sentence, and that long-distance coreference relations help in mid-length contexts. They also show improvements on nearly all question types and across context lengths, with particularly large gains on 'when' and 'how' questions.","pith_inferences":["The attention-scope masks alone, independent of the relation labels, may account for part of the reported gains; a natural control experiment would replace the sense labels with random labels while keeping the same masks and measure whether the improvement persists.","The approach suggests a cheap way to inject structured knowledge into existing transformer checkpoints: instead of retraining with multi-task objectives, one could annotate the input offline and bias attention heads, potentially transferring to other long-document tasks such as summarization or story generation.","The paper's finding that specific question types benefit from specific annotations points to a testable extension: dynamically selecting which annotation types to activate per question, for example routing 'why' questions toward discourse-relation heads, could improve the fine-grained results further.","The authors' oracle answer-mapping procedure, which uses Rouge-L to select training spans, is itself an upper-bound choice; if that selection were biased toward spans that happen to align with SRL or discourse annotations, some of the training-signal gain could be an artifact of the oracle rather than of the model, and this is worth testing."],"forward_implications":["A reading-comprehension system can gain up to +3.4 Rouge-L over the plain QANet baseline by adding discourse-semantic annotations to self-attention heads, without changing the underlying training objective.","Intra-sentential explicit discourse relations and sentence-internal semantic role relations deliver the most consistent gains, suggesting that local, relation-structured attention is especially useful for narrative question answering.","Long-distance coreference relations improve performance mainly in mid-sized contexts, and almost all annotation combinations improve 'when' questions over the baseline.","Because the DASSA block is used only in selected encoder layers and adds only small label-embedding and mask computations, the approach scales to long contexts without significant extra memory."],"supporting_citations":[{"why":"Supplies the multi-head self-attention mechanism that the DASSA encoder extends.","marker":"(Vaswani et al., 2017)"},{"why":"Provides the QANet reading-comprehension model used as the base architecture and baseline.","marker":"(Yu et al., 2018)"},{"why":"Introduces the NarrativeQA dataset and evaluation setup the experiments are run on.","marker":"(Kocisky et al., 2018)"},{"why":"Supplies the semantic role labeling system that produces the SRL annotations for the attention heads.","marker":"(He et al., 2017)"},{"why":"Supplies the discourse relation sense disambiguation system used to generate explicit and non-explicit discourse relation annotations.","marker":"(Mihaylov and Frank, 2016)"},{"why":"Demonstrates that self-attention models benefit from linguistic annotations, the motivation for injecting such information in this work.","marker":"(Strubell et al., 2018)"},{"why":"Defines the PDTB discourse relation sense inventory adopted for the discourse annotations.","marker":"(Prasad et al., 2008)"},{"why":"Provides AllenNLP, the framework containing the QANet and BiDAF implementations used as baselines.","marker":"(Gardner et al., 2018)"}],"fun_headline_variants":["Discourse-aware attention lifts NarrativeQA by 3.4","Self-attention heads for discourse relations boost QA","Narrative comprehension +3.4 with discourse-aware attention","Attention to linguistic relations improves reading QA","Discourse-aware self-attention enhances narrative QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatically generated discourse, semantic-role, and coreference annotations are accurate enough on narrative stories to provide useful training signal, but the paper reports no annotation accuracy or error analysis, so the observed gains could in principle come from the attention-scope masks rather than the semantic content.","fun_headline_variants_meta":{"raw":{"variants":["Discourse-aware attention lifts NarrativeQA by 3.4","Self-attention heads for discourse relations boost QA","Narrative comprehension +3.4 with discourse-aware attention","Attention to linguistic relations improves reading QA","Discourse-aware self-attention enhances narrative QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1436,"prompt_tokens":845,"completion_tokens":591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":517}},"tokens_in":461,"tokens_out":591,"duration_ms":6586,"temperature":1.0,"reasoning_tokens":517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:35:38.676713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the DASSA model with the same attention-scope masks but with relation labels randomly permuted across tokens; if Rouge-L stays at the reported +3.4 improvement over QANet, the semantic content of the annotations is not what drives the gain. Alternatively, measure the accuracy of the SRL, coreference, and discourse-parsing tools on a sample of NarrativeQA summaries; if accuracy is near chance on the spans that carry the answer, the annotations cannot be supplying the claimed signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the NarrativeQA dataset and evaluation setup the experiments are run on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the discourse relation sense disambiguation system used to generate explicit and non-explicit discourse relation annotations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates that self-attention models benefit from linguistic annotations, the motivation for injecting such information in this work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the PDTB discourse relation sense inventory adopted for the discourse annotations."}],"review_version":1}