{"id":"0eea0a1c-9e87-4c5e-95b7-6aedf7744444","arxiv_id":"1908.09137","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A sentence-level graph network with hand-designed connectivity achieves a dev MAP of 0.734 on HotpotQA supporting-sentence detection, outperforming answer-selection baselines.","lead":"This paper presents a graph neural network that links sentences in question-answering passages and propagates information between them to identify the sentences needed to answer a question. The model reports the best supporting-sentence detection scores on the HotpotQA benchmark among sentence-selection models that do not use answer text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Improvement not attributable to graph propagation: no graph-free control, and the §5.4 topology ablations leave the question node connected to every sentence.","rationale":"The reader's weakest assumption identifies the missing non-graph control, and my reading agrees: the central comparison in Table 2 changes the encoder and the loss simultaneously with the graph, so the contribution of the graph itself is not demonstrated. An additional problem strengthens this concern: the topology ablation in Table 5 is not a clean ablation. Because the question node connects to every sentence in the default graph, Type-2 does not remove all inter-passage paths, and Type-3 removes the edges needed by the attention loss, confounding topology with the objective. The paper does report a public code repository, which is a point in its favor, but no code-level or formal verification is present. The 0.032 absolute MAP improvement is plausible and the paper is readable, but the central explanatory claim—that iterative graph propagation across sentences is responsible—requires a control condition that does not exist in the manuscript. This does not overturn the empirical result; it means the paper should be accepted only with the condition that the authors add the missing graph-free baseline and fix the topology ablations. The reader's CONDITIONAL verdict remains appropriate, so I recommend no change.","tokens_in":11451,"tokens_out":4700,"duration_ms":51966,"concrete_test":"Run PS-rnn-elmo with graph aggregation removed (no message passing between sentence nodes) while keeping the same ELMo/GRU encoder and rank loss, and add a non-graph sentence-level attention module over the question and all sentences trained with the same ground-truth labels to replace Eq. (10). Compare dev MAP; if this graph-free model reaches within ~0.01 of 0.734, the graph propagation is not necessary for the reported gain. Also rerun Type-2 with the question node disconnected from one passage's sentences so that cross-passage paths are truly absent, and rerun Type-3 without changing the loss. If the code repository is complete, this can be done by editing the graph construction and loss modules.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that cross-sentence graph propagation produces the MAP gain. That claim is not isolable from the reported experiments. No non-graph baseline uses the same ELMo/GRU encoder and rank loss, so the 0.734 vs 0.702 gap could come from the encoder or from the auxiliary attention loss (Eq. 10), which directly supervises question-to-sentence attention with ground-truth labels. Moreover, the topology ablations in §5.4 do not isolate the graph mechanism. In the default topology, the question node q is connected to every sentence node. Type-2 removes only the first-sentence inter-passage edges, but all sentences remain connected through q, so cross-passage propagation is not actually ablated. Type-3 removes q-sentence edges, but then the attention weights a_qi in Eq. (10) are undefined, so that ablation changes the loss as well as the topology. Consequently, no reported condition tests the contribution of the graph itself. The empirical result is plausible, but the explanatory claim is underdetermined by the experiments as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Propagate-Selector (PS), a graph neural network for detecting supporting sentences in multi-hop question answering. Sentences and the question are represented as graph nodes, edges are added according to a hand-designed topology (within-passage, first-sentence across passages, and question-to-sentence), and information is propagated over several hops using iterative attentive aggregation with a skip connection. The model is trained with a rank loss for supporting-sentence classification and an auxiliary attention loss that supervises question-to-sentence attention weights. Experiments on HotpotQA compare PS with several answer-selection baselines, and the main result is a dev-set MAP improvement from 0.702 (CompClip-LM-LC) to 0.734 (PS-rnn-elmo). Additional experiments analyze the effect of the number of hops, alternative graph topologies, and different node/word representation choices.","tokens_in":11653,"tokens_out":4077,"duration_ms":41846,"significance":"If the empirical result is robust, the paper would make a useful contribution to QA pipelines by showing that sentence-level graph propagation can improve supporting-sentence detection without using answer-span supervision. The paper has several strengths: it targets a standard benchmark (HotpotQA), compares against a range of established answer-selection models, reports ablations of hops, topology, and encoders, and promises to release code. The central mechanism, however, is not isolated by the current experiments: no non-graph control uses the same encoder and the same auxiliary attention loss, and the topology ablations do not cleanly remove cross-sentence propagation. The comparative claim is plausible, but the attribution of the improvement to graph propagation specifically is underdetermined by the evidence as presented.","major_comments":[{"comment":"The paper's central claim that cross-sentence graph propagation drives the MAP improvement from 0.702 to 0.734 is not isolable from the reported experiments. PS differs from the strongest baseline in both the encoder and the training objective: it uses an extra attention loss (Eq. 10) that directly supervises question-to-sentence attention with ground-truth supporting labels, and it uses a GRU sentence encoder in a graph. Please add a graph-free control that uses the same ELMo/GRU encoder and the same rank and attention losses but no sentence-to-sentence edges, and report its MAP/MRR. Without this control, the observed gain cannot be attributed to propagation rather than to the encoder or the attention supervision.","section":"5.2, Table 2; 4.3, Eq. (10)"},{"comment":"The Type-2 ablation does not actually remove all paths for cross-passage information flow. Because the default topology connects the question node q to every sentence node, any two sentences from different passages remain connected through q, so 'removing the connections between the passages' does not ablate the graph mechanism. Please either add a condition that removes all q-sentence edges while appropriately redefining the attention loss, or design a topology that prevents indirect paths through q, and reinterpret the Type-2 comparison accordingly.","section":"5.4, Figure 4(b), Table 5"},{"comment":"The Type-3 ablation removes edges between the question node and sentence nodes, but the attention loss in Eq. (10) is defined in terms of a_qi, the attention weight between the question node q and sentence node i. Removing q-sentence edges therefore changes the loss function as well as the topology, confounding the ablation. The observed degradation from 0.716 to 0.658 MAP could be due to the loss change rather than to the missing question-sentence edges.","section":"5.4, Figure 4(c), Table 5"},{"comment":"The number of propagation hops is selected on the dev set, and no significance tests or multiple-seed results are reported. The reported main improvement is 0.032 MAP, which may be within run-to-run variance; please report means and standard deviations over several random seeds, or a paired significance test, for the main comparison and the hop sweep. In addition, the abrupt collapse at 6 hops (MAP 0.457 in Table 3 and 0.441 in Table 4) is attributed to vanishing gradients without supporting evidence; if this indicates training instability rather than a property of the method, the explanation should be tested or revised.","section":"5.3, Tables 3 and 4"}],"minor_comments":[{"comment":"The cross-entropy loss expression is malformed: 'L = -log sum_i sum_c y_i,c log(y_hat_i,c)' places the negative logarithm outside a sum of nonnegative terms and is not the standard negative log-likelihood. This should be corrected to an average over samples.","section":"4.3, Eq. (3)"},{"comment":"The attention loss formula uses the index i in both the hop summation and the sentence summation, and the text says the loss is defined 'in each hop' while the equation sums over hops. Please clarify the indexing and whether the loss is summed or averaged over hops.","section":"4.3, Eq. (10)"},{"comment":"The softmax denominator is written as a sum over k, but the softmax should be over neighbor nodes u in N(v). Please correct the notation to avoid ambiguity about what is being normalized.","section":"4.2, Eq. (7)"},{"comment":"The caption says 'Different typologies for the graph'; the intended word is 'topologies'.","section":"5.4, Figure 4 caption"},{"comment":"The model naming is inconsistent: the text refers to 'PS-USD T' and 'PS-USD T' in Table 6, while the method is described as 'universal sentence encoding'; please standardize the abbreviation (e.g., PS-USE).","section":"5.1 and 5.5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper shows a plausible gain (0.734 vs 0.702 MAP) for a sentence-level graph attention model on HotpotQA supporting-sentence detection, without using answer spans. The gain looks real, but the paper over-claims the graph's role.\n\nWhat's actually new: applying graph attention to sentence nodes for this sub-task, with a hand-designed topology (same-passage fully connected, first-sentences connected, question connected to all sentences). The building blocks are standard—GAT-style attentive aggregation, GRU encoders, skip connections—but the combination for supporting-sentence detection is not in the cited literature. The paper is clearly written, reports dev and train numbers, includes hop-wise analysis, topology ablations, and node-representation comparisons, and ships code. That is solid work.\n\nWhere it is soft: the central attribution claim. The 3-point gap over CompClip-LM-LC could come from the ELMo/GRU encoder or the auxiliary attention loss (Eq. 10) rather than from cross-sentence propagation. There is no control using the same encoder and both losses but no graph. That is a clean, doable experiment and its absence is the paper's main weakness.\n\nThe topology ablations are also weaker than they look. Type-2 removes first-sentence edges but the question node is still connected to every sentence, so cross-passage paths remain through q. Type-3 removes q-sentence edges, but that also removes the q-to-sentence attention weights used in the attention loss, so it changes the objective, not just the topology. As a result, no reported condition tests the contribution of the graph itself. The hop-4 selection is on dev (transparent, since all hops are reported), and \"significant\" is used without significance tests or multiple seeds. The 6-hop collapse to 0.457 is striking and the vanishing-gradient explanation is a guess.\n\nBottom line: the empirical result is plausible and worth a referee's time, but the explanatory claim should be toned down until an isolated graph-free baseline is run. Authors should add that experiment and clarify what the ablations actually sever. This is a conditional accept for a workshop or conference that values such analyses; for a top venue, the missing control is likely required.\n\nWho this is for: people working on QA sentence selection or multi-hop reasoning components; it is a useful data point and a good starting point for discussion. I would bring it to a reading group and cite it as related work.","headline":"Plausible MAP gain on HotpotQA sentence selection, but the graph itself is never isolated from the encoder and attention loss.","tokens_in":12176,"tokens_out":2975,"would_cite":true,"duration_ms":27237,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that classifying supporting sentences for multi-hop question answering improves when sentences exchange information over a hand-designed graph, and reports that the resulting Propagate-Selector model reaches 0.734 MAP on…","keywords":["question answering","supporting sentence detection","graph neural network","multi-hop reasoning","sentence selection","HotpotQA","attention mechanism","ELMo"],"falsifier":"Train a graph-free ranker that represents each sentence and the question with the same recurrent network over the same pretrained contextual word embeddings, scores with the same rank loss, and adds the same per-hop attention loss with supporting labels but no message passing between sentences. If its dev MAP reaches or exceeds 0.734, the graph propagation is not the source of the improvement; if it falls short, the graph is. A second check: run the 6-hop configuration with gradient clipping or residual scaling; the paper's collapse from 0.734 at 4 hops to 0.457 at 6 hops should be explained by vanishing gradients, so fixing gradient flow should recover near-peak performance if the propagation mechanism itself is sound.","tokens_in":11272,"feed_emoji":"🧠","tokens_out":8779,"duration_ms":80172,"temperature":0.7,"pith_summary":"The paper tries to establish that detecting supporting sentences for multi-hop question answering improves when sentence representations are allowed to exchange information through a graph, rather than scoring each sentence against the question independently. It proposes Propagate-Selector (PS), which builds a graph whose nodes are sentences plus the question, connects nodes by a fixed text-structure topology, and updates each node by attentively aggregating its neighbors over several hops. On the HotpotQA dataset, PS-rnn-elmo raises dev MAP from 0.702 to 0.734 over the previous best answer-selection baseline, which the authors interpret as evidence that modeling intersentential relationships is valuable. The model deliberately does not use answer-span supervision, positioning it as a supporting-sentence subsystem for a full question-answering pipeline.","feed_headline":"A sentence graph lifts supporting-sentence detection to 0.734 MAP","feed_subtitle":"On HotpotQA, letting sentences pass messages outperforms rankers that score each sentence against the question alone.","key_machinery":"The load-bearing machinery is the sentence graph and its propagation rule. Nodes are sentences plus one question node; edges are added offline by three rules: full connectivity among sentences in the same passage, full connectivity among the first sentences of every passage, and an edge from the question node to every sentence node. The update rule is iterative attentive aggregation: in each hop, a node computes attention weights against its neighbors, forms a weighted sum, and combines it with its previous representation through a skip connection. An auxiliary attention loss rewards question-node attention landing on ground-truth supporting sentences at each hop, and this supervised attention is what shapes the propagation. Hop count is a key hyperparameter: performance peaks at four hops and collapses at six.","core_discovery":"The central claim is that supporting-sentence detection in multi-hop question answering is a problem where sentence-level evidence must be combined, and that a graph neural network with a hand-designed topology can perform this combination. PS treats each sentence as a node, fully connects sentences within a passage, connects the first sentences of all passages, and connects each sentence to the question node. It then performs iterative hop-wise updates: at each hop, every node computes attention weights over its neighbors, aggregates their representations, and skip-combines the aggregate with its own representation. A rank loss scores question-sentence pairs, and an auxiliary attention loss supervises the question-to-sentence attention at every hop using the supporting-sentence labels. The paper reports dev MAP 0.734 and MRR 0.853, best among the answer-selection models compared, with ablation results showing that each topology component contributes to the score.","pith_inferences":["Because the paper does not run a graph-free control with the same encoder and the same attention loss, the reader cannot yet conclude that graph propagation itself is the cause of the MAP gain; the improvement could come from the stronger encoder or from the auxiliary supervision.","A natural test is a graph-free ranker using the same recurrent encoder over the same contextual embeddings plus the same per-hop attention loss; matching 0.734 would show the graph is not load-bearing, while falling short would support the propagation claim.","The fixed topology depends on passage structure, so on datasets without explicit passage boundaries or with different document organization, the three edge types may need to be re-derived; the ablations show sensitivity to topology, suggesting the optimal edges are dataset-dependent.","The attention loss uses ground-truth supporting sentences at every hop, a strong training signal that would be unavailable when the model is deployed to predict those labels; weakly supervised or semi-supervised variants are a natural test of whether the propagation still helps.","The 6-hop collapse suggests a gradient-flow problem rather than an inherent limit of the propagation idea; applying gradient clipping or residual scaling could reveal whether more hops help once optimization is stabilized."],"forward_implications":["A supporting-sentence detector can be built without answer-span labels: PS uses only the question, passages, and supporting-sentence labels, and still improves over answer-selection baselines.","Cross-passage connectivity matters: removing edges between the first sentences of passages degrades MAP, indicating that multi-passage reasoning benefits from a path linking passage starts.","The number of propagation hops is not free: performance peaks at four hops, degrades at five, and collapses at six, consistent with vanishing gradients under repeated iterative aggregation.","The graph mechanism is complementary to the choice of sentence encoder: RNN-encoded ELMo nodes outperform average-pooled ELMo, GloVe, and non-fine-tuned BERT node representations under the same graph.","The model is positioned as a component for a full end-to-end MRQA pipeline, to be combined with answer-span models rather than competing with them.","Each element of the graph topology is load-bearing: removing within-passage edges, cross-passage edges, or question-sentence edges all degrade performance in the ablations."],"supporting_citations":[{"why":"Supplies the HotpotQA dataset and the task definition: questions, passages, and human-annotated supporting sentences.","marker":"Yang et al., 2018"},{"why":"Provides the ELMo contextual word representations that the node encoder is built on.","marker":"Peters et al., 2018"},{"why":"Defines the CompAggr compare-aggregate baseline and the attention-comparison-aggregation template that the rank loss builds on.","marker":"Wang and Jiang, 2016"},{"why":"Is the previous best answer-selection baseline (CompClip-LM-LC) that PS is compared against and beats.","marker":"Yoon et al., 2019"},{"why":"Defines the CompAggr-kMax baseline, an extension of CompAggr that is among the strongest non-graph comparisons.","marker":"Bian et al., 2017"},{"why":"Defines the IWAN baseline, an RNN-based sentence-pair matching model representing standard non-graph answer selection.","marker":"Shen et al., 2017a"},{"why":"Defines the sCARNN baseline, another RNN-based sentence-pair matching model in the comparison set.","marker":"Tran et al., 2018"},{"why":"Provides the graph convolutional propagation framework that PS extends with attention weights and skip connections.","marker":"Kipf and Welling, 2017"},{"why":"Provides the graph attention mechanism used as the template for the attentive aggregation in the model's update rule.","marker":"Veličković et al., 2018"}],"fun_headline_variants":["Graph-based sentence selection answers multi-hop questions","Message-passing sentence graph scores 0.734 MAP on HotpotQA","Linking sentences improves supporting-sentence detection","Propagate-Selector: GNN for supporting-sentence detection","Let sentences share context to spot QA evidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gain is never checked against a non-graph model that uses the same pretrained contextual word encoder and the same attention supervision, so the improvement may come from the encoder or the auxiliary loss rather than from propagating information across sentences.","fun_headline_variants_meta":{"raw":{"variants":["Graph-based sentence selection answers multi-hop questions","Message-passing sentence graph scores 0.734 MAP on HotpotQA","Linking sentences improves supporting-sentence detection","Propagate-Selector: GNN for supporting-sentence detection","Let sentences share context to spot QA evidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3048,"prompt_tokens":852,"completion_tokens":2196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":2118}},"tokens_in":468,"tokens_out":2196,"duration_ms":13732,"temperature":1.0,"reasoning_tokens":2118,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:20:37.490749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a graph-free ranker that represents each sentence and the question with the same recurrent network over the same pretrained contextual word embeddings, scores with the same rank loss, and adds the same per-hop attention loss with supporting labels but no message passing between sentences. If its dev MAP reaches or exceeds 0.734, the graph propagation is not the source of the improvement; if it falls short, the graph is. A second check: run the 6-hop configuration with gradient clipping or residual scaling; the paper's collapse from 0.734 at 4 hops to 0.457 at 6 hops should be explained by vanishing gradients, so fixing gradient flow should recover near-peak performance if the propagation mechanism itself is sound.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HotpotQA dataset and the task definition: questions, passages, and human-annotated supporting sentences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ELMo contextual word representations that the node encoder is built on."},{"cited_title":"S., Bui, T., and Jung, K","cited_arxiv_id":null,"evidence_quote":"Is the previous best answer-selection baseline (CompClip-LM-LC) that PS is compared against and beats."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CompAggr-kMax baseline, an extension of CompAggr that is among the strongest non-graph comparisons."},{"cited_title":"H., Lai, T., Haffari, G., Zukerman, I., Bui, T., and Bui, H","cited_arxiv_id":null,"evidence_quote":"Defines the sCARNN baseline, another RNN-based sentence-pair matching model in the comparison set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the graph convolutional propagation framework that PS extends with attention weights and skip connections."}],"review_version":1}