Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Propagate-Selector: Detecting Supporting Sentences for Question Answering via Graph Neural Networks

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that classifying supporting sentences for multi-hop question answering improves when sentences exchange information over a hand-designed graph, and reports that the resulting Propagate-Selector model reaches 0.734 MAP on…

desk verdict Plausible MAP gain on HotpotQA sentence selection, but the graph itself is never isolated from the encoder and attention loss. read the letter →

arxiv 1908.09137 v2 pith:WBHOOQME submitted 2019-08-24 cs.CL cs.AIcs.LG

Seunghyun Yoon , Franck Dernoncourt , Doo Soon Kim , Trung Bui , Kyomin Jung This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords questionansweringsupportingsentencedetectiongraphneuralnetworkmulti-hopreasoningselectionHotpotQAattentionmechanismELMo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that detecting supporting sentences for multi-hop question answering improves when sentence representations are allowed to exchange information through a graph, rather than scoring each sentence against the question independently. It proposes Propagate-Selector (PS), which builds a graph whose nodes are sentences plus the question, connects nodes by a fixed text-structure topology, and updates each node by attentively aggregating its neighbors over several hops. On the HotpotQA dataset, PS-rnn-elmo raises dev MAP from 0.702 to 0.734 over the previous best answer-selection baseline, which the authors interpret as evidence that modeling intersentential relationships is valuable. The model deliberately does not use answer-span supervision, positioning it as a supporting-sentence subsystem for a full question-answering pipeline.

What carries the argument

The load-bearing machinery is the sentence graph and its propagation rule. Nodes are sentences plus one question node; edges are added offline by three rules: full connectivity among sentences in the same passage, full connectivity among the first sentences of every passage, and an edge from the question node to every sentence node. The update rule is iterative attentive aggregation: in each hop, a node computes attention weights against its neighbors, forms a weighted sum, and combines it with its previous representation through a skip connection. An auxiliary attention loss rewards question-node attention landing on ground-truth supporting sentences at each hop, and this supervised attention is what shapes the propagation. Hop count is a key hyperparameter: performance peaks at four hops and collapses at six.

What would settle it

Train a graph-free ranker that represents each sentence and the question with the same recurrent network over the same pretrained contextual word embeddings, scores with the same rank loss, and adds the same per-hop attention loss with supporting labels but no message passing between sentences. If its dev MAP reaches or exceeds 0.734, the graph propagation is not the source of the improvement; if it falls short, the graph is. A second check: run the 6-hop configuration with gradient clipping or residual scaling; the paper's collapse from 0.734 at 4 hops to 0.457 at 6 hops should be explained by vanishing gradients, so fixing gradient flow should recover near-peak performance if the propagation mechanism itself is sound.

Watch

Extended reading notes

Core claim

The central claim is that supporting-sentence detection in multi-hop question answering is a problem where sentence-level evidence must be combined, and that a graph neural network with a hand-designed topology can perform this combination. PS treats each sentence as a node, fully connects sentences within a passage, connects the first sentences of all passages, and connects each sentence to the question node. It then performs iterative hop-wise updates: at each hop, every node computes attention weights over its neighbors, aggregates their representations, and skip-combines the aggregate with its own representation. A rank loss scores question-sentence pairs, and an auxiliary attention loss supervises the question-to-sentence attention at every hop using the supporting-sentence labels. The paper reports dev MAP 0.734 and MRR 0.853, best among the answer-selection models compared, with ablation results showing that each topology component contributes to the score.

Load-bearing premise

The reported gain is never checked against a non-graph model that uses the same pretrained contextual word encoder and the same attention supervision, so the improvement may come from the encoder or the auxiliary loss rather than from propagating information across sentences.

Editorial extensions

If this is right

  • A supporting-sentence detector can be built without answer-span labels: PS uses only the question, passages, and supporting-sentence labels, and still improves over answer-selection baselines.
  • Cross-passage connectivity matters: removing edges between the first sentences of passages degrades MAP, indicating that multi-passage reasoning benefits from a path linking passage starts.
  • The number of propagation hops is not free: performance peaks at four hops, degrades at five, and collapses at six, consistent with vanishing gradients under repeated iterative aggregation.
  • The graph mechanism is complementary to the choice of sentence encoder: RNN-encoded ELMo nodes outperform average-pooled ELMo, GloVe, and non-fine-tuned BERT node representations under the same graph.
  • The model is positioned as a component for a full end-to-end MRQA pipeline, to be combined with answer-span models rather than competing with them.
  • Each element of the graph topology is load-bearing: removing within-passage edges, cross-passage edges, or question-sentence edges all degrade performance in the ablations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper does not run a graph-free control with the same encoder and the same attention loss, the reader cannot yet conclude that graph propagation itself is the cause of the MAP gain; the improvement could come from the stronger encoder or from the auxiliary supervision.
  • A natural test is a graph-free ranker using the same recurrent encoder over the same contextual embeddings plus the same per-hop attention loss; matching 0.734 would show the graph is not load-bearing, while falling short would support the propagation claim.
  • The fixed topology depends on passage structure, so on datasets without explicit passage boundaries or with different document organization, the three edge types may need to be re-derived; the ablations show sensitivity to topology, suggesting the optimal edges are dataset-dependent.
  • The attention loss uses ground-truth supporting sentences at every hop, a strong training signal that would be unavailable when the model is deployed to predict those labels; weakly supervised or semi-supervised variants are a natural test of whether the propagation still helps.
  • The 6-hop collapse suggests a gradient-flow problem rather than an inherent limit of the propagation idea; applying gradient clipping or residual scaling could reveal whether more hops help once optimization is stabilized.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Propagate-Selector (PS), a graph neural network for detecting supporting sentences in multi-hop question answering. Sentences and the question are represented as graph nodes, edges are added according to a hand-designed topology (within-passage, first-sentence across passages, and question-to-sentence), and information is propagated over several hops using iterative attentive aggregation with a skip connection. The model is trained with a rank loss for supporting-sentence classification and an auxiliary attention loss that supervises question-to-sentence attention weights. Experiments on HotpotQA compare PS with several answer-selection baselines, and the main result is a dev-set MAP improvement from 0.702 (CompClip-LM-LC) to 0.734 (PS-rnn-elmo). Additional experiments analyze the effect of the number of hops, alternative graph topologies, and different node/word representation choices.

Significance. If the empirical result is robust, the paper would make a useful contribution to QA pipelines by showing that sentence-level graph propagation can improve supporting-sentence detection without using answer-span supervision. The paper has several strengths: it targets a standard benchmark (HotpotQA), compares against a range of established answer-selection models, reports ablations of hops, topology, and encoders, and promises to release code. The central mechanism, however, is not isolated by the current experiments: no non-graph control uses the same encoder and the same auxiliary attention loss, and the topology ablations do not cleanly remove cross-sentence propagation. The comparative claim is plausible, but the attribution of the improvement to graph propagation specifically is underdetermined by the evidence as presented.

major comments (4)
  1. [5.2, Table 2; 4.3, Eq. (10)] The paper's central claim that cross-sentence graph propagation drives the MAP improvement from 0.702 to 0.734 is not isolable from the reported experiments. PS differs from the strongest baseline in both the encoder and the training objective: it uses an extra attention loss (Eq. 10) that directly supervises question-to-sentence attention with ground-truth supporting labels, and it uses a GRU sentence encoder in a graph. Please add a graph-free control that uses the same ELMo/GRU encoder and the same rank and attention losses but no sentence-to-sentence edges, and report its MAP/MRR. Without this control, the observed gain cannot be attributed to propagation rather than to the encoder or the attention supervision.
  2. [5.4, Figure 4(b), Table 5] The Type-2 ablation does not actually remove all paths for cross-passage information flow. Because the default topology connects the question node q to every sentence node, any two sentences from different passages remain connected through q, so 'removing the connections between the passages' does not ablate the graph mechanism. Please either add a condition that removes all q-sentence edges while appropriately redefining the attention loss, or design a topology that prevents indirect paths through q, and reinterpret the Type-2 comparison accordingly.
  3. [5.4, Figure 4(c), Table 5] The Type-3 ablation removes edges between the question node and sentence nodes, but the attention loss in Eq. (10) is defined in terms of a_qi, the attention weight between the question node q and sentence node i. Removing q-sentence edges therefore changes the loss function as well as the topology, confounding the ablation. The observed degradation from 0.716 to 0.658 MAP could be due to the loss change rather than to the missing question-sentence edges.
  4. [5.3, Tables 3 and 4] The number of propagation hops is selected on the dev set, and no significance tests or multiple-seed results are reported. The reported main improvement is 0.032 MAP, which may be within run-to-run variance; please report means and standard deviations over several random seeds, or a paired significance test, for the main comparison and the hop sweep. In addition, the abrupt collapse at 6 hops (MAP 0.457 in Table 3 and 0.441 in Table 4) is attributed to vanishing gradients without supporting evidence; if this indicates training instability rather than a property of the method, the explanation should be tested or revised.
minor comments (5)
  1. [4.3, Eq. (3)] The cross-entropy loss expression is malformed: 'L = -log sum_i sum_c y_i,c log(y_hat_i,c)' places the negative logarithm outside a sum of nonnegative terms and is not the standard negative log-likelihood. This should be corrected to an average over samples.
  2. [4.3, Eq. (10)] The attention loss formula uses the index i in both the hop summation and the sentence summation, and the text says the loss is defined 'in each hop' while the equation sums over hops. Please clarify the indexing and whether the loss is summed or averaged over hops.
  3. [4.2, Eq. (7)] The softmax denominator is written as a sum over k, but the softmax should be over neighbor nodes u in N(v). Please correct the notation to avoid ambiguity about what is being normalized.
  4. [5.4, Figure 4 caption] The caption says 'Different typologies for the graph'; the intended word is 'topologies'.
  5. [5.1 and 5.5] The model naming is inconsistent: the text refers to 'PS-USD T' and 'PS-USD T' in Table 6, while the method is described as 'universal sentence encoding'; please standardize the abbreviation (e.g., PS-USE).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical comparison on held-out HotpotQA data, with no derivation step that reduces to its inputs.

full rationale

The paper's claimed contribution is a graph-based model for supporting-sentence detection evaluated on the HotpotQA dev set. The prediction target (supporting-sentence labels) is not used to construct the graph topology or the node representations; it is only used as a supervised training signal in the rank loss (Eq. 9) and the auxiliary attention loss (Eq. 10). These are standard supervised losses, and the reported MAP/MRR numbers are computed on held-out dev examples, so the empirical result is not forced by construction. The auxiliary attention loss does directly supervise attention weights a_qi with ground-truth Y_i, which weakens the interpretive claim that attention 'emerges' from propagation (Section 5.3), and the Type-3 ablation removes the q-sentence edges on which Eq. (10) depends, making that ablation conflate topology with loss. These are experimental-validity concerns, not circularity: the dev predictions are not equivalent to the training labels by definition, and no fitted parameter is renamed as a prediction. The baselines include the authors' own prior model CompClip-LM-LC, but that self-citation is only used as a comparison point, not as load-bearing justification for the new model. No self-citation chain, uniqueness theorem, or ansatz smuggled via citation is present. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central empirical claim rests on standard supervised learning assumptions plus the specific graph topology and hop count. No physical or conceptual entities are invented. The main loaded choices are the topology, the hop count, and the loss weight alpha.

free parameters (4)
  • Number of propagation hops (K) = 4
    Selected as best-performing on the dev set among K=1 to 6 (Section 5.3, Table 3).
  • Loss weight alpha = not reported
    Appears in L = alpha * loss_rank + loss_attn (Equation 11); no value or tuning procedure is given.
  • GRU hidden dimension = 200
    Used for sentence and question encodings (Section 5.1).
  • Dropout ratio = 0.7
    Applied to GRU and attention weight matrix (Section 5.1).
assumptions (3)
  • domain assumption The hand-designed graph topology (full same-passage connectivity, cross-passage first-sentence links, question-to-all-sentence links) is an appropriate inductive bias for multi-hop supporting-sentence detection in HotpotQA.
    Section 4.2 specifies the topology; the paper ablates edge types but does not compare to alternative graph construction strategies.
  • domain assumption HotpotQA supporting-sentence annotations are reliable enough to serve as ground truth for training and evaluation.
    The dataset is used as provided (Section 3); no analysis of annotation noise is given.
  • domain assumption Fixed-hop iterative propagation can capture the cross-sentence reasoning needed for the task.
    The model uses K=4 aggregation steps (Section 5.3); no formal or empirical argument establishes that this is sufficient beyond the dev-set search.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Propagate-Selector: Detecting Supporting Sentences for Question Answering via Graph Neural Networks." pith.science (2026). https://pith.science/paper/WBHOOQME

@misc{pith2026190809137,
  author       = {Pith},
  title        = {Pith review of: Propagate-Selector: Detecting Supporting Sentences for Question Answering via Graph Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WBHOOQME}},
  note         = {Machine review of arXiv:1908.09137}
}
read the original abstract

In this study, we propose a novel graph neural network called propagate-selector (PS), which propagates information over sentences to understand information that cannot be inferred when considering sentences in isolation. First, we design a graph structure in which each node represents an individual sentence, and some pairs of nodes are selectively connected based on the text structure. Then, we develop an iterative attentive aggregation and a skip-combine method in which a node interacts with its neighborhood nodes to accumulate the necessary information. To evaluate the performance of the proposed approaches, we conduct experiments with the standard HotpotQA dataset. The empirical results demonstrate the superiority of our proposed approach, which obtains the best performances, compared to the widely used answer-selection models that do not consider the intersentential relationship.

Figures

Figures reproduced from arXiv: 1908.09137 by the authors.

Figure 1
Figure 1. An example of dataset. Detecting supporting sentences is an essential step being able to answer the ques￾tion. signing each sentence to an independent graph node. Then, we connect the undirected edges between nodes using a proposed graph topology (see the discussion in the 4.2.). Next, we allow PS to propagate information between the nodes through iterative hops to perform reasoning across the given sentences. Throu… view at source ↗
Figure 2
Figure 2. Topology of the proposed model. Each node [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Attention weights between the question and sentences in the passages. As the number of hops increases, the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Different typologies for the graph. Type-1 reduce connection within the passage, type-2 remove connection [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 27 canonical work pages

  1. [1]

    Bian, W., Li, S., Yang, Z., Chen, G., and Lin, Z. (2017). A compare-aggregate model with dynamic-clip attention for answer selection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , pages 1987--1990. ACM

    work page 2017

  2. [2]

    S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al

    Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018). Universal sentence encoder. arXiv preprint arXiv:1803.11175

    arXiv 2018

  3. [3]

    and Gardner, M

    Clark, C. and Gardner, M. (2018). Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics , pages 845--855

    work page 2018

  4. [4]

    De Cao, N., Aziz, W., and Titov, I. (2019). Question answering by reasoning across documents with graph convolutional networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2306--2317

    work page 2019

  5. [5]

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4171--4186

    work page 2019

  6. [6]

    Fan, W., Ma, Y., Li, Q., He, Y., Zhao, E., Tang, J., and Yin, D. (2019). Graph neural networks for social recommendation. In The World Wide Web Conference , pages 417--426. ACM

    work page 2019

  7. [7]

    Hamilton, W., Ying, Z., and Leskovec, J. (2017). Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems , pages 1024--1034

    work page 2017

  8. [8]

    Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics , pages 1601--1611

    work page 2017

Show all 33 references
  1. [9]

    Kim, Y. (2014). Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1746--1751

    2014

  2. [10]

    and Ba, J

    Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  3. [11]

    Kipf, T. N. and Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR)

    2017

  4. [12]

    Palm, R., Paquet, U., and Winther, O. (2018). Recurrent relational networks. In Advances in Neural Information Processing Systems , pages 3368--3378

    2018

  5. [13]

    Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532--1543

    2014

  6. [14]

    Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...

    2018

  7. [15]

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2383--2392

    2016

  8. [16]

    M., McClelland, J

    Saxe, A. M., McClelland, J. L., and Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120

    2013 arXiv

  9. [17]

    Seo, M., Kembhavi, A., Farhadi, A., and Hajishirzi, H. (2016). Bidirectional attention flow for machine comprehension. In International Conference on Learning Representations (ICLR)

    2016

  10. [18]

    Shen, G., Yang, Y., and Deng, Z.-H. (2017a). Inter-weighted alignment network for sentence pair modeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 1179--1189

    2017

  11. [19]

    Shen, Y., Huang, P.-S., Gao, J., and Chen, W. (2017b). Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1047--1055. ACM

    2017

  12. [20]

    E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R

    Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research , 15(1):1929--1958

    2014

  13. [21]

    H., Lai, T., Haffari, G., Zukerman, I., Bui, T., and Bui, H

    Tran, Q. H., Lai, T., Haffari, G., Zukerman, I., Bui, T., and Bui, H. (2018). The context-dependent additive recurrent neural net. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ...

    2018

  14. [22]

    Veli c kovi \'c , P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. (2018). Graph attention networks. In International Conference on Learning Representations (ICLR)

    2018

  15. [23]

    and Jiang, J

    Wang, S. and Jiang, J. (2016). A compare-aggregate model for matching text sequences. arXiv preprint arXiv:1611.01747

    2016 arXiv

  16. [24]

    Wang, W., Yang, N., Wei, F., Chang, B., and Zhou, M. (2017). Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics , pages 189--198

    2017

  17. [25]

    Wang, S., Yu, M., Guo, X., Wang, Z., Klinger, T., Zhang, W., Chang, S., Tesauro, G., Zhou, B., and Jiang, J. (2018). R 3: Reinforced ranker-reader for open-domain question answering. In Thirty-Second AAAI Conference on Artificial Intelligence

    2018

  18. [26]

    Weissenborn, D., Wiese, G., and Seiffe, L. (2017). Making neural qa as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) , pages 271--280

    2017

  19. [27]

    Welbl, J., Stenetorp, P., and Riedel, S. (2018). Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association of Computational Linguistics , 6:287--302

    2018

  20. [28]

    Wu, Y., Liu, X., Feng, Y., Wang, Z., Yan, R., and Zhao, D. (2019). Relation-aware entity alignment for heterogeneous knowledge graphs. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 5278--5284. AAAI Press

    2019

  21. [29]

    Xiong, C., Zhong, V., and Socher, R. (2016). Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604

    2016 arXiv

  22. [30]

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. (2018). HotpotQA : A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2369--2380

    2018

  23. [31]

    S., Bui, T., and Jung, K

    Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., and Jung, K. (2019). A compare-aggregate model with latent clustering for answer selection. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management , pages 2093--2096

    2019

  24. [32]

    Zitnik, M., Agrawal, M., and Leskovec, J. (2018). Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics , 34(13):i457--i466

    2018

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.