Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Incorporating External Knowledge into Machine Reading for Generative Question Answering

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A QA model that picks each word's knowledge source beats all compared systems on MS MARCO.

desk verdict A plausible knowledge-aware generative QA architecture with a real but unvalidated fact-retrieval bottleneck; the central gain is directionally supported but the evaluation is too thin to call significant. read the letter →

arxiv 1909.02745 v1 pith:XDCTTIEP submitted 2019-09-06 cs.CL

Bin Bi , Chen Wu , Ming Yan , Wei Wang , Jiangnan Xia , Chenliang Li This is my paper · ORCID
classification cs.CL
keywords knowledge-enrichedanswergenerationgenerativequestionansweringmachinereadingcomprehensionexternalknowledgesourceselectionfactGumbel-SoftmaxMSMARCO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a question-answering system can write better natural-language answers if, at every decoding step, it explicitly chooses which of four sources—the question, the passage, the global vocabulary, or an external knowledge base—should supply the next word. On the MS MARCO answer-generation benchmark, the proposed KEAG model reaches ROUGE-L 51.68 and BLEU-1 45.97, higher than both knowledge-free generative QA models and knowledge-aware baselines. The improvement matters because many questions require commonsense facts that never appear in the passage, and letting the model decide when to consult knowledge is presented as the key to using that knowledge without letting it override the text.

What carries the argument

The central mechanism is the source selector, a learned categorical gate that runs before every generated word and decides which of the four information sources should produce it. Tied to it is a stochastic fact selector that scores the candidate facts by an attention-style function over fact embeddings and the decoder state, sampling a fact with Gumbel-Softmax. These two discrete latent variables are what let the model blend external symbolic knowledge with surface text; the source selector answers 'when to use knowledge' and the fact selector answers 'which fact', and the Gumbel-Softmax relaxation makes both trainable by gradient descent.

What would settle it

Take the MS MARCO dev set and, for each question, check whether the gold answer's key entity appears as the object of any ConceptNet triple whose subject matches the question or passage under the paper's scoring rules; if a large share of correct answers require facts outside the top-1000 candidate pool, the reported gains would not be attributable to knowledge selection but to the retrieval heuristic. Alternatively, replace the heuristic scorer with a simple learned retriever and see whether ROUGE-L changes.

Watch

Extended reading notes

Core claim

KEAG augments a sequence-to-sequence answer generator with a source selector that emits a discrete latent variable $y_t \in \{1,2,3,4\}$ at each timestep to decide whether the next word comes from the question, the passage, the vocabulary, or the knowledge base. When knowledge is chosen, a second discrete latent variable $z_t$ picks one fact from a retrieved set of ConceptNet triples, conditioned on the decoder state, and the object of that fact contributes to the answer. The model is trained by maximizing a variational lower bound on answer likelihood and using the Gumbel-Softmax trick to backpropagate through the discrete choices. The paper reports that this architecture outperforms all compared systems on MS MARCO and that removing the knowledge component, the latent source indicators, or the source selector degrades ROUGE-L to 49.98, 47.61, and 38.33 respectively.

Load-bearing premise

The hand-coded scoring rules that rank candidate facts decide which facts the model can ever see; if the fact needed for a correct answer is not in the top 1000 candidates, the fact selector cannot retrieve it and the knowledge advantage disappears.

Editorial extensions

If this is right

  • If the source selector truly learns when knowledge is useful, then answer generators can be equipped with much larger or multiple knowledge bases without retraining the core decoder, since the selector can learn to ignore irrelevant facts.
  • The per-word source decisions give a built-in provenance signal for each answer word, which could support auditing or explaining why a QA system produced a particular answer.
  • The same architecture transfers to other generative tasks that mix extraction and background knowledge, such as abstractive summarization of technical documents where domain knowledge is not stated in the source text.
  • Because the fact selector picks from a fixed candidate pool, the model's ceiling is set by the quality of the retrieval step; improving retrieval should translate directly into higher answer quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the heuristic fact-scoring rules are themselves learnable; a natural extension would replace the +4/+2/+1 heuristic with a trained retriever and compare ROUGE-L, which would isolate how much of the gain comes from retrieval versus selection.
  • The source selector's per-word choices could be inspected as a weak form of explanation; one could test whether the knowledge source is chosen exactly at the words where the passage lacks the needed fact, as the visualization in Table 5 suggests.
  • Because the model appends the object of the selected fact, it is restricted to using the object as the surface realization; allowing the relation or a paraphrase of the fact to be verbalized might further improve fluency.
  • The comparison set is limited to QA-specific baselines; a stronger test would compare against a large pre-trained language model fine-tuned on the same data with knowledge injected differently.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes KEAG, a generative question-answering model that augments a sequence-to-sequence framework with a source selector over four information sources (question, passage, vocabulary, and external knowledge) and a stochastic fact selector that picks a ConceptNet triple when the knowledge source is chosen. Training uses a variational lower bound on the log-likelihood with Gumbel-Softmax estimators for the discrete source and fact latent variables. Experiments on the MS MARCO V2.1 dev set report ROUGE-L and BLEU-1 results against extractive/generative baselines without knowledge (Table 1) and knowledge-enriched baselines (Table 2), an ablation study (Table 4), and a 100-question human evaluation (Table 3). The central claim is that KEAG improves answer quality over models without knowledge and over existing knowledge-aware models.

Significance. If the reported gains are statistically reliable, the work makes a useful contribution by showing a concrete mechanism for selectively injecting symbolic knowledge into abstractive answer generation, with the added benefit of interpretable source attribution at each decoding step. The paper's strengths are its use of a public benchmark (MS MARCO), comparison with several strong baselines, component ablations, and a human evaluation, as well as a principled treatment of discrete latent variables via Gumbel-Softmax. The main limitations are the absence of significance testing or multiple-seed variance estimates, evaluation restricted to the dev set, and the unvalidated, asymmetric heuristic fact-scoring rule that determines the candidate fact pool. These issues make the quantitative claims plausible but not yet fully established.

major comments (3)
  1. [§5.1, especially the bulleted scoring rules] The fact-extraction scoring is asymmetric in a way that may exclude exactly the connective facts the model needs. A triple such as (Hawaii, LocatedIn, United States), the type needed for the paper's motivating Obama example, has its subject in the passage and its object in the question, so it receives only +1 under the stated rules, the same score as any fact whose subject merely appears in the passage. Since the fact selector can only choose among the top Nf=1000 facts, this ranking can be dominated by within-passage co-occurrence facts and by subject-in-question/object-in-passage triples, while passage-entity-to-question-entity links are not preferentially ranked. The paper does not validate this ranking against a held-out fact-recall gold set, and the Table 4 ablation (removing the whole knowledge module) cannot separate 'knowledge helps' from 'this particular scoring rule helps or hurts.' I request a fact-retrieval evaluation, an ablation or variant of the scoring rule, and at least a discussion of the asymmetry with respect to the motivating example.
  2. [§7.3 and §7.4, Tables 1-4] No statistical significance tests, confidence intervals, or multiple-seed results are reported. The reported margins are small: KEAG's ROUGE-L of 51.68 is only 1.17 above MHPGM's 50.51 in Table 2, and the knowledge-ablation drop in Table 4 is 1.70 ROUGE-L points. Without variance estimates or a paired significance test, these differences may be within noise. Please report results over multiple random seeds (or at least bootstrap confidence intervals) and apply a paired test such as a bootstrap or Wilcoxon signed-rank test for the main comparisons and the knowledge ablation.
  3. [§7.3, Table 3 and surrounding text] The human evaluation is based on only 100 randomly selected questions, with no inter-annotator agreement reported and no significance testing. The text states that KEAG 'significantly outperforms' the compared models on substantive correctness (4.03 vs 3.81 for MHPGM), but this claim is not backed by a statistical test. Please include the number of annotations per item, agreement metrics, and a significance test such as a paired permutation test.
minor comments (6)
  1. [Abstract and §7.3] The phrase 'confirming its effectiveness' is too strong given that the evaluation is on the dev set only and lacks significance tests; 'consistent with its effectiveness' would be more cautious.
  2. [§6, Eq. (9)-(11)] The paper optimizes the variational lower bound, not the exact log-likelihood, and the Gumbel-Softmax estimator introduces bias at nonzero temperature. This is standard, but it would be helpful to state explicitly that the reported objective is a lower bound and that gradients are approximations.
  3. [§5.2] The paper says that when the knowledge source is chosen, the model 'appends the object of the selected fact to the end of the answer.' Please clarify how multi-word objects are handled and whether the fact object's words are emitted one by one or as a single unit; this matters for the decoder's word-level likelihood computation.
  4. [§7.1] The description says hyper-parameters are tuned by cross-validation on the training set, while evaluation is on the dev set; please clarify the relationship between the 'cross-validation' split and the final training set used for the reported dev results.
  5. [Table 5 and §7.5] The running example would be more informative if it identified the specific ConceptNet fact selected and its score under the §5.1 extraction rules, since this directly illustrates the knowledge-injection mechanism that the paper claims.
  6. [Throughout] Minor consistency issues: 'Rouge-L' and 'ROUGE-L' are both used; also the reference list is missing venue information for a few entries (e.g., Bahdanau et al., 2015 and Gumbel, 1954).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: KEAG's claims are evaluated on a held-out external benchmark with an external knowledge source; no prediction reduces to a fitted constant or self-citation.

full rationale

The paper's central claim is that KEAG improves generative QA by integrating external knowledge from ConceptNet. The model is trained on the MS MARCO training split and evaluated on the held-out dev split, with hyper-parameters tuned by cross-validation on the training set (Section 7.1). The knowledge source (ConceptNet) is external, and the fact extraction scoring rules in Section 5.1 are hand-coded heuristics, not parameters fitted to the dev set. The ablation in Table 4 removes the knowledge module and observes a drop in ROUGE-L and BLEU-1; this is an empirical comparison, not a definitional identity. No load-bearing step cites prior work by the same authors, and no 'uniqueness theorem' is invoked. The only notable weakness is that the candidate fact pool depends on the heuristic scoring in Section 5.1, which could limit recall, but this is a correctness and efficacy risk, not circularity, because the scoring rules are not derived from the evaluation targets and the dev-set predictions are not used to set them.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The core contribution is an empirical architecture, so the ledger focuses on hand-set design choices and assumptions in the knowledge pipeline. The most load-bearing item is the heuristic fact scoring, which controls whether the model ever sees the right fact. The Gumbel-Softmax training and the dev-set evaluation protocol are additional assumptions that could change the reported results if altered.

free parameters (4)
  • Fact extraction scoring weights = +4, +2, +1
    Hand-set in Section 5.1 to rank related facts. They determine the candidate fact pool, and the fact selector can only choose from the top Nf facts, so these weights indirectly control the knowledge available to the decoder.
  • Maximum related facts Nf = 1000
    Set in Section 7.2. Caps the candidate pool; if relevant facts are ranked below 1000, they are excluded. The choice is not ablated.
  • Gumbel-Softmax temperature schedule = not specified
    Section 5.2 says the temperature is 'gradually annealed' but gives no schedule, which can affect training convergence and the quality of the learned discrete variables.
  • ConceptNet relation-type frequency threshold = not specified
    Section 7.1 states non-English entities and relation types with few facts are filtered, but the exact threshold is not given, affecting the knowledge base size.
assumptions (6)
  • domain assumption ConceptNet facts are relevant for MARCO queries
    The model uses ConceptNet as the sole external knowledge source in Section 7.1. The paper assumes these commonsense facts help answer MARCO questions; the only support is the dev set improvement.
  • ad hoc to paper The fact scoring rules recover the facts needed for correct answers
    Section 5.1's +4/+2/+1 rules are heuristic. If the needed fact is not in the top Nf, the model cannot use it, so the scoring rules are load-bearing.
  • standard math The variational lower bound is an adequate training objective
    Equations 10-11 use Jensen's inequality to transform the intractable marginal likelihood into a lower bound. This is standard, but the gap is not analyzed.
  • domain assumption Gumbel-Softmax provides effective gradients through discrete choices
    Section 5.2 relies on Gumbel-Softmax to train with discrete latent variables. The estimator is biased at finite temperature, and the paper does not validate that the selected sources or facts are meaningful.
  • domain assumption The dev-set evaluation with concatenated passages is comparable to prior baselines
    Section 7.1 states the dev set is evaluated by concatenating all passages for each question. If prior baselines used different preprocessing, the comparisons in Tables 1-2 may be unfair.
  • ad hoc to paper Appending the object of a selected fact produces an acceptable answer token
    In Section 5.2, when knowledge is chosen, the model appends the object of the sampled fact. This assumes objects are single tokens that fit grammatically into the answer, which may not hold for multi-word or relation-heavy facts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Incorporating External Knowledge into Machine Reading for Generative Question Answering." pith.science (2026). https://pith.science/paper/XDCTTIEP

@misc{pith2026190902745,
  author       = {Pith},
  title        = {Pith review of: Incorporating External Knowledge into Machine Reading for Generative Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDCTTIEP}},
  note         = {Machine review of arXiv:1909.02745}
}
read the original abstract

Commonsense and background knowledge is required for a QA model to answer many nontrivial questions. Different from existing work on knowledge-aware QA, we focus on a more challenging task of leveraging external knowledge to generate answers in natural language for a given question with context. In this paper, we propose a new neural model, Knowledge-Enriched Answer Generator (KEAG), which is able to compose a natural answer by exploiting and aggregating evidence from all four information sources available: question, passage, vocabulary and knowledge. During the process of answer generation, KEAG adaptively determines when to utilize symbolic knowledge and which fact from the knowledge is useful. This allows the model to exploit external knowledge that is not explicitly stated in the given text, but that is relevant for generating an answer. The empirical study on public benchmark of answer generation demonstrates that KEAG improves answer quality over models without knowledge and existing knowledge-aware models, confirming its effectiveness in leveraging knowledge.

Figures

Figures reproduced from arXiv: 1909.02745 by the authors.

Figure 1
Figure 1. An overview of the architecture of KEAG (best viewed in color). A question and a passage both go [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An overview of the fact selection module [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Large Vision-Language Model based Environment Perception System for Visually Impaired People

    cs.CV 2025-04 conditional novelty 4.0 of 10

    A wearable system that feeds segmentation results into a vision-language model's prompt improves scene description accuracy and object retrieval for visually impaired users.

Reference graph

Works this paper leans on

30 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations

    work page 2015

  4. [4]

    Lisa Bauer, Yicheng Wang, and Mohit Bansal. 2018. Commonsense for generative multi-hop question answering tasks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4220--4230

    2018

  5. [5]

    Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1535--1545

    work page 2011

  6. [6]

    Yao Fu and Yansong Feng. 2018. Natural answer generation with heterogeneous memory. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 185--195

    work page 2018

  7. [7]

    Shen Gao, Zhaochun Ren, Yihong Zhao, Dongyan Zhao, Dawei Yin, and Rui Yan. 2019. Product-aware answer generation in e-commerce question-answering. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining

    work page 2019

  8. [8]

    E.J. Gumbel. 1954. Statistical theory of extreme values and some practical applications: a series of lectures. Applied mathematics series. U. S. Govt. Print. Office

    work page 1954

Show all 30 references
  1. [9]

    Shizhu He, Cao Liu, Kang Liu, and Jun Zhao. 2017. Generating natural answers by incorporating copying and retrieving mechanisms in sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 199--208

    2017

  2. [10]

    Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparametrization with gumbel-softmax. In Proceedings of the International Conference on Learning Representations

    2017

  3. [11]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the ACL Workshop: Text Summarization Braches Out 2004, page 10

    2004

  4. [12]

    Teng Long, Emmanuel Bengio, Ryan Lowe, Jackie Chi Kit Cheung, and Doina Precup. 2017. World knowledge for reading comprehension: Rare entity prediction with hierarchical lstms using external descriptions. In Proceedings of the 2017 Conference on Empirical Methods in Natural La...

    2017

  5. [13]

    Mausam, Michael Schmitz, Robert Bart, Stephen Soderland, and Oren Etzioni. 2012. Open language learning for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, page...

    2012

  6. [14]

    Todor Mihaylov and Anette Frank. 2018. Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 821--832

    2018

  7. [15]

    Rajarshee Mitra. 2017. An abstractive approach to question answering. CoRR, abs/1711.06238

    2017 arXiv

  8. [16]

    Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning

    2016

  9. [17]

    Khapra, Anirban Laha, and Balaraman Ravindran

    Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1063--1072. Association fo...

    2017

  10. [18]

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co...

    2016

  11. [19]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311--318

    2002

  12. [20]

    Thomas Pellissier Tanon, Denny Vrande c i\' c , Sebastian Schaffert, Thomas Steiner, and Lydia Pintscher. 2016. From freebase to wikidata: The great migration. In Proceedings of the 25th International Conference on World Wide Web, pages 1419--1428

    2016

  13. [21]

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532--1543

    2014

  14. [22]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics

    2017

  15. [23]

    Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hananneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of the International Conference on Learning Representations

    2017

  16. [24]

    Robert Speer, Joshua Chin, and Catherine Havasi. 2016. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI Conference on Artificial Intelligence

    2016

  17. [25]

    Chuanqi Tan, Furu Wei, Nan Yang, Bowen Du, Weifeng Lv, and Ming Zhou. 2018. S-net: From answer extraction to answer synthesis for machine reading comprehension. In AAAI

    2018

  18. [26]

    Yizhong Wang, Kai Liu, Jing Liu, Wei He, Yajuan Lyu, Hua Wu, Sujian Li, and Haifeng Wang. 2018. Multi-passage machine reading comprehension with cross-passage answer verification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volum...

    2018

  19. [27]

    Dirk Weissenborn. 2017. Dynamic integration of background knowledge in neural NLU systems. CoRR, abs/1706.02596

    2017 arXiv

  20. [28]

    Bishan Yang and Tom Mitchell. 2017. Leveraging knowledge bases in LSTMs for improving machine reading. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1436--1446

    2017

  21. [29]

    Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative question answering. In Proceedings of the Workshop on Human-Computer Question Answering, pages 36--42

    2016

  22. [30]

    Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2018. Improving question answering by commonsense-based pre-training. CoRR, abs/1809.03568

    2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.