{"id":"532fbcd8-618a-48bc-995a-b9f8684e26fd","arxiv_id":"1909.02745","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A model that learns when to draw answer words from a knowledge base instead of the passage can generate higher-quality answers on the MS MARCO benchmark.","lead":"This paper presents a neural question-answering model that mixes text from the question, the passage, a standard vocabulary, and a knowledge base when writing each word of an answer. It reports higher answer-quality scores than existing models on a public benchmark, suggesting that selective use of external knowledge can improve machine-generated answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"§5.1's fact-scoring is directionally asymmetric: a triple whose subject is in the passage and object in the question receives only +1, so the connective facts the model's own 'born in the U.S.' example needs may fall below the N_f=1000 cutoff.","rationale":"The reader identified the §5.1 heuristic fact-scoring pipeline as the weakest assumption, and I agree that it is load-bearing: the fact selector can only choose among the extracted candidate pool, so any systematic exclusion of useful facts directly limits the knowledge contribution. My stress-test sharpens this into a concrete internal weakness rather than a generic 'not validated' concern. The scoring rules give a strong boost only when the subject is in the question and the object is in the passage; the reverse configuration, where a passage entity is the subject and a question entity is the object, receives the same +1 as any fact whose subject appears in the passage. That directionality is exactly what connects a passage to the question in the paper's own motivating example, and it is also the direction most relevant for generating an answer that uses knowledge not stated in the text. The asymmetry is not justified, not tested, and not ablated. Still, this concern does not overturn the central empirical claim. The reported gains are consistent across automatic and human evaluations, and the knowledge-ablation drop in Table 4 suggests the knowledge source contributes positively even under the current heuristic. The right response is to verify and, if needed, repair the extraction scoring rather than reject the work. The reader's CONDITIONAL verdict therefore stands unchanged; the concrete test above would determine whether the asymmetry is actually a bottleneck or merely a theoretical risk.","tokens_in":10548,"tokens_out":9254,"duration_ms":100353,"concrete_test":"Run a diagnostic on MARCO dev: extract candidate facts exactly as in §5.1 with N_f=1000. For a sample of questions whose gold answers require connecting a passage entity to a question entity, compute recall of the relevant ConceptNet triple in the candidate pool, including the paper's own Hawaii/US example. If recall is substantially below 1, the heuristic is a bottleneck. Then retrain or rerun KEAG with a symmetric scoring rule that also gives +4 when subject is in p and object is in q, and compare ROUGE-L/BLEU-1 on the dev set; a change beyond the 1-point knowledge-ablation gap would show the hand-coded directionality matters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the knowledge source being able to retrieve facts that connect the question and passage. Section 5.1 scores each extracted ConceptNet triple with +4 if the subject occurs in q and the object in p, +2 if both subject and object occur in p, and +1 if the subject occurs in q or p. This is not symmetric: a triple such as (Hawaii, LocatedIn, United States) — the type needed for the paper's motivating Obama/Hawaii example — has its subject in the passage and its object in the question, so it receives only +1, the same score as any fact whose subject merely appears in the passage. With N_f=1000, the ranking can therefore be dominated by within-passage lexical matches and by q-subject/p-object triples, while the passage-entity-to-question-entity link that is most diagnostic for answer generation is not preferentially ranked. The paper neither validates this ranking against a held-out fact-retrieval gold set nor ablates it; Table 4 removes the whole knowledge module, which cannot distinguish 'knowledge helps' from 'this particular extraction rule helps or hinders'. Since the fact selector can only pick from this pool, any recall failure directly caps the knowledge signal; the +1.70 ROUGE-L gain from knowledge in Table 4 could be an underestimate or an artifact of which facts happen to survive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KEAG, a generative question-answering model that augments a sequence-to-sequence framework with a source selector over four information sources (question, passage, vocabulary, and external knowledge) and a stochastic fact selector that picks a ConceptNet triple when the knowledge source is chosen. Training uses a variational lower bound on the log-likelihood with Gumbel-Softmax estimators for the discrete source and fact latent variables. Experiments on the MS MARCO V2.1 dev set report ROUGE-L and BLEU-1 results against extractive/generative baselines without knowledge (Table 1) and knowledge-enriched baselines (Table 2), an ablation study (Table 4), and a 100-question human evaluation (Table 3). The central claim is that KEAG improves answer quality over models without knowledge and over existing knowledge-aware models.","tokens_in":10832,"tokens_out":3948,"duration_ms":41803,"significance":"If the reported gains are statistically reliable, the work makes a useful contribution by showing a concrete mechanism for selectively injecting symbolic knowledge into abstractive answer generation, with the added benefit of interpretable source attribution at each decoding step. The paper's strengths are its use of a public benchmark (MS MARCO), comparison with several strong baselines, component ablations, and a human evaluation, as well as a principled treatment of discrete latent variables via Gumbel-Softmax. The main limitations are the absence of significance testing or multiple-seed variance estimates, evaluation restricted to the dev set, and the unvalidated, asymmetric heuristic fact-scoring rule that determines the candidate fact pool. These issues make the quantitative claims plausible but not yet fully established.","major_comments":[{"comment":"The fact-extraction scoring is asymmetric in a way that may exclude exactly the connective facts the model needs. A triple such as (Hawaii, LocatedIn, United States), the type needed for the paper's motivating Obama example, has its subject in the passage and its object in the question, so it receives only +1 under the stated rules, the same score as any fact whose subject merely appears in the passage. Since the fact selector can only choose among the top Nf=1000 facts, this ranking can be dominated by within-passage co-occurrence facts and by subject-in-question/object-in-passage triples, while passage-entity-to-question-entity links are not preferentially ranked. The paper does not validate this ranking against a held-out fact-recall gold set, and the Table 4 ablation (removing the whole knowledge module) cannot separate 'knowledge helps' from 'this particular scoring rule helps or hurts.' I request a fact-retrieval evaluation, an ablation or variant of the scoring rule, and at least a discussion of the asymmetry with respect to the motivating example.","section":"§5.1, especially the bulleted scoring rules"},{"comment":"No statistical significance tests, confidence intervals, or multiple-seed results are reported. The reported margins are small: KEAG's ROUGE-L of 51.68 is only 1.17 above MHPGM's 50.51 in Table 2, and the knowledge-ablation drop in Table 4 is 1.70 ROUGE-L points. Without variance estimates or a paired significance test, these differences may be within noise. Please report results over multiple random seeds (or at least bootstrap confidence intervals) and apply a paired test such as a bootstrap or Wilcoxon signed-rank test for the main comparisons and the knowledge ablation.","section":"§7.3 and §7.4, Tables 1-4"},{"comment":"The human evaluation is based on only 100 randomly selected questions, with no inter-annotator agreement reported and no significance testing. The text states that KEAG 'significantly outperforms' the compared models on substantive correctness (4.03 vs 3.81 for MHPGM), but this claim is not backed by a statistical test. Please include the number of annotations per item, agreement metrics, and a significance test such as a paired permutation test.","section":"§7.3, Table 3 and surrounding text"}],"minor_comments":[{"comment":"The phrase 'confirming its effectiveness' is too strong given that the evaluation is on the dev set only and lacks significance tests; 'consistent with its effectiveness' would be more cautious.","section":"Abstract and §7.3"},{"comment":"The paper optimizes the variational lower bound, not the exact log-likelihood, and the Gumbel-Softmax estimator introduces bias at nonzero temperature. This is standard, but it would be helpful to state explicitly that the reported objective is a lower bound and that gradients are approximations.","section":"§6, Eq. (9)-(11)"},{"comment":"The paper says that when the knowledge source is chosen, the model 'appends the object of the selected fact to the end of the answer.' Please clarify how multi-word objects are handled and whether the fact object's words are emitted one by one or as a single unit; this matters for the decoder's word-level likelihood computation.","section":"§5.2"},{"comment":"The description says hyper-parameters are tuned by cross-validation on the training set, while evaluation is on the dev set; please clarify the relationship between the 'cross-validation' split and the final training set used for the reported dev results.","section":"§7.1"},{"comment":"The running example would be more informative if it identified the specific ConceptNet fact selected and its score under the §5.1 extraction rules, since this directly illustrates the knowledge-injection mechanism that the paper claims.","section":"Table 5 and §7.5"},{"comment":"Minor consistency issues: 'Rouge-L' and 'ROUGE-L' are both used; also the reference list is missing venue information for a few entries (e.g., Bahdanau et al., 2015 and Gumbel, 1954).","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a plausible architecture and a directionally positive empirical result, but the two main load-bearing issues are the unvalidated, asymmetric fact-scoring heuristic that determines the knowledge candidate pool and the absence of any statistical reliability assessment. Both are fixable within the manuscript's scope: the authors can add a fact-recall or ranking evaluation, ablate or repair the scoring rule, and report variance/significance. I do not see a fundamental flaw that would warrant rejection; the contribution is potentially solid but needs stronger evidence before it can be accepted. There is also a mild concern that the comparison to 'gQA' as a simplification of KEAG is somewhat loose, since the source selector and question source are separate contributions; the ablations partially address this."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick read on 1909.02745. The genuinely new piece is the four-way source selector: each generated word can come from the question, the passage, the vocabulary, or directly from an external knowledge-base fact, with a Gumbel-Softmax-trained discrete fact selector. That direct word-from-knowledge path is not in MHPGM or the adapted baselines. The architecture is mostly assembled from known seq2seq/attention/copy pieces, but the combination is reasonable, and the reported gains on MARCO dev are consistent: about +1.2 ROUGE-L and BLEU-1 over the strongest knowledge-aware baseline, and the ablation removing knowledge costs about 1.7 ROUGE-L.\n\nThe soft spots are real but not fatal. The evaluation is single-passage-concatenation on the dev set, with no multiple runs, significance tests, or confidence intervals, so the paper's \"significant improvement\" language is doing more work than the numbers support. No code or data release, which makes it hard to verify the baseline numbers. The human evaluation covers only 100 questions and the scores are close.\n\nThe most interesting soft spot is §5.1's fact scoring. The stress-test note is right: the heuristic is asymmetric. A triple like (Hawaii, LocatedIn, United States)—exactly the connective fact the Obama example needs—gets +1 because the subject is in the passage and the object in the question, while +4 is reserved for the reverse direction. The +1 bucket is huge, so with N_f=1000 the candidate pool can easily fill with within-passage lexical matches and miss the question-passage bridge. The paper neither validates this ranking against a held-out fact-retrieval set nor ablates the scoring rules; Table 4 only removes the whole knowledge module, so it cannot distinguish 'knowledge helps' from 'this particular extraction rule helps.' That does not sink the central claim—the model still gains—but it means the knowledge advantage could be an artifact of which facts happen to survive a rather arbitrary filter.\n\nOverall: the central claim is directionally supported, the novelty is modest but real, and the citation pattern is fair. The paper deserves peer review rather than desk reject, but a serious referee should demand significance testing, code release, and an ablation of the fact-scoring rules. I would not cite it as a primary result in the next year, but I would mention it in related work. For a reading group, maybe—it is a good discussion piece for knowledge integration, not a definitive study.","headline":"A plausible knowledge-aware generative QA architecture with a real but unvalidated fact-retrieval bottleneck; the central gain is directionally supported but the evaluation is too thin to call significant.","tokens_in":11379,"tokens_out":2175,"would_cite":false,"duration_ms":22749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A QA model that picks each word's knowledge source beats all compared systems on MS MARCO.","keywords":["knowledge-enriched answer generation","generative question answering","machine reading comprehension","external knowledge","source selection","fact selection","Gumbel-Softmax","MS MARCO"],"falsifier":"Take the MS MARCO dev set and, for each question, check whether the gold answer's key entity appears as the object of any ConceptNet triple whose subject matches the question or passage under the paper's scoring rules; if a large share of correct answers require facts outside the top-1000 candidate pool, the reported gains would not be attributable to knowledge selection but to the retrieval heuristic. Alternatively, replace the heuristic scorer with a simple learned retriever and see whether ROUGE-L changes.","tokens_in":10318,"feed_emoji":"🧠","tokens_out":5096,"duration_ms":46915,"temperature":0.7,"pith_summary":"The paper argues that a question-answering system can write better natural-language answers if, at every decoding step, it explicitly chooses which of four sources—the question, the passage, the global vocabulary, or an external knowledge base—should supply the next word. On the MS MARCO answer-generation benchmark, the proposed KEAG model reaches ROUGE-L 51.68 and BLEU-1 45.97, higher than both knowledge-free generative QA models and knowledge-aware baselines. The improvement matters because many questions require commonsense facts that never appear in the passage, and letting the model decide when to consult knowledge is presented as the key to using that knowledge without letting it override the text.","feed_headline":"Word-by-word knowledge gate lifts QA answers above baselines","feed_subtitle":"On MS MARCO, KEAG reaches ROUGE-L 51.68 and BLEU-1 45.97, ahead of knowledge-free and knowledge-aware systems.","key_machinery":"The central mechanism is the source selector, a learned categorical gate that runs before every generated word and decides which of the four information sources should produce it. Tied to it is a stochastic fact selector that scores the candidate facts by an attention-style function over fact embeddings and the decoder state, sampling a fact with Gumbel-Softmax. These two discrete latent variables are what let the model blend external symbolic knowledge with surface text; the source selector answers 'when to use knowledge' and the fact selector answers 'which fact', and the Gumbel-Softmax relaxation makes both trainable by gradient descent.","core_discovery":"KEAG augments a sequence-to-sequence answer generator with a source selector that emits a discrete latent variable $y_t \\in \\{1,2,3,4\\}$ at each timestep to decide whether the next word comes from the question, the passage, the vocabulary, or the knowledge base. When knowledge is chosen, a second discrete latent variable $z_t$ picks one fact from a retrieved set of ConceptNet triples, conditioned on the decoder state, and the object of that fact contributes to the answer. The model is trained by maximizing a variational lower bound on answer likelihood and using the Gumbel-Softmax trick to backpropagate through the discrete choices. The paper reports that this architecture outperforms all compared systems on MS MARCO and that removing the knowledge component, the latent source indicators, or the source selector degrades ROUGE-L to 49.98, 47.61, and 38.33 respectively.","pith_inferences":["The paper does not test whether the heuristic fact-scoring rules are themselves learnable; a natural extension would replace the +4/+2/+1 heuristic with a trained retriever and compare ROUGE-L, which would isolate how much of the gain comes from retrieval versus selection.","The source selector's per-word choices could be inspected as a weak form of explanation; one could test whether the knowledge source is chosen exactly at the words where the passage lacks the needed fact, as the visualization in Table 5 suggests.","Because the model appends the object of the selected fact, it is restricted to using the object as the surface realization; allowing the relation or a paraphrase of the fact to be verbalized might further improve fluency.","The comparison set is limited to QA-specific baselines; a stronger test would compare against a large pre-trained language model fine-tuned on the same data with knowledge injected differently."],"forward_implications":["If the source selector truly learns when knowledge is useful, then answer generators can be equipped with much larger or multiple knowledge bases without retraining the core decoder, since the selector can learn to ignore irrelevant facts.","The per-word source decisions give a built-in provenance signal for each answer word, which could support auditing or explaining why a QA system produced a particular answer.","The same architecture transfers to other generative tasks that mix extraction and background knowledge, such as abstractive summarization of technical documents where domain knowledge is not stated in the source text.","Because the fact selector picks from a fixed candidate pool, the model's ceiling is set by the quality of the retrieval step; improving retrieval should translate directly into higher answer quality."],"supporting_citations":[{"why":"Supplies the MS MARCO dataset and the BLEU-1/ROUGE-L evaluation metrics used for all comparisons.","marker":"(Nguyen et al., 2016)"},{"why":"Provides ConceptNet, the external knowledge base from which fact triples are extracted for the knowledge source.","marker":"(Speer et al., 2016)"},{"why":"Gives the Gumbel-Softmax estimator used to differentiate through the discrete source and fact selection variables.","marker":"(Jang et al., 2017)"},{"why":"Defines the attention mechanism the encoders and decoder build on for question and passage context vectors.","marker":"(Bahdanau et al., 2015)"},{"why":"Supplies the coverage mechanism added to attentions to avoid repetitive generated answers.","marker":"(See et al., 2017)"},{"why":"gQA is the generative baseline that KEAG extends and the comparison used to show the benefit of knowledge integration.","marker":"(Mitra, 2017)"},{"why":"MHPGM is the strongest knowledge-aware baseline that KEAG must beat, and differs in that its decoder cannot draw words directly from knowledge.","marker":"(Bauer et al., 2018)"}],"fun_headline_variants":["KEAG: discrete knowledge gate boosts QA answers","QA model learns when and which fact to use","Gumbel-Softmax selector taps ConceptNet for answers","Latent source choice lifts generative QA quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The hand-coded scoring rules that rank candidate facts decide which facts the model can ever see; if the fact needed for a correct answer is not in the top 1000 candidates, the fact selector cannot retrieve it and the knowledge advantage disappears.","fun_headline_variants_meta":{"raw":{"variants":["KEAG: discrete knowledge gate boosts QA answers","QA model learns when and which fact to use","Gumbel-Softmax selector taps ConceptNet for answers","Latent source choice lifts generative QA quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1192,"prompt_tokens":907,"completion_tokens":285,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":224}},"tokens_in":523,"tokens_out":285,"duration_ms":3413,"temperature":1.0,"reasoning_tokens":224,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:40:40.123924+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the MS MARCO dev set and, for each question, check whether the gold answer's key entity appears as the object of any ConceptNet triple whose subject matches the question or passage under the paper's scoring rules; if a large share of correct answers require facts outside the top-1000 candidate pool, the reported gains would not be attributable to knowledge selection but to the retrieval heuristic. Alternatively, replace the heuristic scorer with a simple learned retriever and see whether ROUGE-L changes.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MS MARCO dataset and the BLEU-1/ROUGE-L evaluation metrics used for all comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ConceptNet, the external knowledge base from which fact triples are extracted for the knowledge source."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Gumbel-Softmax estimator used to differentiate through the discrete source and fact selection variables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the attention mechanism the encoders and decoder build on for question and passage context vectors."},{"cited_title":"Liu, and Christopher D","cited_arxiv_id":null,"evidence_quote":"Supplies the coverage mechanism added to attentions to avoid repetitive generated answers."},{"cited_title":"A Generative Approach to Question Answering","cited_arxiv_id":"1711.06238","evidence_quote":"gQA is the generative baseline that KEAG extends and the comparison used to show the benefit of knowledge integration."}],"review_version":1}