{"id":"a5139e27-720d-4951-a881-17a0b8ab4bc2","arxiv_id":"1908.06917","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"QAmp uses unsupervised message passing over knowledge graphs to answer complex questions, achieving higher recall than a prior SPARQL-based baseline on LC-QuAD while showing that question interpretation, not graph reasoning, is the main source of errors.","lead":"This paper presents QAmp, a system that answers complex questions over knowledge graphs by matching words in the question to graph terms and then spreading confidence scores through the graph to find answers. It reports better recall than a prior SPARQL-based system on the LC-QuAD benchmark, and its error analysis shows that most mistakes come from understanding the question, not from the graph search.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Direction-blind message passing contradicts the Abstract's 'always' guarantee: Setup 1 with gold terms still fails on 5% of LC-QuAD test questions that require edge directionality (Section 5.1), so the correctness claim needs a direction-invariance qualifier.","rationale":"The reader's weakest assumption correctly identifies the directionality premise as the most load-bearing issue, and I agree. The baseline comparison in Table 2 is also uncontrolled, since WDAqua's reported numbers are for the full LC-QuAD dataset while QAmp is evaluated on the test split and its parser is trained on the training split; however, that is an empirical comparability problem that can be fixed by rerunning the baseline under identical conditions. The directionality problem is more fundamental because it attacks the paper's central theoretical claim that reasoning is correct once interpretation is correct. Setup 1 is the cleanest available test of that claim: with gold terms and gold question type, 5% of test questions still fail because Algorithm 1 is direction-blind. This is an internal inconsistency between the symmetric adjacency construction in Section 3.2 and the Abstract's unconditional 'always' wording, not merely a disagreement with external consensus. The method remains a solid contribution for direction-invariant or bidirectional questions, and the authors are transparent about the limitation in Section 6, so the appropriate resolution is to narrow the claim while keeping the contribution intact. The concrete test would settle the matter by reproducing the 5% failure count and making the abstract's qualifier explicit.","tokens_in":16212,"tokens_out":10118,"duration_ms":108265,"concrete_test":"Use the gold SPARQL queries in the LC-QuAD test split to identify all questions whose answer set changes when any triple in the query pattern is reversed; this should recover the 53 direction-sensitive questions reported in Section 5.1. Run Algorithm 1 in Setup 1 mode (exact gold entity, property and class URIs, gold question type) on exactly those questions, using the same DBpedia 2016-04 HDT file, and record whether the gold answer is ranked first (or, for COUNT/ASK, matches exactly). If any of the 53 questions fails to return the gold answer, the Abstract's 'always' claim is false; the resulting failure count also makes the required qualifier precise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 builds symmetric adjacency matrices: 'All adjacency matrices are symmetric, because I(q) does not model edge directionality, i.e., it treats K as undirected' (Section 3.2). Consequently, message passing cannot distinguish subject from object in a triple. Section 5.1's Setup 1, which feeds exact ground-truth entity/property/class URIs and the gold question type into the answer inference module, still yields errors on 9% of test questions; the paper attributes 53 of these (5%) to relations whose direction is essential, e.g., dbp:doctoralStudents versus dbp:doctoralAdvisor. For those questions, even a perfectly correct relevance score distribution over terms cannot guarantee a correct answer ranking, because the propagation step conflates the two directions of the relation. This directly contradicts the Abstract's unconditional claim that 'given a correct relevance score distribution, our approach always produces a correct answer ranking.' The guarantee is at best conditional on the question being answer-invariant under edge reversal or on the KG containing both directions. Since the authors themselves identify directionality as future work in Section 6, the Abstract overstates the formal contribution of the answer inference phase.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents QAmp, a question-answering system over knowledge graphs that separates question interpretation from answer inference. The interpretation phase identifies question type and extracts entity, predicate, and class mentions using supervised sequence labeling, then matches them to KG terms via BM25 and FastText embeddings. The answer inference phase performs unsupervised message passing over the subgraph induced by the matched terms, implemented as sparse matrix multiplications, optionally constrained by class filters and question-type aggregation. The authors evaluate QAmp on the LC-QuAD benchmark, reporting an F-measure of 0.33 compared with 0.28 for the WDAqua baseline, and provide an ablation study and error analysis. The central formal claim is that, given a correct relevance score distribution over terms, the message-passing answer inference always produces a correct ranking.","tokens_in":16370,"tokens_out":3282,"duration_ms":36003,"significance":"If properly qualified, this is a useful contribution: it provides a scalable, open-sourced, unsupervised alternative to SPARQL-query-building for complex KGQA, with a clean separation between interpretation and reasoning. The ablation study is informative, clearly identifying predicate matching as the main bottleneck, and the error analysis gives concrete evidence of benchmark and KG inconsistencies. The claim of a guaranteed correct ranking under perfect interpretation is important for the paper's narrative, but as written it is contradicted by the authors' own Setup 1 results, which show residual errors including directionality-sensitive cases. With an explicit qualifier (e.g., invariance under edge reversal), the formal claim becomes defensible for the vast majority of LC-QuAD questions. The end-to-end comparison to WDAqua is also not fully controlled, so the state-of-the-art claim needs to be moderated.","major_comments":[{"comment":"The abstract's claim that 'given a correct relevance score distribution, our approach always produces a correct answer ranking' is contradicted by the paper's own Setup 1 ablation. When the answer inference module receives all ground-truth entity, property, and class URIs plus the gold question type, Table 3 still reports errors on 9% of test questions, with 53 questions (5%) requiring modeling of exact subject-object order, e.g., dbp:doctoralStudents versus dbp:doctoralAdvisor. Because Algorithm 1 constructs symmetric adjacency matrices and explicitly treats the graph as undirected (Section 3.2), the guarantee can hold at most for questions whose answers are invariant under edge reversal or where the KG encodes both directions. The unconditional 'always' statement must be qualified to this setting, and the corresponding sentence should be revised in the abstract, introduction, and Section 5.1.","section":"Abstract; Section 5.1, Table 3"},{"comment":"The claim of outperforming the state of the art is not supported by a controlled comparison. The WDAqua numbers are taken from the published results of Diefenbach et al. rather than re-run under the same evaluation harness, and the WDAqua precision is estimated from the reported precision for answered questions only (footnote in Table 2). Differences in metric computation, answer post-processing, and timeout handling can materially change P, R, and F. The paper should either re-run the baseline with the exact same evaluation script or clearly state that the comparison is indicative, not a controlled benchmark, and soften the 'outperforms the state-of-the-art' phrasing accordingly.","section":"Section 4, Table 2"}],"minor_comments":[{"comment":"The text first says 53 test questions (5%) require directionality but later mentions only 35 questions (3.5%) of the LC-QuAD test split use relations of this type. These numbers should be reconciled or the definitions clarified.","section":"Section 5.1"},{"comment":"The notation is under-specified: the symbol ⊗ for the combination of the 3-dimensional tensor S with property activations is not defined, and the update in line 3 mixes a loop over Pj with the tensor S in a way that is hard to follow. A brief textual explanation of the tensor dimensions and operations would improve reproducibility.","section":"Algorithm 1"},{"comment":"The assumption that all edges are treated as undirected is stated in Algorithm 1's adjacency matrix definition but is only given a detailed motivation in Section 6. Consider stating the assumption and its implications prominently in Section 3.2, since it is central to the method.","section":"Section 3.2"},{"comment":"The sentence 'In this setup, all correct answers according to the benchmark were ranked at the top' is stronger than the reported metrics (P=0.89, R=0.92) justify. The authors should either specify that this holds for a subset of questions or rephrase to 'the correct answers were normally ranked at the top'.","section":"Section 5.1, row 4"},{"comment":"The paper contains several typos and formatting issues (e.g., 'embedded' for 'embedded' in Section 4.2, unbalanced parentheses in the Algorithm 1 description). A careful proofreading pass is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core algorithmic idea is sound and the empirical study is useful, but the headline claims need adjustment. The 'always correct ranking' claim is directly falsified by the authors' own Setup 1 experiment, and the state-of-the-art comparison is not controlled. Both issues are fixable by careful rewording and, ideally, by re-running the baseline. I do not see evidence of fabrication or fundamental unsoundness; the paper's contributions can stand once the claims are aligned with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me cut to the chase. This is a legitimately useful KGQA paper: QAmp is fast, unsupervised, interpretable, and open-sourced, and the ablation cleanly shows that interpretation, not graph reasoning, is the bottleneck. The near-perfect Setup 1 result (F=0.98) when gold URIs and question type are fed in supports the core claim that the message passing does what it says, under the paper's undirected-edge assumption. The error analysis also identifies real issues in LC-QuAD and DBpedia that the community should know. I'd send this out.\n\nThat said, the abstract overstates the guarantee. \"Given a correct relevance score distribution, our approach always produces a correct answer ranking\" is contradicted by the paper's own Setup 1: 9% of test questions still have errors with gold terms, and the authors attribute 5% to relations where direction matters (doctoralAdvisor vs doctoralStudents). Symmetric adjacency matrices cannot distinguish subject from object, so for those questions a perfect interpretation still gives the wrong ranking. The conclusion already admits directionality is future work, so the abstract needs a qualifier like \"for questions that are invariant under edge reversal\" or \"under the undirected edge assumption.\" The claim as written is not true for LC-QuAD's own test set.\n\nSecond soft spot: the WDAqua comparison is not controlled. The baseline numbers were reported on the full LC-QuAD dataset, while QAmp is evaluated on the test split, and QAmp's parser is trained on LC-QuAD's training split. The direction of the bias is probably in QAmp's favor, so \"outperforms state-of-the-art\" should be softened to \"reports improved numbers under a different evaluation setup.\" This is fixable by rerunning WDAqua on the same split or reporting both.\n\nMinor: the free parameters (confidence thresholds, top-k sizes, aggregation weights) are hand-chosen. They aren't fitted to test labels, and the held-out evaluation means this isn't circular, but a sensitivity analysis would help.\n\nVerdict: worth a serious referee. The method is a real alternative to SPARQL query building and the benchmark critique is valuable. But the authors need to narrow the correctness claim and make the baseline comparison apples-to-apples before this is citable as a state-of-the-art result. I would accept for review and ask for those revisions.","headline":"A genuinely useful KGQA paper whose core reasoning is sound, but the abstract's 'always correct' guarantee is false on the paper's own benchmark and the headline baseline comparison is not apples-to-apples.","tokens_in":16988,"tokens_out":1956,"would_cite":true,"duration_ms":20361,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Complex knowledge-graph questions can be answered by propagating confidence scores through the graph, without translating the question into a formal query.","keywords":["question answering","knowledge graphs","message passing","spreading activation","associative retrieval","approximate reasoning","complex question answering","sparse matrix multiplication"],"falsifier":"Take a pair of questions that differ only by reversing a directed relation, for example asking for the doctoral advisor versus the doctoral student of the same person, and feed both with the correct entity and property matches at confidence 1. The symmetric adjacency matrices used in Algorithm 1 cannot distinguish the two directions, so any such pair with a unique answer would violate the claim that correct interpretation always yields a correct answer ranking.","tokens_in":15941,"feed_emoji":"🕸️","tokens_out":8868,"duration_ms":88099,"temperature":0.7,"pith_summary":"The paper proposes that complex question answering over knowledge graphs splits cleanly into two phases: interpreting the natural-language question into graph terms with confidence scores, and pushing those scores through the graph to rank answers. The central claim is that the second phase, the reasoning itself, is exact: whenever the question interpretation is correct, the message-passing procedure places the correct answers at the top. This matters because it would remove query translation and subgraph-isomorphism search from the hard part of the problem, leaving only language interpretation as the source of uncertainty. The paper supports the claim with an end-to-end evaluation on a 5,000-question complex QA benchmark, an ablation that isolates each component's errors, and an error analysis showing that many apparent failures are missing gold answers or inconsistent graph edges.","feed_headline":"Spreading confidence scores answers complex graph questions","feed_subtitle":"A new method ranks the right answer whenever the question terms are interpreted correctly, in under a second on average.","key_machinery":"The load-bearing object is the message-passing update in Algorithm 1: a sparse matrix multiplication routine that mimics joins over small local subgraphs. For each matched property reference, the adjacency matrices of candidate property URIs are weighted by property confidence and summed; entity confidence scores are then propagated over the combined matrix; and the final score of an entity is a normalized combination of the propagated activation sum and the number of entity and property references that activated it. The same update runs hop by hop, so the answer set of one hop seeds the next. The algorithm's symmetry—all adjacency matrices are symmetric—is what makes the propagation order-independent, and it is also the point where the undirected-edge assumption enters.","core_discovery":"QAmp's central discovery is that answer inference for complex KGQA can be a deterministic, unsupervised diffusion over the local subgraph rather than a learned or templated query-construction process. The paper models a question as a sequence of hops, where each hop contributes entity, property, and class references with confidence scores. Inference then builds symmetric adjacency matrices for the matched properties, propagates the entity and property confidences through those matrices, and combines summed activations with counts of non-zero references to score every candidate answer. Because every adjacency matrix is symmetric, the algorithm deliberately treats edges as undirected; the paper reports that only 5% of questions in its benchmark require edge direction. Given that relaxation, the paper argues that a correct relevance score distribution always yields a correct answer ranking, so the residual end-to-end error is attributable to parsing and matching, with predicate matching as the weakest component.","pith_inferences":["The paper's own 5% directionality figure suggests a modest extension: keep the message-passing core but make messages direction-aware, perhaps through two symmetric matrices per relation encoding subject and object roles; the guarantee would then hold on directed questions without a full supervised direction predictor.","The aggregation formula in Algorithm 1 is heuristic; replacing it with calibrated sum-product updates could yield confidence scores that are comparable across questions, at the cost of the paper's closed-form efficiency.","Because inference is independent of the linker, QAmp could be used as a measurement instrument: swap different entity and predicate matchers into the same inference code to attribute end-to-end errors cleanly.","The false-negative cases suggest that precision numbers on this benchmark are inflated and recall numbers underestimated; a canonicalized answer set would re-rank all systems."],"forward_implications":["Given a correct relevance-score distribution, the paper's inference procedure is guaranteed to rank correct answers first, so no supervised ranking or query-generation model is needed at answer time.","Multi-hop compound questions are answered by chaining hops: each hop's answer activations seed the next hop, all within the same matrix-algebra framework.","The approach runs on a billion-triple knowledge graph in about 0.72 seconds per question on average, roughly twice as fast as the SPARQL-based baseline.","End-to-end accuracy is capped by term matching, especially predicate matching, so better relation linking is the highest-leverage improvement.","Because inference is not tied to template queries, the system can surface correct answers that the benchmark's single-URI gold standard missed."],"supporting_citations":[{"why":"Supplies the state-of-the-art SPARQL-based baseline and the reported precision/recall numbers that QAmp is compared against.","marker":"[8]"},{"why":"Provides the 5,000-question benchmark, its train/test splits, ground-truth queries, and the gold reference spans used in the ablation study.","marker":"[45]"},{"why":"Provides the compressed HDT index that lets the implementation access the billion-triple knowledge graph efficiently.","marker":"[10]"},{"why":"Describes the earlier spreading-activation graph query approach that QAmp scales up and replaces with matrix operations.","marker":"[13]"},{"why":"Supplies the word-vector model used for semantic matching of predicate and class references in the matching step.","marker":"[3]"},{"why":"Supplies the pre-trained word vectors used to train the question-type classifier and the reference-extraction sequence tagger.","marker":"[36]"},{"why":"Documents the query-building bottlenecks in component-based KGQA pipelines, motivating the matrix-based inference alternative.","marker":"[42]"}],"fun_headline_variants":["Message passing boosts complex QA over knowledge graphs","Unsupervised message passing cracks complex graph queries","Confidence propagation locates answers on knowledge graphs","Diffuse question confidence to answer complex graph queries","Outperforms on LC-QuAD via unsupervised message passing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes edge direction in the knowledge graph does not matter, so a question whose answer depends on direction—like doctoral advisor versus doctoral student—can get the wrong answer even with perfect interpretation.","fun_headline_variants_meta":{"raw":{"variants":["Message passing boosts complex QA over knowledge graphs","Unsupervised message passing cracks complex graph queries","Confidence propagation locates answers on knowledge graphs","Diffuse question confidence to answer complex graph queries","Outperforms on LC-QuAD via unsupervised message passing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000323,"raw_usage":{"total_tokens":1816,"prompt_tokens":949,"completion_tokens":867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":795}},"tokens_in":565,"tokens_out":867,"duration_ms":7779,"temperature":1.0,"reasoning_tokens":795,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:31:06.729957+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a pair of questions that differ only by reversing a directed relation, for example asking for the doctoral advisor versus the doctoral student of the same person, and feed both with the correct entity and property matches at confidence 1. The symmetric adjacency matrices used in Algorithm 1 cannot distinguish the two directions, so any such pair with a unique answer would violate the claim that correct interpretation always yields a correct answer ranking.","supporting_citations":[{"cited_title":"Towards a Question Answering System over the Semantic Web","cited_arxiv_id":"1803.00832","evidence_quote":"Supplies the state-of-the-art SPARQL-based baseline and the reported precision/recall numbers that QAmp is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 5,000-question benchmark, its train/test splits, ground-truth queries, and the gold reference spans used in the ablation study."},{"cited_title":"Fernández, Miguel A","cited_arxiv_id":null,"evidence_quote":"Provides the compressed HDT index that lets the implementation access the billion-triple knowledge graph efficiently."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the earlier spreading-activation graph query approach that QAmp scales up and replaces with matrix operations."},{"cited_title":"In EMNLP 2018","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained word vectors used to train the question-type classifier and the reference-extraction sequence tagger."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the query-building bottlenecks in component-based KGQA pipelines, motivating the matrix-based inference alternative."}],"review_version":1}