{"id":"57ef88a1-dbac-47fb-90d8-76f3cca486c4","arxiv_id":"1908.00469","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"QUEST builds a noisy graph from facts extracted across web pages and uses group Steiner trees to join evidence from different documents into direct answers for complex questions.","lead":"QUEST is a system that answers multi-part questions by pulling facts from many web pages into a graph and finding the connecting paths between question terms. It needs no training data, so it can handle fresh topics and unusual phrasing, which makes it useful for search and voice assistants.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The edge-weight specification can violate the non-negative cost assumption of the GST algorithm, so the claimed optimal-tree results may not match the described method.","rationale":"The reader's weakest assumption points to the semantic fragility of cornerstone and alignment matching, which is supported by the paper's own error analysis. I agree that this is a real limitation, but I do not think it is the single most load-bearing concern. The paper's error analysis also shows that the answer is present in the top-50 GSTs roughly 90-94% of the time when it is in the quasi KG, so the semantic matching premise is empirically plausible. A more fundamental issue is that the written algorithm is internally inconsistent: the proximity-based triple scores are unbounded, while the GST formulation requires non-negative edge costs and expresses costs as 1 - score. Without a stated normalization, the exact dynamic program's optimality guarantee does not apply to the graph as described. This is checkable from the released code and would settle the matter directly. The conditional verdict remains appropriate; I would not reject the paper on this basis because the code is publicly available and the issue may be an oversight in the description, but it must be verified before the central claim can be accepted.","tokens_in":17912,"tokens_out":9306,"duration_ms":111027,"concrete_test":"Inspect the released source code at the paper's stated URL and locate the edge-cost construction for triple edges. Check whether every triple confidence score is normalized to [0,1] before computing cost = 1 - score. Then rerun QUEST on the CQ-W and CQ-T top-10 corpora in two configurations: (a) raw scores converted directly, and (b) scores min-max normalized to [0,1] before conversion. If the two configurations produce different GSTs or materially different MRR values, the reported experiments do not correspond to the described exact-GST procedure, and the method's central algorithmic claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines triple-edge confidence as the sum over sentences of 1 divided by word distance, with no normalization and no stated upper bound. A subject-predicate pair occurring in two sentences at distance 1 has score 2. Section 4.1 then defines GST edge costs as cost = 1 - score on \"[0,1]-normalized similarity-score weights\" and explicitly assumes edge weights w_ij >= 0. If the raw triple confidence scores are used directly, costs can be negative, which violates the non-negative-weight assumption of the exact GST dynamic program and invalidates the claim that the reported trees are minimum-cost trees. If the implementation instead normalizes triple scores before conversion, that normalization step is absent from the paper and must be documented. Either way, the central claim that QUEST's advantage comes from computing cost-optimal Group Steiner Trees is not fully supported by the written method description.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents QUEST, an unsupervised system for answering multi-document fact-centric complex questions. QUEST retrieves a small document pool for each question, extracts SPO triples with a custom proximity-scored Open IE extractor, and assembles a noisy 'quasi KG' of entity, relation, and type nodes with alignment and type edges. Cornerstones are nodes whose labels match question words; candidate answers are non-terminal nodes of top-k Group Steiner Trees connecting at least one cornerstone from each question-token group. Candidates are filtered by lexical answer type, aggregated by string and alignment redundancy, and ranked by a tree-cost-weighted sum over the trees in which they occur. Experiments on two complex-question collections (CQ-W and CQ-T) compare QUEST with DrQA, BFS, and ShortestPaths under top-10 and stratified corpora, and report consistent MRR gains with ablations, error analyses, and parameter sensitivity.","tokens_in":18091,"tokens_out":11400,"duration_ms":116044,"significance":"QUEST is a coherent and clearly explained unsupervised alternative to neural reading-comprehension QA for a genuinely difficult setting: combining evidence across documents for questions with multiple entities and relations. Its main strengths are the fully unsupervised pipeline, the explicit modeling of multi-document evidence via the quasi KG, the use of Group Steiner Trees for joint disambiguation, and the unusually complete empirical package: two benchmarks, multiple corpus-quality strata, ablations, ranking variants, error analysis, and public data/code/demo. If the comparative results are reproduced, the system is a useful contribution to text-based QA. The main reservations are that both benchmarks were created by the authors' group, the neural baseline set is narrow, and the written description of the edge-weight normalization contains a gap that affects the theoretical claim of optimality.","major_comments":[{"comment":"Section 3.1 defines the triple-edge confidence as a sum over sentences of 1/d(S_i) with no stated upper bound, and Section 3.2 says these confidence scores are used as triple-edge weights. Section 4.1 then assumes '[0,1]-normalized similarity-score weights' and defines edge cost as 1 - score, while explicitly requiring w_ij >= 0. If raw triple scores are used, costs can be negative, violating the non-negative-weight condition of the GST dynamic program and invalidating the claim that the reported trees are minimum-cost trees. If a normalization step exists in the implementation, it is not documented. Please add the exact normalization formula (or define triple-edge scores with a bounded range) and verify that the GST implementation never receives negative costs; without this, the central claim that QUEST's advantage comes from cost-optimal Group Steiner Trees is not fully supported by the paper as written.","section":"Sec. 3.1, Sec. 3.2, Sec. 4.1"},{"comment":"CQ-W is adopted from a prior paper by the authors' group and CQ-T is constructed by the authors for this paper; the only neural baseline is DrQA. The abstract and Section 6 say QUEST 'substantially outperforms state-of-the-art baselines,' but the evidence supports a narrower claim: superiority over DrQA and two graph heuristics on two self-created collections. Please either add an independent evaluation (for example, on a third-party complex multi-document QA benchmark) or rewrite the abstract and Section 6 to state the comparison precisely. This distinction is important because both the multi-document setting and the benchmark construction are controlled by the authors.","section":"Sec. 5.2 and Table 2"},{"comment":"Section 5.3 states that the three thresholds are set to 0.5 and that 'no tuning is involved,' yet Section 7 and Figure 2 report a sensitivity analysis over precisely these thresholds on CQ-W. If the default 0.5 was selected after inspecting results on CQ-W, the comparison in Table 2 is not tuning-free; if it was a priori, the 'no tuning' claim needs an explicit statement. Please specify how the default thresholds were chosen and, ideally, evaluate threshold choices on a development split distinct from the test questions, because the main comparative results depend on these free parameters.","section":"Sec. 5.3 and Sec. 7 (Fig. 2)"}],"minor_comments":[{"comment":"Please present the proximity-based scoring formula as an equation rather than prose; the current text should also state whether distances are normalized by sentence length and how repeated co-occurrences in the same document are aggregated.","section":"Sec. 3.1"},{"comment":"The text alternates between 'word or phrase' and 'per token of the question' for cornerstones; please clarify how multi-word question phrases map to terminal groups in the GST formulation.","section":"Sec. 4.1"},{"comment":"Please specify DrQA's exact configuration, including the retriever index type, the reader checkpoint used, the number of retrieved passages, and the maximum passage length, since DrQA's performance depends strongly on these choices.","section":"Sec. 5.4"},{"comment":"Please define the notation 'A' and explain the binning in the caption; the upper and lower halves of the table are hard to parse without a precise description of how edge contributions from distinct documents are counted.","section":"Table 3"},{"comment":"The error-scenario percentages sum to 101% for CQ-W and 99% for CQ-T; please specify whether the categories are mutually exclusive, clarify the denominator, and account for the rounding.","section":"Table 5"},{"comment":"Statistical significance is reported only for QUEST over DrQA; please state whether the comparisons against BFS and ShortestPaths were tested, and if so, report those p-values or note that no significance claim is made for them.","section":"Sec. 6 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark provenance is the main external-validity concern: both collections come from the authors' own group, so the comparative claim needs either independent data or careful scoping. The edge-weight normalization gap is a verifiable technical issue that should be checked against the released code during revision. The paper is otherwise solid for an IR venue, with a clear contribution and a rich experimental section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"QUEST is worth reading for its core idea: build a per-question noisy graph from Open IE triples over retrieved documents, then extract answer candidates as nodes in low-cost Group Steiner Trees that touch one node from each question-term group. That combination—on-the-fly quasi-KG plus top-k GST—is new as far as I know, and the paper explains it clearly. The ablations and sensitivity analyses are informative, and the authors release code and data. On its own benchmarks, QUEST clearly beats DrQA and two graph baselines, with consistent gains across stratified corpora.\n\nThe soft spots are real but not fatal. Both benchmarks come from the authors' group: CQ-W from an earlier same-group paper, CQ-T built by the authors' students. That creates a circularity risk, especially since parameters like k=50 and the 0.5 thresholds were chosen with these test sets in view. Only DrQA is used as a neural baseline, and significance is reported only against DrQA. There are no confidence intervals for the comparisons against BFS and ShortestPaths.\n\nThere is also a documentation gap in the graph construction. Section 3.1 defines triple-edge confidence as a sum of 1/d over sentences, which can exceed 1. Section 4.1 then says edge costs are 1 − score on '[0,1]-normalized similarity-score weights'. The normalization of triple scores is never described. If raw sums are used, some costs are negative, violating the non-negative weight assumption of the exact GST dynamic program; if the implementation normalizes, that step is missing. This doesn't invalidate the empirical comparison, but it does weaken the 'cost-optimal' explanation.\n\nThe paper is for researchers in open-domain QA and graph-based reasoning over text. It deserves peer review, but not in current form. A revision should document the cost normalization, add an external benchmark or at least a second neural baseline from a different group, and present error bars. With those changes, the contribution would be solid.","headline":"Neat unsupervised QA method with a real benchmark-independence problem and a fixable weight-normalization gap; worth peer review if the authors document the cost conversion.","tokens_in":18613,"tokens_out":4500,"would_cite":true,"duration_ms":39869,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-14T15:54:36.649712+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}