{"id":"92a46b79-60ad-409b-aa49-35a9a82064b5","arxiv_id":"1908.04926","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"The thesis presents a synthesis of previously published work on abductive reasoning for QA, essential-term detection, multi-sentence and temporal commonsense datasets, and a formal framework for reasoning limitations.","lead":"This dissertation compiles the author's prior research on reasoning-driven question answering, including ILP-based systems, essential-term detection, and two challenge datasets. A generalist should read it to see how explicit linguistic structure and constrained optimization can be combined to improve QA systems.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Chapter 8's formal framework and impossibility results are absent from the provided text, so the strongest theoretical claim is unverifiable here.","rationale":"Read in good faith, the dissertation's empirical parts are coherent and the published results are consistent with the thesis statement. The difficulty is allocation of weight: the abstract's strongest and most novel claim is the formal framework plus impossibility theorems, yet that part is absent from the supplied text. I concur with the reader's UNVERDICTED verdict, but not exactly with the stated weakest assumption. Static benchmark bias (Section 2.3.1) is acknowledged by the author and affects all empirical NLP claims; it does not bear specifically on the theoretical claim. The missing Chapter 8 is more load-bearing because it is the only unverifiable, non-peer-reviewed novelty. I would keep the verdict UNCHANGED: still UNVERDICTED, not because benchmarks might be unrepresentative, but because the central proof cannot be checked. If the complete chapter and appendix are obtained and the proof re-derives cleanly, the verdict could move to ACCEPT with respect to the theoretical claim. This is a review-state limitation, not an allegation of error.","tokens_in":50583,"tokens_out":4033,"duration_ms":39214,"concrete_test":"Retrieve the complete Chapter 8 (e.g., from the UPenn dissertation or companion arXiv:1901.02522) and independently re-derive the main limitation theorem stated in Section 8.6 using the construction in Definition 9. Verify that the lower bound provably holds over the full parameter range charted in Figure 29 (ε+ = 0.7, λ = 3, and p− varying from 0 toward 1), and that no proof step silently assumes a clean meaning-symbol interface (e.g., ε+ = 0 or p− = 0). If the proof requires additional assumptions, the claim of 'fundamental limitations' must be qualified; if the proof checks out cleanly, the concern dissolves.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's third contribution—a 'first formal framework for multi-step reasoning algorithms' and proofs of 'fundamental limitations'—is the most distinctive claim in the thesis, and it cannot be inspected in the provided text. Chapter 8 appears only in the table of contents; the supplied body stops before it, and Appendix A.2 ('Supplementary Details for Chapter 8') is listed but not included. Section 3.5 explicitly defers the brittleness of multi-step reasoning to Chapter 8, and Section 2.5.4 says Chapter 8 'proposes a formalism to study limits of chaining long-range information.' The entire theoretical layer on which the strongest claim rests is therefore missing. This is load-bearing because Part III is the only part not already published as peer-reviewed empirical work; the reader is asked to accept an impossibility result without seeing the meaning-symbol interface, the noise model (ε+, p−), the cut construction of Definition 9, or the proof in Section 8.6. The static-benchmark concern in Section 2.3.1 is real but secondary: even if benchmarks are biased, the theory is meant to provide independent grounding, and that grounding is absent. Treating the empirical chapters as sound does not close the gap; it leaves the headline claim unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This dissertation-style manuscript investigates natural language understanding (NLU) through the question answering (QA) task and is organized into three parts. Part I develops reasoning-driven QA systems: TableILP, which casts QA as an integer linear program over semi-structured tabular knowledge and is evaluated on elementary science exams; SemanticILP, which generalizes this formulation to raw text by reasoning over semantic abstractions from off-the-shelf NLP tools; and a supervised essential-term classifier that identifies critical question words and is shown to improve an IR-based solver. Part II introduces two challenge datasets: MultiRC, requiring multi-sentence reading comprehension, and TacoQA, requiring temporal commonsense reasoning. Part III announces a formal framework for multi-step reasoning algorithms and claims to prove fundamental limitations of such algorithms under properties of language use such as incompleteness and ambiguity. The abstract and introduction present this theoretical contribution as a central claim, but the corresponding Chapter 8 is absent from the provided text; only its table-of-contents entry and references to it in earlier chapters are visible. The empirical chapters present clear experimental designs with baselines, ablations, and significance tests, and several datasets and code bases are released publicly.","tokens_in":50900,"tokens_out":6864,"duration_ms":64057,"significance":"If the results hold, the thesis offers a coherent and useful body of work: an ILP-based reasoning framework that outperforms structured baselines on science QA with limited training data; a crowd-sourced dataset and classifier for essential question terms; two QA datasets that push beyond single-sentence and static-benchmark settings; and, potentially, a formal analysis of the limits of reasoning algorithms in natural language. The empirical chapters are carefully presented: they include baselines, ablations, statistical significance tests, hidden test sets, and public releases of code and datasets, which are strengths that should be acknowledged. The most distinctive claim, however, is the theoretical framework of Chapter 8, which cannot be inspected in the current manuscript because the chapter body and Appendix A.2 are missing. Since Part III is the only part of the thesis that is not already published as peer-reviewed empirical work, the headline theoretical claim is unverified as submitted, and the overall significance of the thesis cannot be fully assessed without it.","major_comments":[{"comment":"The central theoretical claim of the dissertation—presenting 'the first formal framework for multi-step reasoning algorithms' and proving 'fundamental limitations' for reasoning algorithms—is stated in the abstract and introduction, and Chapters 2 and 3 explicitly defer to Chapter 8 (e.g., Section 3.5 says the brittleness of multi-step reasoning is studied in Chapter 8). However, the submitted text contains only the table-of-contents entry for Chapter 8; the chapter body and its supplementary appendix (A.2) are not included. I therefore cannot inspect the meaning-symbol interface, the noise model (epsilon+, p-), Definition 9, or the proof in Section 8.6. This is a load-bearing omission because Part III is the only part of the thesis that is not already published as peer-reviewed empirical work. The authors must supply the complete text of Chapter 8 and Appendix A.2 (or the full content of the corresponding publication) so that the theoretical claims can be evaluated.","section":"Chapter 8 / Appendix A.2"},{"comment":"The text states that the ET classifier 'has a 5% higher AUC (area under the curve)' relative to baselines, but Table 18 reports AUC of 0.79 for the ET Classifier, equal to PropSurf and lower than PropLem's 0.80. This numerical inconsistency contradicts the table as printed. The sentence must be corrected and the AUC claim reworded to match the reported data; the F1 and MAP advantages remain supported by the tables, but the AUC statement is not.","section":"Section 5.3.1 / Table 18"},{"comment":"The demonstration that essentiality information improves the TableILP solver is based on only 12 curated questions (the QR set). The reported '41.7% error reduction' corresponds to correcting 5 of 12 errors made by vanilla TableILP. This sample is too small to support the general conclusion that the ET cascade helps TableILP cope with distracting terms. The section should either be expanded with a larger evaluation or explicitly framed as a small case study with limited statistical power, so that readers are not misled about the strength of the evidence.","section":"Section 5.4.2"}],"minor_comments":[{"comment":"Several internal cross-references are inconsistent with the actual chapter numbering. The thesis outline says Chapter 5 presents MultiRC and Chapter 6 presents TacoQA, but the actual chapters are 6 and 7 respectively; Section 2.4.5 refers to 'Chapter 6' for temporal commonsense reasoning, which is actually Chapter 7. These should be corrected throughout.","section":"Section 1.4 and cross-references"},{"comment":"The sentence 'In Chapter 2, 3 we use elementary-school science tests' appears to contain a typo; it should presumably read 'In Chapters 3 and 4, we use elementary-school science tests.'","section":"Section 2.3.1"},{"comment":"There are grammatical errors in the final paragraph: 'a major portion of our understanding come is only implied from text' and 'lack explicit explicit attention' contain typos and should be proofread.","section":"Section 4.5"},{"comment":"The phrase 'Binomial 10 exact test' appears to be a typo; it should read 'binomial exact test.'","section":"Section 5.3.1"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the absence of Chapter 8 and Appendix A.2 from the submitted manuscript. If this is an artifact of the arXiv upload rather than the actual dissertation, please ask the authors to provide the full chapter and appendix in the revision. Without them, the thesis's headline theoretical contribution cannot be vetted. The empirical portions are otherwise solid and, if the missing material is supplied, the thesis may be suitable for publication as a dissertation summary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a PhD thesis that collects five of the author's earlier papers plus one EMNLP paper. The empirical content is real and generally well done: the ILP solvers on science questions, the essential-terms classifier, and two datasets (MultiRC, TacoQA) that have been used by others. If you work on QA, you will recognize the contributions.\n\nThe second thing: the abstract promises 'the first formal framework for multi-step reasoning algorithms' with proofs of 'fundamental limitations.' That chapter (Chapter 8) is the only part of the thesis not already published in the listed papers, and it is absent from the text I was given. The table of contents lists it and Appendix A.2 has supplementary details, but the body stops before it. So the headline claim—the one that would make this more than a compilation—is unverifiable from this document. The paper it is based on (arXiv:1901.02522) exists, so the theory may be fine, but a referee cannot evaluate it from this submission.\n\nThe empirical chapters are sound. There are ablations, baselines, significance tests where appropriate, and the datasets are public. The 12-question curated set in Section 5.4.2 is small, but it is explicitly a pilot and the error reduction is plausible. The static-benchmark concern in Section 2.3.1 is real—the author himself notes datasets are biased toward simplicity—but that is a field-wide issue, not a specific flaw in these experiments. The ILP hyperparameters are numerous but they were tuned on dev sets and the systems win on held-out exams.\n\nCitation pattern: self-citation is inherent in a thesis compilation. The prior work is peer-reviewed, so no red flag there.\n\nBottom line: as a snapshot of a research program it is a useful document, but as a new research contribution it hinges on the missing Chapter 8. I would only send this to peer review with the full theory included. For a reading group, the datasets and the synthesis might be worth discussion, but the novelty is thin.","headline":"A solid compilation of previously published empirical work whose only new piece—the formal theory chapter—is missing from the supplied text, leaving the strongest claim unverified.","tokens_in":51389,"tokens_out":3170,"would_cite":false,"duration_ms":32814,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"QA systems that reason over semantic abstractions beat retrieval and neural baselines on science and biology exams, and a formal model shows why multi-step reasoning has limits.","keywords":["natural language understanding","question answering","abductive reasoning","support graph optimization","integer linear programming","semantic abstractions","temporal common sense","reasoning limitations"],"falsifier":"One concrete check is to assemble a benchmark whose instances are independently verified to require multi-sentence chaining and temporal common sense, and then show that a model with no explicit reasoning component, trained on enough data, matches or exceeds human accuracy; that outcome would contradict the thesis's claim that reasoning over abstractions and world knowledge is needed for QA progress.","tokens_in":50379,"feed_emoji":"🧠","tokens_out":14687,"duration_ms":133011,"temperature":0.7,"pith_summary":"This thesis tries to establish that the bottleneck in natural language understanding (NLU) is not data volume but the ability to reason over abstract representations of meaning. It defends the claim with three moves: constrained-optimization QA systems that chain facts across tables and semantic graphs outperform retrieval and neural baselines on science and biology questions; two new datasets force multi-sentence and temporal-common-sense reasoning and expose large gaps to human performance; and a formal model of the meaning-symbol interface proves that multi-step reasoning algorithms face fundamental limits when symbols are noisy, incomplete, and ambiguous. If the thesis is right, progress in NLU comes from building reasoning and grounding into QA systems rather than from scaling corpora or models alone.","feed_headline":"Explicit reasoning lifts QA systems and exposes their limits","feed_subtitle":"A thesis ties ILP-based solvers, multi-sentence QA benchmarks, and a formal meaning-symbol model into one case for reasoning.","key_machinery":"The load-bearing object is the support graph: a subgraph of an augmented graph whose nodes are question constituents, answer options, and knowledge units (table cells or semantic-graph nodes), with edges weighted by entailment or similarity. An ILP formulation selects the support graph that maximizes weighted alignments while enforcing connectivity, evidence-chaining, and semantic-relation constraints; the same template powers the two system chapters, over tables and over semantic graphs. For the limitations result, the carrying object is the meaning-symbol interface: a two-layer model in which a clean, unique meaning space is observed only through a noisy, incomplete, variable symbol space. The proof of limits uses a cut-based construction that separates meaning pairs that are connected from those that are disconnected, showing that any algorithm relying on local symbol-graph distances must fail in the noisy regime.","core_discovery":"On the author's own terms, the central claim is that question answering should be treated as abductive reasoning: the system must find the best support graph connecting a question to exactly one answer through available knowledge, where 'best' is defined by structural constraints and soft preferences over alignments. Casting this search as an integer linear program lets the same machinery operate over curated tables, relation-extraction tuples, and multi-view semantic graphs, and it beats the retrieval and neural baselines on unseen exam questions. The theoretical companion claim is that reasoning over natural language happens in a noisy symbol space that only approximates a clean meaning space; the thesis constructs a connectivity-reasoning model and proves both when accurate recovery of meaning-space connectivity is possible and when it is provably impossible. This is offered as the first formal framework for multi-step reasoning algorithms under incompleteness, ambiguity, and variability.","pith_inferences":["Editorial inference: the formal limitation suggests a lower-bound-style claim—within the model, more training data alone cannot remove the ambiguity injected by the symbol space, so progress will require grounding or additional structured world knowledge.","Editorial inference: the support-graph/ILP formulation can be hybridized with modern neural models by using neural similarity scores as edge weights and keeping the ILP as a trainable inference layer; the thesis provides a clean interface for that combination.","Editorial inference: the essential-terms study points to a cheaper supervision signal—annotating which terms matter instead of full answers—that could transfer to other NLU tasks and could be used to audit neural attention mechanisms.","Editorial inference: the thesis's own warning that static benchmarks are biased toward simplicity implies a testable research program: build evaluation sets by construction, verifying that each instance requires multi-step chaining, and use those sets to measure progress rather than relying on sampled corpora."],"forward_implications":["A QA system that explicitly chains evidence through structured abstractions can outperform broad-coverage retrieval and a specialized neural reader on small-data reasoning domains, with 2–6 percent absolute gains on science exams and near-parity with a domain-specific biology system.","Because the same optimization template is applied to tables, tuples, and semantic graphs, new semantic abstractions can be added to the framework without changing the reasoning machinery.","Forcing solvers to use question terms that are predicted to be essential makes them more robust to distractors; the thesis reports up to 5 percent absolute gains for a retrieval solver and a 41.7 percent error reduction on a curated hard set.","The proposed multi-sentence and temporal-common-sense benchmarks imply that existing datasets understate the difficulty of NLU, and systems trained on current benchmarks should show a large gap to human performance on these new instances.","The formal limitation result implies that brittleness in multi-step reasoning over language is not only an engineering problem: within the model's assumptions, no symbol-space algorithm can always recover meaning-space connectivity."],"supporting_citations":[{"why":"Supplies the science-exam question sets and the IR/PMI/ensemble baselines against which TableILP and SemanticILP are measured.","marker":"Clark et al. (2016)"},{"why":"Provides the probabilistic-logic structured reasoner that TableILP outperforms by 14 absolute points on unseen science questions.","marker":"Khot et al. (2015)"},{"why":"Defines the tuple-based ILP reasoning system that SemanticILP generalizes and compares against.","marker":"Khot et al. (2017)"},{"why":"Provides the neural reading model whose failures on paraphrased knowledge motivate abstraction-based reasoning.","marker":"Seo et al. (2016)"},{"why":"Provides the ProcessBank biology paragraphs and domain-specific baselines that test cross-domain generalization.","marker":"Berant et al. (2014)"},{"why":"Provides the large extractive QA dataset used to train the neural baseline, illustrating why big-data approaches transfer poorly.","marker":"Rajpurkar et al. (2016)"},{"why":"Documents adversarial perturbations that break high-scoring QA models, motivating the robustness experiments.","marker":"Jia and Liang (2017)"},{"why":"Provides MCTest, an earlier multi-sentence reading comprehension dataset and a source of fiction paragraphs for MultiRC.","marker":"Richardson et al. (2013)"},{"why":"Provides the RACE dataset and the estimate that only about a quarter of its questions require multi-sentence reasoning, the gap MultiRC targets.","marker":"Lai et al. (2017)"}],"fun_headline_variants":["QA as abductive reasoning: best support wins","Thesis: QA needs abductive reasoning, not just retrieval","Reasoning-driven QA: from ILP to provable limits","Abductive reasoning for QA: better answers, known limits","Why natural language QA hits a reasoning wall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that accuracy on static question-answering benchmarks measures real progress toward natural language understanding, even though the thesis itself concedes that such benchmarks are skewed toward simplicity and give a biased estimate of the space of questions.","fun_headline_variants_meta":{"raw":{"variants":["QA as abductive reasoning: best support wins","Thesis: QA needs abductive reasoning, not just retrieval","Reasoning-driven QA: from ILP to provable limits","Abductive reasoning for QA: better answers, known limits","Why natural language QA hits a reasoning wall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2609,"prompt_tokens":991,"completion_tokens":1618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":1540}},"tokens_in":607,"tokens_out":1618,"duration_ms":11509,"temperature":1.0,"reasoning_tokens":1540,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:28:09.572895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete check is to assemble a benchmark whose instances are independently verified to require multi-sentence chaining and temporal common sense, and then show that a model with no explicit reasoning component, trained on enough data, matches or exceeds human accuracy; that outcome would contradict the thesis's claim that reasoning over abstractions and world knowledge is needed for QA progress.","supporting_citations":[],"review_version":1}