{"id":"c99b31a4-66b7-44ec-ae36-581c1bfd4539","arxiv_id":"1908.03645","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A generate-validate QA method that scores template-generated English hypotheses with BERT/ESIM textual entailment reaches 76.63% on QUAREL, beating QUASP+ by 7.93 points.","lead":"This paper describes a question-answering system that, instead of parsing stories into formal logic, generates candidate English statements and uses textual entailment models to pick the best answer. On the QUAREL benchmark it reports 76.63% accuracy, about eight points above the previous state of the art.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Step 3's givenscore comparison is the dominant failure mode; the paper's own error analysis confirms the assumption, so the 7.93% margin lacks robustness evidence.","rationale":"I identified the same weakest link as the reader: the givenscore-based answer selection in Section 3, Step 3. The paper's own error analysis (Section 6) confirms this as the dominant source of the 129 test failures, with examples where claimA*/claimB* are correct but givenscore chooses wrongly. The training procedure in Section 4.2 attempts to teach qualitative-relation transfer via QRKB-derived pairs, but this is exactly the unvalidated assumption: plausible, but not demonstrated to generalize beyond templated train stories. The 76.63% result is reported without code, data, error bars, or significance tests, so the margin over QUASP+ is credible but not robust. A leave-one-relation-out retraining of f_given would directly test whether the transfer is real. This does not change the reader's CONDITIONAL assessment; the concern is significant and addressable, but not grounds for rejection.","tokens_in":10834,"tokens_out":7477,"duration_ms":75909,"concrete_test":"Retrain f_given on the QUAREL-generated data with all QRKB-derived positive/negative pairs for one relation (e.g., q+(friction,heat)) removed, keeping direct given-fact pairs and SNLI. Evaluate gvQPS on test questions whose gold reasoning uses that relation. If accuracy on those questions drops materially relative to the full model, the Step 3 assumption is not supported as general QRKB transfer; if it holds, the assumption is credible. Also report a breakdown of the 129 test failures by givenscore versus claim-selection errors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the Step 3 answer rule: choose option A iff givenscore(claimA*) > givenscore(claimB*), where the paper explicitly assumes 'the givenscore will learn to capture the qualitative relationship' (Section 3, Step 3). Section 4.2 attempts to teach this by adding QRKB-derived positive/negative training pairs, but this only demonstrates transfer on the templated QUAREL training distribution. The paper's own error analysis states that the majority of the 129 test failures are due to givenscore errors, and both given examples show claimA*/claimB* identified correctly while givenscore picks the wrong world (e.g., 'lynn's hair has more strength' scores 0.01 vs 'nell has more strength' 0.00003). Thus the assumption is not a peripheral detail; it is the main observed failure mode. Because no code, generated data, error bars, or significance tests are provided, the robustness of the 7.93% improvement over QUASP+ is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes gvQPS, a generate-validate solver for the QUAREL dataset of qualitative relationship questions. Instead of training a semantic parser to produce logical forms, the method generates natural-language hypotheses from 46 hand-authored templates instantiated over noun phrases in the story and options, scores each hypothesis with two textual entailment functions (a given-fact scorer and a claim scorer), and selects the answer by comparing given-fact scores of the best-scoring claim hypotheses. The entailment functions are trained on automatically generated premise-hypothesis pairs derived from QUAREL's logical-form annotations and the QRKB, optionally augmented with SNLI, using ESIM or BERT. The best configuration, BERT for the given-fact scorer and ESIM for the claim scorer, reaches 76.63% test accuracy, which the authors report as a 7.93-point improvement over the previous state of the art QUASP+ (68.7%). The paper also provides a small error analysis attributing most of the 129 test failures to the given-fact scorer.","tokens_in":11066,"tokens_out":4120,"duration_ms":45469,"significance":"If the empirical claims hold, the paper makes a useful contribution: it shows that a generate-validate pipeline can outperform direct semantic parsing on QUAREL and that NLI-style training with SNLI and BERT can transfer to this task. The proposal to replace target-language learning with natural-language hypothesis generation and entailment scoring is interesting and arguably more general than the specific QUAREL application. The paper is also honest in stating its key assumption and in presenting negative evidence in the error analysis. However, the significance of the headline result is currently limited by the absence of variance estimates, significance tests, and a direct evaluation of the assumed transfer mechanism. The paper ships no code, data, or training details, which makes the empirical claim difficult to verify.","major_comments":[{"comment":"The headline accuracy of 76.63% is reported for a single run of the best of 16 model combinations selected on the development set, with no variance, confidence intervals, or significance tests. Because the configuration was chosen by taking the maximum dev accuracy over 16 combinations, the reported 7.93-point improvement over QUASP+ (68.7%) could be inflated by selection noise. Please report mean and standard deviation over multiple seeds, provide a significance test against the QUASP+ baseline, and either fix the model-selection rule before evaluation or use cross-validation to reduce selection bias.","section":"Section 6, Table 5"},{"comment":"The answer-selection rule is load-bearing: the final decision compares givenscore(claimA*) and givenscore(claimB*), and the paper explicitly states, \"we assume that the givenscore will learn to capture the qualitative relationship.\" The paper's own error analysis says that the majority of the 129 test failures are due to givenscore errors, and both given examples show correct identification of claimA* and claimB* but an incorrect givenscore ordering. Please quantify the error breakdown (for example, how many failures are caused by the givenscore comparison after correct claim identification), report failures by property and direction, and discuss whether an alternative decision rule (such as using claim scores alone or a learned combination) would be more robust.","section":"Section 3, Step 3"},{"comment":"The QRKB-derived positive and negative pairs for f_given are intended to teach the model to transfer qualitative relations (for instance, from friction to heat), but the paper provides no direct analysis of whether this transfer actually occurs on test stories. The end-to-end accuracy does not separate template-scoring quality from relation transfer. Please add an ablation that removes the QRKB-generated pairs, report per-property or per-relation accuracies, and ideally test generalization to qualitative relations held out during training. Without such evidence, the mechanism asserted in Step 3 is not directly supported.","section":"Section 4.2"},{"comment":"The empirical claims cannot be independently verified because the paper does not release code, model checkpoints, hyperparameters, random seeds, or the automatically generated premise-hypothesis datasets (only their sizes are given). Since the method depends on template generation and automatically labeled entailment pairs, even a careful reader cannot reproduce Table 5. Please make the code and generated data available, or provide a sufficiently detailed training and evaluation protocol, including the exact oversampling procedure and any hyperparameter tuning.","section":"General (reproducibility)"}],"minor_comments":[{"comment":"The caption reads \"Example problems form the QUAREL corpus\"; \"form\" should be \"from\".","section":"Table 1 caption"},{"comment":"In the example list, \"trail has more friction\" should presumably be \"trial has more friction\", matching the noun phrase \"trial\" listed earlier.","section":"Section 3, Step 1"},{"comment":"The sentence \"For the problem II in table 1, world1 literal = “carpet” and world1 literal = “skin”\" repeats \"world1 literal\"; the second occurrence should be \"world2 literal\".","section":"Section 4.2"},{"comment":"The notation is inconsistent: the text lists \"f^{BERT}_{TrainQUAREL_Fact}\" and \"f^{ESIM}_{TrainESIM_Fact}\" where the subscript should be \"TrainQUAREL_Given\" (or G1/G2 as in Table 5). Please standardize the subscripts to match the table.","section":"Section 6, notation"},{"comment":"In Error Example I, the hypothesis \"lynn’s hair has more strength\" uses a lowercase initial while the story uses \"Lynn’s hair\"; this is likely a template-instantiation artifact, but it should be noted or fixed for clarity.","section":"Error Analysis examples"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern about Step 3 is confirmed by the paper's own error analysis: the given-fact scorer is the dominant failure mode, and the current paper does not quantify or mitigate this. I do not see circularity or label leakage; the training data come from the train split and the selection rule is fixed. The main barrier to acceptance is evidential robustness rather than novelty or correctness of the core idea. If the authors provide multi-seed results, significance tests, an ablation of the QRKB-derived pairs, and a detailed error breakdown, the paper could be acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—bottom line: this paper makes a single strong empirical claim—generate-validate with hand-authored templates and two trained entailment scorers beats the best semantic-parser baseline on QUAREL by 7.93 points—and the claim is plausible but not nailed down. The novelty is real: unlike the prior generate-validate work (Mitra et al., AAAI 2019), this version uses the framework for both question understanding and given-fact verification, and shows SNLI/BERT pretraining transfers for the givenscore. The architecture is clearly explained, the template set is explicit, and the authors report a full 4x4 grid of model/dataset combinations on dev and test, which is more transparent than many papers.\n\nThe soft spots are in verification, not in the idea. The headline 76.63% comes from a single unseeded run of the best of 16 configurations selected on dev, with no variance or significance testing. No code, no generated data, no trained model release. And the step-3 answer rule rests on an explicitly stated assumption—\"we assume that the givenscore will learn to capture the qualitative relationship\"—which the paper's own error analysis shows is the dominant failure mode: in the majority of 129 failures, claimA* and claimB* are identified correctly but givenscore picks the wrong one. That makes the 7.93% margin fragile: the method could be right for the wrong reason on a substantial fraction of test items. The stress-test note is on target here; this is not a peripheral detail.\n\nI would not call the paper unsound. The train/dev/test split is clean, the training labels come only from the train split, and the QRKB-derived pairs are a legitimate use of the dataset's annotation machinery. The concern is robustness evidence, not circularity or fabrication. A serious referee should ask for multiple seeds, confidence intervals, a significance test against QUASP+, and at least the generated hypothesis sets or a small release.\n\nWho's this for? Anyone working on qualitative reasoning, or on alternatives to semantic parsing in closed-domain QA. It deserves peer review, not desk rejection; an editor should send it out, and a referee should push on reproducibility rather than reject the premise.","headline":"Generate-validate beats semantic parsing on QUAREL by 8 points, but the margin is not yet robust and the key assumption is the main failure mode.","tokens_in":11577,"tokens_out":1902,"would_cite":true,"duration_ms":19592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing semantic parsing with generate-validate scoring lifts accuracy on qualitative QA from 68.7% to 76.63%.","keywords":["qualitative relationships","generate-validate","semantic parsing","textual entailment","natural language inference","QUAREL","transfer learning","BERT"],"falsifier":"Train the given-fact scorer on a version of the QUAREL-generated dataset from which all pairs involving a particular QRKB relation (for example, q+(friction, heat)) have been removed, then test on held-out problems whose answers require that relation; if accuracy on those problems does not drop markedly relative to the full-data model, the claim that givenscore learns qualitative relationships is falsified, whereas a sharp drop would confirm that the training pairs are what carry the transfer.","tokens_in":10616,"feed_emoji":"📈","tokens_out":10808,"duration_ms":88584,"temperature":0.7,"pith_summary":"This paper claims that a long-standing bottleneck in question answering about qualitative relationships—converting text into a logical form via semantic parsing—can be bypassed. Instead of parsing, the authors generate natural-language descriptions of every plausible logical fact using 46 templates, then score each description by whether the story text entails it, using two trained textual-entailment functions. On the QUAREL benchmark, this generate-validate approach reaches 76.63% test accuracy, beating the previous best semantic-parser system (QUASP+) by 7.93 percentage points and the unmodified parser (QUASP) by 20.53 points. The stated reason is that generate-validate opens a channel for transfer learning from resources like SNLI and pretrained models such as BERT, which direct semantic parsing cannot exploit as easily.","feed_headline":"Generate-validate tops semantic parsers by 8 points on qualitative QA","feed_subtitle":"The 76.63% accuracy beats the prior best QUASP+ by 7.93 points on qualitative-relationship questions.","key_machinery":"The mechanism has three parts. A generator builds a set of $46 \\times n$ natural-language hypotheses from 46 manually written templates, one per (qualitative property, direction) pair—for example, (Friction, high) yields 'X has more friction'—instantiating the variable X with every noun phrase in the problem. Two textual entailment functions carry the validation: givenscore ($f^{given}_{TE}$) scores each hypothesis against the full story text, while claimAscore and claimBscore ($f^{claim}_{TE}$) score hypotheses against the question concatenated with each answer option. Both scorers are trained on automatically generated premise-hypothesis pairs built from QUAREL’s logical-form annotations and the QRKB qualitative-relations knowledge base, with SNLI and BERT contributing transfer learning for the given scorer. The final answer rule selects the highest-scoring hypothesis for each option and chooses the option whose chosen hypothesis has the higher givenscore.","core_discovery":"The central discovery is that, for QUAREL’s multiple-choice qualitative-reasoning stories, a generate-validate solver outperforms state-of-the-art semantic parsers, and the gain comes specifically from replacing direct logical-form generation with entailment-based scoring plus transfer learning. The solver, gvQPS, enumerates hypotheses such as 'Carpet has more friction' by instantiating 46 templates over noun phrases from the story; one entailment function scores whether each hypothesis is a given fact from the story text, and another scores whether it follows from the question together with each answer option. The answer is chosen by comparing the given-scores of the two highest-scoring claim hypotheses. The best configuration—BERT trained on the given-fact data augmented with SNLI, and ESIM trained on the claim data—yields 76.63% accuracy, versus 68.7% for QUASP+ and 56.1% for QUASP, with no delexicalization preprocessing.","pith_inferences":["If the generate-validate decomposition is the real source of the gain, then the same two-scorer architecture should transfer to other multiple-choice story benchmarks with implicit world-to-property relations, provided a template set covers their predicates; this is a testable hypothesis the paper does not run.","Because most errors trace to givenscore, a relation-aware given scorer that explicitly chains QRKB links (for example, inferring heat from friction) could close the remaining gap beyond what raw textual entailment achieves.","The 46 templates are hand-authored for the 19 qualitative properties in QUAREL; learning or paraphrasing templates automatically would show whether the method scales to open-domain qualitative questions without re-engineering per domain."],"forward_implications":["Any future improvement in natural-language-inference models should transfer directly to qualitative-reasoning QA, because the solver reduces the task to two entailment-scoring calls.","The generate-validate recipe can be applied to other semantic-parsing applications where raw text is available, without retraining a parser for each new output logical form, as the paper argues in its conclusion.","The given-fact scorer benefits from adding SNLI to its training data, while the claim scorer does not; the two entailment functions therefore require differently distributed training sets, a finding that should guide dataset construction for such solvers.","Manual delexicalization, which was needed for QUASP+’s best result, is not required in this approach; the entailment scorers absorb the world-reference distinctions from the generated hypotheses."],"supporting_citations":[{"why":"Supplies the QUAREL dataset, its logical-form annotations, the QRKB knowledge base, and the QUASP/QUASP+ baselines that the method must beat.","marker":"Tafjord et al., 2018"},{"why":"Introduces the generate-validate framework that this paper applies to both given-fact and question understanding.","marker":"Mitra et al., 2019"},{"why":"Provides SNLI, the natural-language-inference corpus used to augment the given-fact training data, which the best configuration relies on.","marker":"Bowman et al., 2015"},{"why":"Provides BERT, the pretrained model used in the given-fact scorer of the best configuration.","marker":"Devlin et al., 2018"},{"why":"Provides ESIM, the entailment model used for the claim scorer in the best configuration.","marker":"Chen et al., 2016"},{"why":"Provides the type-constrained neural semantic parser used inside the QUASP baseline, the direct-parsing approach the paper argues against.","marker":"Krishnamurthy et al., 2017"}],"fun_headline_variants":["Generate-validate beats semantic parsers by 8 points on qualitative QA","No semantic parser needed: generate-validate wins qualitative QA","Generate-validate outscores semantic parsers by 7.93% on QUAREL","Qualitative QA: generate-validate beats semantic parsers by 8","Entailment-based validation beats semantic parsing on qualitative QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The answer-selection rule assumes that the given-fact entailment scorer will learn the qualitative relationships themselves—so that a hypothesis like 'skin has less friction' is scored close to 'less heat is generated on skin'—because the final comparison uses only the given-scores of the two chosen claim hypotheses; if the automatically generated training pairs do not teach this transfer of QRKB relations to new stories, the selection step fails.","fun_headline_variants_meta":{"raw":{"variants":["Generate-validate beats semantic parsers by 8 points on qualitative QA","No semantic parser needed: generate-validate wins qualitative QA","Generate-validate outscores semantic parsers by 7.93% on QUAREL","Qualitative QA: generate-validate beats semantic parsers by 8","Entailment-based validation beats semantic parsing on qualitative QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3369,"prompt_tokens":942,"completion_tokens":2427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":2333}},"tokens_in":558,"tokens_out":2427,"duration_ms":16877,"temperature":1.0,"reasoning_tokens":2333,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:06:51.188497+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the given-fact scorer on a version of the QUAREL-generated dataset from which all pairs involving a particular QRKB relation (for example, q+(friction, heat)) have been removed, then test on held-out problems whose answers require that relation; if accuracy on those problems does not drop markedly relative to the full-data model, the claim that givenscore learns qualitative relationships is falsified, whereas a sharp drop would confirm that the training pairs are what carry the transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the generate-validate framework that this paper applies to both given-fact and question understanding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the type-constrained neural semantic parser used inside the QUASP baseline, the direct-parsing approach the paper argues against."}],"review_version":1}