A Generate-Validate Approach to Answering Questions about Qualitative Relationships

Arindam Mitra; Aurgho Bhattacharjee; Chitta Baral; Ishan Shrivastava

REVIEW 4 major objections 5 minor 19 references

A Generate-Validate Approach to Answering Questions about Qualitative Relationships

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing semantic parsing with generate-validate scoring lifts accuracy on qualitative QA from 68.7% to 76.63%.

desk verdict Generate-validate beats semantic parsing on QUAREL by 8 points, but the margin is not yet robust and the key assumption is the main failure mode. read the letter →

arxiv 1908.03645 v1 pith:IY5KBFVS submitted 2019-08-09 cs.CL cs.AI

Arindam Mitra , Chitta Baral , Aurgho Bhattacharjee , Ishan Shrivastava This is my paper · ORCID

classification cs.CLcs.AI

keywords qualitativerelationshipsgenerate-validatesemanticparsingtextualentailmentnaturallanguageinferenceQUARELtransferlearningBERT

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a long-standing bottleneck in question answering about qualitative relationships—converting text into a logical form via semantic parsing—can be bypassed. Instead of parsing, the authors generate natural-language descriptions of every plausible logical fact using 46 templates, then score each description by whether the story text entails it, using two trained textual-entailment functions. On the QUAREL benchmark, this generate-validate approach reaches 76.63% test accuracy, beating the previous best semantic-parser system (QUASP+) by 7.93 percentage points and the unmodified parser (QUASP) by 20.53 points. The stated reason is that generate-validate opens a channel for transfer learning from resources like SNLI and pretrained models such as BERT, which direct semantic parsing cannot exploit as easily.

What carries the argument

The mechanism has three parts. A generator builds a set of $46 \times n$ natural-language hypotheses from 46 manually written templates, one per (qualitative property, direction) pair—for example, (Friction, high) yields 'X has more friction'—instantiating the variable X with every noun phrase in the problem. Two textual entailment functions carry the validation: givenscore ($f^{given}_{TE}$) scores each hypothesis against the full story text, while claimAscore and claimBscore ($f^{claim}_{TE}$) score hypotheses against the question concatenated with each answer option. Both scorers are trained on automatically generated premise-hypothesis pairs built from QUAREL’s logical-form annotations and the QRKB qualitative-relations knowledge base, with SNLI and BERT contributing transfer learning for the given scorer. The final answer rule selects the highest-scoring hypothesis for each option and chooses the option whose chosen hypothesis has the higher givenscore.

What would settle it

Train the given-fact scorer on a version of the QUAREL-generated dataset from which all pairs involving a particular QRKB relation (for example, q+(friction, heat)) have been removed, then test on held-out problems whose answers require that relation; if accuracy on those problems does not drop markedly relative to the full-data model, the claim that givenscore learns qualitative relationships is falsified, whereas a sharp drop would confirm that the training pairs are what carry the transfer.

Watch

Extended reading notes

Core claim

The central discovery is that, for QUAREL’s multiple-choice qualitative-reasoning stories, a generate-validate solver outperforms state-of-the-art semantic parsers, and the gain comes specifically from replacing direct logical-form generation with entailment-based scoring plus transfer learning. The solver, gvQPS, enumerates hypotheses such as 'Carpet has more friction' by instantiating 46 templates over noun phrases from the story; one entailment function scores whether each hypothesis is a given fact from the story text, and another scores whether it follows from the question together with each answer option. The answer is chosen by comparing the given-scores of the two highest-scoring claim hypotheses. The best configuration—BERT trained on the given-fact data augmented with SNLI, and ESIM trained on the claim data—yields 76.63% accuracy, versus 68.7% for QUASP+ and 56.1% for QUASP, with no delexicalization preprocessing.

Load-bearing premise

The answer-selection rule assumes that the given-fact entailment scorer will learn the qualitative relationships themselves—so that a hypothesis like 'skin has less friction' is scored close to 'less heat is generated on skin'—because the final comparison uses only the given-scores of the two chosen claim hypotheses; if the automatically generated training pairs do not teach this transfer of QRKB relations to new stories, the selection step fails.

Editorial extensions

If this is right

Any future improvement in natural-language-inference models should transfer directly to qualitative-reasoning QA, because the solver reduces the task to two entailment-scoring calls.
The generate-validate recipe can be applied to other semantic-parsing applications where raw text is available, without retraining a parser for each new output logical form, as the paper argues in its conclusion.
The given-fact scorer benefits from adding SNLI to its training data, while the claim scorer does not; the two entailment functions therefore require differently distributed training sets, a finding that should guide dataset construction for such solvers.
Manual delexicalization, which was needed for QUASP+’s best result, is not required in this approach; the entailment scorers absorb the world-reference distinctions from the generated hypotheses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the generate-validate decomposition is the real source of the gain, then the same two-scorer architecture should transfer to other multiple-choice story benchmarks with implicit world-to-property relations, provided a template set covers their predicates; this is a testable hypothesis the paper does not run.
Because most errors trace to givenscore, a relation-aware given scorer that explicitly chains QRKB links (for example, inferring heat from friction) could close the remaining gap beyond what raw textual entailment achieves.
The 46 templates are hand-authored for the 19 qualitative properties in QUAREL; learning or paraphrasing templates automatically would show whether the method scales to open-domain qualitative questions without re-engineering per domain.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague—bottom line: this paper makes a single strong empirical claim—generate-validate with hand-authored templates and two trained entailment scorers beats the best semantic-parser baseline on QUAREL by 7.93 points—and the claim is plausible but not nailed down. The novelty is real: unlike the prior generate-validate work (Mitra et al., AAAI 2019), this version uses the framework for both question understanding and given-fact verification, and shows SNLI/BERT pretraining transfers for the givenscore. The architecture is clearly explained, the template set is explicit, and the authors report a full 4x4 grid of model/dataset combinations on dev and test, which is more transparent than many papers.

The soft spots are in verification, not in the idea. The headline 76.63% comes from a single unseeded run of the best of 16 configurations selected on dev, with no variance or significance testing. No code, no generated data, no trained model release. And the step-3 answer rule rests on an explicitly stated assumption—"we assume that the givenscore will learn to capture the qualitative relationship"—which the paper's own error analysis shows is the dominant failure mode: in the majority of 129 failures, claimA* and claimB* are identified correctly but givenscore picks the wrong one. That makes the 7.93% margin fragile: the method could be right for the wrong reason on a substantial fraction of test items. The stress-test note is on target here; this is not a peripheral detail.

I would not call the paper unsound. The train/dev/test split is clean, the training labels come only from the train split, and the QRKB-derived pairs are a legitimate use of the dataset's annotation machinery. The concern is robustness evidence, not circularity or fabrication. A serious referee should ask for multiple seeds, confidence intervals, a significance test against QUASP+, and at least the generated hypothesis sets or a small release.

Who's this for? Anyone working on qualitative reasoning, or on alternatives to semantic parsing in closed-domain QA. It deserves peer review, not desk rejection; an editor should send it out, and a referee should push on reproducibility rather than reject the premise.

Referee Report

4 major / 5 minor

Summary. The paper proposes gvQPS, a generate-validate solver for the QUAREL dataset of qualitative relationship questions. Instead of training a semantic parser to produce logical forms, the method generates natural-language hypotheses from 46 hand-authored templates instantiated over noun phrases in the story and options, scores each hypothesis with two textual entailment functions (a given-fact scorer and a claim scorer), and selects the answer by comparing given-fact scores of the best-scoring claim hypotheses. The entailment functions are trained on automatically generated premise-hypothesis pairs derived from QUAREL's logical-form annotations and the QRKB, optionally augmented with SNLI, using ESIM or BERT. The best configuration, BERT for the given-fact scorer and ESIM for the claim scorer, reaches 76.63% test accuracy, which the authors report as a 7.93-point improvement over the previous state of the art QUASP+ (68.7%). The paper also provides a small error analysis attributing most of the 129 test failures to the given-fact scorer.

Significance. If the empirical claims hold, the paper makes a useful contribution: it shows that a generate-validate pipeline can outperform direct semantic parsing on QUAREL and that NLI-style training with SNLI and BERT can transfer to this task. The proposal to replace target-language learning with natural-language hypothesis generation and entailment scoring is interesting and arguably more general than the specific QUAREL application. The paper is also honest in stating its key assumption and in presenting negative evidence in the error analysis. However, the significance of the headline result is currently limited by the absence of variance estimates, significance tests, and a direct evaluation of the assumed transfer mechanism. The paper ships no code, data, or training details, which makes the empirical claim difficult to verify.

major comments (4)

[Section 6, Table 5] The headline accuracy of 76.63% is reported for a single run of the best of 16 model combinations selected on the development set, with no variance, confidence intervals, or significance tests. Because the configuration was chosen by taking the maximum dev accuracy over 16 combinations, the reported 7.93-point improvement over QUASP+ (68.7%) could be inflated by selection noise. Please report mean and standard deviation over multiple seeds, provide a significance test against the QUASP+ baseline, and either fix the model-selection rule before evaluation or use cross-validation to reduce selection bias.
[Section 3, Step 3] The answer-selection rule is load-bearing: the final decision compares givenscore(claimA*) and givenscore(claimB*), and the paper explicitly states, "we assume that the givenscore will learn to capture the qualitative relationship." The paper's own error analysis says that the majority of the 129 test failures are due to givenscore errors, and both given examples show correct identification of claimA* and claimB* but an incorrect givenscore ordering. Please quantify the error breakdown (for example, how many failures are caused by the givenscore comparison after correct claim identification), report failures by property and direction, and discuss whether an alternative decision rule (such as using claim scores alone or a learned combination) would be more robust.
[Section 4.2] The QRKB-derived positive and negative pairs for f_given are intended to teach the model to transfer qualitative relations (for instance, from friction to heat), but the paper provides no direct analysis of whether this transfer actually occurs on test stories. The end-to-end accuracy does not separate template-scoring quality from relation transfer. Please add an ablation that removes the QRKB-generated pairs, report per-property or per-relation accuracies, and ideally test generalization to qualitative relations held out during training. Without such evidence, the mechanism asserted in Step 3 is not directly supported.
[General (reproducibility)] The empirical claims cannot be independently verified because the paper does not release code, model checkpoints, hyperparameters, random seeds, or the automatically generated premise-hypothesis datasets (only their sizes are given). Since the method depends on template generation and automatically labeled entailment pairs, even a careful reader cannot reproduce Table 5. Please make the code and generated data available, or provide a sufficiently detailed training and evaluation protocol, including the exact oversampling procedure and any hyperparameter tuning.

minor comments (5)

[Table 1 caption] The caption reads "Example problems form the QUAREL corpus"; "form" should be "from".
[Section 3, Step 1] In the example list, "trail has more friction" should presumably be "trial has more friction", matching the noun phrase "trial" listed earlier.
[Section 4.2] The sentence "For the problem II in table 1, world1 literal = “carpet” and world1 literal = “skin”" repeats "world1 literal"; the second occurrence should be "world2 literal".
[Section 6, notation] The notation is inconsistent: the text lists "f^{BERT}_{TrainQUAREL_Fact}" and "f^{ESIM}_{TrainESIM_Fact}" where the subscript should be "TrainQUAREL_Given" (or G1/G2 as in Table 5). Please standardize the subscripts to match the table.
[Error Analysis examples] In Error Example I, the hypothesis "lynn’s hair has more strength" uses a lowercase initial while the story uses "Lynn’s hair"; this is likely a template-instantiation artifact, but it should be noted or fixed for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the held-out test accuracy is produced by a standard train/evaluate pipeline over a fixed external knowledge base.

full rationale

The paper's reported 76.63% test accuracy is obtained by training the textual entailment scorers on TrainQUAREL_Given/Claim (plus SNLI) and then evaluating end-to-end on the QUAREL test split. No test labels are used to train f_given or f_claim, and the QRKB-derived entailment pairs in Section 4.2 are constructed from the train-split annotations only. At test time, the hypotheses are generated from the test story text and options using fixed templates, and the answer is selected by comparing givenscore(claimA*) and givenscore(claimB*); this is a trained inference rule, not a quantity fitted to the test answer. The explicit assumption in Step 3 that givenscore will learn qualitative relationships is a modeling assumption, and the paper's own error analysis exposes its failure modes, which is evidence that the result is not forced by construction. The self-citation to Mitra et al. (2019) introduces the generate-validate framework as background methodology; the present paper's contribution is a new application, and the citation is not used to forbid alternatives or to justify the empirical accuracy. Therefore no circular step can be exhibited with specific equations or fitted parameters renamed as predictions.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or formal entities. Its load-bearing ingredients are the manual templates, the QUAREL annotations, and the QRKB relations, all treated as axioms here. No scalar parameters are fitted to the test labels; the design choices are mostly categorical and are captured as assumptions rather than free parameters.

assumptions (5)

domain assumption QUAREL gold logical-form annotations and world literals are correct and exhaustive for every training, dev, and test problem.
Section 4 generates all textual entailment training labels from these annotations; errors or omissions in the annotations would propagate into the trained scorers.
domain assumption QRKB contains the correct and complete qualitative relations among the 19 properties for the QUAREL domain.
Section 4.2 uses QRKB relations such as q+(friction, heat) to create positive and negative training pairs for the given-fact entailment function; a missing or wrong relation would prevent givenscore from inferring the correct claim.
ad hoc to paper For each (property, direction) pair, at least one of the 46 manually authored templates in Table 4 generates a natural-language hypothesis that matches the story's intended meaning.
The templates define the entire hypothesis space. If a property is expressed in the story with phrasing not covered by a template, the correct claim may never be generated.
ad hoc to paper The given-fact entailment scorer learns to capture qualitative relationships, so comparing givenscore(claimA*) and givenscore(claimB*) selects the correct answer.
Stated explicitly in Section 3, Step 3: "we assume that the givenscore will learn to capture the qualitative relationship." The fixpoint of the arithmetic by the other scores.
domain assumption Spacy noun phrase extraction yields all relevant world descriptions, including world1 literal, world2 literal, and useful distractor phrases.
Candidate hypotheses are built only from extracted noun phrases; if a true world description is missed by the parser, no correct hypothesis is generated and the answer can only be wrong.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Generate-Validate Approach to Answering Questions about Qualitative Relationships." pith.science (2026). https://pith.science/paper/IY5KBFVS

@misc{pith2026190803645,
  author       = {Pith},
  title        = {Pith review of: A Generate-Validate Approach to Answering Questions about Qualitative Relationships},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IY5KBFVS}},
  note         = {Machine review of arXiv:1908.03645}
}

read the original abstract

Qualitative relationships describe how increasing or decreasing one property (e.g. altitude) affects another (e.g. temperature). They are an important aspect of natural language question answering and are crucial for building chatbots or voice agents where one may enquire about qualitative relationships. Recently a dataset about question answering involving qualitative relationships has been proposed, and a few approaches to answer such questions have been explored, in the heart of which lies a semantic parser that converts the natural language input to a suitable logical form. A problem with existing semantic parsers is that they try to directly convert the input sentences to a logical form. Since the output language varies with each application, it forces the semantic parser to learn almost everything from scratch. In this paper, we show that instead of using a semantic parser to produce the logical form, if we apply the generate-validate framework i.e. generate a natural language description of the logical form and validate if the natural language description is followed from the input text, we get a better scope for transfer learning and our method outperforms the state-of-the-art by a large margin of 7.93%.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages

[1]

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533--1544

work page 2013
[2]

Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1499--1510

work page 2014
[3]

Daniel G Bobrow. 2012. Qualitative reasoning about physical systems, volume 24. Elsevier

work page 2012
[4]

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326

arXiv 2015
[5]

Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. arXiv preprint arXiv:1609.06038

arXiv 2016
[6]

Peter Clark. 2015. Elementary school science and math tests as a driver for ai: take the aristo challenge! In Twenty-Seventh IAAI Conference

work page 2015
[7]

Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1--220

2013
[8]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018

Show all 19 references

[9]

Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523--533

2014
[10]

Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516--1526

2017
[11]

Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 271--281

2014
[12]

Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2011. Lexical generalization in ccg grammar induction for semantic parsing. In Proceedings of the conference on empirical methods in natural language processing, pages 1512--1523. Association for Computati...

2011
[13]

Iddo Lev, Bill MacCartney, Christopher Manning, and Roger Levy. 2004. Solving logic puzzles: From robust processing to precise semantics. In Proceedings of the 2nd Workshop on Text Meaning and Interpretation

2004
[14]

Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning

2012
[15]

Arindam Mitra, Peter Clark, Oyvind Tafjord, and Chitta Baral. 2019. Declarative question answering over knowledge bases containing natural language text with answer set programming. In AAAI 2019

2019
[16]

Siva Reddy, Mirella Lapata, and Mark Steedman. 2014. Large-scale semantic parsing without question-answer pairs. Transactions of the Association for Computational Linguistics, 2:377--392

2014
[17]

Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2018. Quarel: A dataset and models for answering questions about qualitative relationships. arXiv preprint arXiv:1811.08048

2018 arXiv
[18]

Daniel S Weld and Johan De Kleer. 2013. Readings in qualitative reasoning about physical systems. Morgan Kaufmann

2013
[19]

John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050--1055

1996

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533--1544

work page 2013

[2] [2]

Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1499--1510

work page 2014

[3] [3]

Daniel G Bobrow. 2012. Qualitative reasoning about physical systems, volume 24. Elsevier

work page 2012

[4] [4]

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326

arXiv 2015

[5] [5]

Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced lstm for natural language inference. arXiv preprint arXiv:1609.06038

arXiv 2016

[6] [6]

Peter Clark. 2015. Elementary school science and math tests as a driver for ai: take the aristo challenge! In Twenty-Seventh IAAI Conference

work page 2015

[7] [7]

Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1--220

2013

[8] [8]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018