Pith. sign in

REVIEW 4 major objections 4 minor 29 references

Let's CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NLI models, including large language models, do not reliably infer presuppositions embedded in conditional sentences, and fine-tuning on existing presupposition datasets does not transfer to this structure.

desk verdict A useful new NLI benchmark for conditional presuppositions, but the headline claim leans on a contested Type 5 labeling choice that the paper itself admits is unsettled. read the letter →

arxiv 2506.06133 v1 pith:D5RLPSWW submitted 2025-06-06 cs.CL

Tara Azin , Daniel Dumitrescu , Diana Inkpen , Raj Singh This is my paper
classification cs.CL
keywords naturallanguageinferencepresuppositionconditionalreasoningprojectionProvisoProblempragmaticNLIbenchmarklargemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that current natural language inference (NLI) models, including large language models, fail to handle presuppositions when they are embedded in conditional sentences, and that training on existing presupposition datasets does not teach this skill. It introduces CONFER, a dataset of 18,000 NLI pairs built from 3,600 conditional templates spanning five logical relations between the antecedent and the presupposition, to isolate where reasoning breaks down. If the claim is right, current NLI benchmarks and fine-tuning recipes are blind to a core piece of pragmatic inference, and progress will require structured data aimed specifically at conditional projection. The paper itself notes that how humans interpret such conditionals remains an open question, so the benchmark's labels carry a theoretical commitment that the results depend on.

What carries the argument

The carrying object is CONFER itself: semi-automatically generated conditionals of the form If $A$, $B_p$, where the consequent $B_p$ contains a presupposition trigger (the iterative adverb 'again' or a possessive construction) and $p$ is the presupposition. Each conditional appears unembedded and inside negation, interrogation, and factive embedding, paired with $p$ as the hypothesis and labeled Entailment, Neutral, or Contradiction. The five-type design encodes the logical relation between $A$ and $p$: equivalence, $A$ entailing $p$, $p$ entailing $A$, $A$ strongly suggesting $p$, and independence; this type structure is what lets the paper attribute model failures to specific projection conditions, especially the Type 5 case tied to the Proviso Problem.

What would settle it

A human-subject experiment asking readers whether 'If Lisa finishes her meeting early, she'll never drive a sports car again' commits to 'Lisa has driven a sports car before' would settle it: if most readers say no, the Type 5 entailment labels are wrong and the model errors are not evidence of a general deficit.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that models score well on Neutral and Contradiction labels but poorly on Entailment for conditionals, and that the hardest case is Type 5, where the antecedent $A$ and presupposition $p$ are logically and probabilistically independent: the correct label is Entailment, yet LLMs mostly answer Neutral. Fine-tuning transformer models on the existing IMPPRES and NOPE datasets and then testing on CONFER produces near-chance transfer, while training on CONFER itself yields high precision. The paper interprets this as evidence that presuppositional reasoning in conditionals is a distinct, under-learned inference type rather than a surface lexical trigger problem.

Load-bearing premise

The load-bearing premise is that the Type 5 labels are correct, namely that when a conditional's antecedent and the presupposition of its consequent are independent, human readers really do infer the full presupposition, since the paper itself says this interpretation is an open question.

Editorial extensions

If this is right

  • Good performance on IMPPRES or NOPE cannot be taken as evidence of general presupposition reasoning, because conditionals are a separate diagnostic that those datasets do not cover.
  • Fine-tuning on non-conditional or simple-conditional presupposition data can leave performance on conditionals unchanged or worse, so benchmark improvement requires targeted conditional data.
  • The five-type error profile gives model developers a concrete failure map, with Entailment and Type 5 as the cells that separate models from the benchmark's intended reasoning.
  • Few-shot prompting is not a reliable fix: it helps some LLMs slightly, hurts at least one, and leaves Type 5 accuracy very low across all evaluated models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Type 5 labels are correct, current models are missing a core piece of human pragmatic competence; if the labels are contested, the hardest cell of CONFER is better read as a site for human judgment experiments than as pure model failure.
  • The same five-type schema could be extended to other presupposition triggers, such as factive verbs and change-of-state verbs, to test whether the Type 5 deficit is trigger-specific or structural.
  • Because the five types encode competing answers to the Proviso Problem, model error patterns on CONFER could be used to compare which projection rule a model approximately implements, turning benchmark errors into evidence about the learned semantics of conditionals.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CONFER, a semi-automatically generated NLI dataset of 18,000 conditional sentence pairs (13,500 used in the experiments) designed to test presupposition projection in conditionals. The dataset covers five types of antecedent-presupposition relations and two presupposition triggers (the iterative adverbagain and possessive constructions). The authors evaluate four NLI models (GloVe, InferSent, RoBERTa-large-MNLI, DeBERTa-large-MNLI) under fine-tuning with existing presupposition datasets and with CONFER itself, and four LLMs (GPT-4o, Llama-3-8B-Instruct, Gemma-2B-it, DeepSeek-R1-Distill-Qwen-1.5B) in zero-shot and few-shot prompting settings. Their central claim is that NLI models, including LLMs, struggle with presuppositional reasoning in conditionals, and that fine-tuning on existing NLI datasets does not transfer well to this structure. The paper also releases the dataset, prompts, code, and experimental results.

Significance. If the benchmark is valid, CONFER fills a genuine gap: no existing NLI dataset focuses specifically on presupposition projection in conditional sentences. The work has several concrete strengths: the dataset is publicly released, the template-based generation with 2,139 lexical items provides controlled lexical diversity, the annotator agreement is very high (99.86%), and the evaluation spans both fine-tuned encoder models and several LLM families. The central empirical claim, however, depends on the gold labels for the five conditional types, and in particular on the Type 5 labels, which encode a contested resolution of the Proviso Problem. The paper itself acknowledges in Section 2 that how humans interpret presuppositions in conditionals remains an open question. Consequently, the headline finding that models fail at presuppositional reasoning is not cleanly separable from the authors' theoretical commitment about Type 5. As a resource paper, the contribution is potentially useful, but the validity of the benchmark's central claim needs additional support.

major comments (4)
  1. [Section 4, Table 3; Section 2] The Entailment labels assigned to Type 5 pairs presuppose a specific resolution of the Proviso Problem: that the full unconditional presupposition p projects from a conditional with an independent antecedent. The paper itself states in Section 2 that "How humans interpret presuppositions, particularly in complex structures such as conditionals, remains an open question." Under the alternative conditional reading ("If A then p"), many Type 5 hypotheses would be Neutral rather than Entailment. Since Type 5 is the class where models perform worst (Tables 7 and 8, recall 0.03-0.54), the central claim that NLI models fail at presuppositional reasoning in conditionals is not separable from this contested labeling decision. Please provide non-expert human validation for Type 5 labels, or report the analysis with Type 5 excluded and discuss both outcomes.
  2. [Section 5.1, footnote 5] The manuscript says that 13,500 of the 18,000 pairs were selected "to ensure a balanced distribution across sentence types and labels" but does not document the selection mechanism. Every accuracy and recall figure in Tables 4-8 depends on this undocumented subset. Please specify whether the selection was random, stratified, or manually filtered, and report the resulting distribution by type, trigger, embedding environment, and label, along with the random seed used.
  3. [Section 6, Tables 4-8] All experimental results are single point estimates with no confidence intervals, bootstrap intervals, or significance tests. This is a problem for conclusions phrased comparatively, such as fine-tuning "often results in performance degradation" and the small differences between zero-shot and few-shot GPT-4o (0.64 vs. 0.66 in Table 6). Please add variance estimates across multiple seeds or runs, or temper the comparative claims to what a single run can support.
  4. [Section 4, annotation paragraph] The reported agreement of 99.86% is between annotators applying the same theoretical labeling scheme; it does not validate the scheme against independent human inference judgments. Since the paper's central claim is that models fail at human-like presuppositional reasoning, a comparison with non-expert human judgments on a sample of items would strengthen the benchmark substantially.
minor comments (4)
  1. [Section 3] The dataset name "HellaSW AG" contains a typo and should be "HellaSWAG."
  2. [Section 5.1] The corpus name "IMPRESS" appears once and should be "IMPPRES."
  3. [Footnote 4] The independence condition is written as Pr(q|r, K) = Pr(q|K), but the preceding text says "the probability of r does not change by learning q"; either the formula or the text should be corrected to use Pr(r|q, K) = Pr(r|K).
  4. [Section 6.2, Tables 7-8] The text in Section 6.2 discusses accuracy for type-level results, while Tables 7 and 8 report recall; please make the metric names consistent throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's model evaluations are measured against theory-derived, human-annotated labels rather than fitted to the labels, and the only self-citation is a non-load-bearing overview reference.

full rationale

The paper does not fit any parameter to data and then rename that fit as a prediction; there is no derivation chain in which an output is equivalent to an input by construction. CONFER's gold labels are generated from standard presupposition-projection theories (Karttunen 1973, Stalnaker 1973), independently of the model outputs, and model scores are then measured against those labels. The contested Type-5 labeling (full projection of p when A and p are independent) is a substantive theoretical assumption about human inference, but it is not circular in the technical sense: the labels are not derived from model behavior, and the paper's conclusion would be invalidated by wrong labels rather than shown to be a tautology. The only self-citation is [9], Singh's 'recent overview' of the Proviso Problem, which is not load-bearing; the projection assumptions used for labeling trace to [1] and [2], not to the authors' own prior results. The annotator-agreement statistic (99.86%) reflects consistency among annotators applying the same theory, and while this bears on benchmark validity, it does not constitute a circular derivation. Accordingly, no circular step can be exhibited under the required standard, and the central empirical claim retains independent content.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claim rests on linguistic assumptions about presupposition projection, not on fitted numeric parameters. No new physical or theoretical entities are postulated; the dataset and taxonomy are constructed artifacts.

assumptions (6)
  • domain assumption Presuppositions project through entailment-canceling environments (negation, questions, factive embeddings), as in Karttunen and Stalnaker.
    Used to design the three embeddings and to label pairs; Section 2, Table 1.
  • domain assumption In conditionals of the form 'If A, Bp', the projected presupposition is either A implies p or p, depending on the logical relationship between A and p (Proviso Problem).
    This is the basis for the five-type taxonomy and the Entailment/Neutral/Contradiction labels; Section 2, Proviso Problem paragraph.
  • ad hoc to paper Type 5 conditionals, where A and p are logically and probabilistically independent, project the full presupposition p, so the label is Entailment.
    The paper assigns Entailment to Type 5 pairs (Table 3), but this is exactly the contested Proviso Problem; no human behavioral evidence is provided.
  • domain assumption NLI labels can be used to test presupposition projection: a projected presupposition counts as entailment, a non-projected one as neutral or contradiction.
    The task design maps p to hypothesis and uses standard NLI labels; Section 4.
  • domain assumption Statistical independence of A and p is defined as Pr(q|r,K) = Pr(q|K).
    Footnote 4 of the paper; used to define Type 5.
  • domain assumption GPT-3.5 generated sentences are grammatical and natural after linguist review.
    Generation quality is not quantitatively evaluated beyond a single linguist's review; Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Let's CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition." pith.science (2026). https://pith.science/paper/D5RLPSWW

@misc{pith2026250606133,
  author       = {Pith},
  title        = {Pith review of: Let's CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D5RLPSWW}},
  note         = {Machine review of arXiv:2506.06133}
}
read the original abstract

Natural Language Inference (NLI) is the task of determining whether a sentence pair represents entailment, contradiction, or a neutral relationship. While NLI models perform well on many inference tasks, their ability to handle fine-grained pragmatic inferences, particularly presupposition in conditionals, remains underexplored. In this study, we introduce CONFER, a novel dataset designed to evaluate how NLI models process inference in conditional sentences. We assess the performance of four NLI models, including two pre-trained models, to examine their generalization to conditional reasoning. Additionally, we evaluate Large Language Models (LLMs), including GPT-4o, LLaMA, Gemma, and DeepSeek-R1, in zero-shot and few-shot prompting settings to analyze their ability to infer presuppositions with and without prior context. Our findings indicate that NLI models struggle with presuppositional reasoning in conditionals, and fine-tuning on existing NLI datasets does not necessarily improve their performance.

Figures

Figures reproduced from arXiv: 2506.06133 by the authors.

Figure 2
Figure 2. IMPPRES NOPE CONFER Entailment Neutral Contradiction 0.26 0.47 0.87 0.46 0.32 0.85 0.41 0.00 0.92 GloVe IMPPRES NOPE CONFER 0.18 0.47 0.93 0.50 0.02 0.92 0.54 0.00 0.97 InferSent IMPPRES NOPE CONFER 0.00 0.00 0.98 0.56 0.50 0.96 0.00 0.45 0.96 DeBERTa IMPPRES NOPE CONFER 0.10 0.00 0.97 0.42 0.22 0.95 0.00 0.34 0.96 RoBERTa 0.0 0.2 0.4 0.6 0.8 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 1
Figure 1. Precision of models trained on IMPPRES, NOPE, and CONFER (tested on CONFER). [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Confusion matrices showing label prediction errors for RoBERTa and DeBERTa fine-tuned on IMPPRES, [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 24 canonical work pages

  1. [1]

    Presuppositions of compound sentences.Linguistic Inquiry, 4(2):169–193, 1973

    Lauri Karttunen. Presuppositions of compound sentences.Linguistic Inquiry, 4(2):169–193, 1973

    work page 1973

  2. [2]

    Presuppositions.Journal of Philosophical Logic, 2(4):447–457, 1973

    Robert Stalnaker. Presuppositions.Journal of Philosophical Logic, 2(4):447–457, 1973

    work page 1973

  3. [3]

    Are natural language inference models imppressive? learning implicature and presupposition

    Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. Are natural language inference models imppressive? learning implicature and presupposition. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online, 2020. Association for Computational Linguistics

    work page 2020

  4. [4]

    Nope: A corpus of naturally-occurring presuppositions in english

    Alyssa Parrish, Sebastian Schuster, Alex Warstadt, et al. Nope: A corpus of naturally-occurring presuppositions in english. InProceedings of the 25th Conference on Computational Natural Language Learning, pages 349–366, 2021

    work page 2021

  5. [5]

    Presupposition

    David Ian Beaver. Presupposition. In Johan Van Benthem and Alice Ter Meulen, editors,Handbook of Logic and Language, chapter 17, pages 939–1008. North-Holland, 1997. 10 Let’s CONFER: A Dataset for NLI Models on Conditional Inference

    work page 1997

  6. [6]

    Presupposition and implicature

    Christopher Potts. Presupposition and implicature. InThe Handbook of Contemporary Semantic Theory, pages 168–202. John Wiley & Sons, Ltd, 2015

    work page 2015

  7. [7]

    Scorekeeping in a language game.Journal of Philosophical Logic, 8(1):339–359, 1979

    David Lewis. Scorekeeping in a language game.Journal of Philosophical Logic, 8(1):339–359, 1979

    work page 1979

  8. [8]

    Local satisfaction guaranteed: A presupposition theory and its problems.Linguistics and Philosophy, 19:259–294, 1996

    Bart Geurts. Local satisfaction guaranteed: A presupposition theory and its problems.Linguistics and Philosophy, 19:259–294, 1996

    work page 1996

Show all 29 references
  1. [9]

    R. Singh. Matrix and embedded presuppositions. InThe Wiley Blackwell Companion to Semantics, pages 1–42. Wiley-Blackwell, 2020

    2020

  2. [10]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computation...

    2015

  3. [11]

    A broad-coverage challenge corpus for sentence un- derstanding through inference

    Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence un- derstanding through inference. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,...

    2018

  4. [12]

    Robin Cooper, Richard Crouch, Jan Van Eijck, Chris Fox, Josef Van Genabith, Jan Jaspers, Hans Kamp, Manfred Pinkal, Massimo Poesio, and Stephen Pulman et al.FraCaS: A framework for computational semantics. 1994. Deliverable D6

    1994

  5. [13]

    Swag: A large-scale adversarial dataset for grounded commonsense inference

    Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104. Association for Computational Linguist...

    2018

  6. [14]

    Abductive commonsense reasoning

    Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. Abductive commonsense reasoning. InProceedings of the 2020 International Conference on Learning Representations (ICLR), 2020

    2020

  7. [15]

    The role of veridicality and factivity in clause selection

    Aaron Steven White and Kyle Rawlins. The role of veridicality and factivity in clause selection. InProceedings of the 48th Annual Meeting of the North East Linguistic Society, Amherst, MA, USA, 2018. GLSA Publications

    2018

  8. [16]

    The commitmentbank: Investigating projection in naturally occurring discourse

    Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. InProceedings of Sinn und Bedeutung, volume 23, pages 107–124, 2019

    2019

  9. [17]

    Investigating the distribution of some (but not all) implicatures using corpora and web-based methods.Semantics and Pragmatics, 8:11–1, 2015

    Judith Degen. Investigating the distribution of some (but not all) implicatures using corpora and web-based methods.Semantics and Pragmatics, 8:11–1, 2015

    2015

  10. [18]

    Harnessing the richness of the linguistic signal in predicting pragmatic inferences

    Sebastian Schuster, Yuxing Chen, and Judith Degen. Harnessing the richness of the linguistic signal in predicting pragmatic inferences. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020

    2020

  11. [19]

    Let’s do it “again”: A first computational approach to detecting adverbial presupposition triggers

    Andre Cianflone, Yulan Feng, Jad Kabbara, and Jackie Chi Kit Cheung. Let’s do it “again”: A first computational approach to detecting adverbial presupposition triggers. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

  12. [20]

    Do you know that florence is packed with visitors? evaluating state-of-the-art models of speaker commitment

    Nanjiang Jiang and Marie-Catherine de Marneffe. Do you know that florence is packed with visitors? evaluating state-of-the-art models of speaker commitment. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4208–4213, Florence, It...

    2019

  13. [21]

    Alexis Ross and Ellie Pavlick. How well do nli models capture verb veridicality? InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240, 2019

    2019

  14. [22]

    Paloma Jeretiˇc, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, and Samuel R. Bowman. Investigating bert’s knowledge of language: Five analysis methods with npis. InProceedings of EMNLP-IJCNLP, pages 2870–2880, 2019. 11 Let’s CONFER: A Dataset for N...

    2019

  15. [23]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. Blimp: The benchmark of linguistic minimal pairs for english.Transactions of the Association for Computational Linguistics, 8:377–392, 2020

    2020

  16. [24]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. InEmpirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, 2014

    2014

  17. [25]

    Supervised learning of universal sentence representations from natural language inference data

    Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–6...

    2017

  18. [26]

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint,

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint,

  19. [27]

    Deberta: Decoding-enhanced bert with disentan- gled attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentan- gled attention. InInternational Conference on Learning Representations, 2021

    2021

  20. [28]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint, 2017. arXiv:1701.06538. 12

    2017 arXiv

  21. [2018]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.