REVIEW 3 major objections 6 minor 11 references
Math Natural Language Inference: this should be easy!
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Recent LLMs match human labels 88% of the time when unanimous on math NLI, yet still fail basic self-entailment.
desk verdict Useful new Math NLI corpus and a robust sanity-check finding, but the abstract's 'majority vote' claim is not what the experiments actually test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the Math NLI pair, a premise, hypothesis, and gold label triple with labels Entailment, Contradiction, or Neutral, and the unanimity statistic, the label shared by all models in a group when they agree. The seed corpus fixes 279 pairs from a mathematics-journal sentence corpus, designed so inferences are uncontroversial and mathematical concepts are treated as black boxes. A second corpus of 1,157 GPT-4-generated hypotheses from the same premises yields 89 pairs that humans checked. Three sanity checks carry the argument: self-entailment, symmetry of contradiction, and a hypothesis-only baseline with a dummy premise. The unanimity statistic is what converts individual model noise into the paper's positive result.
What would settle it
Re-annotate the seed corpus with several independent mathematicians who never saw model outputs, then recompute per-model accuracy and the unanimous-agreement rate against those fresh labels; if the independent human labels disagree with one another as often as models disagree with the original labels, or if unanimous-model agreement with fresh labels falls well below 88%, the central equivalence claim collapses.
Extended reading notes
Core claim
The paper's central claim is that Math NLI, deciding whether a hypothesis is entailed by, contradicts, or is neutral with respect to a mathematical premise, is not easier for LLMs than ordinary-language NLI. On a 279-pair seed corpus built from category-theory abstracts, ten models score between 71% and 91% accuracy, with contradiction pairs handled best and neutral pairs the consistent failure point. On a separate 89-pair subset generated by GPT-4 and checked by human mathematicians, the unanimous label of the newer model group matches the human label in 88% of cases, which the paper reads as evidence that collective LLM agreement can approximate human annotation in some settings. The negative side is documented by sanity checks: a model can judge an identical premise and hypothesis as neutral, violating A to A self-entailment, and models on the GPT-generated corpus frequently relabel neutral pairs as entailments. The paper concludes that self-consistency, not advanced mathematical reasoning, is the bottleneck for Math NLI.
Load-bearing premise
The 279 seed labels were written by one team member each and treated as gold by construction, with no inter-annotator agreement measured, and pairs that models disagreed with were later re-discussed and some removed as controversial; if those labels are idiosyncratic or the deletions are biased, every accuracy and agreement number in the paper shifts, including the 88% headline.
Editorial extensions
If this is right
- On the human-written seed corpus, ten LLMs score between 71% and 91% accuracy, with contradiction pairs easiest and neutral pairs the main source of errors.
- When all models in the recent group agree on a GPT-generated pair, their unanimous label matches the human label 88% of the time, so LLM unanimity is a plausible weak substitute for gold labels in this domain.
- GPT-4 generates usable entailment and contradiction pairs but is not self-consistent, since it later rejudges 41.4% of its own neutral pairs as entailments.
- Current models pass the hypothesis-only sanity check: with a dummy true premise, all four recent models essentially classify every hypothesis as neutral, unlike older NLI systems that relied on dataset artifacts.
- The A implies A sanity check is nearly error-free, with only 3 failures in 2,312 pairs, but the failure explanations reveal a real reasoning-process error, undercutting the assumption that math inference is easy for LLMs.
Reading between the lines
- The abstract's majority vote wording is not what the experiments measure: the 88% figure comes from unanimous agreement among recent models, and majority-rule behavior over the ten models is never directly tested.
- The deliberate removal of controversial pairs after seeing model outputs means the reported agreement rates likely overstate how often an arbitrary new LLM would match the human labels, since the hardest disagreements were deleted.
- Because the corpora are perfectly balanced at one-third entailment, contradiction, and neutral, the headline accuracy and unanimity numbers would shift on a test set with a naturalistic distribution of labels, so the equal balance should be read as a favorable condition.
- The A implies A failure offers a cheap screening test: a model that cannot reliably certify identical premise and hypothesis pairs should not be trusted on more complex mathematical inferences.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Math NLI, the task of natural language inference over mathematical text, and presents two corpora: a seed set of 279 premise-hypothesis-label triples created from category-theory abstracts (labels written by a single team member each), and a larger GPT-4-generated corpus of 1,157 pairs with a human-evaluated subset of 89. Ten LLMs are evaluated on both corpora, with additional sanity checks for A→A entailment, symmetry of contradictions, and hypothesis-only artifacts. The authors report positive findings (LLMs' unanimous labels agree with a human label on 88% of the unanimous subset of the GPT-generated corpus) and negative findings (LLMs occasionally fail even on identical-premise-hypothesis pairs). They release the corpora to support future work.
Significance. The paper targets an underexplored and relevant extension of NLI to research-level mathematics, and the release of the corpora is a concrete community resource. Its sanity checks go beyond reporting accuracy: the A→A failure example with its explanation is a valuable, falsifiable piece of evidence about LLM reasoning, and the symmetry check addresses an under-reported phenomenon. The main quantitative claim of the abstract, however, is not supported as stated: majority voting is never computed, and the 88% figure is an agreement rate between unanimous model labels and a permissive human-agreement criterion on a subset of the data. The post-hoc filtering of 'controversial' pairs also weakens the validity of the seed-corpus numbers. With a corrected analysis and a more cautious framing, the paper would be a useful contribution.
major comments (3)
- [Abstract; Section 5, Table 6] The abstract's positive finding—'in some settings, using a majority vote of LLMs is approximately equivalent to using human-labeled data'—is not what the experiments measure. The 88% figure in Table 6 is the fraction of the 65/89 group-2 pairs (73%) where the five models are unanimous and their unanimous label agrees with at least one human annotator (57/65). A majority vote is never computed: the 24 non-unanimous pairs are excluded, and there is no analysis of what a majority vote on those pairs would yield. In addition, the permissive gold standard of 'agrees with at least one human' inflates the result; the same table shows 'agrees with all human' is only 50/65 (76.9%). The authors should either compute a genuine majority-vote agreement over all 89 pairs (or over a principled subset), or revise the abstract and conclusion to describe unanimous-model agreement on the unanimous subset.
- [Section 2.1; Section 3.1] The seed corpus labels are single-author by construction, and no inter-annotator agreement is reported; the 279 labels are thus not demonstrated to be reliable gold labels. More seriously, Section 3.1 reports that after evaluating the five initial LLMs, the authors re-discussed pairs and removed any that were considered 'controversial' by the mathematicians. Because this filtering happens after seeing model disagreements, it can be correlated with model behavior and can bias every accuracy and agreement number in Tables 3 and 5. The authors should report inter-annotator agreement on the full 279 pairs before any filtering, and should analyze the sensitivity of their headline numbers to the inclusion or exclusion of the removed pairs.
- [Section 4.2; Table 6] The human evaluation of the GPT-generated corpus uses 'agrees with at least one human annotator' as a success criterion, but the annotators themselves disagree on 9 of the 89 pairs (80/89 full agreement). This permissive criterion is particularly problematic for the positive claim, since a model can 'agree' with a minority human label. The authors should report the primary analysis against a consensus gold label (e.g., the majority human label, with full-agreement pairs as a sensitivity subset), and should present the 50/65 and 57/65 figures with their denominators and confidence intervals.
minor comments (6)
- [Section 3.2] The sentence 'And in 271 of the 290 pairs (including ones where the models were not unanimous)' should read '279 pairs'; 290 is inconsistent with the seed corpus size stated in Section 2.1.
- [Section 5] The sentence 'In group 1, for 50 of these 65 pairs (87%)' is numerically inconsistent: group 1 has 57 unanimous pairs, so 50/57 is the relevant fraction; the same paragraph uses 65 for both groups.
- [Table 5] The column header 'agree w/ a human' is ambiguous: in the unanimous rows it counts unanimous labels matching a human label, while 'some agree w/ human' counts pairs with at least one model agreeing with the human. Rename the columns (e.g., 'unanimous label matches human' vs. 'at least one model matches human') and state the denominators.
- [Section 6.3] The hypothesis-only check uses a single dummy premise ('Right adjoints preserve limits') and reports that models classify all hypotheses as N. This does not rule out the possibility that models default to N for any premise; using a small set of varied dummy premises would make the test more informative.
- [Section 2.2] The 'red pairs' are introduced as 'some 20 pairs,' but no precise count or criterion is given there; later, Section 3.2 reports 8 pairs where no initial model agrees with the human label. Clarify how the 20 red pairs relate to the 8 pairs and to the pairs removed in Section 3.1.
- [Limitations] The statement in the Limitations section that the authors 'do not expect significant differences when we pivot to other branches of mathematics' is unsupported, since all premises come from category-theory abstracts; this should be framed as an empirical assumption.
Circularity Check
No circular derivation: the paper is an empirical evaluation whose reported agreements are measured, not forced by construction; the only self-citation (TAC corpus) supplies raw premise text and is not load-bearing.
full rationale
This is an empirical evaluation paper rather than a derivation chain. There is no fitted parameter, no equation, and no uniqueness theorem that reduces a reported result to an input. The human seed labels (Section 2.1) were created before any model outputs, and the phrase "gold labels by construction" describes corpus design, not the evaluation outcome. The headline claim about "majority vote" (abstract) is an overstatement relative to the conditional actually reported in Section 5: "If we take the unanimous labels from the group 2 models to simply be the gold label, then this label is the same as the human label 88% of the time." That is a measured agreement on the 65/89 unanimous pairs, not a quantity forced by the definition of the labels. The majority-vote mismatch is a correctness/scope concern, not circularity. The post-hoc removal of "controversial" pairs (Section 3.1) is a data-quality concern that could inflate accuracy, but it is transparently reported and does not make the conclusion equivalent to its input. The only noticeable self-citation is the TAC corpus (Collard et al., 2022, with author overlap), used as the source of premise sentences. That citation supplies raw input text, and the paper's claims are evaluated against human judgments and sanity checks, so the self-citation is not an argument that reduces the conclusion to itself. No circular step can be quoted with a specific reduction; the appropriate finding is no significant circularity, with score 1 reflecting only the minor, non-load-bearing self-citation.
Assumptions & free parameters
assumptions (4)
- domain assumption The 432 Goldilocks sentences from TAC abstracts are representative of research-level mathematical text.
- domain assumption Single-author 'gold labels by construction' are reliable ground truth for the seed corpus.
- domain assumption The keyword-counting parser recovers each LLM's E/C/N judgment from the first sentence of the explanation.
- ad hoc to paper Removing pairs deemed 'controversial' after seeing model disagreements does not bias the reported agreement.
Cite this review
Pith. "Pith review of Math Natural Language Inference: this should be easy!." pith.science (2026). https://pith.science/paper/QNKQGUNK
@misc{pith2026250723063,
author = {Pith},
title = {Pith review of: Math Natural Language Inference: this should be easy!},
year = {2026},
howpublished = {\url{https://pith.science/paper/QNKQGUNK}},
note = {Machine review of arXiv:2507.23063}
}
read the original abstract
We ask whether contemporary LLMs are able to perform natural language inference (NLI) tasks on mathematical texts. We call this the Math NLI problem. We construct a corpus of Math NLI pairs whose premises are from extant mathematical text and whose hypotheses and gold labels were provided by people with experience in both research-level mathematics and also in the NLI field. We also investigate the quality of corpora using the same premises but whose hypotheses are provided by LLMs themselves. We not only investigate the performance but also the inter-group consistency of the diverse group of LLMs. We have both positive and negative findings. Among our positive findings: in some settings, using a majority vote of LLMs is approximately equivalent to using human-labeled data in the Math NLI area. On the negative side: LLMs still struggle with mathematical language. They occasionally fail at even basic inferences. Current models are not as prone to hypothesis-only "inference" in our data the way the previous generation had been. In addition to our findings, we also provide our corpora as data to support future work on Math NLI.
Reference graph
Works this paper leans on
-
[1]
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. https://www.aclweb.org/anthology/D15-1075/ A large annotated corpus for learning natural language inference . In Proceedings of EMNLP
work page 2015
-
[2]
Jacob Collard, Valeria de Paiva, Brendan Fong, and Eswaran Subrahmanian. 2022. http://arxiv.org/abs/2208.13830 Extracting mathematical concepts from text
work page Pith review arXiv 2022
-
[3]
Holliday, Matthew Mandelkern, and Cedegao E
Wesley H. Holliday, Matthew Mandelkern, and Cedegao E. Zhang. 2024. http://arxiv.org/abs/2401.17169 Conditional and modal reasoning in large language models
arXiv 2024
-
[4]
Aikaterini-Lida Kalouli, Valeria de Paiva, and Livy Real. 2017. https://aclanthology.org/W17-7205 Correcting contradictions . In Proceedings of the Computing Natural Language Inference Workshop
work page 2017
-
[5]
Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. 2024. http://arxiv.org/abs/2305.00050 Causal reasoning and large language models: Opening a new frontier for causality
arXiv 2024
-
[6]
Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, and Dieuwke Hupkes. 2024. Lost in inference: Rediscovering the role of natural language inference for large language models. arXiv preprint arXiv:2411.14103
arXiv 2024
-
[7]
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. https://doi.org/10.18653/v1/S18-2023 Hypothesis only baselines in natural language inference . In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180--191, New Orleans, Louisiana. Association for Computational Linguistics
-
[8]
Anastasiia Sedova, Robert Litschko, Diego Frassinelli, Benjamin Roth, and Barbara Plank. 2024. https://aclanthology.org/2024.findings-emnlp.1003 To know or not to know? analyzing self-consistency of large language models under ambiguity . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 17203--17217, Miami, Florida, USA. Ass...
work page 2024
Show all 11 references
-
[9]
Trinh, Yuhuai Wu, Quoc V
Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. 2024. https://api.semanticscholar.org/CorpusID:267032902 Solving olympiad geometry without human demonstrations . Nature, 625:476 -- 482
2024
-
[10]
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A B road- C overage C hallenge C orpus for S entence U nderstanding through I nference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec...
2018
-
[11]
Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2024. http://arxiv.org/abs/2306.09841 Are large language models really good logical reasoners? a comprehensive evaluation and beyond
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.