REVIEW 3 major objections 3 minor 12 references
Controlled Reformulation Testing for Logical Consistency in Large Language Models
T0 review · 3 major / 3 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read A model can score 98.9% on logical questions yet agree with itself only 60.3% of the time when the same question is rewritten in a logically equivalent form.
desk verdict Useful benchmark and transformation typology, but the main 'consistency' metric is actually joint accuracy, so the self-contradiction claim overreaches. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the question family plus the family-consistency metric. Each family consists of a base question and four reformulations, each generated by applying exactly one controlled logical transformation (contrapositive, double negation, negation flip, passive voice, De Morgan, or quantifier rewrite). Family consistency requires the model to answer all five versions correctly; one wrong answer makes the family inconsistent. This design converts abstract logical invariance into a measurable per-operation property, allowing the authors to attribute failures to specific logical operations rather than to surface phrasing.
What would settle it
Independently rate the CRTBench reformulations for grammaticality, logical fidelity, and naturalness; or, more directly, re-run the evaluation with human-written paraphrases of the same logical forms. If the transformation-specific failure pattern (e.g., contrapositive 72.4%) does not reproduce with human paraphrase templates, the causal claim about logical operations would be an artifact of template phrasing.
Extended reading notes
Core claim
The central claim is the existence of an 'accuracy–consistency gap': models can be individually correct on most questions yet systematically contradict themselves across logically equivalent reformulations of the same question. The paper demonstrates this with CRTBench, a template-generated benchmark of 350 families (1,750 questions) spanning seven reasoning categories, using six controlled transformations: contrapositive rewriting, double negation, negation flipping, De Morgan's laws, quantifier rewriting, and passive voice. It finds that failures are transformation-specific: negation-heavy operations (contrapositive, double negation) drive inconsistency, whereas surface rephrasing (passive
Load-bearing premise
The attribution of failures to specific logical operations assumes each generated reformulation is a clean, natural instance of exactly one transformation — a claim validated only by the authors' manual audit, with no formal inter-annotator agreement.
Editorial extensions
If this is right
- Accuracy-based leaderboards overstate logical ability; adding a consistency-required metric would change model rankings (e.g., GPT-5.4-mini drops from near-top accuracy to bottom consistency).
- Applications requiring multi-query coherence (legal, medical, knowledge bases) should validate answers across paraphrases before deployment.
- Reasoning effort is not a universal fix: it can introduce new failure modes (quantifier hedging) while repairing others.
- The benchmark protocol can be extended to open-ended or multi-step reasoning by adapting the transformation set.
- Per-transformation scores give a diagnostic handle for targeted mitigation, e.g., training on contrapositive and double-negation variants.
Reading between the lines
- The quantifier collapse under high reasoning effort looks like an overcautious hedging bias ('Cannot be determined' overuse) rather than a logical reversal; if real, it suggests reasoning fine-tuning can induce abstention biases on universal/existential statements.
- The template-based method could be sharpened by pairing it with fully natural human paraphrases to separate logical invariance from linguistic generalization; without that, the transformation-level attribution is bounded by template fidelity.
- One testable extension: fine-tune a model on consistency (rather than accuracy) across CRTBench families and measure whether base accuracy is preserved; the paper hints at this direction in its conclusion.
- The gap between a small and a reasoning-optimized model (60.3% vs 96.9%) suggests consistency is a separable capability that may track training data diversity of operator-level rewrites.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CRTBench, a 350-family benchmark (1,750 questions) that evaluates whether LLMs answer logically equivalent reformulations consistently. Each family consists of a base question plus four reformulations drawn from six transformations: contrapositive, double negation, De Morgan, negation flip, passive voice, and quantifier rewriting. The authors report a large accuracy–family-consistency gap (e.g., GPT-5.4-mini: 98.9% base accuracy vs. 60.3% family consistency), transformation-level differences (contrapositive and double negation hardest, passive voice easiest), and a reasoning-ablation finding in which high reasoning effort improves several categories for GPT-5.4 but collapses quantifier reasoning, leaving overall family consistency unchanged. The paper concludes that accuracy alone is insufficient for evaluating logical reasoning in LLMs.
Significance. The question of whether LLMs are logically invariant to semantically equivalent surface forms is timely and practically important. The benchmark design is thoughtful: it is category-balanced, uses controlled transformations rather than arbitrary paraphrases, and attempts to attribute failures to specific logical operations. The paper also provides useful quantitative detail, including the per-category reasoning ablation and the quantifier abstention analysis, and it is transparent about the lack of formal inter-annotator agreement. If the central accuracy–consistency gap were properly separated from answer-disagreement consistency, the contribution would be valuable for the evaluation of reasoning models. The main weakness is that the headline metric conflates joint correctness with self-contradiction, so the central interpretive claim needs reanalysis before the results can be accepted as stated.
major comments (3)
- [Section 2, Metrics; Abstract; Section 3.2] Family consistency is defined as requiring 'the base answer and all four reformulation answers to match their respective labels.' This is joint correctness, not logical consistency. A model can answer all five versions with the same wrong categorical answer—e.g., 'No' to both 'Is it true that cats are mammals?' and 'Is it NOT the case that it is NOT true that cats are mammals?'—and be perfectly logically invariant while still counted as inconsistent. The abstract's claim that the model 'contradicts itself ... two-fifths of the time' and the Section 3.2 statement 'contradicting themselves across reformulations' do not follow from the reported metric. As the skeptic's example shows, even a model with 90% accuracy on every reformulation would exhibit a family-consistency rate near 0.9^5 ≈ 59% under this metric, with zero self-contradiction. The authors must report a direct consistency metri
- [Section 2 vs. Table 7] The text states that not all transformations apply to every category and gives explicit counts (contrapositive 290, passive 300, De Morgan 50, quantifier rewrite 60, double negation and negation flip 350 each). Table 7, however, appears to mark every transformation as applicable to every category. This contradiction makes it impossible to reconstruct which four reformulations each family actually received and undermines the reproducibility of the slot accounting. The table should be corrected to reflect the actual per-category transformation sets.
- [Section 2, Validation protocol; Section 4, Limitations] The transformation-level attribution—the claim that failures cluster on logical operations rather than surface phrasing—depends on each reformulation being a clean, natural instance of exactly one transformation. The paper states that validation was a manual audit by the authors and that 'We did not collect formal inter-annotator agreement.' Given that the causal attribution is a central contribution, the authors should provide at least a small human-annotation study or a second independent audit of a random sample, or explicitly downgrade the transformation-level attribution to a hypothesis. The current limitation statement is honest but does not resolve the concern.
minor comments (3)
- [Throughout] Several typographical issues: 'V alidation protocol' should be 'Validation protocol'; Table 3 caption has 'Double. Neg.' with an unnecessary period; Table 6's description says 'answered both the base question and its reformulation correctly,' which is ambiguous when more than one reformulation is involved. Clarify that each cell is computed for the base–reformulation pair only.
- [Table 1/Table 6] The confidence interval footnote gives ±5.1pp for n=350, but some reported cells have much smaller sample sizes (e.g., quantifier rewrite n=60, De Morgan n=50). The paper should provide the corresponding intervals or at least note the wider uncertainty in the tables.
- [Section 3.3] The quantifier-collapse analysis is described as based on 'Inspection of the outputs.' The quantitative claim (119 incorrect abstentions, etc.) is reproducible, but the interpretive claim about 'overcautious hedging' should be supported by at least a few representative example outputs in the appendix.
Circularity Check
No circularity: the paper reports direct benchmark measurements; the metric-labeling caveat is a validity issue, not a derivation loop.
full rationale
CRTBench is an empirical measurement study, not a derivation. The headline numbers (98.9% base accuracy, 60.3% family consistency, transformation-level rates, and reasoning-ablation deltas) are read directly from API outputs; no parameter is fitted from a subset of data and then used to predict a closely related quantity, and there is no equation chain whose conclusion is equivalent to its input by construction. There are no self-citations: the references (e.g., [3] Lin et al., [2] Kaup et al.) are external related work or prior demonstrations of LLM inconsistency, and the paper provides its own independent experiments for its central claims. The one definitional shortcut is semantic: 'family consistency' is defined as joint correctness on all five reformulations, and the text glosses an inconsistent family as the model 'contradicting itself' across reformulations. That gloss is a construct-validity/soundness issue—the metric does not directly measure answer disagreement—rather than a circular derivation. The paper also explicitly acknowledges its validation caveats ('We did not collect formal inter-annotator agreement'; 'claims about linguistic realism should be interpreted conservatively'), which further confirms that the template-fidelity concern is an empirical validation limitation, not a logical circularity. The observed gap magnitude is not forced by the metric definition: with independent per-question errors, 98.9% accuracy would imply roughly 94.6% family consistency, far above the observed 60.3%, so the central empirical finding is genuine measured behavior. Score 0.
Assumptions & free parameters
assumptions (5)
- domain assumption Each template-generated reformulation is a clean instance of exactly one logical transformation, isolating that operation without confounding phrasing artifacts.
- domain assumption 'Family consistency' (all five labeled answers correct) is a valid measure of cross-reformulation logical invariance.
- domain assumption Regex-based extraction with fallback correctly recovers model answers ('Answer:' patterns, last-sentence scans); the 5 UNCLEAR outputs are negligible.
- domain assumption The reported API calls (gpt-5.4, gpt-5.4-mini, claude-sonnet-4-20250514, gemini-2.5-flash, o4-mini; April 2026; T=0) occurred as described and are representative.
- standard math Binomial CI formula applies to family-consistency proportions over 350 independent families.
Cite this review
Pith. "Pith review of Controlled Reformulation Testing for Logical Consistency in Large Language Models." pith.science (2026). https://pith.science/paper/ZPY7JDQJ
@misc{pith2026260714528,
author = {Pith},
title = {Pith review of: Controlled Reformulation Testing for Logical Consistency in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZPY7JDQJ}},
note = {Machine review of arXiv:2607.14528}
}
abstract
Large language models (LLMs) frequently contradict themselves when the surface form of a logically equivalent question changes. We present a benchmark of 350 question families (1,750 total questions) for Controlled Reformulation Testing (CRTBench) to evaluate logical invariance. In this benchmark, we investigate LLMs' ability to maintain consistent answers across controlled reformulations, which include contrapositive rewriting, double negation, negation flipping, and passive voice. We evaluate several frontier LLMs and observe an accuracy-consistency gap where GPT-5.4-mini achieves $98.9\%$ base accuracy but only $60.3\%$ family-level consistency, while reasoning-optimized o4-mini achieves $96.9\%$ consistency. From our experiments, we observe that failures cluster around logically nontrivial transformations such as contrapositive rewriting ($72.4\%$ for GPT-5.4-mini) and double negation ($84.6\%$), while surface-level rephrasing remains robust ($94-100\%$). Increasing reasoning effort improves GPT-5.4-mini to $85.4\%$ consistency, but leaves GPT-5.4 unchanged overall because gains on nested negation are offset by failures on quantifier families. These results show that accuracy alone is not enough for evaluating logical reasoning in LLMs.
Figures
Reference graph
Works this paper leans on
-
[1]
Finding the Cracks: Improving
Shi, Weili and Guo, Dongliang and Yang, Lehan and Wang, Tianlong and Yuan, Hanzhang and Li, Sheng , journal=. Finding the Cracks: Improving
-
[2]
Existing
Lin, Zhenru and Tao, Jiawen and Yuan, Yang and Yao, Andrew Chi-Chih , journal=. Existing
-
[3]
Empowering
Cheng, Fengxiang and Li, Haoxuan and Liu, Fenrong and van Rooij, Robert and Zhang, Kun and Lin, Zhouchen , booktitle=. Empowering
-
[4]
Wei, Anjiang and Wu, Yuheng and Wan, Yingjia and Suresh, Tarun and Tan, Huanmi and Zhou, Zhanke and Koyejo, Sanmi and Wang, Ke and Aiken, Alex , booktitle=
-
[5]
arXiv preprint arXiv:2503.08679 , year=
Chain-of-Thought Reasoning In The Wild Is Not Always Faithful , author=. arXiv preprint arXiv:2503.08679 , year=
-
[6]
Advances in Neural Information Processing Systems (NeurIPS) , volume=
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=
-
[7]
Proceedings of the International Conference on Learning Representations (ICLR) , year=
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=
-
[8]
Liu, Jian and Cui, Leyang and Liu, Hanmeng and Huang, Dandan and Wang, Yile and Zhang, Yue , booktitle=
Show all 12 references
-
[9]
Han, Simeng and Schoelkopf, Hailey and Zhao, Yilun and Qi, Zhenting and Riddell, Martin and Zhao, Wenfei and Coady, James and Peng, David and Qiao, Yujie and Benson, Luke and others , journal=
-
[10]
ICML 2025 Workshop on Reliable and Responsible Foundation Models , year=
Consistency in Language Models: Current Landscape, Challenges, and Future Directions , author=. ICML 2025 Workshop on Reliable and Responsible Foundation Models , year=
2025
-
[11]
Higher Level Language Processes in the Brain: Inference and Comprehension Processes , editor=
The Experiential View of Language Comprehension: How is Negation Represented? , author=. Higher Level Language Processes in the Brain: Inference and Comprehension Processes , editor=. 2007 , publisher=
2007
-
[12]
2024 , address =
Parmar, Mihir and Patel, Nisarg and Varshney, Neeraj and Nakamura, Mutsumi and Luo, Man and Mashetty, Santosh and Mitra, Arindam and Baral, Chitta , booktitle =. 2024 , address =
2024
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.