{"id":"b93cb8ed-3996-4ab4-8db0-2cb51f4480bc","arxiv_id":"2607.14528","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs show an accuracy–consistency gap under controlled logical reformulations (e.g., GPT-5.4-mini: 98.9% base accuracy vs. 60.3% family consistency), with negation-heavy rewrites hardest and high reasoning effort collapsing quantifier families in GPT-5.4.","lead":"This paper releases CRTBench, a 350-family benchmark that rewrites each logical question four ways — contrapositive, double negation, negation flip, and passive voice — and finds that frontier LLMs with ~99% accuracy still contradict themselves across equivalent formulations in up to 40% of families. It also reports that turning on reasoning fixes most transformations but collapses quantifier reasoning in GPT-5.4, a warning for consistency-critical deployments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Family-consistency metric measures joint correctness, not logical invariance; reported 'contradicts itself' gap may be artifact of reformulation difficulty.","rationale":"The reader identified template fidelity as the weakest assumption, but I believe a more fundamental measurement-validity issue precedes it: the paper never actually measures consistency. Family consistency as defined is joint correctness of base and reformulations, not answer agreement. The paper's own example—'one wrong answer out of five would mean the family is inconsistent'—makes this explicit. The abstract's 'contradicts itself roughly two-fifths of the time' interprets family inconsistency as logical contradiction, which is a non sequitur. A model could be perfectly invariant across forms yet score 0% family consistency if all answers are the same wrong label. The central contribution of the paper is the accuracy–consistency gap; if the metric does not isolate consistency, the contribution is overstated. The template-fidelity concern is real and relevant to transformation-level attribution, but it presupposes that the consistency metric is valid. A clean template with a flawed metric still fails to support the central claim. The proposed concrete test—computing actual answer agreement—decisively resolves whether the gap is due to contradiction or to reformulation difficulty. Therefore the paper should be accepted only conditional on this re-analysis. I partially agree with the reader because they noted the conflation in passing but did not treat it as load-bearing.","tokens_in":7361,"tokens_out":7201,"duration_ms":81254,"concrete_test":"Recompute Tables 1–6 using a true consistency metric: for each family, take the model's categorical answer to the base question (regardless of correctness) and check, for each of the four reformulations, whether the model's answer equals the base answer for equivalence-preserving transformations (contrapositive, double negation, De Morgan, passive, quantifier rewrite) and equals the complement (Yes↔No, CBD unchanged) for negation flip. Report (a) the proportion of families with full 4/4 answer agreement, (b) per-transformation agreement, and (c) the decomposition of family inconsistency into 'answer disagreement' versus 'consistent-but-wrong.' If full agreement is ≥95% while all-correct family consistency is 60.3%, the paper's consistency-gap narrative collapses; if agreement is ≈60%, the gap is real and the current metric, though mislabeled, preserves the qualitative conclusion.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that models 'contradict themselves' across logically equivalent reformulations—is not actually measured by the paper's primary metric. Section 2 defines family consistency as requiring 'the base answer and all four reformulation answers to match their respective labels.' Thus a family is declared inconsistent even when the model gives the same categorical answer to every version (e.g., 'No' to both 'Is it true that cats are mammals?' and 'Is it NOT the case that it is NOT true that cats are mammals?'). Such a model is perfectly logically invariant but counts as inconsistent because all five answers are wrong. The 98.9% base accuracy versus 60.3% family consistency gap therefore conflates two distinct phenomena: (i) per-question accuracy on reformulations and (ii) genuine cross-form answer disagreement. The abstract's claim that a model with these numbers 'contradicts itself ... two-fifths of the time' is unsupported. Even a perfectly consistent model with, say, 90% accuracy on every reformulation would show a large 'accuracy–family-consistency gap' (0.9^5 ≈ 59%). To sustain the central conclusion, the paper must report a direct consistency metric that compares the model's own answers across versions, not their joint correctness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CRTBench, a 350-family benchmark (1,750 questions) that evaluates whether LLMs answer logically equivalent reformulations consistently. Each family consists of a base question plus four reformulations drawn from six transformations: contrapositive, double negation, De Morgan, negation flip, passive voice, and quantifier rewriting. The authors report a large accuracy–family-consistency gap (e.g., GPT-5.4-mini: 98.9% base accuracy vs. 60.3% family consistency), transformation-level differences (contrapositive and double negation hardest, passive voice easiest), and a reasoning-ablation finding in which high reasoning effort improves several categories for GPT-5.4 but collapses quantifier reasoning, leaving overall family consistency unchanged. The paper concludes that accuracy alone is insufficient for evaluating logical reasoning in LLMs.","tokens_in":7429,"tokens_out":4024,"duration_ms":40200,"significance":"The question of whether LLMs are logically invariant to semantically equivalent surface forms is timely and practically important. The benchmark design is thoughtful: it is category-balanced, uses controlled transformations rather than arbitrary paraphrases, and attempts to attribute failures to specific logical operations. The paper also provides useful quantitative detail, including the per-category reasoning ablation and the quantifier abstention analysis, and it is transparent about the lack of formal inter-annotator agreement. If the central accuracy–consistency gap were properly separated from answer-disagreement consistency, the contribution would be valuable for the evaluation of reasoning models. The main weakness is that the headline metric conflates joint correctness with self-contradiction, so the central interpretive claim needs reanalysis before the results can be accepted as stated.","major_comments":[{"comment":"Family consistency is defined as requiring 'the base answer and all four reformulation answers to match their respective labels.' This is joint correctness, not logical consistency. A model can answer all five versions with the same wrong categorical answer—e.g., 'No' to both 'Is it true that cats are mammals?' and 'Is it NOT the case that it is NOT true that cats are mammals?'—and be perfectly logically invariant while still counted as inconsistent. The abstract's claim that the model 'contradicts itself ... two-fifths of the time' and the Section 3.2 statement 'contradicting themselves across reformulations' do not follow from the reported metric. As the skeptic's example shows, even a model with 90% accuracy on every reformulation would exhibit a family-consistency rate near 0.9^5 ≈ 59% under this metric, with zero self-contradiction. The authors must report a direct consistency metri","section":"Section 2, Metrics; Abstract; Section 3.2"},{"comment":"The text states that not all transformations apply to every category and gives explicit counts (contrapositive 290, passive 300, De Morgan 50, quantifier rewrite 60, double negation and negation flip 350 each). Table 7, however, appears to mark every transformation as applicable to every category. This contradiction makes it impossible to reconstruct which four reformulations each family actually received and undermines the reproducibility of the slot accounting. The table should be corrected to reflect the actual per-category transformation sets.","section":"Section 2 vs. Table 7"},{"comment":"The transformation-level attribution—the claim that failures cluster on logical operations rather than surface phrasing—depends on each reformulation being a clean, natural instance of exactly one transformation. The paper states that validation was a manual audit by the authors and that 'We did not collect formal inter-annotator agreement.' Given that the causal attribution is a central contribution, the authors should provide at least a small human-annotation study or a second independent audit of a random sample, or explicitly downgrade the transformation-level attribution to a hypothesis. The current limitation statement is honest but does not resolve the concern.","section":"Section 2, Validation protocol; Section 4, Limitations"}],"minor_comments":[{"comment":"Several typographical issues: 'V alidation protocol' should be 'Validation protocol'; Table 3 caption has 'Double. Neg.' with an unnecessary period; Table 6's description says 'answered both the base question and its reformulation correctly,' which is ambiguous when more than one reformulation is involved. Clarify that each cell is computed for the base–reformulation pair only.","section":"Throughout"},{"comment":"The confidence interval footnote gives ±5.1pp for n=350, but some reported cells have much smaller sample sizes (e.g., quantifier rewrite n=60, De Morgan n=50). The paper should provide the corresponding intervals or at least note the wider uncertainty in the tables.","section":"Table 1/Table 6"},{"comment":"The quantifier-collapse analysis is described as based on 'Inspection of the outputs.' The quantitative claim (119 incorrect abstentions, etc.) is reproducible, but the interpretive claim about 'overcautious hedging' should be supported by at least a few representative example outputs in the appendix.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The central empirical finding is likely recoverable, but the current manuscript overstates it. The authors have the model outputs; reanalyzing them to separate answer-agreement consistency from joint correctness is essential. If the reanalysis shows a genuine self-contradiction gap, the paper would be a strong candidate. If not, the contribution reduces to a joint-accuracy benchmark, which is less novel. I also recommend asking for the corrected Table 7 and a small human validation sample."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this one with interest. CRTBench is a thoughtful instrument: seven balanced categories, four controlled reformulations per family, and a transformation typology that actually isolates operation types — a real improvement over paraphrase-based probes and accuracy-only benchmarks like LogicBench. The empirical story has substance: passive voice holds at 94–100% across models while contraposition and double negation drop sharply for the weaker models (GPT-5.4-mini at 72.4% and 84.6%), and the quantifier collapse under high reasoning effort on GPT-5.4 is a specific, checkable pattern that deserves follow-up.\n\nThe problem is the paper's central metric. 'Family consistency' is defined as all five answers matching their gold labels. That is joint accuracy, not consistency. A model that answers 'No' to both the base and its double negation is perfectly logically invariant but scores as inconsistent because both answers are wrong. The text says the gap means models 'contradict themselves two-fifths of the time' — that inference does not follow. The gap could simply be lower accuracy on the reformulated phrasings. The stress-test note presses the right point, though its 0.9^5 example is slightly off (if errors are perfectly correlated, a consistent model would show a smaller gap). The fix is simple and imperative: report a direct self-agreement metric — for each family, compare the model's own base answer to each reformulation (treating the negation-flip as complement), and define a family as consistent when the model's answers agree, independent of gold labels. Per-transformation numbers should be recomputed the same way.\n\nElsewhere the paper is careful. The arithmetic checks out, the validation protocol is honestly described — manual audit, no inter-annotator agreement, no release yet, all acknowledged in the Limitations — and the category-level analysis is transparent. The quantifier abstention analysis is the best part of the empirical work.\n\nWho will get value: researchers building consistency or paraphrastic-robustness benchmarks; they'll want the template typology and the per-operation results even if the metric gets fixed. I would not cite the paper in its current form, because the headline claim is not what is measured. With the self-agreement metric added, I would.\n\nRecommendation: accept for peer review, but the reviewers should insist on the direct agreement metric. Without it, the paper's own conclusion is overstated.","headline":"Useful benchmark and transformation typology, but the main 'consistency' metric is actually joint accuracy, so the self-contradiction claim overreaches.","tokens_in":8107,"tokens_out":5239,"would_cite":false,"duration_ms":55850,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model can score 98.9% on logical questions yet agree with itself only 60.3% of the time when the same question is rewritten in a logically equivalent form.","keywords":["logical consistency","LLM evaluation","controlled reformulation","negation","contrapositive","accuracy-consistency gap","benchmark","reasoning effort"],"falsifier":"Independently rate the CRTBench reformulations for grammaticality, logical fidelity, and naturalness; or, more directly, re-run the evaluation with human-written paraphrases of the same logical forms. If the transformation-specific failure pattern (e.g., contrapositive 72.4%) does not reproduce with human paraphrase templates, the causal claim about logical operations would be an artifact of template phrasing.","tokens_in":7036,"feed_emoji":"⚖️","tokens_out":3415,"duration_ms":30199,"temperature":0.7,"pith_summary":"The paper introduces CRTBench, a benchmark of 350 question families, each with a base question and four controlled logical reformulations, and shows that frontier LLMs are far less consistent than their accuracy suggests. For example, GPT-5.4-mini answers 98.9% of base questions correctly but is consistent across an entire family only 60.3% of the time, with failures concentrated on contrapositive rewrites (72.4%) and double negation (84.6%), while passive-voice rephrasing stays near 99%. Turning on explicit reasoning boosts GPT-5.4-mini to 85.4% consistency but leaves GPT-5.4 unchanged at 86.6%, because gains on nested negation are cancelled by a collapse in quantifier reasoning (92% to 20%). The paper argues that accuracy alone is a misleading measure of logical reasoning, and that inconsistency should be diagnosed at the level of specific logical operations.","feed_headline":"98.9% accuracy, 60.3% consistency on equivalent logic","feed_subtitle":"Frontier LLMs contradict themselves on logically equivalent rewrites; negation-heavy transformations cause the most failures.","key_machinery":"The load-bearing object is the question family plus the family-consistency metric. Each family consists of a base question and four reformulations, each generated by applying exactly one controlled logical transformation (contrapositive, double negation, negation flip, passive voice, De Morgan, or quantifier rewrite). Family consistency requires the model to answer all five versions correctly; one wrong answer makes the family inconsistent. This design converts abstract logical invariance into a measurable per-operation property, allowing the authors to attribute failures to specific logical operations rather than to surface phrasing.","core_discovery":"The central claim is the existence of an 'accuracy–consistency gap': models can be individually correct on most questions yet systematically contradict themselves across logically equivalent reformulations of the same question. The paper demonstrates this with CRTBench, a template-generated benchmark of 350 families (1,750 questions) spanning seven reasoning categories, using six controlled transformations: contrapositive rewriting, double negation, negation flipping, De Morgan's laws, quantifier rewriting, and passive voice. It finds that failures are transformation-specific: negation-heavy operations (contrapositive, double negation) drive inconsistency, whereas surface rephrasing (passive","pith_inferences":["The quantifier collapse under high reasoning effort looks like an overcautious hedging bias ('Cannot be determined' overuse) rather than a logical reversal; if real, it suggests reasoning fine-tuning can induce abstention biases on universal/existential statements.","The template-based method could be sharpened by pairing it with fully natural human paraphrases to separate logical invariance from linguistic generalization; without that, the transformation-level attribution is bounded by template fidelity.","One testable extension: fine-tune a model on consistency (rather than accuracy) across CRTBench families and measure whether base accuracy is preserved; the paper hints at this direction in its conclusion.","The gap between a small and a reasoning-optimized model (60.3% vs 96.9%) suggests consistency is a separable capability that may track training data diversity of operator-level rewrites."],"forward_implications":["Accuracy-based leaderboards overstate logical ability; adding a consistency-required metric would change model rankings (e.g., GPT-5.4-mini drops from near-top accuracy to bottom consistency).","Applications requiring multi-query coherence (legal, medical, knowledge bases) should validate answers across paraphrases before deployment.","Reasoning effort is not a universal fix: it can introduce new failure modes (quantifier hedging) while repairing others.","The benchmark protocol can be extended to open-ended or multi-step reasoning by adapting the transformation set.","Per-transformation scores give a diagnostic handle for targeted mitigation, e.g., training on contrapositive and double-negation variants."],"fun_headline_variants":["98.9% accuracy, 60.3% consistency: LLM logic gap","LLMs ace questions, then contradict themselves","High accuracy hides logical contradictions in LLMs","Logic rewrites expose LLM answer inconsistency"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The attribution of failures to specific logical operations assumes each generated reformulation is a clean, natural instance of exactly one transformation — a claim validated only by the authors' manual audit, with no formal inter-annotator agreement.","fun_headline_variants_meta":{"raw":{"variants":["98.9% accuracy, 60.3% consistency: LLM logic gap","LLMs ace questions, then contradict themselves","High accuracy hides logical contradictions in LLMs","Logic rewrites expose LLM answer inconsistency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000347,"raw_usage":{"total_tokens":1738,"prompt_tokens":750,"completion_tokens":988,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":924}},"tokens_in":494,"tokens_out":988,"duration_ms":7980,"temperature":1.0,"reasoning_tokens":924,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T01:50:22.934950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently rate the CRTBench reformulations for grammaticality, logical fidelity, and naturalness; or, more directly, re-run the evaluation with human-written paraphrases of the same logical forms. If the transformation-specific failure pattern (e.g., contrapositive 72.4%) does not reproduce with human paraphrase templates, the causal claim about logical operations would be an artifact of template phrasing.","supporting_citations":[],"review_version":1}